add sitemap to robots.txt file

Trying To Add Sitemap To Robots.txt File? Read This!

If you are ever confused or have forgotten how to add Sitemap to Robots.txt file on your WordPress site, worry no more.

In this post I’ll show you the easiest way to do just that.

In addition, I’ll explain to you the importance of the Robots.txt file and Sitemap. Also, you get to understand if it is a safe practice to add sitemap to Robots.txt file.

What is the Importance of Robots.txt File?

The primary function of the Robot.txt file is to communicate to Search Engine crawlers where or where not to access on your site.

Without the Robots.txt files,

  1. Your website server could be overwhelmed by request from Search Engine crawlers and this could have a negative impact on your SEO
  2. Search Engine crawlers will have access to unimportant or duplicate URLs on your website e.g Login URL and others.

Therefore, it is very important to have a proper Robot.txt file.

How Should A WordPress Robots.txt file look like?

Typically, for WordPress sites, your Robots.txt file should look like this:

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/
 
Sitemap: https://yoursite.com/post-sitemap.xml
Sitemap: https://yoursite.com/page-sitemap.xml

All you have to do is replace sitemap URL here with your actual post and page sitemap URL.

If you do not know how to edit or create a Robots.txt file, read on.

Also, if you do not know how to find your WordPress Sitemap, read this if you use Yoast or check this guide if you use RankMaths.

CSGO - How To Download & Instal...
CSGO - How To Download & Install Custom Maps

What’s The Function of Sitemap to a Website?

The primary function of a Sitemap is to communicate the structure of your website to the Search Engines.

A sitemap helps to organize all frontend URLs of your website and categorize them in a logical structure such as URLs for posts, pages, images, videos, Authors etc.

Although you do not need a Sitemap to be indexed by the Search Engines, having one simply means that your website URLs would be indexed faster.

Difference Between Robots.txt File and Sitemap

In reality, the Robots.txt file and Sitemap are two different things but are however similar. Both communicate to Search Engines.

Load WordPress Sites in as fast as 37ms!

The major difference between Robots.txt file and Sitemap is that Robots.txt file authorizes Search Engine crawlers what URLs on your site should be crawled or not while Sitemap only informs Search Engines the structure of your site.

One gives authorizes or gives access to… the other informs. The difference is clear.

Load WordPress Sites in as fast as 37ms!

However, to illustrate, Robots.txt file is like a parent who instructs a child of where to go or not go. On the other hand, Sitemap is like a map that guides a child to his exact location when his parents allow him to go out.

Should You Add Sitemap to Robots.txt File?

The truth is that you do not have to add sitemap to your Robots.txt file. Even if you do not, Search Engines would still crawl and index your site regularly.

The practice of adding sitemap to Robots.txt file originated in the mid 2000s when sitemaps where newly introduced and Search Engines found it difficult to read them.

Now, almost all Search Engines allows you to upload your sitemap in their console because they are now sophisticated enough to read and interpret it.

Therefore, it is no longer necessary to add sitemap to Robots.txt file. However, it never harms your site to do it.

One important reason why you may still choose to add your sitemap to Robots.txt file is to be double sure that Search Engines have access to your site.

So, I ask again; should you add sitemap to your Robots.txt file? It is entirely up to you. Even though it is not required, neither does it harm your site.

In fact, in the early 2000s this is required for your site URLs to be indexed by Search Engines.

How To Add Sitemap to Robots.txt File on WordPress Using Yoast SEO Plugin

It is actually very simple to add sitemap to Robots.txt file on WordPress. To do this, follow the steps below.

  • Login to your WordPress dashboard
  • Locate Yoast SEO
  • Click “Tools”
how to add Sitemap to Robots.txt File on Yoast SEO Plugin for WordPress
  • Click “File Editor”
Sample WordPress Robots.txt with Sitemap

How To Add Sitemap to Robots.txt File Via File Client Manager

If the first method above did not work for you or you do not use a WordPress site, this method is for you.

It is important to mention that this second approach is a bit more technical than the first one. However, it is still very easy to do if you follow these steps.

  • Access your site files via A Client Manager or use a plugin to do that.
  • Open a .txt document
  • Copy, paste and modify the sample Robots.txt code above. Sitemap is already added to the code.
  • Save the .txt file as “Robots”. (The .txt extension will be automatically added.)
  • Close the .txt document
  • Drag or copy and paste the newly created Robots.txt file into the root directory of your site.

After completing these steps above, it is important to test if you did it correctly.

To test, simply visit https://yoursite.com/robots.txt (replace “yoursite” with your actual domain name.)

Conclusion

As you may have observed, it is very easy to add sitemap to Robots.txt file.

Two methods were highlighted:

  • 1. Using Yoast SEO Plugin (very easy)
  • 2. Via File Manager

I strongly recommend that you use the first method to add Sitemap to your Robots txt file if you use Yoast SEO plugin.

Also, as you may have learnt, you do not have to add your sitemap to Robots file. But it is still safe for your site even if you do it.

Therefore, it is entirely your choice to decide if you want to add your sitemap to your Robots txt file.

If you found value in this article, maybe you should also educate yourself about Invalid Traffic, or learn the SEO meaning of H1, H2 H3, H4 and H5 or discover how to easily convert a blog post to a video.

Leave a Comment

Your email address will not be published.