fbpx

Trying To Add Sitemap To Robots.txt File? Read This!

If you are ever confused or have forgotten how to add Sitemap to Robots.txt file on your WordPress site, worry no more.

I’ll show you the easiest way to do that in this post.

In addition, I’ll explain to you the importance of the Robots.txt file and Sitemap. Also, you get to understand if it is a safe practice to add a sitemap to Robots.txt file.

What is the Importance of Robots.txt File?

The primary function of the Robot.txt file is to communicate to Search Engine crawlers where or where not to access on your site.

Without the Robots.txt files,

  1. Your website server could be overwhelmed by requests from Search Engine crawlers, and this could hurt your SEO
  2. Search Engine crawlers will have access to unimportant or duplicate URLs on your website e.g Login URL and others.

Therefore, it is very important to have a proper Robot.txt file.

How Should A WordPress Robots.txt file look like?

Typically, for WordPress sites, your Robots.txt file should look like this:

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/
 
Sitemap: https://yoursite.com/post-sitemap.xml
Sitemap: https://yoursite.com/page-sitemap.xml

All you have to do is replace sitemap URL here with your actual post and page sitemap URL.

Read on if you do not know how to edit or create a Robots.txt file.

Also, if you do not know how to find your WordPress Sitemap, read this if you use Yoast or check this guide if you use RankMaths.

What’s The Function of Sitemap to a Website?

The primary function of a Sitemap is to communicate the structure of your website to Search Engines.

A sitemap helps to organize all frontend URLs of your website and categorize them in a logical structure, such as URLs for posts, pages, images, videos, Authors etc.

Although you do not need a Sitemap to be indexed by Search Engines, having one means your website URLs would be indexed faster.

Difference Between Robots.txt File and Sitemap

In reality, the Robots.txt file and Sitemap are different but similar. Both communicate with Search Engines.

The major difference between the Robots.txt file and Sitemap is that the Robots.txt file authorizes Search Engine crawlers what URLs on your site should be crawled or not, while Sitemap only informs Search Engines of the structure of your site.

One gives or gives access to… the other informs. The difference is clear.

However, to illustrate, the Robots.txt file is like a parent instructing a child on where to go or not go. On the other hand, Sitemap is like a map that guides a child to his exact location when his parents allow him to go out.

Should You Add Sitemap to Robots.txt File?

The truth is that you do not have to add a sitemap to your Robots.txt file. Even if you do not, Search Engines would still regularly crawl and index your site.

Adding sitemaps to Robots.txt files originated in the mid-2000s when sitemaps were newly introduced, and Search Engines found it difficult to read them.

Almost all Search Engines allow you to upload your sitemap in their console because they are now sophisticated enough to read and interpret it.

Therefore, adding a sitemap to the Robots.txt file is no longer necessary. However, it never harms your site to do it.

One important reason why you may still choose to add your sitemap to Robots.txt file is to be double sure that Search Engines have access to your site.

So, I ask again; should you add a sitemap to your Robots.txt file? It is entirely up to you. Even though it is not required, neither does it harm your site.

In fact, in the early 2000s, this was required for your site URLs to be indexed by Search Engines.

How To Add Sitemap to Robots.txt File on WordPress Using Yoast SEO Plugin

It is very simple to add a sitemap to the Robots.txt file on WordPress. To do this, follow the steps below.

  • Login to your WordPress dashboard
  • Locate Yoast SEO
  • Click “Tools”
how to add Sitemap to Robots.txt File on Yoast SEO Plugin for WordPress
  • Click “File Editor”
Sample WordPress Robots.txt with Sitemap

How To Add Sitemap to Robots.txt File Via File Client Manager

If the first method above did not work for you or you do not use a WordPress site, this method is for you.

It is important to mention that this second approach is a bit more technical than the first one. However, it is still very easy to do if you follow these steps.

  • Access your site files via A Client Manager or use a plugin to do that.
  • Open a .txt document.
  • Copy, paste and modify the sample Robots.txt code above. Sitemap is already added to the code.
  • Save the .txt file as “Robots”. (The .txt extension will be automatically added.)
  • Close the .txt document.
  • Drag or copy and paste the newly created Robots.txt file into your site’s root directory.

After completing these steps above, it is important to test if you did it correctly.

To test, visit https://yoursite.com/robots.txt (replace “yoursite” with your actual domain name.)

Conclusion

As you may have observed, it is very easy to add a sitemap to Robots.txt file.

Two methods were highlighted:

  • 1. Using Yoast SEO Plugin (very easy)
  • 2. Via File Manager

I strongly recommend using the first method to add Sitemap to your Robots txt file if you use Yoast SEO plugin.

Also, as you may have learnt, you do not have to add your sitemap to Robots file. But it is still safe for your site even if you do it.

Therefore, it is entirely your choice to decide if you want to add your sitemap to your Robots txt file.

If you found value in this article, maybe you should also educate yourself about Invalid Traffic, or learn the SEO meaning of H1, H2 H3, H4 and H5 or discover how to easily convert a blog post to a video.

Leave a Comment