WordPress Robots.txt file for SEO

Having your WordPress Robots.txt properly setup for SEO will improve your rankings in the search results and get you more traffic.

WordPress, straight out of the box is an excellent platform that is feature rich, easy to use and highly customizable. But guess what, it comes with no WordPress robots.txt file and it’s navigation structure is absolutely horrible in the eyes of search engines because many of the features that make user navigation so easy and intuitive end up having the exact opposite effect for search engine spiders/bots. The ability to easily navigate to posts/pages by multiple links of various names can produce a pile of the ever-dreaded duplicate content that search engines such as Google will penalize your site for. Unless you want your pages in the supplemental index, read on to learn about how you can properly use a WordPress robots.txt file as one more SEO “must-do” to keep your site SERP’ing strong.

By properly utilizing a WordPress robots.txt file, we can tell search engine bots where to look and where to not waste their time.

The process of making a WordPress robots.txt file is quite easy, you just have to instruct  the bot by providing it a WordPress robots.txt file telling it to not look in any places where they won’t find any valuable content such as the wp-admin, wp-content etc folders. The WordPress robots.txt file itself should go in your root directory and guess what, it should be named robots.txt (I know, too obvious)

Here is a sample WordPress robots.txt file that you can use for your wordpress powered site, because guess what… WordPress doesn’t come with a WordPress robots.txt file by default, yep thats right, unless you have added a WordPress robots.txt file there will not be one there at all. So, here you go (this one will work just fine but you can obviously customize it to suite your specific needs by adding more directories to disallow access to.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/<a href="http://admindaily.com/recommends/studiopress" style="" target="_blank" rel="nofollow" onmouseover="self.status='themes';return true;" onmouseout="self.status=''">themes</a>
Disallow: /trackback
Disallow: /<a href="http://admindaily.com/my/feed" style="" target="_blank" rel="nofollow" onmouseover="self.status='feed';return true;" onmouseout="self.status=''">feed</a>
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */<a href="http://admindaily.com/my/feed" style="" target="_blank" rel="nofollow" onmouseover="self.status='feed';return true;" onmouseout="self.status=''">feed</a>
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads
# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*
# Google <a href="http://admindaily.com/recommends/keyword_elite" style="" target="_blank" rel="nofollow" onmouseover="self.status='adsense';return true;" onmouseout="self.status=''">AdSense</a>
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /
# digg mirror
User-agent: duggmirror
Disallow: /

Having a WordPress robots.txt file is by no means the only step that should be taken to optimize your WordPress site, but it is one good step in the right direction. I personally believe a combination of the proper use of WordPress SEO Meta Tags in addition to a WordPress robots.txt file is the best overall approach, the WordPress robots.txt file should be used primarily as a mechanism to restrict what folders are indexed but can also be used to show bots what areas they should specifically look in as well, I will write more on that topic another time. I hope you find this article useful, please feel free to comment and share any best practices you have in this area or questions you may have about WordPress robots.txt files.

Add your comment here or use Facebook below.

Add your comment on this post using Facebook!