Get 25% off any hosting plan at Hostgator. Use coupon code wphosting9 at Hostgator checkout
Use robots.txt Hack to Speed Up WordPress
WordPress Sites generally get rated as slow by Google. How do I know? I have inspected the Google Webmaster Tools ‘Site Performance’ of over 100 WordPress Websites. The great news is that what is causing WordPress to be perceived as slow is, in fact, extremely simple to rectify.
Google measures the Speed of your Website as an aggregate of your Page load speeds. What this means is that every page indexed by Google is included in the speed score calculation. There are various means of identifying and reducing the pages that are Slow to load however a generalised ‘blanket’ approach is probably the first step;
Identify why Google sees WordPress as Slow
I come from a Management background and am a bit of a fan of the Theory of Constraints (TOC). This whole WordPress speed dilemma is a great candidate for using one of the core principals of TOC; identify and eliminate the significant bottleneck. To identify the significant bottleneck, as identified within the measurement tool which, in this case, is Google Webmaster Tools we need merely look at Labs -> Site Performance in the menu of Google WebMaster Tools. Under ‘Example pages’ you’ll see the browser load speeds of various Pages. If you use WordPress you’ll notice that Pages beginning wp-admin are generally the slowest loading.
Do any Pages beginning wp-admin need to be indexed by the Search Engines? Almost always the answer is that there is no need for wp-admin pages to be indexed. From a Search Engine Optimization (SEO) perspective it makes sense to do something to remove the non value adding wp-admin from being indexed and identified as a bottleneck.
The Solution
The solution, very fortunately, is incredibly simple; we need to let Google (and Yahoo, Bing etc.) know not to index anything on your Site starting with wp-admin. The means of doing this is through the robots.txt file (this is far more simple and effective than trying to use rel=”nofollow” on all wp-admin links). Below are instructions on two methods of achieving the result of advising Google et.al. not to index wp-admin pages:
1. WordPress robots.txt Plugin
WordPress creates a virtual robots.txt. This means that you won’t see a file named robots.txt in the root folder of your fileserver. When you go to http://yourdomainname.com/robots.txt in your Internet Browser you’ll see the dynamically generated (virtual) robots.txt file created by WordPress. There are various WordPress Plugins which assist with changing the virtual robots.txt and all you need do is instruct the Plugin to exclude /wp-admin/ files from being indexed. I like the KB Robots.txt Plugin for its simplicity and effectiveness. All you need do is enter the following in the Robots.txt Editor (found at Settings->KB Robots.txt):
User-agent: *
Disallow: /wp-admin/
2. Static robots.txt file
It is incredibly simple to create a static robots.txt file (which will override the virtual one). Create a file named robots.txt, add the two lines of text as below, save and then copy the file to the root of your fileserver (the root folder of your domain):
User-agent: *
Disallow: /wp-admin/
Explanation of robots.txt Content
The information being added to the robots.txt file in both options above is as follows:
User-agent: * is saying that all Bots may index the Site
Disallow: /wp-admin/ is instructing the Bots not to index and content in the wp-admin folder
Shouldn’t I have a link to my sitemap in robots.txt
Having a reference/link to your sitemap.xml file in your robots.txt is unnecessary as the Bots anyway look for sitemap.xml (or sitemap.xml.gz) in the root of your domain just as the Bots look for robots.txt. WordPress Plugins such as the excellent Google XML Sitemaps have the option to ‘add the sitemap URL to the virtual robots.txt file’ but this is not needed (if however you do want your sitemap.xml URL in the robots.txt then just add Sitemap: http://wordpresspartner.com/sitemap.xml.gz to a new line in your robots.txt Plugin or robots.txt file (obviously change my domain to yours and remove the .gz if your sitemap.xml isn’t gzipped)).
What next for WordPress Speed
There are many things that may be done to improve WordPress load speeds. The tutorial above is but one in numerous activities to improve the speed of WordPress. Keep at improving Page load speeds … it’s good SEO!
Related posts:
[...] performance is becoming increasingly important as a Search Engine Optimization task. Although Page load speeds only have a small bearing on the SERPs at present, all indicators are that Website performance will [...]