Linkrain Articles

After changing my robots.txt to disallow the Google Bot to spider some duplicate pages, I finally see the first results. A while back I wrote about how to get your site out of Google’s supplemental result index and since than I made only some small changes to my robots.txt file. After one month, my supplemental pages went down from 1.200 to 1.030. This does not seem to be a lot, but one month ago Linkrain was counting 150 pages less than now (new links added in the directory, new categories, new articles). So that’s a total of 320 pages less in the supplemental index. Also Google’s webmaster tools start to show first results about pages that were not allowed to be spidered by Google.

Once Google will spider all my pages again, supplemental results should decrease even more. In fact I’ve seen that many pages that I recently disallowed to be spidered, are still in Google’s cache.

In the article written one month ago, I listed some webmasters that had a great search engine traffic increment by tweaking the robots.txt. Actually I’m experiencing an increment too. Search engine traffic increased by 70%, and this only by getting a few hundred pages removed. There are many big websites with thousands of pages and thousand of supplemental results. These websites could achieve some incredible results by getting rid of some supplemental pages. I will now monitor my search engine traffic every day along with the number of pages listed in the supplemental result index to collect some more information about this topic