Tag Archives: use robots.txt

Using Robots.txt to tell search engines what you want them to index

Lately, I’ve been checking my blog at Google a lot. To check my blog, I use this search query.

site:silkenhut.com/blog

I use this to check if All in One SEO Plug-in has taken effect and also to see which pages are indexed. When I saw my results, I noticed several pages that I don’t want to be indexed but are indexed by Google. Aside from my individual post pages, these are the other things that Google has indexed from the blog.

  • My stats page was indexed and included with it are the links for every person who have commented. (http://silkenhut/stats/?stats_author=Elizar)
  • My feeds and comments are being indexed. (http://silkenhut/back-up-wordpress-using-a-plug-in/feed/)
  • My archives and pages (http://silkenhut/page/3/) are also being indexed.

This is not a major problem but personally, I want to make sure that the pages being indexed are the useful ones such as the blog articles and not the feeds, comments, archives etc…
Continue reading Using Robots.txt to tell search engines what you want them to index