Hi! I'm Allen and I welcome you to my blog. I blog mainly about my blogging experiences where I get to learn how to earn money online through blogging and have fun doing so. To learn more about my blog, you may want to subscribe to my RSS feed. If you have some questions, feel free to ask me by using my contact form. Thank and have a nice day.

If you are on free hosting such as wordpress.com or blogspot, you don’t need to do this since they have already made this file for you.

However, if you are on paid hosting, you need to make a robots.txt file.

What is this robots.txt file?

I have limited knowledge about this so let me allow wikipedia to define what’s robots.txt.

The robots exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is, otherwise, publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. (Wikipedia)

Is this important?

Yes. Search engines use robots to crawl your sites. That means, it browses your files in your hosting. The robot crawls every bit of your site, public and private files all together. That will pose a big security risk for you since confidential data will be made available by the robots and be placed into the search engines. Imagine your password files can be found on google searches!

People visit your website through your web pages. What they are seeing are a result of html code that has been parsed/processed by your browser. Robots don’t have browser so they just crawl the files one by one. You have to restrict their access to your files. Robots should be limited only the files that you allow them to crawl.

So how do you limit these robots? By creating a robots.txt file.

How to create a robots.txt file?

• You can write the contents of the robots.txt file yourself if you know how to code it.
• You can check these websites out that will generate the contents for you.
  • http://www.mcanerin.com/EN/search-engine/robots-txt.asp
  • http://www.1-hit.com/all-in-one/tool-robots.txt-generator.htm
• Use an external program to create the file
  • http://www.softsland.com/oven_fresh_robots_txt_maker.html
• If you are using wordpress in your blog, these posts might prove useful.
  • Wordpress Robots.txt optimized for SEO
  • Best Robots.txt for wordpress
  • Creating a wordpress robots.txt to improve seo

The robots.txt file I’m using is what I copied from enblogopedia. http://www.enblogopedia.com/robots.txt

Popularity: 1% [?]