If you are on free hosting such as wordpress.com or blogspot, you don’t need to do this since they have already made this file for you.
However, if you are on paid hosting, you need to make a
What is this robots.txt file?
I have limited knowledge about this so let me allow wikipedia to define what’s
The robots exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is, otherwise, publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. (Wikipedia)
Is this important?
Yes. Search engines use robots to crawl your sites. That means, it browses your files in your hosting. The robot crawls every bit of your site, public and private files all together. That will pose a big security risk for you since confidential data will be made available by the robots and be placed into the search engines. Imagine your password files can be found on google searches!
People visit your website through your web pages. What they are seeing are a result of html code that has been parsed/processed by your browser. Robots don’t have browser so they just crawl the files one by one. You have to restrict their access to your files. Robots should be limited only the files that you allow them to crawl.
So how do you limit these robots? By creating a
How to create a
- You can write the contents of the
robots.txtfile yourself if you know how to code it.
- You can check these websites out that will generate the contents for you.
- Use an external program to create the file
- If you are using wordpress in your blog, these posts might prove useful.
robots.txt file I’m using is what I copied from enblogopedia. http://www.enblogopedia.com/robots.txt