Discussion in 'SEO Forum' started by cyberdesignz, Aug 3, 2010.
I want to know What Is Robot Txt in SEO?
Waiting for Effective Reply
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.
The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example: http://www.yourwebsite.com/robots.txt
robot txt file is used for the search engines to notify them about the pages you want them to crawl and index, and also the do not want pages.. with this file all unwanted pages of your site can be stopped from crawling and indexing..
Using robots.txt you can allow or ban search engine spiders to crawl your website.You can also allow only some part of your website to be indexed and can remove admin part on your website being indexed in search results.
For Effective & Informative Replies :
Although the robots file is a very important file if you want to have a good ranking on search engines, many Websites don't offer this file.
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages.
Robot.txt is a file, used to prevent search engine to crawl some specific file on web server.
Great Replys :
Thanks for Providing me Sufficient Information, Good Work by All.
Robots.txt file is a file that can ban crawlers from searching as well as investigative your website, for instance not to index convinced pages. This file can be formed with a notepad.
Robots.txt file is a position of information for visiting robots (spiders) that index the substance of your web site pages.
Although the robots register is a real significant file if you want to have a good rating on search engines, lots Websites don't extend this file.
When Web spider crawl site, you don't want some pages crawl. Robot.txt is such a file which tell websiyour der do not crawl those links which are in it.
Then what xml sitemap, and why is it used?`
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index.
You can use Google Webmaster tools to create and check your sites robot.txt file.
Robot txt is a simple text -HTML file. It provides a set of instruction for web crawlers that index the content of host's web site pages. Robot txt file store in root directory.
Generating a robot file with this tool is ideal if you wish to block certain directories or files from search engines.
That means if you put robot.txt in your unwanted files, Google will not be able to crawl it.
robots.txt is used for preventing the crawlers or robots from some particular pages which are mentioned in the file by the webmaster. It is just like a small conversation in webmaster and crawlers.
Separate names with a comma.