1. I've started a massive SPAM purge, any accounts that have spammed are being removed, also for a short period new user signups are disallowed until the SPAM cleaning is finished.

What Is Robot Txt in SEO?

Discussion in 'SEO Forum' started by cyberdesignz, Aug 3, 2010.

  1. cyberdesignz

    cyberdesignz Member

    Joined:
    Jun 17, 2010
    Messages:
    83
    Likes Received:
    0
    Trophy Points:
    6
    Hi Friends,
    I want to know What Is Robot Txt in SEO?

    Waiting for Effective Reply
    Loading...
  2. sousen

    sousen New Member

    Joined:
    Aug 4, 2010
    Messages:
    73
    Likes Received:
    0
    Trophy Points:
    0
    When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

    The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example: http://www.yourwebsite.com/robots.txt
  3. sheril123

    sheril123 New Member

    Joined:
    Jun 22, 2010
    Messages:
    77
    Likes Received:
    0
    Trophy Points:
    0
    robot txt file is used for the search engines to notify them about the pages you want them to crawl and index, and also the do not want pages.. with this file all unwanted pages of your site can be stopped from crawling and indexing..
  4. mark6789

    mark6789 New Member

    Joined:
    May 25, 2010
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    US
    Using robots.txt you can allow or ban search engine spiders to crawl your website.You can also allow only some part of your website to be indexed and can remove admin part on your website being indexed in search results.
  5. cyberdesignz

    cyberdesignz Member

    Joined:
    Jun 17, 2010
    Messages:
    83
    Likes Received:
    0
    Trophy Points:
    6
    Thanks friends, :)
    For Effective & Informative Replies :
  6. pinkvinny

    pinkvinny Member

    Joined:
    Jul 21, 2010
    Messages:
    269
    Likes Received:
    1
    Trophy Points:
    16
    Although the robots file is a very important file if you want to have a good ranking on search engines, many Websites don't offer this file.
  7. hanna_depp

    hanna_depp New Member

    Joined:
    Aug 24, 2010
    Messages:
    21
    Likes Received:
    0
    Trophy Points:
    0
    The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages.
  8. nehacct

    nehacct New Member

    Joined:
    Aug 23, 2010
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    0
    Robot.txt is a file, used to prevent search engine to crawl some specific file on web server.
  9. cyberdesignz

    cyberdesignz Member

    Joined:
    Jun 17, 2010
    Messages:
    83
    Likes Received:
    0
    Trophy Points:
    6
    Great Replys : :)
    Thanks for Providing me Sufficient Information, Good Work by All.
  10. strauss01

    strauss01 New Member

    Joined:
    Jan 11, 2010
    Messages:
    34
    Likes Received:
    0
    Trophy Points:
    0
    Robots.txt file is a file that can ban crawlers from searching as well as investigative your website, for instance not to index convinced pages. This file can be formed with a notepad.
  11. vendarjonshan

    vendarjonshan New Member

    Joined:
    Mar 30, 2010
    Messages:
    17
    Likes Received:
    0
    Trophy Points:
    0
    Robots.txt file is a position of information for visiting robots (spiders) that index the substance of your web site pages.
  12. norbertwarne

    norbertwarne New Member

    Joined:
    Oct 13, 2010
    Messages:
    13
    Likes Received:
    0
    Trophy Points:
    0
    Although the robots register is a real significant file if you want to have a good rating on search engines, lots Websites don't extend this file.
  13. paulwalker

    paulwalker New Member

    Joined:
    Oct 17, 2010
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    0
    When Web spider crawl site, you don't want some pages crawl. Robot.txt is such a file which tell websiyour der do not crawl those links which are in it.
  14. newbie2010

    newbie2010 New Member

    Joined:
    Oct 20, 2010
    Messages:
    18
    Likes Received:
    0
    Trophy Points:
    0
    Then what xml sitemap, and why is it used?`
  15. lolserver

    lolserver New Member

    Joined:
    Sep 7, 2011
    Messages:
    31
    Likes Received:
    1
    Trophy Points:
    0
    The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index.
  16. Farbyte

    Farbyte Member

    Joined:
    Jun 21, 2011
    Messages:
    36
    Likes Received:
    2
    Trophy Points:
    8
    Location:
    UK
    You can use Google Webmaster tools to create and check your sites robot.txt file.
  17. kareelasmith

    kareelasmith New Member

    Joined:
    Sep 7, 2011
    Messages:
    41
    Likes Received:
    0
    Trophy Points:
    0
    Robot txt is a simple text -HTML file. It provides a set of instruction for web crawlers that index the content of host's web site pages. Robot txt file store in root directory.
  18. wtfcode

    wtfcode New Member

    Joined:
    Sep 21, 2011
    Messages:
    26
    Likes Received:
    0
    Trophy Points:
    0
    Generating a robot file with this tool is ideal if you wish to block certain directories or files from search engines.
  19. transcribe

    transcribe New Member

    Joined:
    Oct 26, 2010
    Messages:
    29
    Likes Received:
    0
    Trophy Points:
    1
    That means if you put robot.txt in your unwanted files, Google will not be able to crawl it.
  20. neo127

    neo127 Member

    Joined:
    Mar 17, 2011
    Messages:
    187
    Likes Received:
    8
    Trophy Points:
    18
    robots.txt is used for preventing the crawlers or robots from some particular pages which are mentioned in the file by the webmaster. It is just like a small conversation in webmaster and crawlers.

Share This Page