1. I've started a massive SPAM purge, any accounts that have spammed are being removed, also for a short period new user signups are disallowed until the SPAM cleaning is finished.

What is Crawler?

Discussion in 'SEO Forum' started by JhonSmit, Feb 1, 2018.

  1. JhonSmit

    JhonSmit Member

    Joined:
    Sep 27, 2012
    Messages:
    116
    Likes Received:
    0
    Trophy Points:
    16
    Gender:
    Male
    Loading...
  2. rajdeep

    rajdeep Member

    Joined:
    Aug 10, 2014
    Messages:
    497
    Likes Received:
    0
    Trophy Points:
    16
    Gender:
    Male
    A web crawler is a relatively simple automated program, or script, that methodically scans or "crawls" through Internet pages to create an index of the data it's looking for; these programs are usually made to be used only once, but they can be programmed for long-term usage as well. There are several uses for the program, perhaps the most popular being search engines using it to provide webs surfers with relevant websites. Crawler programs can be purchased on the Internet, or from many companies that sell computer software, and the programs can be downloaded to most computers.
  3. anirban09P

    anirban09P Member

    Joined:
    Jun 30, 2014
    Messages:
    221
    Likes Received:
    0
    Trophy Points:
    16
    Gender:
    Female
    A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot." Crawlers are typically programmed to visit sites that have been submitted by their owners as new or updated. Entire sites or specific pages can be selectively visited and indexed. Crawlers apparently gained the name because they crawl through a site a page at a time, following the links to other pages on the site until all pages have been read.

Share This Page