Spider: A search engine spider is a program that crawls the web, visiting web pages to collect information to add to or update a search engine’s index. The major search engines on the web all have such a program, which is also known as a “crawler” or a “bot.”
General Process for a Spider:
