Synonymous with a crawler, this is a program that searches the Internet and attempts to locate new, publicly accessible resources, such as WWW documents, files available in public FTP archives, and Gopher documents.
Also called wanderers or bots, spiders contribute their discoveries to a database that Internet users can search by using a search engine. Spider technology is necessary because the rate at which people are creating new Internet documents greatly exceeds any manual indexing capacity (although search directories prefer the manual approach).
NetLingo Classification: Net Technology
Subscribe to Word of the Day - Email this Definition to a Friend