Skip to content
Equizzing
Home
Quiz Categories
Linux Entry Level
LPI Level 1
LPI Level 2
LPI Level 3
Contact
WWW wanderers or spiders are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages Search engines like Google frequently spider web pages for indexing
How will you stop web spiders from crawling certain directories on your website?