Spider Identification

Spider identification. Check your referrer codes in your log files. You can ban spiders by User-Agent using robots.txt. Banning spiders is at your own risk. Only ban spiders if you're sure you don't want any traffic from that search engine.

Fast PartnerSite Crawler - Spider for FAST, used for AlltheWeb index.
Googlebot/2.1 - GoogleBot
htdig/3.1.5
Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d05 - WiseNut
Mozilla/5.0 (Slurp/si; slurp@inktomi.com) - Inktomi
msnbot/0.11 - MSNBot
Scooter/3.2 - Altavista Scooter

A very active Robot to the game is the new Yahoo Slurp. Now using the former Inktomi engine, Yahoo Search will leave a user agent of Yahoo Slurp with an address of help.yahoo.com.

Look closely for this one, because its means you're in Yahoo's expanding index.

The Newest GoogleBot is Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

If robots follow the Robots Exclusion Standards you can restrict all or parts of your website using the robots.txt file.