|
Don't get caught by search engine spiders creeping on your site under wraps! |
|
|
|
• Link Popularity Program
• Search Engine Newsletter
• fantomas spiderSpy™
• Search Engine Optimization
• Webmaster Resources
• 450+ Search Engines
• LinkTopper™ Directory
The leading weblog newsletter on cloaking, search engine optimization, spider IPs, breaking news, etc.
|
|
Determining search engine spiders' identity beyond reasonable doubt is a rather complex and sophisticated process fraught with uncertainty and risky decisions. The procedure involved is a mix of tools, various data sources and expertise: Whois, NSLookup, our own extensive dabases (literally growing by the hour), log file research all day and all night, lots of proprietary programs, spider traps, experience, plus, last not least, fantastic client support – it all adds up. Spiders can be detected by IP (the most reliable procedure), by UserAgent (highly unreliable) and by behavior (not very reliable, either). One of many complications lies in the fact that not all spiders are created equal: these days, there are hundreds of spiders (crawler, snoopbots) around which are not related to any public search engines. They may or may not honor the Robots Exclusion Standard (robots.txt) convention, which lets you determine which specific parts of a site shall be accessible to which search engine spider. So if a spider crawls your site and first thing hunts for the robots.txt file, this may or may not indicate search engine activity. While this is no general problem with most major search engines, there are those that will only heed your robots.txt entries erratically if at all, one prominent example being Google. Be as comprehensive as possible. Full log file entry excerpts are best. (You may, however, substitute your domain or file name to preserve anonymity.) Please don't off topic questions such as “What are Inktomi's spiders?” or “How many spiders does AltaVista have?” This is a search engine spider verification service, remember? It is designed to help you clarify weird log entries, check up on unfamiliar UserAgents, get the scoop on particularly aggressive crawling from specific engines or understand the general mechanics of search engine spider activity. Where can I get a comprehensive spider list? If you want the world's most comprehensive search engine spider database, check out our very own fantomas spiderSpy™ service, which is updated no less than six times a day. It covers literally thousands of spiders with currently over 8,000 entries. More search engine material here …
|
|
||
|
|