User-agent:ABCdatos BotLink
User-agent:Acme.Spider
User-agent:Ahoy! The Homepage Finder
User-agent:Alkaline
User-agent:Anthill
User-agent:Walhello appie
User-agent:Arachnophilia
User-agent:Arale
User-agent:Araneo
User-agent:AraybOt
User-agent:ArchitextSpider
User-agent:Aretha
User-agent:ARIADNE
User-agent:arks
User-agent:ASpider (Associative Spider)
User-agent:ATN Worldwide
User-agent:Atomz.com Search Robot
User-agent:AURESYS
User-agent:BackRub
User-agent:unnamed
User-agent:BBot
User-agent:Big Brother
User-agent:Bjaaland
User-agent:BlackWidow
User-agent:Die Blinde Kuh
User-agent:Bloodhound
User-agent:Borg-Bot
User-agent:BoxSeaBot
User-agent:bright.net caching robot
User-agent:BSpider
User-agent:Calif
User-agent:Cassandra
User-agent:Digimarc Marcspider/CGI
User-agent:Checkbot
User-agent:ChristCrawler.com
User-agent:churl
User-agent:cIeNcIaFiCcIoN.nEt
User-agent:CMC/0.01
User-agent:Collective
User-agent:Combine System
User-agent:ConfuzzledBot
User-agent:CoolBot
User-agent:Web Core / Roots
User-agent:XYLEME Robot
User-agent:Internet Cruiser Robot
User-agent:Cusco
User-agent:CyberSpyder Link Test
User-agent:CydralSpider
User-agent:Desert Realm Spider
User-agent:DienstSpider
User-agent:DownLoad Express
User-agent:DragonBot
User-agent:DWCP (Dridus' Web Cataloging Project)
User-agent:e-collector
User-agent:EbiNess
User-agent:ELFINBOT
User-agent:Emacs-w3 Search Engine
User-agent:ananzi
User-agent:esculapio
User-agent:FastCrawler
User-agent:Fluid Dynamics Search Engine robot
User-agent:Felix IDE
User-agent:Wild Ferret Web Hopper #1, #2, #3
User-agent:FetchRover
User-agent:fido
User-agent:Hämähäkki
User-agent:KIT-Fireball
User-agent:Fish search
User-agent:Fouineur
User-agent:Robot Francoroute
User-agent:Freecrawl
User-agent:GCreep
User-agent:GetBot
User-agent:GetURL
User-agent:Golem
User-agent:Googlebot
User-agent:Grapnel/0.01 Experiment
User-agent:Griffon 
User-agent:Gromit
User-agent:Northern Light Gulliver
User-agent:Gulper Bot
User-agent:HamBot
User-agent:Harvest
User-agent:havIndex
User-agent:HI (HTML Index) Search
User-agent:Hometown Spider Pro
User-agent:HTMLgobble
User-agent:Hyper-Decontextualizer
User-agent:iajaBot
User-agent:IBM_Planetwide
User-agent:Popular Iconoclast
User-agent:Ingrid
User-agent:Imagelock
User-agent:IncyWincy
User-agent:Informant
User-agent:InfoSeek Robot 1.0
User-agent:Infoseek Sidewinder
User-agent:InfoSpiders
User-agent:Inspector Web
User-agent:IntelliAgent
User-agent:Iron33
User-agent:JavaBee
User-agent:JCrawler
User-agent:AskJeeves
User-agent:JoBo Java Web Robot
User-agent:Jobot
User-agent:JoeBot
User-agent:JumpStation
User-agent:image.kapsi.net
User-agent:Katipo
User-agent:KDD-Explorer
User-agent:KO_Yappo_Robot
User-agent:LabelGrabber
User-agent:larbin
User-agent:Link Validator
User-agent:LinkScan
User-agent:LinkWalker
User-agent:logo.gif Crawler
User-agent:Lycos
User-agent:Mac WWWWorm
User-agent:Magpie
User-agent:marvin/infoseek
User-agent:MediaFox
User-agent:MerzScope
User-agent:MindCrawler
User-agent:MOMspider
User-agent:Monster
User-agent:Motor
User-agent:MSNBot
User-agent:Muncher
User-agent:Muninn
User-agent:Mwd.Search
User-agent:NDSpider
User-agent:NetCarta WebMap Engine
User-agent:NetMechanic
User-agent:HKU WWW Octopus
User-agent:OntoSpider
User-agent:Openfind data gatherer
User-agent:Orb Search
User-agent:pegasus
User-agent:PerlCrawler 1.0
User-agent:Phantom
User-agent:PiltdownMan
User-agent:Pimptrain.com's robot
User-agent:Pioneer
User-agent:Portal Juice Spider
User-agent:PlumtreeWebAccessor
User-agent:PortalB Spider
User-agent:psbot
User-agent:GetterroboPlus Puu
User-agent:The Python Robot
User-agent:RoadHouse Crawling System
User-agent:RixBot
User-agent:Road Runner: The ImageScape Robot
User-agent:ComputingSite Robi/1.0
User-agent:RoboCrawl Spider
User-agent:RoboFox
User-agent:Robozilla
User-agent:Roverbot
User-agent:Scooter
User-agent:Sleek
User-agent:SearchProcess
User-agent:Senrigan
User-agent:SG-Scout
User-agent:ShagSeeker
User-agent:Shai'Hulud
User-agent:Sift
User-agent:SiteTech-Rover
User-agent:Skymob.com
User-agent:SLCrawler
User-agent:Inktomi Slurp
User-agent:Smart Spider
User-agent:spider_monkey
User-agent:SpiderBot
User-agent:Spiderline Crawler
User-agent:SpiderMan
User-agent:SpiderView(tm)
User-agent:Site Searcher
User-agent:Suke
User-agent:suntek search engine
User-agent:Sven
User-agent:Sygol
User-agent:Tarantula
User-agent:tarspider
User-agent:Tcl W3 Robot
User-agent:TechBOT
User-agent:Templeton
User-agent:TitIn
User-agent:TITAN
User-agent:TLSpider
User-agent:UCSD Crawl
User-agent:UdmSearch
User-agent:vision-search
User-agent:void-bot
User-agent:Voyager
User-agent:VWbot
User-agent:The NWI Robot
User-agent:W3M2
User-agent:WallPaper (alias crawlpaper)
User-agent:WebBandit Web Spider
User-agent:WebCatcher
User-agent:WebCopy
User-agent:WebLinker
User-agent:WebQuest
User-agent:Digimarc MarcSpider
User-agent:WebReaper
User-agent:webs
User-agent:Websnarf
User-agent:WebSpider
User-agent:WebVac
User-agent:WebWalker
User-agent:WhoWhere Robot
Disallow:/ftp
Disallow:/maildir
Disallow:/test
Disallow:/stats
Disallow:/log
Disallow:/_fpclass
Disallow:/_notes
Disallow:/_vti_log
Disallow:/aspnet_client
Disallow:/cgi_bin
Disallow:/fpdb
Disallow:/db
Disallow:/log