Robots

I write robots, spiders, and screen scrapers for a variety of purposes, for fun and for professional applications. From time to time, I will post sample code and tutorials about the bots I build. If you have any questions about my bots or how they affect your website, please contact me. I will be glad to answer any questions you may have! The following is a list of bots I employ, along with a short description of what they do.

SirBoticus

I frequently use this bot as a general spider and/or screen scraper, depending on the situation. If I have precompiled a list of links, then I use it to scrape information. If my task requires the spidering of links, this bot is versatile enough to accomodate that as well. SirBoticus utilizes Perl & LWP, PHP, Bash scripts, MySQL databases, and lives on an Ubuntu-based server. SirBoticus uses the LWP::RobotUA Perl Module to broadcast its presence, to automaticaly throttle its requests, and to obey the robots.txt of any server it hits.