Robots
I write robots, spiders, and screen scrapers for a variety of purposes, for fun and for professional applications. From time to time, I will post sample code and tutorials about the bots I build. If you have any questions about my bots or how they affect your website, please contact me. I will be glad to answer any questions you may have! The following is a list of bots I employ, along with a short description of what they do.
BabbleBot
BabbleBot is a quick php/cURL bot I wrote to post odd ramblings to my twitter account. Though twitter provides an API, I found it easier to cURL directly to my account. It uses a set of small word lists that I compiled from various sources around the net. Until I decided to use my twitter account for non-automated ramblings, BabbleBot was scheduled to tweet at random intervals throughout the day. This bot could easily be extended to create random text for all sorts of purposes, but I throttled to post messages 140 characters or less as per twitter's limits. I have provided the source code in .zip format for anyone who is interested. For the positively bored, I have additionally provided BabbleBot's log transmissions for most of the tweets it made during its 5 month run. Maybe it will make good bot food for your own project!
SirBoticus
I frequently use this bot as a general spider and/or screen scraper, depending on the situation. If I have precompiled a list of links, then I use it to scrape information. If my task requires the spidering of links, this bot is versatile enough to accomodate that as well. SirBoticus utilizes Perl & LWP, PHP, Bash scripts, MySQL databases, and lives on an Ubuntu-based server. SirBoticus uses the LWP::RobotUA Perl Module to broadcast its presence, to automaticaly throttle its requests, and to obey the robots.txt of any server it hits.
