Your request is off topic for license-discuss. There are many free/open source WWW indexing programs in the rocketaware index: http://www.rocketaware.com/spec/infotool/www/robot/