Hello employer, I have much experience in creating custom crawlers, a dirty work indeed. I can solve your problems in 5 days (through planning with you; writing; testing; deploying). I plan to develop everything in PHP (crawling with other languages can become painful), PHP has many tools to deal with page crawling (top one is DOMDocument which is really fast). Anyway I might require more time to deal with this problem, depending on how many sites you need to crawl (since each crawler is diferent). Also, to get a glance of architecture I would like to build a "center" crawler to handle all the work, and add little (1 file crawling script) classes as plugins to the main one. In this way you will be able to control each crawler differently. Since you asked also for a "Shell script", I can make you able to command the crawler I will be building under CLI (you will be needing a PHP builtin server (so version 5.3 or higher)). Also (the most difficult part) we can obtain ID3 information just by downloading a few bits. Regards and feel free to contact me for whatever you need.