Spidering to Populate the Directories

As earlier discussed, Data Mining with a Spider is a way to populate directories.  The plan is to supply an URL and get back the Title, Description and Keywords.  Looking around at the available spiders led me to two excellent articles by James Bruce. How To Build A Basic Web Crawler To Pull Information From A Website (Part 1) and How To Build A Basic Web Crawler To Pull Information From A Website (Part 2).  These two articles and a little bit of PHP scripting and I had enough to make a first pass on populating the directory websites.

The little bit of PHP I got from putting together the open source spider code gave me the example code to put together a little mySQL db to help me save the URLs and other data collected while I was surfing the web.

I bought a email extractor to extract emails.  The websites being given on topic links may want to enhance their listing with features like banners, multiple category listing, featuring, etc.

Progress continues...