Link Dump Spider Needed
Hello everyone, I am looking for someone to code me either a desktop application, or a server based applicate, depending on your expertise and your recommendation.
Some quick info: I am using a Link Dump program. It uses a database that is customizable. I am looking for an automated way to actually update my own database without having to look on the major search engines for the newest ones.
I need a utility that will be able to 'spider' google for specific string(s). Then i need it to actually spider to indexed pages from the search engine. I need it to pull certain strings from the 'spidered page'.
The Database is extremely simple here is an example:
#linkdump.zxlog.com#/index.php#title=TXT_SITENAME&url=TXT_LINK&Submit=Dump#200 OK
The above string basically stands for this:
[login to view URL] = the link domain
/[login to view URL] = the php script location
The Link Dump Software uses the other strings.
Do you think you can program something to help me out? Even if the hole thing is not compeltely automated, maybe all need to do is sit at the computar and click my mouse. I just need a simplier way to add the link dumps to a database.
Thank you for reading.
## Deliverables
What I Need:
A program/script that can crawl the major search engines via a custom string, and crawl the search results (depending on the number of results i want to crawl)
I would like the "crawled" sites to be added to a database, and if duplicated, remove.
I need the addon script to crawl the search results and look for certain Strings and put them in this format:
#www.adultlinkdump.com#/[login to view URL] received. Will be posted once confirmed!
I have the Text Documentations on how to add the new link dump into the database, so that will probably be the best thing for you to read.
But Simple:
Addon or Script/Program that can crawl the major search engines, view the source, and see if the link dump uses a specific link dumping script (adult link dumps don't use many scripts, they are mostly the same ones). I need to have Google, MSN, Yahoo, [login to view URL], as options. When the "addon" comes in contact with a link dump site that has the correct strings in the source, i would like for the program to read the source page, and record down the specific strings that my Link Dumping Software will use. I would also like the program to keep the Link Dumps in a database, and i would like the program to be able to remove duplicates. I need the program to be able to export its database in TXT Form. Any other details can be worked out between us, the script shouldnt be too hard, but i know this is life and real problems happen. I also would like to have it be "licenseable", i will lend this out to close networking buddies in the same work i am in, but if it gets leaked ill be screwed since my program would be able to be used by everyone.
I am here, i am human, so lets work on this!