Create a robot to get information from public pages for a market study - (Phyton)

We require a robot that supports multithread processing, dynamic IPs & User Agents management, handling captchas / ReCaptachas (using a library to be provided by us), recover from web page timeouts & bad responses, etc. Phyton language (with BeautifulSoup, Pandas libraries), MySQL DB, running on AWS or Linux hosting (TBD). [Open to discuss adjustments if needed.]

The provider must develop a base framework that handles multiple threads, IPs, User Agents and Captchas; plus the web scrapping process itself for two jobs:

A) Site with 1 query web page with two input fields and a captcha + 1 response web page with a table of 2 columns or a "No results" message.

B) Site with 1 query web page with two input fields and a ReCaptcha + 1 response web page with a table of 2 columns or a "No results" message.

To process both jobs, the software will read an "input table" with the list of values to be used in the query web page input fields, and will write in an "output table" the results obtained, including some additional process-related information, such as time stamp, IP used, user agent used, time to process the record, success / error, type of error, and others you may think are relevant.

Each job will be defined in a configuration file indicating the input and output tables used, web page url to be scrapped, and other relevant parameters you may consider.

The software must have the capability to add additional web pages in the future (with the corresponding parsing programming, of course) without needing to modify the framework [login to view URL] future jobs may require different tables and logic, and are not part of the scope of this work.

Parameters for managing the framework should be modifiable with config files (Ex: # of threads, general web page timeout, time between requests, time wait before each query submit, etc.).

The base framework also should be updating statistics for each process being executed in a table to be designed for this purpose. Columns should be something like: Job name, start date/time, status, # active threads, # of successful scrappings so far, # of failed scrappings so far, estimated time to complete, and others that you may think are relevant to monitor the progress.

The provider must deliver source code and documentation for usage, and leave it installed and running in an AWS environment or Linux hosting (TBD) to be provided by us.

Please kindly estimate the cost and time required.

Веб-дизайн HTML

ID Проекту: #25887912

Про проект