Scrapy Jobs

Regular comprehensive snapshot. There are 3,000 products. 20 columns for each product. Page by page. I’m looking for a repeatable, fully automated workflow. A Python-based stack (Scrapy, BeautifulSoup, Selenium, Playwright, or an equivalent you prefer). Robustness is key: the crawler should cope with pagination, JavaScript-rendered. Clear, well-commented code is part of the deliverable so my team can review and rerun it internally. Each quarterly hand-off must include: • Cleaned CSV or JSON containing the structured product records • The raw HTML or a compressed WARC snapshot for auditing • The executable script(s) plus a brief change log highlighting any site-structure updates you handled Please outline your proposed tool chain, an example of a large scrape yo...

I have a growing list of websites from which I routinely pull business details, and I’m looking for a single, reusable script that lets me swap in a new URL, hit “run,” and walk away with clean data. The fields I always need are: Name, Address, Phone number, Email, Website, and a short description of Services. Python 3 is my first choice because I’m already set up with it locally, and I’m comfortable installing BeautifulSoup, Requests, Scrapy, or Selenium if the page structure calls for it. If you feel another stack (e.g., Node-JS with Cheerio or Puppeteer) would shorten development time or handle tricky JavaScript sites better, let me know—flexibility is more important than sticking to a single library. What matters to me is that: • The URL (or...

I have a curated list of specific company websites and I need an automated solution that extracts complete contact information from each one. The goal is to turn every URL into a clean, ready-to-use lead. WEBSITE : The scraper should capture: • Email addresses • Phone numbers • Mailing addresses • LinkedIn profile link • Location (city / state / country) • First and last name • Occupation / job title • Company name • Company website A well-structured CSV or Excel file is the preferred output, with each field in its own column. I am comfortable with your choice of tech—Python with BeautifulSoup, Scrapy, or Selenium are all fine—as long as the script runs reliably and respects and rate limits where required. Ac...

I need a small, always-on scraper that keeps an eye on a popular second-hand marketplace and alerts me the moment any Electronics listing matching my keywords appears. My priority is speed—ideally I hear about a new post within seconds, certainly no longer than a minute after it goes live. Here’s what the script must do: • Crawl the marketplace continuously without being blocked, parse every new listing, and filter it against a configurable set of electronics keywords. • Extract and store the Price and Condition fields so I can track changes and avoid duplicates. • Push an instant notification (email, SMS, or Slack—whichever you prefer to wire up) each time a fresh match is found. I’m comfortable with a Python 3 stack—think Requests/...

PROJECT TITLE Web Scraping Developer for Global Legal & Regulatory Data Collection PROJECT OVERVIEW We are looking for a developer who can build an automated system to collect legal and regulatory documents from multiple global sources. The goal is to create a scalable automated pipeline that can gather legal data across multiple jurisdictions and regulatory domains. DATA COLLECTION SCOPE The system will collect information related to: - Medical law and healthcare regulation - Medical advertising regulation - Corporate formation and company governance laws - Investment regulation (stocks, cryptocurrency, real estate) - Tax law and administrative tax rulings - Beauty and cosmetic regulation - Medical and cosmetic manufacturing compliance - Import and export law - Customs and tariff...

Project Overview I am seeking an expert Magento 2 Developer with strong data extraction (scraping) capabilities to build out a comprehensive medical supply catalog. The previous developer started a sample set but is no longer on the project. I need a professional to scrape high-volume product data and perform a structured, complex import into my Magento 2 staging site. Scope of Work 1. Data Scraping & Extraction Source: Target website (medical supply industry) to be provided. Requirements: Full extraction of Product Titles, High-Res Images, Long/Short Descriptions, SKUs, and Technical Specifications. Variation Mapping: Correctly identify and link "Simple" products to their "Configurable" parents (e.g., mapping different sizes or packaging options to a single prod...

Thanks for looking. I urgently need data for a set of local businesses in and around Berkshire and London UK. We will pay per 2k list of the industries we will send upon acceptance. Email addresses must not be role based or trip any spam traps. I need a one-time extraction of verified email addresses from reputable online business directories. No other data fields are required—just the clean list of emails. Please choose whatever approach you prefer—Python with Scrapy/BeautifulSoup, browser automation with Selenium, or a similar tool chain—as long as the result is accurate and the scraping respects each site’s terms of service and rate limits. Deliverable • A CSV or XLSX file containing every unique email address you capture, de-duplicated and ready ...

I need a robust yet easy-to-maintain web scraper that pulls player statistics from four different sports sites—a blend of official league pages, sports news outlets, and a couple of well-known fan forums. All scraped data should flow into a single database and surface through a lightweight web dashboard where I can search by player, season, and team, compare numbers side by side, and export results to CSV. My ideal flow looks like this: enter or schedule the URLs, run or auto-run the scraper, watch progress logs, and then immediately view fresh stats inside the dashboard—no command-line work once everything is deployed. If any source changes its HTML, the scraper should fail gracefully and flag the issue in the UI so I can react quickly. Tech stack is flexible; Python with Be...

I need a robust AI-driven scraper that reliably pulls product text, images, SKUs, and pricing from selected e-commerce websites and our own admin-controlled catalog portals. The script must handle pagination, dynamic content, and, where required, authenticated sessions without manual intervention. Core expectations • Extract and store: product titles/descriptions, all associated images, SKU codes, and current prices. • Export: structured CSV or JSON for data, separate folder (or S3 bucket) for images, with clear file naming that links each image back to its SKU. • Tech stack: Python with libraries such as Scrapy, Playwright/Selenium, BeautifulSoup, or a comparable approach—whatever you can prove is most efficient and resilient. Basic computer-vision or OCR hoo...

Need experienced dev to fix my claude-coded python playwright automation script. Note that I don't know any coding. So need your help to review and then fix. Details will be provided. Reply with "claude" so I know you've really read this very short job description.

I need a clean, well-structured scrape of vendor details from jiii.ng. The goal is straightforward: pull each vendor’s name and contact information and hand the results back to me in a single Excel workbook. Categories The focus is on the main marketplace sections. I will confirm the exact categories before you begin, but expect the usual high-traffic areas (Electronics, Real Estate, Automobiles) to be included. Data points • Vendor name • All available contact details (phone, email, WhatsApp, or any other channel the site exposes) Delivery You’ll give me: 1. The Excel file, one sheet per category, neatly labeled and deduplicated. 2. The script or notebook you used (Python with requests / BeautifulSoup, Scrapy, or a comparable tool is fine) so I c...

I need an automated service that visits eight real-estate platforms every day—, Zillow (FSBO only), , Hubzu, HomePath, HUDHomestore, InvestorLift, and Crexi—pulls both property details and price data, and drops the results into a single clean dataset. Each site has its own search criteria that I will supply (zip codes, status flags, auction dates, etc.). The scraper must respect those individual rules and still be easy for me to adjust later without having to rewrite code—think a clearly labeled config section or similarly simple mechanism. Key expectations • Runs on a schedule without manual intervention and retries gracefully if a site is temporarily unavailable. • Captures all standard listing fields (address, beds, baths, square footage, list/asking ...

I need every piece of publicly available data pulled from three separate websites and packaged so I can work with it offline. The moment the contract is awarded I’ll hand you the URLs; from there I expect you to spin up an automated solution—Python with Scrapy, BeautifulSoup, Selenium, or a comparable stack—that captures all records, follows pagination, and handles any lazy-loaded content. Time is tight: the full job must be wrapped up within 3 calendar days. My total budget is fixed at US $300, so please be sure your approach fits comfortably inside that limit. Deliverables • A clean, well-structured file for each site (CSV or Excel, whichever you prefer) containing every scraped field. • The runnable scripts or notebooks you used, clearly commented so I ...