Create a dynamic script to scrape multiple websites.

Create a dynamic script in NodeJS to scrape 15+ sites using configurable json.


-Should be able to scrape lazy loading sites

-Download images

-Go through all pagination pages and scrape the child pages

-If element is a product link "click" that link and scrape that child page

-Get all below elements and retrieve correctly

See below for understanding how to set it up.

//Example configurable object


site1: {

url: "[login to view URL]",

parent: {

productList: "div.product-list",

product-link: "a.product-url",

product-name: ".product-info .product-title",

product-image: ".product-images img[src attribute]",

product-price: {

list-price: ".product-price span.old-price",

sale-price: ".product-price [login to view URL]"


pagination: "button.load-more-btn",

product-page: {

name: "form#product h1",

price: {

list-price: ".price .old",

sale-price: ".price new"


colors: "#options-articles li",

sizes: "#options-variants li",

description: ".description p",

images: {

main: "#images .product-image-box img[src attribute]",

thumbs: "#images .thumbs .product-thumbs li"






site2: {

Similar to above Site1






//Product Class

Class Product() {

Product(name, link, image, price, colors, sizes, description) {

[login to view URL] = name;

[login to view URL] = image;

[login to view URL] = price;

[login to view URL] = colors;

[login to view URL] = sizes;

[login to view URL] = description;




//Main Scraper Class

Class Scraper() {

Scraper(site) {

[login to view URL] = site;


getName(attr) {

return name value;


getLink(attr) {

return link value;


getMainImage(attr) {

return main value;


getThumbImages(attr) {

return thumb array;


getPrice(attr) {

return Price Object(list, sale);


getColor(attr) {

return Colors array;


getSizes(attr) {

return Sizes array;


getName(attr) {

return desc value;


hasPagination() {

if([login to view URL] element exist)

//scrape all pagination pages to the last page


downloadImage() {

if([login to view URL] element exist)

//download image

src = getImage();



getProductInfo(product) {

//Go to product page and scrape using "product-page" attribute from json sample.

return {

name: String,

price {

old: String

new: String


images : {

main: String src,

thumbs: [array]


sizes: [array],

colors: [array],

description: String



getListOfProducts() {

//Use [login to view URL] to get listOfProducts

ArrayOfProducts = [];

for( items in listOfProducts ){

//create new product

product = new Product( getProductInfo() );

//add it to list

[login to view URL]( product );




Навички: Java, Javascript, node.js, Python, Веб-скрапінг

Деталі: I like someone who can scrape data from multiple websites and arrange it in excel, python script scrape websites, magento create multiple websites, web scraping dynamic content, python scrapy example, python scrapy vs beautifulsoup, python web scraping dynamic content, dynamic web scraping python, beautifulsoup dynamic content, scrapy tutorial pdf, scrape javascript generated content python, java, javascript, web scraping, create multiple websites magento, scrape data multiple websites, scrape multiple websites, web scrape multiple websites, create multiple websites single magento, create multiple websites magento windows

Про роботодавця:
( 3 відгуки(-ів) ) San Jose, United States

ID Проекту: #15659228



I have changed my bid. Relevant Skills and Experience don't worry Proposed Milestones $500 USD - 100%

$500 USD за 10 дні(-в)
(20 відгуків(-и))

21 фрілансерів(-и) у середньому готові виконати цю роботу за $425


Dear sir, This is Lin and I am scraping expert, I have did many scraping projects, please check my reviews then you wil know. I have checked all details for your project, but there are some details that i wan Більше

$448 USD за 6 дні(-в)
(300 відгуків(и))

Dear,[login to view URL] you for reviewing me. I have just read your project description carefully. I am suitable for this work and have enough power . I will provide the best product to you as you wanted. Relevant Skills and Більше

$555 USD за 10 дні(-в)
(36 відгуків(и))

Greeting, I have understood your Create a dynamic script to scrape multiple websites. task and can do it with your 100% satisfaction. Please ping me for more discussion. Relevant Skills and Experience I have more th Більше

$300 USD за 5 дні(-в)
(49 відгуків(и))
$555 USD за 10 дні(-в)
(18 відгуків(и))

Over 8 +years experience writing almost exclusively web scraping code. I've done it all. I can scrape all LinkedIn profile My languages in order of experience and use is Python, JavaScript, PHP. Python libraries ( sele Більше

$250 USD за 5 дні(-в)
(15 відгуків(и))

hello,we are team of experts and can do this project ,so lets discuss about it in detail and then start the project Relevant Skills and Experience Java, Javascript, node.js, Python, Web Scraping Proposed Milestones $ Більше

$555 USD за 8 дні(-в)
(18 відгуків(и))

*** I check this website and the code. I can do it. I have experience writing many scrap project using nodejs. You can see my portfolio on software development. Relevant Skills and Experience scrap, nodejs Proposed M Більше

$300 USD за 10 дні(-в)
(36 відгуків(и))
$526 USD за 10 дні(-в)
(15 відгуків(и))

I am an IITK graduate and I have 11 years of experience in software development. I have 100% completion rate and I have finished all the projects with the highest level of customer satisfaction. Relevant Skills and Ex Більше

$555 USD за 10 дні(-в)
(24 відгуків(и))

Hello , My Name is Sam , I am Full Stack web and app developer . I'd read the Project details extremely carefully and I am absolutely sure that I can do the It very well . Relevant Skills and Experience To Know idea a Більше

$555 USD за 10 дні(-в)
(18 відгуків(и))

I have gone up through the job description I completely understood your all requirement, So I can easily Create a dynamic script to scrape multiple websites and able to start right now. Relevant Skills and Experience Більше

$555 USD за 10 дні(-в)
(11 відгуків(и))

Hi, I have gone through the job description I completely understood your all requirement and I am available with complete expertise in Python, node.js and past experience to meet your job requirement Relevant Skills Більше

$555 USD за 10 дні(-в)
(3 відгуків(и))

I have briefly read the description on scrap, and I can deliver as per the requirements however I need us to discuss for more clarity on the details, deadline and budget as well. Relevant Skills and Experience python Більше

$250 USD за 4 дні(-в)
(9 відгуків(и))

I'm guessing you want to scrape these data to build a product or price comparison aggregator. I just finished a similar project although for Airline booking service using Nodejs. Relevant Skills and Experience Have b Більше

$366 USD за 10 дні(-в)
(1 відгук)

Hey, I'm interested in your project. Please send me a message so that we can discuss more. I can do trial work before hire me sir , will wait for your reply, Thanks, Syed

$250 USD за 5 дні(-в)
(6 відгуків(и))

Hi, I have extensive knowledge in java ,j2ee,spring,hibernate,json,javafx.I have already completed 1 project in Scrapping. Thanks Relevant Skills and Experience Below are the projects for your reference :- 1 http://w Більше

$555 USD за 10 дні(-в)
(3 відгуків(и))

Been developing with MERN stack for 3+ years and would be able to finish this within 2 days easily. Let me know when to start Relevant Skills and Experience Been developing with MERN stack for 3+ years and would be ab Більше

$388 USD за 2 дні(-в)
(1 відгук)
$250 USD за 10 дні(-в)
(1 відгук)

hi Relevant Skills and Experience I have experience about this task if You give me this task i will surely finish i can try to keep the best relationship with you Proposed Milestones $250 USD - 1

$250 USD за 10 дні(-в)
(0 відгуків(и))
$400 USD за 3 дні(-в)
(0 відгуків(и))