Find Jobs
Hire Freelancers

Django/web2py price comparison website with scrapy scrapper

$250-750 USD

Ditutup
Dibuat sekitar 8 tahun yang lalu

$250-750 USD

Dibayar ketika dikirim
Hello I need someone to develop a website in Django/web2py for a price comparison site using Scrapy (or something better) & Selenium or beautifulsoup - code must be documented in English. It should allow Scrapy/alternative to crawl a "variable" number of separate sites (using a number of "spiders") that can pull out product details such as Product ID, Title, Price, Vendor, Description, Image, URL and Stock Position etc. This information should then be placed in a PostgrSQL/Mongodb database to be displayed using Web2py/Django. There should also be a way of the URL to the products be changed to affiliate links. This is an easy project for someone who has done this before, if you have examples of previous work this will go in your favour so please reference them. Additionally if you have advice on the a better architecture/solution I am open to ideas. Expected Features: a) The Products Table in the server database to be automatically populated by the scraper. The required fields are Product ID, Title, Price, Vendor, Stock Position, Payment Options, Delivery Time b) Easy extensibility (with some python coding) to add more sites in future. c) To meet the above, the scraper to be implemented as two modules. The "Scraper Module" and the "Parameter Module". d) The "Scraper Module" would do the actual scraping of multiple sites (based on parameters read from the Parameters Module), and also automatically populate the Products Table in the database server. For sites with content rendered in JavaScript, Scrapy to be used with Selenium for effective scraping. e) The "Parameters Module" would include a Form through which scrape parameters such as the primary URL, scraping rules for each field to be scraped, format of data to be extracted, and whether to use simple crawl (for sites without JavaScript) or complex crawl (for sites with content rendered in JavaScript). These parameters would be stored in a table, and accessed by the "Scraper Module" at run time. f) The scraped URLs (referred by the primary URL) to be saved in a Database Table with "processed flag", so that these can be skipped if scraping needs to be resumed after interruption. g) Primary URLs also to be saved with the date of last successful scraping, to enable scheduling of periodic repeat scrapings. h) While executing scraping, only those fields that have changed since last scrape are to be extracted and the original table entry for the product to be "updated", as required. In case of new products, the details to be "inserted" as a new row in the Products Table. i) Scrapy to be used with Selenium for effective scraping of sites with heavy JavaScript content. j) Performance must be adequate to enable scraping of the sites in order to generate the Products database k) There should also be a way of the URL to the comparison products within the website to be updated changed to affiliate links. Expected Skills: Web Scraping, Scrapy, Selenium,beautifulsoup,Python, Data Mining, Javascript, MySQL,Mongodb Budget: USD 200 to USD 300
ID Proyek: 9423766

Tentang proyek

8 proposal
Proyek remot
Aktif 8 tahun yang lalu

Ingin menghasilkan uang?

Keuntungan menawar di Freelancer

Tentukan anggaran dan garis waktu Anda
Dapatkan bayaran atas pekerjaan Anda
Uraikan proposal Anda
Gratis mendaftar dan menawar pekerjaan
8 freelancer menawar dengan rata-rata $537 USD untuk pekerjaan ini
Avatar Pengguna
Hello I am very interested with your project I have good skills and have many experiences in this area. I can start it immediately and can do it for you perfectly I wish to work for you Thanks
$1.030 USD dalam 10 hari
5,0 (141 ulasan)
8,1
8,1
Avatar Pengguna
Looking forward to discuss further and deliver the project to your specifications and requirements .
$555 USD dalam 10 hari
4,9 (73 ulasan)
6,9
6,9
Avatar Pengguna
Hello, We are a Web Design and Development company with expertise in latest and legacy technologies to create an awesome product for you. We have he best project management practices to ensure that your project is completed with 100% on time with 100% within budget and more than 100% exceeding quality.
$263 USD dalam 18 hari
0,0 (0 ulasan)
0,0
0,0
Avatar Pengguna
Hi Sir, I am having experience on developing price comparison tool, 1)Scrapping data from sites like flipkart ,snapdeal,amazon..etc and storing data in mysql. 2)Having experience on good tools that scraps data from websites that stores data in database or directly gives us API or REST service. 3)How to add affiliate links in product urls dynamically. 4)How to earn money from comparison script tool. 5)Good ui skills with bootstrap,jQuery and angularjs for users. We can provide the above features for applicationa dn try to make your application go live and give good business ideas for this application as we have good experience on this platform. We are not bidding this project for money, as we are trying for new project in freelancer , satisfying client is first requirement for us . Please give us an opportunity to work with you. Thanks, Sreekanth.
$666 USD dalam 15 hari
0,0 (0 ulasan)
0,0
0,0
Avatar Pengguna
I have mor than 10 years of experience on Web development, and about 3 years using Python and Django in projects for real estate, marketing, crm, web sites and others web applications.
$666 USD dalam 10 hari
0,0 (0 ulasan)
0,0
0,0

Tentang klien

Bendera SAUDI ARABIA
Saudi Arabia
0,0
0
Anggota sejak Mei 23, 2008

Verifikasi Klien

Terima kasih! Kami telah mengirim Anda email untuk mengklaim kredit gratis Anda.
Anda sesuatu yang salah saat mengirimkan Anda email. Silakan coba lagi.
Pengguna Terdaftar Total Pekerjaan Terpasang
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Memuat pratinjau
Izin diberikan untuk Geolokasi.
Sesi login Anda telah kedaluwarsa dan Anda sudah keluar. Silakan login kembali.