Find Jobs
Hire Freelancers

Import 10 million records 2GB .csv (5 parts) every hour to MySQL

$30-250 USD

Ditutup
Dibuat lebih dari 7 tahun yang lalu

$30-250 USD

Dibayar ketika dikirim
Import .csv files from here to MySQL database: [login to view URL] Must be fast, easy to run, cheap, complete the process in under 5mins. It updates the CSVs' every hour and makes note of the data that changes. What kind of server/mysql setup is needed on AWS? Let me know your estimations so I can judge how suitable you are for the job.
ID Proyek: 11724502

Tentang proyek

23 proposal
Proyek remot
Aktif 7 tahun yang lalu

Ingin menghasilkan uang?

Keuntungan menawar di Freelancer

Tentukan anggaran dan garis waktu Anda
Dapatkan bayaran atas pekerjaan Anda
Uraikan proposal Anda
Gratis mendaftar dan menawar pekerjaan
23 freelancers are bidding on average $166 USD for this job
Avatar Pengguna
Hello! I am ready to help you with your task! Do you need an unattended script to do this import job? Please do not hesitate to contact me via messenger!
$250 USD dalam 20 hari
5,0 (19 ulasan)
5,7
5,7
Avatar Pengguna
Hi, I read carefully your project, and I wanna do it for you. If you wanna work together, send me a message to discuss details. I am an experienced Web Developer and Web Designer, and have worked with wide variety of skills like PHP, MySQl, jQuery / Prototype, Javascript, HTML, CSS and more for many years. I also have worked over many Wordpress projects. Have good communication skills and great responsibility.
$150 USD dalam 11 hari
4,9 (71 ulasan)
5,5
5,5
Avatar Pengguna
Hello, I have extensive experience with MySQL and importing large datasets on set interval basis. From your information in the project, you mentioned that when the csv's are updated, there are notes of the data that changes, does this mean that the original import will be a full set and then there will be ongoing hourly delta (changes only) updates? Also, can you clarify if you already have a MySQL server instance running or are looking to establish a new one (AWS)? If you can send me some more information, I am sure we can work together for a solution. Thank you Darren
$250 USD dalam 5 hari
4,8 (13 ulasan)
5,5
5,5
Avatar Pengguna
hi, i can help you with your project. According to the site, notifies the date of the last update, so I do not see that you need to run a process every hour, but I can do. send me more information about via freelancer.com chat. regards
$333 USD dalam 5 hari
5,0 (14 ulasan)
4,1
4,1
Avatar Pengguna
there could be easier ways to do this.
$111 USD dalam 5 hari
5,0 (1 ulasan)
4,0
4,0
Avatar Pengguna
Hi I have been in this industry for 5 years and such jobs are my daily practice.I can assure you that if you work with me once, you will always work with me for these kind of projects.
$250 USD dalam 3 hari
5,0 (5 ulasan)
3,5
3,5
Avatar Pengguna
Hello, Thanks for reading my propusal. First of all let me tell you that loading such amount of data won't be fast. Since you need to do it in steps to prevent problems in DB. Secondly I would suggest you to read the link you sent. It says all info is updated ONCE a month every fifth day of month updating last month data. So doing this task every couple of hours is not really needed and won't be usefull. Beside from that importing CSV to MySQL database is not difficult. Let me know if you are interested and we discuss about real price for this kind of job. Greetings, Pablo
$100 USD dalam 20 hari
5,0 (3 ulasan)
3,1
3,1
Avatar Pengguna
Hi There, We can setup aws RDS instance which is highly available and the best performance database for mysql. Initially we would setup micro instance db if the queries are taking time to process then we can upgrade immediately. Lets get started. Regards, Imran Ahmed
$60 USD dalam 2 hari
5,0 (6 ulasan)
2,4
2,4
Avatar Pengguna
I will deliver a satisfactory job that will impress you. I crave for a chance to prove to you my worth and abilities.
$100 USD dalam 2 hari
0,0 (0 ulasan)
0,0
0,0
Avatar Pengguna
I noticed your project and I am interested in this work. I can do 100% quality with effectively. Kindly contact me. Thanks, G Anburaj
$166 USD dalam 10 hari
0,0 (0 ulasan)
0,0
0,0
Avatar Pengguna
A proposal has not yet been provided
$111 USD dalam 3 hari
0,0 (0 ulasan)
0,0
0,0
Avatar Pengguna
I have the experience in work load of large files with management in record time. Also I have experience handling government databases with extensive confidentiality.
$111 USD dalam 2 hari
0,0 (0 ulasan)
0,0
0,0
Avatar Pengguna
Hello. Assuming data integrity is good (fields consistency and no junks) this could be done fast. Standard MySQL setup is sufficient (we are not talking about data access and processing here, right?) as long as the AWS server instance processing power is set fast enough and the memory is big enough too. We could do this with simple scripting and run the script through cron scheduling. The note could be written to simple text file or a formatted one (.csv). Finally, remote SSH connection should be provided so I can enter the server and work remotely. I've done bulk data scraping and importing of millions of records through scripting and custom programming. I could do this job for you. Thanks.
$155 USD dalam 3 hari
0,0 (0 ulasan)
0,0
0,0
Avatar Pengguna
I am an expert of full-stack web development and have similar experience with your requirement. although I am new to here freelancer and earn achievements for future so i will bid reasonable price. My planning is below. 1. aws structure only one EC2 instance is enough i can install Linux,MySQL and some script language (php or python if we need) on instance. 2. how to run ill use cron-damon for kicking import script so you don't need to manipulate for importing data. 3. details of script from bash script i will call mysql command namely "Load data" which will import data from specific csv file. i have similar script written by php so i can apply this to this project if we need more complicated requirement. 4. in under 5mins regulation i have experience that import two hundred million data in csv to Mysql in under 1 hour, so i estimate i can import the whole records in under 5mins. also we need to consider other functions about download csv from websites and extract zip files. how much time we need to those functions are depend on EC2 spec, my recommendation is we will select t2.* Instance which is appropriate for this case. thank you.
$35 USD dalam 3 hari
0,0 (0 ulasan)
0,0
0,0
Avatar Pengguna
I would recommend to use: an rds with mysql or aurora as database with 20.000 provisioned IOps ebs volume (r3 instance family or I family depending on the needs of computing power of the queries). To import the file can then be used an application server running a python script or datapipeline + s3
$250 USD dalam 3 hari
0,0 (0 ulasan)
0,0
0,0

Tentang klien

Bendera UNITED KINGDOM
London, United Kingdom
5,0
37
Memverifikasi Metode pembayaran
Anggota sejak Mar 14, 2015

Verifikasi Klien

Terima kasih! Kami telah mengirim Anda email untuk mengklaim kredit gratis Anda.
Anda sesuatu yang salah saat mengirimkan Anda email. Silakan coba lagi.
Pengguna Terdaftar Total Pekerjaan Terpasang
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Memuat pratinjau
Izin diberikan untuk Geolokasi.
Sesi login Anda telah kedaluwarsa dan Anda sudah keluar. Silakan login kembali.