ETL Jobs

I need an experienced data engineer to design and implement fully automated data pipelines that pull from our production databases and deliver clean, query-ready tables to our analytics environment. The work is focused on Data Pipelines only—no warehousing architecture or ad-hoc analysis is required. Here is the scope I have in mind: • Connect to multiple relational databases (currently MySQL and PostgreSQL, with the possibility of others) and set up reliable, incremental extractions. • Transform and load the data so that downstream analysts can query consistent, well-documented tables. • Embed resilience features such as retry logic, data-quality checks, and schema-change handling. • Orchestrate everything on a modern scheduling platform (Airflow or an e...

## Microsoft Fabric / Power BI Expert Required We are looking for a highly experienced Microsoft Fabric / Power BI expert to enhance and optimize an enterprise analytics platform with advanced reporting, predictive analytics, and AI-driven insights. ### Scope of Work * Advanced Power BI dashboard development and optimization * Microsoft Fabric Data Lake / Data Warehouse implementation * RBAC, Multi-Tenant architecture, and security implementation * Performance tuning for Fabric F2 capacity * Predictive Analytics, RCA, Anomaly Detection, and forecasting * Integration with APIs, ERP, HRMS, IoT, and external systems * Enterprise-grade scalable architecture and governance setup ### Mandatory Requirements * Proven hands-on experience in Microsoft Fabric and Power BI enterprise implementati...

Hiring: Sr. Data Engineer Job ID: GKS1805_3 --- Role: Senior Data Engineer Mode: Work From Home Budget: ₹50,000 – ₹65,000/month Availability: Immediate Joiners Only Open Positions: 1 Interview Process: AI Interview → L1 → 2 Client Rounds --- Role Overview We are looking for a high-impact Senior Data Engineer to design and build scalable, multi-tenant data systems on AWS. You will work across data engineering, backend systems, APIs, and ML workflows, contributing to end-to-end product delivery. --- Key Responsibilities - Design & build scalable multi-tenant systems on AWS (Glue, Lambda, Step Functions, ECS, S3, DynamoDB, SageMaker) - Develop robust data pipelines (ETL/ELT) for high-volume datasets - Build Lambda-based microservi...

I currently run an in-house spreadsheet application that has outgrown its limits. I want to migrate everything into a purpose-built database that can accept multiple incoming feeds—CSV files arriving by batch, live API calls, and occasional manual entries (often straight from Excel). Before any record lands in the tables it needs to pass through an AI layer that cleans, classifies, and enriches the data. Once stored, different user groups must see only what their privilege level allows, so role-based security is essential. My short-term goal is to generate a print-ready publication from the data; longer term I will also push the same content to a website and, potentially, a mobile app. Deliverables • A relational schema that mirrors the existing spreadsheet logic whil...

I need a cloud-native ETL pipeline built end-to-end on AWS, coded in PySpark and designed for production reliability. The pipeline will ingest data from three sources—databases, APIs, and file systems—then standardise and load it into an analytics-ready destination. Source files arrive in a mix of CSV, JSON, and Parquet, so the job must include automatic format detection, schema inference, and efficient column-wise writes. Beyond raw transformation, I want solid engineering practices: parameter-driven jobs, modular Spark code, unit tests, logging, alerting, and retry logic. Leveraging AWS native services such as Glue, EMR, Lambda, and S3 is expected, but I’m open to other AWS components if they shorten development time or lower cost. Candidates must have expertise in da...

I’m setting up an intensive, expert-level course for a team that already designs and operates complex AWS workloads. The objective is to deepen their Data Engineering skills on AWS, so we need a trainer who can move straight past fundamentals and into real-world architecture, scaling strategies, and cost-efficient design. Core focus: Data Engineering with AWS, specifically • Data Storage Solutions • Data Processing & ETL • Big Data Analytics Your role is to craft and deliver a hands-on program that blends concise theory, architectural white-boarding, and live labs. Glue, EMR, Redshift, Lake Formation, Kinesis, Lambda, Step Functions, and related services should feature prominently, but I’m open to your proven toolset and workflows. Deliverables 1...

Over the coming weeks I want to master three core AWS services—EC2, S3 glue and Lambda—and immediately apply them to real-world ETL testing scenarios. The entire program needs to be delivered as live online classes so I can ask questions in real time and work through hands-on labs while you guide me. On the ETL side, the agenda should give equal attention to data validation, data transformation and performance testing. I’m aiming to finish the course able to design, automate and benchmark an end-to-end AWS-based ETL pipeline with confidence. Please structure the engagement around practical demonstration and practice rather than slides alone, using the AWS console, CLI or Terraform where appropriate. To keep us both on track, here’s what I expect to receive: &bull...

Looking for BlackRock Aladdin Experts with Implementation experience. Data engineer, technical Business Analyst, please detail your specific experience. I need hands-on practitioners who have delivered full Aladdin Implementations, with a sharp focus on Performance and FactSet integration. Your background must show you have handled Data Mapping, carried out precise System Configuration, and driven rigorous Testing and Validation for this specific stream. Please reference concrete projects where you owned or significantly contributed to these areas, noting any challenges you solved around data pipelines, integration touchpoints, and post-go-live support. Include the tools and techniques you relied on inside the Aladdin ecosystem.

I need to move an entire set of database records from a proprietary legacy archive into a new cloud-based content management system. The source isn’t MySQL, PostgreSQL, or Oracle—it’s a home-grown format—so the job starts with understanding and reverse-engineering its schema and export routines. From there we’ll design an ETL pipeline that transforms each field cleanly into the structures required by the target CMS while preserving every relation, timestamp, and audit trail. Deliverables • Technical mapping document that explains each field’s journey from legacy tables to the cloud schema • Re-usable migration scripts or jobs (Python, SQL, or the ETL tool of your choice) stored in a version-controlled repo • One full test migration ...

Dear Data Engineers, We are currently looking for mutliple Data Engineers for one of the leading tech companies in Europe. The position is 100% remote and long-term. QUICK FACTS: - Freelancing/Contracting - Start: ASAP - 100% remote - Duration: long-term 12+ months - Capacity: full-time, part-time also possible - Language: English (no german needed!!) - German business hours REQUIREMENTS: - Experience in data engineering, ideally in Microsoft Azure environments. - Knowledge of Databricks, Apache Spark, and Medallion Architecture. - Experience with Delta Lake / Delta Tables, including concepts such as Row-Level Security. - Ability to work with Spark DataFrames, Spark SQL, and UDFs. - Good Python skills for data processing, transformation, and pipeline development. Are you interested? I...

I need a clean, well-tested Python pipeline that listens to our MongoDB (DocumentDB) product collection, transforms each incoming SKU document, and pushes the result into Apache Solr so the search index is always in lock-step with the source data. What the transform must do • Enrich taxonomy on two fronts: category as well as attribute level. • Keep only English content for now, but structure the code so Spanish or French could be switched on with minimal effort. • Filter out the fields we don’t want surfaced—specifically the SKU number, product description and price—while leaving all other data intact. • Merge duplicates that share the same SKU logic we will provide. Runtime behaviour The service should behave like a sink connector: near...

I’m looking for an experienced ETL specialist (10 + years) who can step in as my on-call partner during my working hours, 8 am – 3 pm CST, all online and paid monthly. I work in a finance environment that relies heavily on Snowflake and DBT, so the focus is on keeping daily data pipelines healthy, performant, and well-tested. During each session you will help me: • Extract, transform, and load data from multiple sources. • Configure, tune, and troubleshoot Snowflake warehouses. • Design, refactor, and test DBT models. The tech stack you’ll need to be fully comfortable with includes: • SQL for complex queries and performance tuning. • Python for ad-hoc scripts, utilities, and automation. • Apache Spark for the occasional large-scale ...