Senior Data Engineer

Data Science UA Responds quickly
$$$$
Product

Data Science UA is a service company with strong data science and AI expertise. Our journey began in 2016 with uniting top AI talents and organizing the first Data Science tech conference in Kyiv. Over the past 9 years, we have diligently fostered one of the largest Data Science & AI communities in Europe.

About the client: 
The company is a trailblazer in the world of data-driven advertising, known for its innovative approach to optimizing ad placements and campaign effectiveness through advanced analytics and machine learning techniques. Its mission is to revolutionize the advertising sector by enabling brands to reach their audiences more effectively.

About the role:
We are looking for an experienced and dynamic Data Engineer with a curious and creative mindset to join our client’s team. The ideal candidate will have a strong background in Python, SQL and large-scale data pipelines. 

 

As a Data Engineer you will design, build and operate the data platform that powers the company: pipelines that ingest and transform very large datasets from external data partners, data models that the whole company queries, and services that make that data available to internal products. 
 

Location: Ukraine is mandatory.

Responsibilities:
-Design and build batch data pipelines that ingest, validate and transform multi-billion-row datasets from external data providers and internal systems;
- Model complex real-world data: dimensional models, reference data, and temporal data whose attributes change over time (e.g. slowly changing dimensions), and evolve those models safely as upstream sources change their schemas and semantics;
- Develop and operate workloads on our lakehouse platform (Databricks / Spark / Delta) and our data warehouse (Redshift), including migrating existing pipelines from the warehouse to the lakehouse;
- Orchestrate pipelines with Airflow: scheduling, dependencies, retries, backfills and alerting;
- Prove correctness, not just completion: design parity checks and reconciliation queries when replacing an existing pipeline, run large historical backfills, and investigate data discrepancies down to the row level;
- Build and maintain Python services and REST APIs that serve data to internal products;
- Optimize for performance and cost: query tuning, table design, workload management and right-sizing compute;
- Own what you ship: monitor production pipelines, participate in incident triage and root-cause analysis, and harden systems so the same failure does not happen twice;
- Collaborate cross-functionally with product managers, data scientists and stakeholders across the company to deliver on product roadmap;
- Work within an Agile team that releases cutting-edge new features regularly;
- Take a high degree of ownership and freedom to experiment with new technologies to improve our software.

Requirements:
- Python as your primary language for pipelines and services in recent years;
- 3+ years of hands-on Spark (PySpark) in production batch pipelines: you wrote the jobs, deployed them through CI/CD and kept them running at scale;
- Production experience with Databricks: Jobs/Workflows, Delta Lake, and deploying through CI/CD. Notebook-only experience does not count. A strong candidate without Databricks must show deep production PySpark on AWS EMR instead;
- Expert-level SQL: writing, reading and tuning complex analytical queries against very large tables, and debugging why two result sets disagree;
- Amazon Redshift in production is strongly preferred. Otherwise, deep hands-on tuning of another MPP columnar warehouse: distribution and sort keys, workload management, and diagnosing slow queries from the query plan;
- Built and owned production Airflow DAGs: dependencies, retries, backfills and alerting;
- Ability to design complex data models: normalized, dimensional and temporal (slowly changing dimensions, effective-dated records, point-in-time correctness);
- Experience integrating third-party data feeds: schema drift, late or missing deliveries, vendor data-quality defects and versioned reference data;
- Experience building REST services in Python (FastAPI, Flask, etc.);
- Experience developing, maintaining and debugging large server-side code bases;
- Working knowledge of AWS (S3, IAM, ECS) and Docker;
- Good engineering practices: unit and integration tests, code review, CI/CD;
- The desire to take a high level of ownership, learn quickly, keep a high bar for quality and be pragmatic;
- Able to communicate effectively with U.S.-based teams and work 11:00 AM – 8:00 PM EEST (11:00–20:00).

 

Tech Stack:
- Almost everything we run is on AWS (S3, ECS, EMR, RDS and more)
- Python is our primary language; SQL is everywhere;
- Databricks (Spark, Delta Lake) is our lakehouse platform; - Redshift and Postgres are our warehouses and operational databases;
- Airflow orchestrates our pipelines;
- Docker for packaging; GitHub Actions and Jenkins for CI/CD;
- Grafana, Sentry and OpenSearch for observability;
- Datasets measured in billions of rows.

Required languages

English B2 - Upper Intermediate
Ukrainian Native
Published 30 October
133 views Β· 19 applications
Response activity: Very high
Last responded 1 day ago
See stats of candidates who applied for this job πŸ‘€
To apply for this and other jobs on Djinni login or signup.
Loading...