Jobs Data Engineer

Position Description:
As a Data Engineer, you will play a key role in designing and maintaining scalable data pipelines and platforms.
You’ll work closely with data scientists, analysts, and product teams to ensure high-quality, reliable, and scalable data is always available. This role requires deep technical expertise, strong analytical skills, and exposure to emerging AI/LLM technologies
Location(s): Germany & EU
Working Model: remote

Role Overview
As a Senior Data Engineer, you will play a key role in designing and maintaining scalable data pipelines and platforms. You’ll work closely with
data scientists, analysts, and product teams to ensure high-quality, reliable, and scalable data is always available. This role requires deep
technical expertise, strong analytical skills, and exposure to emerging AI/LLM technologies.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Python and PySpark.
• Build, manage, and optimize big data solutions leveraging Google Cloud Platform (BigQuery, DataProc, Dataflow).
• Collaborate with cross-functional teams to translate business requirements into efficient data models and workflows.
• Ensure data accuracy, consistency, and availability across multiple systems and applications.
• Implement best practices for data governance, quality, and security.
• Stay updated with advancements in big data, cloud, and AI technologies, recommending innovative solutions.
• Mentor junior engineers and establish best practices within the team.

Requirements for "Professional Skills"
All professional skills required to fulfill the following general task description:
Works as a developer and is responsible for the development and programming of software, tools, systems, etc., as well as the associated troubleshooting and error analysis. This role requires comprehensive knowledge of various programming languages as well as appropriate development tools, methodologies, and development processes.

Works on the integration of systems, applications, or databases with one another or into other systems/software. This role requires exceptional knowledge of scripting and various programming languages, along with relevant integration tools, methods, and processes. It also requires a strong understanding of the technologies or systems being integrated.

Also works as an engineer and documents operational requirements in the form of system performance parameters and system configurations. He/she is additionally responsible for ensuring the seamless integration of architecture and development.

This also includes system integration or network integration activities, for example, SDM.

Projects using Agile development methodology:
If the contractor works within a Scrum team, they possess extensive knowledge of Scrum methodology and have experience in developing software within such a framework.

Requirements for "Social Skills"
All social skills required to fulfill the previously mentioned general task description as well as the following requirements:
The employee has intermediate-level experience, typically between 2 and 5 years, in their role or in the tasks performed by the group. Alternatively, the role and responsibilities require solid, but not exceptional, expertise.
The activities are associated with a moderate level of complexity and responsibility and require the ability to work independently.
The project may be carried out at either a national or international level.

Other Requirements & Relativizations
Skill Sets
Core Technical Skills

• Python (expert-level coding, modular programming, debugging, performance optimization)
• PySpark (distributed processing, tuning, handling large-scale data, performance optimization)
• Google Cloud Platform (GCP):
• BigQuery – advanced SQL, optimization, partitioning, clustering, cost management
• DataProc – Spark/Hadoop cluster management, job orchestration
• Dataflow – streaming & batch pipelines, real-time processing

Big Data & Cloud Skills

• Strong understanding of big data technologies (distributed computing, Hadoop/Spark ecosystem, data lakes, Kafka, batch & stream processing)
• Experience with data modeling and designing scalable data pipelines
• Familiarity with cloud-native architectures and serverless processing
•  

Analytical & Problem-Solving Skills:

• Ability to analyze complex data sets and derive insights
• Strong logical and critical thinking
• Experience optimizing query performance and pipeline efficiency
• Comfortable troubleshooting large-scale data workflows
• Preferred / Nice-to-Have Skills
• Experience with CI/CD pipelines for data engineering workflows
• Workflow orchestration: Airflow, Cloud Composer, Dagster
• Data governance, data quality frameworks, and compliance standards
• AI & Advanced Analytics Exposure
• Familiarity with Large Language Models (LLMs) and their applications in data workflows
• General understanding of AI/ML concepts and how they integrate with data engineering pipelines
• Exposure to MLOps tools or frameworks for deploying and monitoring AI models

Soft Skills

• Strong communication and collaboration with cross-functional teams
• Ability to work in Agile environments
• Mentoring and guiding junior engineers
• Self-driven, proactive learner, and adaptable to emerging technologies

We're looking for a Computer Vision Engineer to build visual intelligence features for a product that runs across connected devices and mobile apps.

This is a hands-on engineering role, not a research one: you'll take CV models from training to production, with a strong focus on making them run fast on resource-constrained devices.

What you'll do:

• Design, train, and deploy CV models - object detection, classification, segmentation, visual analysis;
• Optimize models for edge and mobile inference (quantization, pruning, distillation);
• Build data pipelines for image and video processing, including annotation and dataset versioning;
• Set up monitoring and retraining workflows for models in production;
• Work with backend and mobile teams to integrate model outputs into user-facing features.

Requirements:

• 3+ years in applied computer vision or ML with a strong CV focus;
• Solid deep learning background: CNNs, detection and segmentation architectures (YOLO, Faster R-CNN, U-Net), vision transformers;
• Python, PyTorch or TensorFlow, OpenCV;
• Experience deploying and maintaining CV models in production;
• Hands-on with on-device inference: ONNX, TensorRT, TensorFlow Lite, Core ML, or OpenVINO;
• Cloud platforms (GCP, AWS, or Azure) and container-based deployments;
• English: Intermediate or higher.

Nice to have:

• MLOps tools (MLflow, Airflow, Vertex AI);
• Real-time video processing at scale (FFmpeg, GStreamer, Kafka);
• Edge AI or mobile ML experience.

A second hands-on senior engineering role alongside Senior Data Engineer #1, building and maintaining production data pipelines on one cloud platform — typically covering a different workstream or cloud stack within the same data engineering team.

Key Responsibilities:

• Design, build, and maintain scalable ETL/ELT pipelines on AWS or Azure
• Optimize pipeline performance, reliability, and cost-efficiency
• Implement data-quality checks, monitoring, and alerting for production pipelines
• Collaborate with the Data Engineering Lead and Architect on pipeline design and standards
• Support data modeling and schema design together with the Data Modelling Analyst
• Troubleshoot and resolve data pipeline issues in production
• Participate in code reviews and contribute to engineering best practices

Requirements:

• 5+ years of hands-on data engineering experience
• Deep expertise in one cloud platform: AWS (Glue, S3, Redshift, EMR, Lambda, Kinesis) or Azure (Data Factory, Synapse, ADLS, Databricks)
• Strong SQL and Python (or Scala) skills; experience with orchestration tools (Airflow, ADF, etc.)
• Experience with ETL/ELT design, data-quality checks, and performance optimization
• Familiarity with distributed data processing frameworks (Spark)
• Experience operating production data pipelines at scale
• Solid understanding of data warehousing and dimensional modeling concepts

Nice to Have:

• Familiarity with CI/CD and infrastructure-as-code (Terraform)
• Experience with streaming data pipelines (Kafka, Kinesis, Event Hubs)
• Exposure to dbt or similar transformation tooling

Leads a team of data engineers delivering pipelines and platform components across AWS and Azure, balancing hands-on technical work with people leadership. Partners with the Data Architect and business stakeholders to keep the team's roadmap aligned with platform architecture and delivery priorities.

Key Responsibilities:

• Lead and mentor a team of data engineers delivering pipelines across AWS and Azure
• Own technical design and implementation decisions for data pipeline architecture
• Establish and enforce engineering best practices: code quality, testing, CI/CD, documentation
• Plan and prioritize the team's roadmap in collaboration with architecture and business stakeholders
• Troubleshoot and resolve complex data pipeline and platform issues
• Drive adoption of orchestration, monitoring, and data-quality tooling
• Partner with the Data Architect to keep delivered pipelines aligned with platform architecture

Requirements:

• 7+ years of data engineering experience, including 2+ years in a technical leadership role
• Hands-on expertise with both AWS (Glue, EMR, Redshift, Lambda, Kinesis) and Azure (Data Factory, Synapse, Databricks)
• Strong background in pipeline architecture, orchestration (Airflow/ADF), and CI/CD for data
• Experience mentoring engineers and setting technical best practices across a team
• Solid SQL and Python/Scala skills, with experience in distributed data processing (Spark)
• Experience with data quality, testing, and monitoring frameworks
• Strong communication and planning skills; comfortable working directly with stakeholders

Nice to Have:

• Experience with dbt or similar transformation frameworks
• Familiarity with infrastructure-as-code (Terraform, CloudFormation, Bicep)
• Prior experience in a consulting or client-facing delivery environment

Owns the end-to-end data architecture across both AWS and Azure — from storage and ingestion patterns to governance and security. Works closely with engineering leads and business stakeholders to define a target-state architecture and evaluate the tools and services that support it, and provides architectural oversight as the platform is built out.

Key Responsibilities:

• Define and own the target-state data architecture across AWS and Azure environments
• Design data lake / data warehouse solutions, including storage layers, ingestion patterns, and processing frameworks
• Establish data governance, security, and access-control standards across both cloud platforms
• Translate business requirements into scalable, well-documented technical designs
• Evaluate and select tools/services across AWS and Azure, balancing cost, performance, and maintainability
• Provide architectural guidance and review to engineering teams during implementation
• Support cloud migration and multi-cloud integration initiatives

Requirements:

• 8+ years in data architecture/engineering, including senior/lead-level design responsibility
• Hands-on architecture experience with both AWS (S3, Redshift, Glue, Lake Formation, Athena) and Azure (ADLS, Synapse, Data Factory, Databricks)
• Proven track record designing large-scale data lakes/warehouses and enterprise data governance frameworks
• Strong grasp of data modeling, ETL/ELT design, and distributed data processing (Spark)
• Experience with data security, compliance, and access management across cloud platforms
• Track record leading multi-cloud or cloud-migration data initiatives
• Excellent stakeholder communication; able to present architecture decisions to technical and business audiences

Nice to Have:

• Cloud certifications (AWS Certified Solutions Architect, Azure Solutions Architect Expert)
• Experience with data mesh or domain-driven data architecture
• Background in a regulated industry (finance, healthcare, etc.)

About Deep Knowledge Group

Deep Knowledge Group (https://www.dkv.global/) builds large-scale analytical databases, investment-intelligence systems, industry ecosystem platforms, competitiveness indices and interactive dashboards across frontier industries including AI, Longevity, HealthTech, DeepTech and QuantumTech.

The Data Science department builds and maintains the industry intelligence databases behind those products — organisations, people, products and the relationships between them, across multiple sectors and regions. Every one of them starts as messy public data and has to end up as a clean, deduplicated, verifiable dataset that analysts and downstream products can rely on.
 

About the Role

We currently build each database the way the last one was built. Collectors are source-specific scripts, scheduling is manual, quality is whatever the person who built it checked, and nothing links one database to another. That works at our current size and will not work at the next one.

We are looking for a Senior Data Engineer to own the shared platform that replaces this: one collection and enrichment framework that every database runs on, with orchestration, monitoring, provenance and entity resolution as properties of the system rather than as things individual engineers remember to do.

This is an ownership role. You will work directly with the Head of Data Science on architecture, and you will be the person other engineers bring design decisions to. We are not looking for an extra pair of hands; we are looking for a second point of technical decision-making in the department.
 

Key Responsibilities

• Design and own the shared collection and enrichment framework: what is reusable, what stays source-specific, and where the line between them sits.
• Own orchestration end to end — scheduling, dependencies, retries, backfills, incremental and idempotent re-runs.
• Make entity resolution a permanent service rather than a per-project effort: blocking, matching, thresholds, and a defensible account of why a merge was made.
• Design the provenance model: every value traceable to its source, its retrieval date and the logic that transformed it, six months after the fact.
• Establish the staging-and-promotion rule and enforce it: nothing writes to production tables directly.
• Build monitoring and observability for data, not only for jobs — freshness, coverage, null rates, distribution drift, source schema changes.
• Define the quality standard: accuracy, completeness, duplicate rate, freshness, provenance — and the instrument that measures it.
• Set the schemas, conventions and review standards that the rest of the department works to.
• Design the deterministic backbone of our agent network, and decide where an agent is warranted and where plain code is cheaper, faster and testable.
• Migrate existing databases onto the shared framework alongside the engineers who built them.
• Mentor and unblock other engineers; take architectural decisions off the Head of Department's desk.
   

Required Skills

• Strong Python — you write services and pipelines, not notebooks.
• You have owned a data platform, not just tasks on one — orchestration, scheduling, monitoring, backfills.
• Airflow, Dagster or Prefect in production, with a considered view on why that one.
• SQL and PostgreSQL in depth — indexing, query plans, schema design, migrations.
• Entity resolution or record linkage across sources that disagree with each other, including how you measured whether it worked.
• You have set the standards other people then worked to: schemas, conventions, review.
• Docker, CI/CD, and comfort running production systems on plain infrastructure rather than a managed platform.
• Practical experience with LLM APIs and with validating non-deterministic output before it reaches a production table.
• Data quality discipline: you can state a dataset's duplicate rate or field staleness with evidence, not by inspection.
• The ability to write a design down clearly enough that someone can argue with it.
   

Strong Advantages

• Production collection from public web sources: anti-bot, JS-rendered pages, undocumented APIs, source drift.
• Knowledge graphs, ontologies and canonical entity models across multiple domains.
• Data lineage and governance tooling in production, used by people other than its author.
• Agent orchestration frameworks, and a clear view of their limits.
• Vector search and RAG pipelines.
• Experience turning a team's ad-hoc scripts into a framework that team then adopted.
• OSINT methodology and source verification.
   

What We Will Evaluate

Two things, mostly.

You question your own data. The engineers who work out well here are the ones who notice that a source silently changed its schema, that a 97% match rate is hiding a systematic bias, that a field is populated but wrong. Volume without verification is worth nothing to us.

You explain your reasoning. Much of this work involves judgment calls that nobody can check quickly — which record wins a merge, what confidence threshold to use, when to stop enriching. We need those decisions written down and defensible, not buried in a script.

The hiring process is an intro call, a short practical task on real data, a technical review of your solution, and an offer. We evaluate the task on judgment, not polish: reproducibility, honest handling of edge cases, and clear reasoning about what you chose not to do count for more than a clever one-liner.

Project Description:

This is a strategic data engineering engagement with our client to architect and plan the migration of their entire data processing and ETL estate from Matillion to AWS Glue — a foundational shift in how one of the world's largest financial market infrastructure companies handles its data pipelines.
During this Mobilisation Phase, you'll work jointly with client's engineering teams to reverse-engineer the existing landscape, design the target-state architecture across every layer (infrastructure, data processing, workflows, dependencies, and operating model), and build the detailed delivery blueprint that will greenlight the full-scale migration.
The work is technically rich and highly collaborative: you'll review and validate job inventories spanning hundreds of ETL workflows, define reusable migration patterns and templates, design a validation and reconciliation framework, run a proof-of-concept to stress-test the approach, and navigate client's rigorous internal governance — from Architectural Significance Assessments through Architectural Review Boards to a formal Gate 1 decision.
This is the kind of engagement where your recommendations directly shape a multi-phase, multi-million-pound programme: the target-state framework you produce here becomes the blueprint that a larger delivery team will execute against. Perfect opportunity for combining deep data engineering knowledge with architecture leadership, stakeholder influence, and structured delivery planning inside a Tier 1 financial services environment

Responsibilities:

- Design, build, and maintain ETL/ELT pipelines on AWS
- Migrate existing data pipelines and workloads from legacy/on-prem systems to AWS
- Develop and optimize data models for data warehouse/data lake (Redshift, S3)
- Build and orchestrate data workflows (Glue, Step Functions, Lambda, EMR)
- Implement data quality checks, validation, and reconciliation processes
- Optimize pipeline performance and manage compute/storage costs
- Ensure data security and access control (IAM, KMS, encryption)
- Monitor and troubleshoot pipeline failures (CloudWatch, logging)
- Collaborate with data analysts, BI developers, and architects on data requirements
- Document data pipelines, architecture, and operational runbooks

Mandatory Skills Description:

- 5+ years of experience
- Hands-on experience building data pipelines on AWS (Glue, EMR, Lambda, Step Functions)
- Strong SQL and experience with data warehousing (Redshift, dimensional modeling)
- Proficiency in Python or Scala for data engineering
- Experience with S3-based data lake architecture (partitioning, cataloging, formats like Parquet)
- Experience migrating data pipelines from on-prem or other cloud platforms
- Understanding of data governance, security, and access control on AWS
- Experience with CI/CD for data pipelines

Nice-to-Have Skills Description:

- AWS certification (Data Analytics Specialty or Data Engineer Associate)
- Experience with streaming data (Kinesis, MSK/Kafka)
- Familiarity with Infrastructure as Code (Terraform/CloudFormation)
- Experience with orchestration tools (Airflow, dbt)
- Experience in financial domain

Languages:

English: C2 Proficient

Project Description:

This is a strategic data engineering engagement with our client to architect and plan the migration of their entire data processing and ETL estate from Matillion to AWS Glue — a foundational shift in how one of the world's largest financial market infrastructure companies handles its data pipelines.
During this Mobilisation Phase, you'll work jointly with client's engineering teams to reverse-engineer the existing landscape, design the target-state architecture across every layer (infrastructure, data processing, workflows, dependencies, and operating model), and build the detailed delivery blueprint that will greenlight the full-scale migration.
The work is technically rich and highly collaborative: you'll review and validate job inventories spanning hundreds of ETL workflows, define reusable migration patterns and templates, design a validation and reconciliation framework, run a proof-of-concept to stress-test the approach, and navigate client's rigorous internal governance — from Architectural Significance Assessments through Architectural Review Boards to a formal Gate 1 decision.
This is the kind of engagement where your recommendations directly shape a multi-phase, multi-million-pound programme: the target-state framework you produce here becomes the blueprint that a larger delivery team will execute against. Perfect opportunity for combining deep data engineering knowledge with architecture leadership, stakeholder influence, and structured delivery planning inside a Tier 1 financial services environment

Responsibilities:

- Design, build, and maintain ETL/ELT pipelines on AWS
- Migrate existing data pipelines and workloads from legacy/on-prem systems to AWS
- Develop and optimize data models for data warehouse/data lake (Redshift, S3)
- Build and orchestrate data workflows (Glue, Step Functions, Lambda, EMR)
- Implement data quality checks, validation, and reconciliation processes
- Optimize pipeline performance and manage compute/storage costs
- Ensure data security and access control (IAM, KMS, encryption)
- Monitor and troubleshoot pipeline failures (CloudWatch, logging)
- Collaborate with data analysts, BI developers, and architects on data requirements
- Document data pipelines, architecture, and operational runbooks

Mandatory Skills Description:

- 5+ years of experience
- Hands-on experience building data pipelines on AWS (Glue, EMR, Lambda, Step Functions)
- Strong SQL and experience with data warehousing (Redshift, dimensional modeling)
- Proficiency in Python or Scala for data engineering
- Experience with S3-based data lake architecture (partitioning, cataloging, formats like Parquet)
- Experience migrating data pipelines from on-prem or other cloud platforms
- Understanding of data governance, security, and access control on AWS
- Experience with CI/CD for data pipelines

Nice-to-Have Skills Description:

- AWS certification (Data Analytics Specialty or Data Engineer Associate)
- Experience with streaming data (Kinesis, MSK/Kafka)
- Familiarity with Infrastructure as Code (Terraform/CloudFormation)
- Experience with orchestration tools (Airflow, dbt)
- Experience in financial domain

• This is a strategic data engineering engagement with our client to architect and plan the migration of their entire data processing and ETL estate from Matillion to AWS Glue — a foundational shift in how one of the world's largest financial market infrastructure companies handles its data pipelines.
  During this Mobilisation Phase, you'll work jointly with client's engineering teams to reverse-engineer the existing landscape, design the target-state architecture across every layer (infrastructure, data processing, workflows, dependencies, and operating model), and build the detailed delivery blueprint that will greenlight the full-scale migration.
  The work is technically rich and highly collaborative: you'll review and validate job inventories spanning hundreds of ETL workflows, define reusable migration patterns and templates, design a validation and reconciliation framework, run a proof-of-concept to stress-test the approach, and navigate client's rigorous internal governance — from Architectural Significance Assessments through Architectural Review Boards to a formal Gate 1 decision.
  This is the kind of engagement where your recommendations directly shape a multi-phase, multi-million-pound programme: the target-state framework you produce here becomes the blueprint that a larger delivery team will execute against. Perfect opportunity for combining deep data engineering knowledge with architecture leadership, stakeholder influence, and structured delivery planning inside a Tier 1 financial services environment

   

• Responsibilities:

  - Agentic AI Migration Framework development — design and build an agentic AI framework to automate and accelerate migration workflowsж
  - Pattern library build — develop and maintain a library of reusable migration patterns/templates for common conversion scenarios;
  -Automated conversion pipeline — build pipelines that automatically convert/transform code and configurations (e.g., Matillion jobs) as part of the migration process;
  -GitHub Copilot integration for code generation — integrate GitHub Copilot into the migration workflow to accelerate code generation and conversion tasks;
  -Support factory pods — provide technical support, tooling, and troubleshooting to migration factory pods executing migrations at scale

   

• Mandatory Skills Description:

  - 2 month project with posibility of extentention

  - Python — strong hands-on Python development experience for building tools and automation
  - GenAI/LLM integration — experience integrating GenAI/LLM capabilities into applications, workflows, or tooling
  - Prompt engineering — hands-on experience designing, testing, and optimizing prompts for LLM-based tools
  - AWS Bedrock — hands-on experience with AWS Bedrock for building GenAI-powered applications
  - GitHub Copilot — practical experience using and/or integrating GitHub Copilot into development workflows
  - Automation frameworks — experience designing and building automation frameworks/tooling from scratch
  - Testing — strong experience with test automation and validation frameworks to ensure conversion accuracy
  - Matillion JSON parsing — experience parsing, interpreting, and transforming Matillion job definitions (JSON format)

   

• Nice-to-Have Skills Description:

  Experience in financial domain

   

• Languages:
  • English: C2 Proficient

Project Description:

This is a strategic data engineering engagement with our client to architect and plan the migration of their entire data processing and ETL estate from Matillion to AWS Glue — a foundational shift in how one of the world's largest financial market infrastructure companies handles its data pipelines.
During this Mobilisation Phase, you'll work jointly with client's engineering teams to reverse-engineer the existing landscape, design the target-state architecture across every layer (infrastructure, data processing, workflows, dependencies, and operating model), and build the detailed delivery blueprint that will greenlight the full-scale migration.
The work is technically rich and highly collaborative: you'll review and validate job inventories spanning hundreds of ETL workflows, define reusable migration patterns and templates, design a validation and reconciliation framework, run a proof-of-concept to stress-test the approach, and navigate client's rigorous internal governance — from Architectural Significance Assessments through Architectural Review Boards to a formal Gate 1 decision.
This is the kind of engagement where your recommendations directly shape a multi-phase, multi-million-pound programme: the target-state framework you produce here becomes the blueprint that a larger delivery team will execute against. Perfect opportunity for combining deep data engineering knowledge with architecture leadership, stakeholder influence, and structured delivery planning inside a Tier 1 financial services environment

Responsibilities:

- Agentic AI Migration Framework development — design and build an agentic AI framework to automate and accelerate migration workflowsж
- Pattern library build — develop and maintain a library of reusable migration patterns/templates for common conversion scenarios;
-Automated conversion pipeline — build pipelines that automatically convert/transform code and configurations (e.g., Matillion jobs) as part of the migration process;
-GitHub Copilot integration for code generation — integrate GitHub Copilot into the migration workflow to accelerate code generation and conversion tasks;
-Support factory pods — provide technical support, tooling, and troubleshooting to migration factory pods executing migrations at scale

Mandatory Skills Description:

- Python — strong hands-on Python development experience for building tools and automation
- GenAI/LLM integration — experience integrating GenAI/LLM capabilities into applications, workflows, or tooling
- Prompt engineering — hands-on experience designing, testing, and optimizing prompts for LLM-based tools
- AWS Bedrock — hands-on experience with AWS Bedrock for building GenAI-powered applications
- GitHub Copilot — practical experience using and/or integrating GitHub Copilot into development workflows
- Automation frameworks — experience designing and building automation frameworks/tooling from scratch
- Testing — strong experience with test automation and validation frameworks to ensure conversion accuracy
- Matillion JSON parsing — experience parsing, interpreting, and transforming Matillion job definitions (JSON format)

Nice-to-Have Skills Description:

Experience in financial domain

This is a strategic data engineering engagement with our client to architect and plan the migration of their entire data processing and ETL estate from Matillion to AWS Glue — a foundational shift in how one of the world's largest financial market infrastructure companies handles its data pipelines.
During this Mobilisation Phase, you'll work jointly with client's engineering teams to reverse-engineer the existing landscape, design the target-state architecture across every layer (infrastructure, data processing, workflows, dependencies, and operating model), and build the detailed delivery blueprint that will greenlight the full-scale migration.
The work is technically rich and highly collaborative: you'll review and validate job inventories spanning hundreds of ETL workflows, define reusable migration patterns and templates, design a validation and reconciliation framework, run a proof-of-concept to stress-test the approach, and navigate client's rigorous internal governance — from Architectural Significance Assessments through Architectural Review Boards to a formal Gate 1 decision.
This is the kind of engagement where your recommendations directly shape a multi-phase, multi-million-pound programme: the target-state framework you produce here becomes the blueprint that a larger delivery team will execute against. Perfect opportunity for combining deep data engineering knowledge with architecture leadership, stakeholder influence, and structured delivery planning inside a Tier 1 financial services environment

• Responsibilities:

  - Design, build, and maintain ETL/ELT pipelines on AWS
  - Migrate existing data pipelines and workloads from legacy/on-prem systems to AWS
  - Develop and optimize data models for data warehouse/data lake (Redshift, S3)
  - Build and orchestrate data workflows (Glue, Step Functions, Lambda, EMR)
  - Implement data quality checks, validation, and reconciliation processes
  - Optimize pipeline performance and manage compute/storage costs
  - Ensure data security and access control (IAM, KMS, encryption)
  - Monitor and troubleshoot pipeline failures (CloudWatch, logging)
  - Collaborate with data analysts, BI developers, and architects on data requirements
  - Document data pipelines, architecture, and operational runbooks

   

• Mandatory Skills Description:

  - 5+ years of experience

  - 2 month project with posibility of extentention
  - Hands-on experience building data pipelines on AWS (Glue, EMR, Lambda, Step Functions)
  - Strong SQL and experience with data warehousing (Redshift, dimensional modeling)
  - Proficiency in Python or Scala for data engineering
  - Experience with S3-based data lake architecture (partitioning, cataloging, formats like Parquet)
  - Experience migrating data pipelines from on-prem or other cloud platforms
  - Understanding of data governance, security, and access control on AWS
  - Experience with CI/CD for data pipelines

   

• Nice-to-Have Skills Description:

  - AWS certification (Data Analytics Specialty or Data Engineer Associate)
  - Experience with streaming data (Kinesis, MSK/Kafka)
  - Familiarity with Infrastructure as Code (Terraform/CloudFormation)
  - Experience with orchestration tools (Airflow, dbt)
  - Experience in financial domain

• Languages:
  • English: C2 Proficient

Project Description:

This is a strategic data engineering engagement with our client to architect and plan the migration of their entire data processing and ETL estate from Matillion to AWS Glue — a foundational shift in how one of the world's largest financial market infrastructure companies handles its data pipelines.
During this Mobilisation Phase, you'll work jointly with engineering teams to reverse-engineer the existing landscape, design the target-state architecture across every layer (infrastructure, data processing, workflows, dependencies, and operating model), and build the detailed delivery blueprint that will greenlight the full-scale migration.
The work is technically rich and highly collaborative: you'll review and validate job inventories spanning hundreds of ETL workflows, define reusable migration patterns and templates, design a validation and reconciliation framework, run a proof-of-concept to stress-test the approach, and navigate rigorous internal governance — from Architectural Significance Assessments through Architectural Review Boards to a formal Gate 1 decision.

Responsibilities:

-Target-state architecture design — define and own the target-state AWS architecture across compute, data, and orchestration layers
-HLD/LLD ownership — produce and own High-Level Design and Low-Level Design documentation for migration workloads
-ARB submissions — prepare and present architecture proposals to the Architecture Review Board for approval
-Technical decisions — make and document key technical decisions across the migration program, balancing trade-offs (cost, performance, scalability)
Pattern definition — define reusable architecture and migration patterns for use across teams/pods
-Agentic AI framework architecture — design the architecture for the Agentic AI migration framework, including its integration with the broader technical landscape

Mandatory Skills:

AWS Database Migration Service

Mandatory Skills Description:

- 5+ years experience
- hands-on experince with AWS Migration
- AWS Glue, Step Functions, Lambda, EventBridge — deep hands-on experience designing solutions using these core AWS services
- IaC (Terraform) — strong experience defining and managing infrastructure as code with Terraform
- Data platform architecture — proven experience architecting data platforms (data lakes, warehouses, pipelines) at scale
- PySpark — hands-on experience with PySpark for large-scale data processing
- CI/CD — experience designing and implementing CI/CD pipelines for infrastructure and data workflows
- AI/ML frameworks — working knowledge of AI/ML frameworks and their application within architecture design
- GitHub Copilot — practical experience using GitHub Copilot within development/architecture workflows

Nice-to-Have Skills Description:

- AWS certification (Data Analytics Specialty or Solutions Architect)
- Experience with BI tools (QuickSight, Tableau, Power BI)
- Infrastructure as Code experience (Terraform/CloudFormation)
- Industry experience relevant to your business
- Exposure to streaming data (Kinesis) or ML pipelines (SageMaker)
- experince in financial domain

Languages:

English: C2 Proficient

Project description

This is a strategic data engineering engagement with our client to architect and plan the migration of their entire data processing and ETL estate from Matillion to AWS Glue — a foundational shift in how one of the world's largest financial market infrastructure companies handles its data pipelines.
During this Mobilisation Phase, you'll work jointly with engineering teams to reverse-engineer the existing landscape, design the target-state architecture across every layer (infrastructure, data processing, workflows, dependencies, and operating model), and build the detailed delivery blueprint that will greenlight the full-scale migration.
The work is technically rich and highly collaborative: you'll review and validate job inventories spanning hundreds of ETL workflows, define reusable migration patterns and templates, design a validation and reconciliation framework, run a proof-of-concept to stress-test the approach, and navigate rigorous internal governance — from Architectural Significance Assessments through Architectural Review Boards to a formal Gate 1 decision.

Responsibilities

-Target-state architecture design — define and own the target-state AWS architecture across compute, data, and orchestration layers

-HLD/LLD ownership — produce and own High-Level Design and Low-Level Design documentation for migration workloads

-ARB submissions — prepare and present architecture proposals to the Architecture Review Board for approval

-Technical decisions — make and document key technical decisions across the migration program, balancing trade-offs (cost, performance, scalability)

Pattern definition — define reusable architecture and migration patterns for use across teams/pods

-Agentic AI framework architecture — design the architecture for the Agentic AI migration framework, including its integration with the broader technical landscape

Skills

Must have

5+ years experience

hands-on experince with AWS Migration

AWS Glue, Step Functions, Lambda, EventBridge — deep hands-on experience designing solutions using these core AWS services

IaC (Terraform) — strong experience defining and managing infrastructure as code with Terraform

Data platform architecture — proven experience architecting data platforms (data lakes, warehouses, pipelines) at scale

PySpark — hands-on experience with PySpark for large-scale data processing

CI/CD — experience designing and implementing CI/CD pipelines for infrastructure and data workflows

AI/ML frameworks — working knowledge of AI/ML frameworks and their application within architecture design

GitHub Copilot — practical experience using GitHub Copilot within development/architecture workflows

Nice to have

AWS certification (Data Analytics Specialty or Solutions Architect)

Experience with BI tools (QuickSight, Tableau, Power BI)

Infrastructure as Code experience (Terraform/CloudFormation)

Industry experience relevant to your business

Exposure to streaming data (Kinesis) or ML pipelines (SageMaker)

experince in financial domain

Other

Languages

English: C2 Proficient

• This is a strategic data engineering engagement with our client to architect and plan the migration of their entire data processing and ETL estate from Matillion to AWS Glue — a foundational shift in how one of the world's largest financial market infrastructure companies handles its data pipelines.
  During this Mobilisation Phase, you'll work jointly with client's engineering teams to reverse-engineer the existing landscape, design the target-state architecture across every layer (infrastructure, data processing, workflows, dependencies, and operating model), and build the detailed delivery blueprint that will greenlight the full-scale migration.
  The work is technically rich and highly collaborative: you'll review and validate job inventories spanning hundreds of ETL workflows, define reusable migration patterns and templates, design a validation and reconciliation framework, run a proof-of-concept to stress-test the approach, and navigate client's rigorous internal governance — from Architectural Significance Assessments through Architectural Review Boards to a formal Gate 1 decision.
  This is the kind of engagement where your recommendations directly shape a multi-phase, multi-million-pound programme: the target-state framework you produce here becomes the blueprint that a larger delivery team will execute against. Perfect opportunity for combining deep data engineering knowledge with architecture leadership, stakeholder influence, and structured delivery planning inside a Tier 1 financial services environment

• Responsibilities:

  - Design, build, and maintain ETL/ELT pipelines on AWS
  - Migrate existing data pipelines and workloads from legacy/on-prem systems to AWS
  - Develop and optimize data models for data warehouse/data lake (Redshift, S3)
  - Build and orchestrate data workflows (Glue, Step Functions, Lambda, EMR)
  - Implement data quality checks, validation, and reconciliation processes
  - Optimize pipeline performance and manage compute/storage costs
  - Ensure data security and access control (IAM, KMS, encryption)
  - Monitor and troubleshoot pipeline failures (CloudWatch, logging)
  - Collaborate with data analysts, BI developers, and architects on data requirements
  - Document data pipelines, architecture, and operational runbooks

• Mandatory Skills Description:

  - 5+ years of experience
  - Hands-on experience building data pipelines on AWS (Glue, EMR, Lambda, Step Functions)
  - Strong SQL and experience with data warehousing (Redshift, dimensional modeling)
  - Proficiency in Python or Scala for data engineering
  - Experience with S3-based data lake architecture (partitioning, cataloging, formats like Parquet)
  - Experience migrating data pipelines from on-prem or other cloud platforms
  - Understanding of data governance, security, and access control on AWS
  - Experience with CI/CD for data pipelines

• Nice-to-Have Skills Description:

  - AWS certification (Data Analytics Specialty or Data Engineer Associate)
  - Experience with streaming data (Kinesis, MSK/Kafka)
  - Familiarity with Infrastructure as Code (Terraform/CloudFormation)
  - Experience with orchestration tools (Airflow, dbt)
  - Experience in financial domain

This is a strategic data engineering engagement with our client to architect and plan the migration of their entire data processing and ETL estate from Matillion to AWS Glue — a foundational shift in how one of the world's largest financial market infrastructure companies handles its data pipelines.
During this Mobilisation Phase, you'll work jointly with client's engineering teams to reverse-engineer the existing landscape, design the target-state architecture across every layer (infrastructure, data processing, workflows, dependencies, and operating model), and build the detailed delivery blueprint that will greenlight the full-scale migration.
The work is technically rich and highly collaborative: you'll review and validate job inventories spanning hundreds of ETL workflows, define reusable migration patterns and templates, design a validation and reconciliation framework, run a proof-of-concept to stress-test the approach, and navigate client's rigorous internal governance — from Architectural Significance Assessments through Architectural Review Boards to a formal Gate 1 decision.
This is the kind of engagement where your recommendations directly shape a multi-phase, multi-million-pound programme: the target-state framework you produce here becomes the blueprint that a larger delivery team will execute against. Perfect opportunity for combining deep data engineering knowledge with architecture leadership, stakeholder influence, and structured delivery planning inside a Tier 1 financial services environment

• Responsibilities:

  - Agentic AI Migration Framework development — design and build an agentic AI framework to automate and accelerate migration workflowsж
  - Pattern library build — develop and maintain a library of reusable migration patterns/templates for common conversion scenarios;
  -Automated conversion pipeline — build pipelines that automatically convert/transform code and configurations (e.g., Matillion jobs) as part of the migration process;
  -GitHub Copilot integration for code generation — integrate GitHub Copilot into the migration workflow to accelerate code generation and conversion tasks;
  -Support factory pods — provide technical support, tooling, and troubleshooting to migration factory pods executing migrations at scale

• Mandatory Skills Description:

  - Python — strong hands-on Python development experience for building tools and automation
  - GenAI/LLM integration — experience integrating GenAI/LLM capabilities into applications, workflows, or tooling
  - Prompt engineering — hands-on experience designing, testing, and optimizing prompts for LLM-based tools
  - AWS Bedrock — hands-on experience with AWS Bedrock for building GenAI-powered applications
  - GitHub Copilot — practical experience using and/or integrating GitHub Copilot into development workflows
  - Automation frameworks — experience designing and building automation frameworks/tooling from scratch
  - Testing — strong experience with test automation and validation frameworks to ensure conversion accuracy
  - Matillion JSON parsing — experience parsing, interpreting, and transforming Matillion job definitions (JSON format)