Jobs Data Engineer

Hi there! AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.

Why join us
If you’re looking for a place to grow, make an impact, and work with people who care, we’d love to meet you! :)

About the role
As a Data Steward, you will play a key role in ensuring high-quality, reliable product data that directly impacts sales performance across the EMEA market. This role sits at the intersection of business and technology, shaping how product information flows through PIM systems and reaches external channels. You’ll have real influence on data standards, processes, and visibility, while collaborating with diverse stakeholders in a structured yet supportive environment. It’s an opportunity to grow your expertise in data stewardship, product information management, and cross-functional collaboration within a global setup.

What you will do
● Validate that mandatory product data provided by business teams is accurate and complete;
● Upload product data into the PIM system and resolve data gaps, including coordinating with multiple stakeholders to complete missing information;
● Syndicate product data into channel-specific templates and resolve any related issues;
● Support business teams when they encounter issues using product data for the channel;
● Clarify and communicate data issues and requirements to stakeholders to ensure data quality and accuracy;
● Create, track, and report on the status of product data requests;
● Reconfigure and maintain syndication templates in inRiver.

Must haves
● Minimum of 3 years of experience in data management, with solid knowledge of core principles and concepts;
● Good understanding of the data lifecycle;
● Ability to use multiple tools to create and maintain data stewardship processes, inputs, and outputs;
● Hands-on experience with Inriver is required; alternatively, experience with Salsify, Informatica, or Akeneo (in complex implementations) will be considered;
● Experience working with retailers and marketplaces, including PIM syndication setup and management;
● Strong Excel skills;
● Ability to work efficiently under tight and ongoing deadlines;
● Strong communication skills, with the ability to interact with both business and technical stakeholders;
● Fluent written and spoken English.

Perks and benefits
● Professional growth: Accelerate your professional journey with mentorship, TechTalks, and personalized growth roadmaps
● Competitive compensation: We match your ever-growing skills, talent, and contributions with competitive USD-based compensation and budgets for education, fitness, and team activities
● A selection of exciting projects: Join projects with modern solutions development and top-tier clients that include Fortune 500 enterprises and leading product brands
● Flextime: Tailor your schedule for an optimal work-life balance, by having the options of working from home and going to the office — whatever makes you the happiest and most productive.


Meet Our Recruitment Process
Asynchronous stage — An automated, self-paced track that helps us move faster and give you quicker feedback:
● Short online form to confirm basic requirements
● 30–60 minute skills assessment
● 5-minute introduction video
Synchronous stage — Live interviews
● Technical interview with our engineering team (scheduled at your convenience)
● Final interview with your future teammates
If it’s a match—you’ll get an offer!

We are looking for a Senior Data Engineer to design and build scalable data platforms that process large volumes of data for analytics, reporting, and business-critical applications.

You will work with distributed data processing systems, cloud infrastructure, streaming platforms, and modern data pipelines. We are looking for an engineer who can independently own production solutions and solve complex performance and architecture challenges.

Mandatory Skills

Programming & Data Processing

• Strong Python development skills
• Advanced SQL (complex queries, optimization, execution plans)
• Strong commercial experience with Apache Spark / PySpark
• Experience optimizing Spark jobs (partitioning, shuffle, joins, skew handling)

Data Engineering

• Experience building production-grade ETL / ELT pipelines
• Strong experience with Apache Airflow
• Strong hands-on experience with Apache Kafka
• Understanding of batch and streaming data processing architectures

Infrastructure & Storage

• Experience with relational databases (PostgreSQL, MySQL or similar)
• Data modeling and query optimization
• Hands-on experience with at least one cloud platform (AWS, Azure or GCP)
• Experience working with Docker, Linux, Git, and basic CI/CD pipelines

Desired Skills

• Databricks, Snowflake or similar cloud data platforms
• Kubernetes
• CDC pipelines (Debezium or similar)
• Modern analytical/storage systems (ClickHouse, Trino, Iceberg, Delta Lake)
• Monitoring tools (Grafana, Prometheus)

Responsibilities

• Build scalable batch and streaming data pipelines
• Develop and optimize Spark/PySpark data processing jobs
• Build Kafka-based ingestion and event-driven pipelines
• Design and maintain Airflow orchestration workflows
• Optimize SQL queries and improve pipeline performance
• Investigate production issues and improve system reliability
• Participate in architecture and technical design discussions

Senior-Level Expectations

• Independently design scalable data pipelines
• Debug and optimize distributed production systems
• Solve performance bottlenecks in Spark and SQL workloads
• Make technical decisions with minimal supervision
• Take ownership of production solutions end-to-end

Soft Skills

• Strong problem-solving skills
• Ownership and accountability
• Ability to work independently
• Good communication and collaboration skills
• Proactive engineering mindset

Summary

# Job Title

Product-Minded Full-Stack / AI Engineer for Web Data Intelligence Platform

# Job Description

We are building an AI-powered platform that collects, structures, and analyzes information from public websites and documents.

The system needs to process messy real-world data from many organizations, extract useful structured information, store it in a database, and present it through a user-facing dashboard.

This is not a basic scraping project. We are looking for someone who can think through system architecture, data pipelines, database structure, AI extraction workflows, and product design.

The right person should be able to take an existing MVP, understand the logic behind it, and help redesign it into a more scalable and configurable system.

# What You’ll Work On

You will help design and build a platform that can:

* Discover and process public web pages and documents.
* Extract structured information from unstructured text and PDFs.
* Normalize information into a database.
* Track updates over time.
* Detect duplicates and changes.
* Classify and score records based on configurable rules.
* Match structured records to user-defined profiles.
* Support a dashboard for review, filtering, and workflow management.
* Make the system configurable across different use cases.

# Required Skills

Strong fit if you have experience with:

* Python and/or Node.js
* Web crawling / scraping
* PDF and document extraction
* LLM-based extraction and classification
* Database design
* Full-stack development
* Data pipelines
* AI workflow design
* Prompt engineering
* Handling messy real-world data

Bonus if you have worked with:

* Public web data
* Search or monitoring systems
* Workflow dashboards
* Vector databases
* Supabase / Postgres
* Playwright
* OpenAI / Claude APIs
* LangChain / LlamaIndex or similar tools

# Important

We are not looking for someone who only knows how to scrape websites.

We are looking for someone who can help turn an MVP into a structured, scalable intelligence platform.

You should be comfortable thinking about:

* Data models
* Extraction quality
* Source reliability
* Change detection
* Deduplication
* Configurable scoring logic
* Human review workflows
* Product usability

# What We Are Looking For

A strong candidate will think in terms of:

* Sources
* Documents
* Records
* Evidence
* Configurations
* Profiles
* Matches
* Processing runs
* Review workflows
* Data quality
* Deduplication
* Change tracking
* Explainable scoring

A weak candidate will only talk about scraping pages.

# Engagement

This will start as a short test project. If successful, it can become an ongoing engineering role.

We are looking for someone who can eventually take ownership of the platform architecture and development.

# To Apply

Please answer the following questions:

1. Describe a similar system you have built or would know how to build.
2. How would you separate use-case-specific logic from the core platform?
3. How would you detect whether a record is new, updated, duplicate, stale, or no longer relevant?
4. What database schema would you start with?
5. What would you automate, and where would you keep human review?
6. Please share examples of relevant work if available.

Applications without thoughtful answers will not be considered.

Job Description
Extensive software development experience, including experience working on complex backend or data infrastructure projects.
Strong hands-on experience with Apache Iceberg .

Experience with Scala.

Hands-on experience developing backend services, data-processing engines, infrastructure components.
Programming experience in C /or C++ is highly desirable / preferred, the role is primarily focused on backend/system-level engineering rather than full-stack application development.
Deep understanding of distributed systems, data processing, storage engines, or data infrastructure.

Strong knowledge of data consistency, transactions, schema evolution, partitioning, compaction, and performance optimization.

Proven ability to troubleshoot complex production and customer issues.

Strong ownership mindset and the ability to operate effectively in areas with limited documentation or historical knowledge.

Excellent communication and collaboration skills.

Preferred Qualifications: 
Big Data Frameworks: Hands-on experience with distributed data-processing technologies such as Apache Spark, Flink, Trino, or Presto.
Cloud Object Storage: Experience working with cloud storage platforms, including Amazon S3, Google Cloud Storage (GCS), or Azure Data Lake Storage (ADLS).
Legacy Modernization: Proven track record of modernizing or gradually improving a mature codebase without disrupting existing customers.
Job Responsibilities
Architect & Build: Lead the architecture, design, and implementation of highly scalable, complex backend systems and robust data infrastructure using Iceberg.
Data Lake Engineering: Design and manage enterprise-grade data lake solutions leveraging Apache Iceberg or equivalent table formats, ensuring optimal data consistency, transactions, schema evolution, partitioning, and compaction.
Performance Optimization: Maximize system throughput, minimize latency, and optimize resource utilization across large-scale distributed data processing systems.
Production Excellence: Systematically diagnose, troubleshoot, and resolve complex, high-impact production and customer-facing infrastructure issues.
Codebase Modernization: Champion the gradual improvement and modernization of a mature, production-critical codebase without causing disruption to active customers or services.
Technical Leadership & Collaboration: Act as a technical authority, fostering strong communication and collaboration across engineering teams to align on technical objectives.
Navigate Ambiguity: Exercise a strong ownership mindset to operate effectively and deliver high-quality outcomes in areas with limited documentation or historical knowledge.
Department/Project Description
We are seeking an expert Iceberg Engineer to design, optimize, and scale our next-generation multi-cloud data lakehouse architecture across AWS and Azure. The ideal candidate is a backend/system engineer with strong experience building engines, infrastructure, or high-performance backend components. In this role, you will leverage client's Open Lakehouse (integrated with Client's Talend Cloud) to establish a low-overhead, high-throughput data foundation. You will manage automated data ingestion, implement continuous table optimization, handle cross-cloud data mirroring, and ensure seamless multi-engine access without vendor lock-in.
 

We are looking for a talented Backend Engineer to help build the infrastructure that secures the world’s software supply chains.

You’ll work alongside top-tier engineers to design and deliver high-performance, secure, and scalable systems. This role is ideal for someone who thrives in a fast-paced environment, enjoys solving complex technical challenges, and wants to build products that make a meaningful impact.
 

Responsibilities

• Design, develop, and maintain scalable and reliable backend services and APIs.
• Build high-performance systems capable of handling large volumes of data and traffic.
• Develop and improve backend infrastructure using Node.js.
• Work with both SQL and NoSQL databases, including data modeling and performance optimization.
• Design and implement solutions within a microservices architecture.
• Build and maintain cloud-based services and infrastructure.
• Collaborate closely with product, engineering, and other cross-functional teams to deliver new features.
• Troubleshoot, debug, and resolve complex production issues.
• Improve system performance, reliability, scalability, and security.
• Participate in code reviews and contribute to engineering best practices.
• Write clean, maintainable, well-tested, and documented code.
• Work closely with customers and internal stakeholders to understand technical requirements and translate them into effective solutions.

What We’re Looking For

• Bachelor’s degree in Computer Science, Software Engineering, or a related field.
• 2–4 years of experience in backend development, with a focus on building scalable applications and services.
• Strong proficiency in Node.js.
• Hands-on experience with database technologies, including both SQL and NoSQL databases.
• Familiarity with cloud services and microservices architecture.
• Experience working directly with customers or within a product-focused company/environment.
• Excellent problem-solving and analytical skills.
• Strong commitment to writing clean, maintainable, and high-quality code.

Our client is a performance rights organization in the United States. It collects blanket license fees from businesses that use music, entitling those businesses to play or sync any songs from the client’s repertoire of over 22.4 million musical works. On a regular basis, the client distributes the money to songwriters, composers, and music publishers as royalties to those members whose works have been performed.

Requirements:

• Databricks Expertise: Minimum of 3+ years of proven, previous Databricks administration experience in a complex, multi-tenant enterprise environment.
• Advanced Terraform: Strong Terraform skills. You must have hands-on experience building and maintaining reusable modules specifically utilizing the Databricks Terraform Provider.
• Strong understanding of cloud-native IaC (Infrastructure as Code) like Terraform for automating instance provisioning
• Cloud IAM Mastery: Deep, practical knowledge of Cloud Identity and Access Management (IAM), including writing secure IAM policies, managing roles, cross-account access, and secrets management.
• Robust cloud networking experience
• Scripting: Proficiency in Bash

Job responsibilities:

• Environment Ownership: Own the technical implementation of all Databricks environment configuration changes across multi-workspace environments.
• Platform Lifecycle: Manage Databricks runtimes, cluster policies, and coordinate seamless platform upgrades.
• Infrastructure as Code (IaC): Utilize strong Terraform skills to programmatically provision, update, and manage all Databricks resources (workspaces, clusters, tokens, policies) and cloud infrastructure, eliminating manual configurations.
• Cloud Security & Identity (IAM): Architect and enforce secure access models. Manage Cloud Identity and Access Management (IAM) integrations for users, groups, and service principals.
• Secure Cloud Networking: Design, implement, and maintain secure cloud networking architectures for Databricks.
• Cost Optimization: Actively review current environment setups—including cluster configurations, instance types, and data storage parameters—to identify and execute cost-optimization opportunities.
• Efficiency Strategies: Provide expert input on automation, auto-scaling, and monitoring strategies to drastically reduce ongoing compute and storage costs.

📍 Location: EU-based, Remote
💼 Cooperation type: Full-time

We’re looking for a skilled Data Engineer to join our Analytics team. 

We are building the data platform that serves both our analytics and operational data needs. In this role you will own significant parts of the platform architecture — from design through deployment — and build the pipelines and services that make the company’s data flow reliably between systems. You will work with large, complex datasets and high-throughput pipelines, and you will own a major greenfield project connecting our production systems with our CRM.

About us:
Stape is a global product-driven IT company and the #1 leader in the server-side tracking market. We’re building a powerful, technically complex product that simplifies server-side tracking for marketers and website owners. Our platform processes over 5 billion requests daily, helping improve tracking accuracy and data privacy for more than 200, 000 clients worldwide. We work closely with top tech partners, including Meta, LinkedIn, Google, Snapchat, and TikTok to provide advanced tracking capabilities.


Skills and experience
 

• Minimum 4 years of experience as a data engineer (platform/pipeline building, not only report maintenance).
• At least an upper-intermediate level of English (written and spoken).
• Bachelor’s degree in Computer Science, Engineering, or another related technical field.
• Extensive knowledge of SQL and Python.
• Experience organizing, monitoring, and maintaining data infrastructure: managing ETL/ELT processes, writing tests for data pipelines, and monitoring data consistency.
• Practical experience with GCP: BigQuery, Cloud Logging, Cloud Storage, Cloud Functions, Cloud Run, Datastream, Pub/Sub.
• Practical experience with Dataform (or dbt).
• Experience with a workflow orchestrator (Apache Airflow or similar) — scheduling, dependency management (DAGs), retries, and backfills.
• Experience with both batch and streaming data processing.
• Ability to build custom Python services and server-to-server API integrations (e.g. FastAPI).
• Understanding of data modeling for analytics — including star schema and lakehouse/medallion (bronze/silver/gold) design patterns.
• Experience with Jira or similar project management tools.

Will be a plus
 

• Familiarity with containerization (Docker); Kubernetes.
• Experience integrating CRM or third-party SaaS platforms via their APIs (e.g. HubSpot, Salesforce) — building bidirectional data syncs, handling webhooks, and reconciling records across systems.
• Change Data Capture (CDC) experience for near-real-time replication.
• Stream processing (Apache Flink, Dataflow) and open table formats (Apache Iceberg).
• Owning data platform architecture end-to-end — technology selection and implementation standards.
• Experience with MySQL.

Duties and responsibilities
 

• Build and maintain infrastructure for reliable extraction, transformation, and loading of data from a variety of sources.
• Design and build batch and streaming pipelines that meet business requirements.
• Own the design and delivery of a bidirectional sync between our production systems and our CRM.
• Conduct SQL performance tuning for existing queries in cloud databases (primarily BigQuery).
• Create and maintain optimized data models and schemas (including star schema and medallion layers) for analytics.
• Evaluate data for integrity and accuracy; work with teams to backfill gaps where found.
• Design and implement data security and governance measures.
• Collaborate with analysts to improve data collection and processing.
• Work with stakeholders on data-related technical issues and support their infrastructure needs.
• Participate in strategic, data-related decision-making.

We’re looking for a Data Engineer with strong hands-on experience in PySpark and Databricks to join an Insurance domain project. You’ll work on building scalable data processing pipelines and end-to-end data & analytics solutions using Azure, PySpark, Databricks, and Palantir Foundry.

Responsibilities:

• Design and develop data objects and data processing jobs in Azure Data Platform.
• Design and implement E2E analytics solutions using Palantir Foundry Platform, including: various types of data sources and data connections, scalable data transformation pipelines in PySpark.
• Optimize and automate data processing jobs and transformation pipelines through reviewing of logs and query plan, partitioning and code simplification.
• Ensure data quality, security (access concept) and governance across the whole flow of data & analytics solution.
• Collaborate with cross-functional teams and stakeholders to understand and translate. business requirements into technical specifications and actionable plans, ensuring successful delivery.

• Troubleshoot and resolve functional and technical data issues for stakeholders, actively contributing to shared goals and supporting peers in solving technical and analytical challenges.

   

Requirements:

• 4+ years of professional experience in data engineering roles.
• Strong hands-on experience with data transformation jobs/pipelines and complex data & analytics solution.
• Expertise in Python/PySpark and experience with distributed computing frameworks like Spark.
• Proficiency/advanced in SQL.
• Proficiency with Git.
• Experience with cloud platforms (Azure), specifically Azure Databricks, and Palantir Foundry is a significant advantage.
• Proven ability to work across business and technical teams, supporting requirement definition and solution design.
• Agile mindset and practical experience with Scrum or other agile frameworks, Azure DevOps is a plus.
• Strong team player with excellent communication skills, proactive mindset, and a collaborative approach.
• English at B2 or higher.

• Bonus: knowledge of REST API or Power BI.

   

Our Benefits:

• Professional growth: Individual development plan, mentorship, reimbursement for professional certifications and English lessons, access to professional courses in Corporate Learning Management System.
• Community: Tech community and knowledge-sharing events, English speaking club, corporate library and book club, volunteering and charity initiatives.
• Wellbeing: Medical insurance, regular medical check-ups, sport reimbursement, paid vacation and sick leave, mental health support and events.
• Work environment: Fully-equipped offices, top-notch equipment, flexible work format, activities both in-office and online, Y-bucks and access to the Yalantis store.

Mobilunity is looking for a Senior Data Engineer. 
 

The company is building an internal data warehouse and BI capability to turn data into actionable insights and automate data analysis. As a Senior Data Engineer, you will be part of a dedicated remote team, helping build and evolve the data infrastructure supporting business intelligence and data-driven decision-making.
 

Responsibilities:

• Design and build the foundational data warehouse and BI infrastructure from the ground up. 
• Design, build, and maintain scalable data pipelines that support streaming, viewer, and content analytics.
•  Work with data from the platform’s existing MySQL or MariaDB databases and integrate it with AWS-based infrastructure. 
• Consolidate and process web tracking and event data collected through tools such as Google Tag Manager or Segment. 
• Automate data analysis processes that are currently done manually, to surface user experience issues and patterns faster. 
• Develop and optimize ETL/ELT processes across multiple data sources.
• Ensure data quality, consistency, and reliability across systems. 
• Collaborate with the backend and frontend teams, as well as future data product managers, to align data pipelines with the platform’s architecture. 
• Write clean, testable, well-documented code for data processing and automation tasks. 
• Monitor pipeline performance and troubleshoot issues in production.
  
  Requirements:
• 5+ years of experience as a Data Engineer or in a similar role.
• Experience setting up a data warehouse or BI infrastructure.
• Strong experience building and maintaining ETL/ELT pipelines.
• Solid experience with relational databases, preferably MySQL or MariaDB.
• Proficiency with AWS.
• Strong skills in Python or PHP for data processing and automation.
• Experience working with web tracking or analytics platforms, such as Google Tag Manager or Segment.
• Understanding of microservices architecture and how data flows across distributed systems.
• Experience writing clean, testable, well-documented code.
• Proficiency with Git.
• Upper-Intermediate English or higher, for direct communication with the client’s team.
  
  Nice to have:
• Experience with ad tech data, such as Google Ad Server reporting.
• Experience in media, streaming, or broadcast companies.
• Comfortable working in an early-stage, results-driven project where impact needs to be demonstrated to stakeholders.
  
  In return we offer:
• The friendliest community of like-minded IT-people
• Open knowledge-sharing environment – exclusive access to a rich pool of colleagues willing to share their endless insights into the broadest variety of modern technologies
• Mobilunity Medical Insurance program designed to attend our teams’ needs
• Paid vacations and sick leaves, including 5 paid days per year that don’t require a sick note
• Perfect office location in the city-center (900m from Lukyanivska metro station with a green and spacious neighborhood) or remote mode engagement: you can choose a convenient one for you, with a possibility to fit together both
• No open-spaces setup – separate rooms for every team’s comfort and multiple lounge and gaming zones
• English classes in 1-to-1 & group modes with elements of gamification

• Neverending fun: sports events, tournaments, music band, multiple affinity groups
   

  Come on board, and let’s grow together!

About Our Client

Our client is a leading enterprise data platform company building an open, high-performance data lakehouse for AI and analytical workloads. The platform combines an intelligent SQL query engine, an AI-ready semantic layer, and an open catalog built on Apache Iceberg — enabling Fortune 500 companies across finance, energy, manufacturing, and logistics to unify, query, and govern data at massive scale across cloud and on-premise sources.

About the Role

We are looking for a Senior Software Engineer for the data catalog and enterprise data management layer of a large-scale lakehouse platform — the services that automate data ingestion and autonomously optimize Apache Iceberg tables. You will own features through the full development cycle, from design through deployment, in a multithreaded, distributed environment where scalability, performance, and always-on availability are the baseline expectations.

This is a systems-level, backend engineering role focused on data infrastructure internals — not application development or CRUD services.

Responsibilities

• Own the full software development cycle — inception, design, development, testing, deployment — for catalog and data-management services
• Build and operate services for automated ingestion and autonomous optimization of Apache Iceberg tables
• Reason about concurrency and parallelization to deliver scalability and performance in multithreaded, distributed systems
• Drive performance tuning and system-efficiency improvements
• Work with Apache Foundation open-source projects, contributing upstream where agreed
• Participate in code and design reviews, upholding a high engineering bar
• Collaborate with product managers, support teams, and solution architects on feature requests and design updates
• Partner with US-based engineering leads on technical decisions

Required Qualifications

• B.S., M.S., or PhD in Computer Science or a related field
• 5+ years of software engineering experience, preferably focused on database systems or related fields
• Strong object-oriented programming skills in Java or C++
• Solid grounding in data structures and algorithms
• Hands-on experience with multithreaded and asynchronous programming patterns for scalable, performant systems
• A passion for engineering quality — zero-downtime upgrades, availability, resiliency, and uptime as first-class concerns
• Comfort with a fast-moving environment; strong ownership; the confidence to defend a technical position and the openness to be mentored
• Comfortable with AI-assisted development workflows — using modern AI tools productively while critically validating their output
• English Upper-Intermediate or higher (B2+) — daily written and verbal communication with a US-based engineering team
• Availability to work EU business hours shifted 2–3 hours later for daily overlap with US West Coast mornings

Desired Skills

• Database internals and query planning
• Distributed systems: concurrency control, data replication, storage systems
• Apache Iceberg or other open table formats (Delta Lake, Hudi)
• Messaging systems: Kafka, NATS, or cloud pub/sub services
• Contributions to open-source data infrastructure (Apache projects especially valued)

Details

• Engagement: Long-term contract
• Location: Europe (EU / EEA / UK), remote
• Working hours: EU business hours, shifted 2–3 hours later for daily overlap with the US West Coast team

Level: Senior Engagement: Long-term contract Location: Europe (EU / EEA / UK), remote Working hours: EU business hours, shifted 2–3 hours later to ensure daily overlap with US West Coast (PST/PDT) mornings

About the Client

Our client is a leading enterprise data platform company building an open, high-performance data lakehouse for AI and analytical workloads. The platform combines an intelligent SQL query engine, an AI-ready semantic layer, and an open catalog built on Apache Iceberg — enabling Fortune 500 companies across finance, energy, manufacturing, and logistics to unify, query, and govern data at massive scale across cloud and on-premise sources.

About the Role

We are looking for a Senior Software Engineer to design, build, and own significant components of the core query engine of a large-scale distributed data platform. You will lead work across query planning, optimization, and execution — driving architectural decisions, mentoring middle engineers, and pushing the platform's performance and scalability boundaries across all major clouds.

This is a systems-level, backend engineering role focused on distributed data processing internals — not application development or CRUD services.

Responsibilities

• Architect, design, and implement significant components of the query engine — planner, optimizer, execution operators, memory management, and data access layers.
• Write and review high-quality, performance-critical code in Java and/or C++.
• Own end-to-end delivery of features from technical design through production rollout.
• Drive performance optimization work — profiling, benchmarking, and eliminating bottlenecks in latency- and throughput-sensitive paths.
• Integrate deeply with columnar formats, open table formats, and connectivity drivers.
• Mentor middle engineers, lead code reviews, and set technical standards for the team.
• Contribute to CI/CD architecture and quality processes across Jenkins, containerization, and multi-cloud deployment (GCP, AWS, Azure via Kubernetes / Docker).
• Debug complex, cross-layer issues spanning query planning, distributed execution, memory management, and I/O.
• Partner with US-based engineering leads on design reviews, architectural decisions, and roadmap execution.

Required Qualifications

• Education: B.S. or M.S. in Computer Science, Computer Engineering, or a related technical field.
• Programming: Strong proficiency in Java or C++, with deep understanding of OOP, systems design, memory management, and concurrency. Candidates strong in both are especially valued.
• SQL & Data: Advanced SQL skills and strong understanding of query execution internals — planning, optimization, and operator implementation.
• Data processing systems experience: Demonstrated experience building or extending data processing systems — query engines, distributed databases, ETL/ELT frameworks, or analytical platforms.
• Engineering experience: 5+ years in backend/systems software engineering, with a track record of owning significant components.
• CI/CD & DevOps: Solid experience with Jenkins and modern engineering workflows.
• Containers & Orchestration: Strong working knowledge of Docker and Kubernetes.
• Cloud: Hands-on experience with at least one major cloud (GCP, AWS, or Azure); exposure to more than one is a strong plus.
• Version Control: Confident with Git / GitHub workflows and rigorous code review practices.
• English: Upper-Intermediate or higher (B2+) — daily written and verbal communication with a US-based engineering team.
• Availability: Able to work EU hours with a 2–3 hour shift toward US West Coast time to ensure daily overlap with the client team.
• Leadership: Prior experience mentoring engineers, leading technical designs, or owning components end-to-end.

Desired Skills

• Deep experience with Apache Arrow (columnar in-memory format).
• Hands-on experience with Apache Calcite (SQL parsing, planning, optimization framework).
• Experience with Gandiva or LLVM-based expression compilation and code generation.
• Experience with open table formats — Apache Iceberg, Delta Lake, or Hudi.
• Deep familiarity with distributed computing frameworks (e.g., Apache Spark, Kafka) and MPP SQL query engines (e.g., Presto, Trino, or similar).
• Deep understanding of query planning, optimization (cost-based, rule-based), and execution internals.
• Experience with data connectivity drivers: JDBC, ODBC, Arrow Flight.
• Kubernetes on managed services (GKE / EKS / AKS) and multi-cloud exposure.
• IaC tools such as Terraform.
• Track record of performance optimization at the systems level — profiling, vectorization, cache/memory-conscious design.
• Prior experience contributing to open-source data infrastructure projects.
  Please note: We can consider only candidates currently residing in the EU, EEA, or the UK

Level: Senior Engagement: Long-term contract Location: Europe (EU / EEA / UK), remote Working hours: EU business hours, shifted 2–3 hours later to ensure daily overlap with US West Coast (PST/PDT) mornings

About the Client

Our client is a leading enterprise data platform company building an open, high-performance data lakehouse for AI and analytical workloads. The platform combines an intelligent SQL query engine, an AI-ready semantic layer, and an open catalog built on Apache Iceberg — enabling Fortune 500 companies across finance, energy, manufacturing, and logistics to unify, query, and govern data at massive scale across cloud and on-premise sources.

About the Role

We are looking for a Senior SDET to lead the design of automated test frameworks and testing strategy for a large-scale distributed data platform. You will own the quality of the query engine, connectivity layer, and backend services end-to-end — from framework architecture through CI/CD infrastructure across all major clouds — and mentor middle engineers on the team.

This is a data-heavy, backend-focused role. We are not looking for web/UI QA engineers — the work centers on validating distributed query execution, data correctness at scale, connectivity drivers, and backend microservices.

Responsibilities

• Architect, design, and evolve automated test frameworks in Python / pytest for backend services, REST APIs, and distributed data components.
• Define testing strategy for new features and platform components — coverage, risk assessment, and quality gates.
• Own end-to-end, integration, performance, and regression testing covering SQL query execution, data correctness at scale, and platform APIs.
• Lead performance and load testing efforts with JMeter, including workloads over JDBC / ODBC / Arrow Flight drivers.
• Design and validate large-scale data testing scenarios — query plans, result correctness across heterogeneous data sources, metadata consistency, and behavior under high concurrency.
• Own CI/CD test pipelines in Jenkins — architect, maintain, and continuously improve.
• Provision and manage test environments in Kubernetes (GKE / EKS / AKS) across GCP, AWS, and Azure using Docker; drive automation and reproducibility.
• Investigate complex, cross-layer failures and drive root-cause analysis with engineering.
• Mentor middle SDETs, review test designs, and set quality standards across the team.
• Partner with US-based development leads on shift-left practices, testability, and release readiness.

Required Qualifications

• Education: B.S. or M.S. in Computer Science, Computer Engineering, or a related technical field.
• Programming: Strong proficiency in Python, including pytest, with deep understanding of OOP, software design principles, and test framework architecture.
• SQL & Data: Advanced SQL skills and strong understanding of relational and analytical data systems, including query execution internals.
• Data-intensive testing experience: Demonstrated experience testing data-intensive systems — query engines, ETL/ELT pipelines, streaming platforms, or analytical databases. Candidates with only web/UI QA backgrounds are not a fit.
• Testing experience: 5+ years in backend/system test automation, with demonstrated ownership of test strategy and infrastructure.
• CI/CD & DevOps: Solid experience architecting and maintaining Jenkins pipelines and test environments.
• Containers & Orchestration: Strong working knowledge of Docker and Kubernetes (running workloads, debugging pods, deploying complex environments).
• Cloud: Hands-on experience with at least one major cloud (GCP, AWS, or Azure); exposure to more than one is a strong plus.
• Version Control: Confident with Git / GitHub workflows and code review practices.
• English: Upper-Intermediate or higher (B2+) — daily written and verbal communication with a US-based engineering team.
• Availability: Able to work EU hours with a 2–3 hour shift toward US West Coast time to ensure daily overlap with the client team.
• Leadership: Prior experience mentoring engineers, defining test strategy, or leading a QA/SDET function.

Desired Skills

• Deep experience testing REST APIs and backend microservices at scale.
• Hands-on with distributed computing frameworks (e.g., Apache Spark, Kafka) and MPP SQL query engines (e.g., Presto, Trino, or similar).
• Strong understanding of modern data lakehouse concepts, open table formats (Apache Iceberg), and data warehousing.
• Experience with data connectivity drivers: JDBC, ODBC, Arrow Flight.
• Performance testing with JMeter or comparable load-testing tools at production scale.
• Kubernetes on managed services (GKE / EKS / AKS) and multi-cloud exposure.
• IaC tools such as Terraform.
• Deep understanding of query plan generation, query acceleration / materializations, and metadata integrity in distributed data systems.
• Prior experience leading a QA/SDET function in a data-platform or database engineering environment.
  Please note: We can consider only candidates currently residing in the EU, EEA, or the UK.

Level: Middle Engagement: Long-term contract Location: Europe (EU / EEA / UK), remote Working hours: EU business hours, shifted 2–3 hours later to ensure daily overlap with US West Coast (PST/PDT) mornings

About the Client

Our client is a leading enterprise data platform company building an open, high-performance data lakehouse for AI and analytical workloads. The platform combines an intelligent SQL query engine, an AI-ready semantic layer, and an open catalog built on Apache Iceberg — enabling Fortune 500 companies across finance, energy, manufacturing, and logistics to unify, query, and govern data at massive scale across cloud and on-premise sources.

About the Role

We are looking for a Middle-level SDET to design and maintain automated tests for a large-scale distributed data platform. You will validate the correctness, performance, and reliability of the query engine, connectivity layer, and backend services across all major clouds.

This is a data-heavy, backend-focused role. We are not looking for web/UI QA engineers — the work centers on validating distributed query execution, data correctness at scale, connectivity drivers, and backend microservices.

Responsibilities

• Develop and maintain automated tests in Python / pytest for backend services, REST APIs, and distributed data components.
• Write end-to-end, integration, and regression tests covering SQL query execution, data correctness, and platform APIs.
• Support performance and load testing with JMeter, including workloads over JDBC / ODBC / Arrow Flight drivers.
• Validate data-intensive scenarios — query plans, result correctness across data sources, metadata consistency, and behavior under concurrency.
• Contribute to CI/CD test pipelines in Jenkins — configure, maintain, and troubleshoot.
• Provision and manage test environments in Kubernetes (GKE / EKS / AKS) across GCP, AWS, and Azure using Docker.
• Investigate failures across the stack — query engine, distributed services, drivers, infrastructure — and drive them to resolution with engineering.
• Collaborate with US-based developers on testability and quality gates.

Required Qualifications

• Education: B.S. or M.S. in Computer Science, Computer Engineering, or a related technical field.
• Programming: Strong proficiency in Python, including pytest, with a solid grasp of OOP and software design principles.
• SQL & Data: Strong SQL skills and solid understanding of how relational and analytical data systems work.
• Data-intensive testing experience: Hands-on experience testing data-intensive systems — query engines, ETL/ELT pipelines, streaming platforms, or analytical databases. Candidates with only web/UI QA backgrounds are not a fit.
• Testing experience: 3+ years in backend/system test automation or SDET roles.
• CI/CD & DevOps basics: Practical experience with Jenkins pipelines and deploying/maintaining test environments.
• Containers: Working knowledge of Docker and basic Kubernetes (running workloads, debugging pods, kubectl fluency).
• Cloud: Comfortable operating in at least one major cloud (GCP, AWS, or Azure).
• Version Control: Confident with Git / GitHub workflows.
• English: Upper-Intermediate or higher (B2+) — daily written and verbal communication with a US-based engineering team.
• Availability: Able to work EU hours with a 2–3 hour shift toward US West Coast time to ensure daily overlap with the client team.

Desired Skills

• Experience testing REST APIs and backend microservices at scale.
• Familiarity with distributed computing frameworks (e.g., Apache Spark, Kafka) and MPP SQL query engines (e.g., Presto, Trino, or similar).
• Understanding of modern data lakehouse concepts, open table formats (Apache Iceberg), and data warehousing.
• Experience with data connectivity drivers: JDBC, ODBC, Arrow Flight.
• Performance testing with JMeter or comparable load-testing tools.
• Kubernetes on managed services (GKE / EKS / AKS) and multi-cloud exposure.
• IaC tools such as Terraform.
• Understanding of query plan generation, query acceleration / materializations, and metadata integrity in distributed data systems.
  Please note: We can consider only candidates currently residing in the EU, EEA, or the UK

Level: Middle Engagement: Long-term contract Location: Europe (EU / EEA / UK), remote Working hours: EU business hours, shifted 2–3 hours later to ensure daily overlap with US West Coast (PST/PDT) mornings

About the Client

Our client is a leading enterprise data platform company building an open, high-performance data lakehouse for AI and analytical workloads. The platform combines an intelligent SQL query engine, an AI-ready semantic layer, and an open catalog built on Apache Iceberg — enabling Fortune 500 companies across finance, energy, manufacturing, and logistics to unify, query, and govern data at massive scale across cloud and on-premise sources.

About the Role

We are looking for a Middle-level Software Engineer to work on the core query engine of a large-scale distributed data platform. You will develop features across query planning, optimization, and execution, contribute to performance-critical components, and help evolve the platform's ability to process massive analytical workloads across heterogeneous data sources.

This is a systems-level, backend engineering role focused on distributed data processing internals — not application development or CRUD services.

Responsibilities

• Develop and maintain features across the query engine — planning, optimization, execution, and data access layers.
• Write high-quality, performance-conscious code in Java and/or C++.
• Work with SQL semantics, query plans, and execution operators over large-scale distributed data.
• Integrate with columnar formats, open table formats, and connectivity drivers used across the platform.
• Contribute to CI/CD pipelines and automated testing — participate in build, test, and release workflows in Jenkins.
• Deploy and validate work in Kubernetes (GKE / EKS / AKS) across GCP, AWS, and Azure using Docker.
• Debug complex issues spanning query planning, distributed execution, memory management, and I/O.
• Collaborate closely with US-based engineering teams on design reviews, code reviews, and technical decisions.

Required Qualifications

• Education: B.S. or M.S. in Computer Science, Computer Engineering, or a related technical field.
• Programming: Strong proficiency in Java or C++, with solid OOP and software design fundamentals. Candidates comfortable with both are especially valued.
• SQL & Data: Strong SQL skills and solid understanding of relational and analytical data systems, including query execution concepts.
• Data processing systems experience: Hands-on experience with data processing systems — query engines, distributed databases, ETL/ELT frameworks, or analytical platforms.
• Engineering experience: 3+ years in backend/systems software engineering.
• CI/CD & DevOps basics: Practical experience with Jenkins pipelines and modern development workflows.
• Containers: Working knowledge of Docker and basic Kubernetes (running workloads, debugging pods, kubectl fluency).
• Cloud: Comfortable operating in at least one major cloud (GCP, AWS, or Azure).
• Version Control: Confident with Git / GitHub workflows.
• English: Upper-Intermediate or higher (B2+) — daily written and verbal communication with a US-based engineering team.
• Availability: Able to work EU hours with a 2–3 hour shift toward US West Coast time to ensure daily overlap with the client team.

Desired Skills

• Experience with Apache Arrow (columnar in-memory format).
• Familiarity with Apache Calcite (SQL parsing, planning, optimization framework).
• Exposure to Gandiva or LLVM-based expression compilation.
• Experience with open table formats — Apache Iceberg, Delta Lake, or Hudi.
• Familiarity with distributed computing frameworks (e.g., Apache Spark, Kafka) and MPP SQL query engines (e.g., Presto, Trino, or similar).
• Understanding of query planning, optimization, and execution internals.
• Experience with data connectivity drivers: JDBC, ODBC, Arrow Flight.
• Kubernetes on managed services (GKE / EKS / AKS) and multi-cloud exposure.
• IaC tools such as Terraform.
• Experience with performance profiling and optimization of latency-sensitive systems.
  Please note: We can consider only candidates currently residing in the EU, EEA, or the UK.
  Please note: We can consider only candidates currently residing in the EU, EEA, or the UK

Description

The Digital Health organization is technology team which focused on next generation Digital Health capabilities which deliver on the Medicine mission and vision to deliver Insight Driven Care. This role will operate within the Digital Health Applications & Interoperability subgroup of the broader Digital Health team, focused on patient engagement, care coordination, AI, healthcare analytics & interoperability amongst other advanced technologies which enhance our product portfolio with new services, while improving clinical & patient experiences.

The project is a cloud-based PaaS Ecosystem built with a privacy by design centric approach to provide a centralized cloud-based platform to store, classify, and control access to federated datasets in a scalable, secure, and efficient manner.

The ecosystem will allow Customer Operating Units (medical device departments) to store federated data sets of varying sizes and formats and control access to those data sets through Data steward(s). Source data sets can be exposed to qualified use cases and workflows through different project types.

The Healthcare Data Platform ecosystem will provide ML/AI project capabilities for streamlined development processes and a ML/AI workbench to enhance data exploration, wrangling, and model training.

OU’s: Currently, Nuero, Cardio, Diabetes is the OU that HDP data platform is working with, but there could be more OU’s coming up with requirements in future.

GL Role: is to work on the enhancement of current capabilities, including taking over the work that AWS proserve team is doing, and develop new requirements that will keep coming from different OU’s in the future.

Requirements

As a System Engineer within the Company Research & Systems Engineering team, you will have the opportunity to work on cloud-based solutions with a strong focus on AWS, Data Engineering, and scalable data platforms. You will collaborate with cross-functional engineering teams to design, develop, integrate, and maintain reliable cloud-native systems and data workflows.

In this role, you will contribute to building and evolving AWS-based architectures, data processing pipelines, and integrations across distributed systems. You will develop a strong understanding of end-to-end system architecture, data flows, infrastructure, and operational requirements, helping to drive technical implementation and delivery throughout the software development lifecycle.

You will work closely with software engineers, data engineers, architects, and other stakeholders to ensure that solutions are scalable, secure, maintainable, and aligned with both technical and business requirements.

• 3+ years of experience in System Engineering, Software Engineering, Cloud Engineering, Data Engineering, or a related technical role.
• Strong software development experience with Python.
• Hands-on experience designing, developing, or supporting solutions on AWS.
• Experience with data engineering concepts and technologies, including data ingestion, ETL/ELT, transformation, validation, and data processing pipelines.
• Strong SQL skills and experience working with relational and/or NoSQL databases.
• Experience with REST APIs, service integrations, and distributed application architectures.
• Understanding of cloud-native architecture principles and modern software development practices.
• Experience troubleshooting applications and distributed cloud systems and performing root cause analysis.
• Experience with Git and modern CI/CD development workflows.
• Strong analytical and problem-solving skills with the ability to understand complex systems and data flows.
• Ability to work effectively with software, data, cloud, and machine learning engineering teams.
• BA/BS in Computer Science, Software Engineering, Information Technology, or a related technical field, or equivalent practical experience.

Preferred Qualifications

• Experience with AWS data services such as S3, Glue, Athena, EMR, or similar technologies.
• Experience with Amazon SageMaker and machine learning development or deployment workflows.
• Experience with Infrastructure as Code using AWS CloudFormation, Terraform, or AWS CDK.
• Experience with AWS monitoring and observability services such as CloudWatch and CloudTrail.
• Experience working in Agile/Scrum development environments.

Job responsibilities

• Work with engineering and cross-functional stakeholders to gather, analyze, clarify, and document business, functional, and non-functional requirements, translating them into clear system and technical requirements.
• Define system requirements, interfaces, data flows, integration requirements, acceptance criteria, and technical constraints for cloud-based software components, data platforms, and integrations.
• Maintain requirements throughout the development lifecycle, ensuring traceability between business needs, system requirements, technical design, implementation, and validation.
• Identify gaps, dependencies, assumptions, and potential technical risks in requirements and work with stakeholders to resolve them before and during implementation.
• Design and contribute to scalable AWS-based architectures supporting data ingestion, processing, transformation, storage, and distribution.
• Analyze data and system behavior to support architecture, implementation, performance, and product decisions.
• Collaborate with Data Engineering, Software Engineering, Cloud Engineering, DevOps, and Architecture teams to design and integrate end-to-end cloud solutions.
• Support operational monitoring, logging, alerting, and troubleshooting of cloud applications and data processing workflows.
• Contribute to system architecture and technical design discussions, including scalability, reliability, security, performance, and cost considerations.
• Participate in technical and requirements reviews to ensure solutions meet defined requirements, engineering standards, cloud best practices, and established software development processes.
• Support continuous improvement of cloud infrastructure, data pipelines, deployment processes, requirements management, and engineering practices throughout the software development lifecycle.