Jobs Data Engineer

N-iX is looking for Senior DataOps Engineer to join the team.

Client Overview:

Our client is an Azerbaijani telecommunications company, the largest mobile network operator in Azerbaijan. The main products are: Fixed telephony, Mobile telephony, Internet services, Wireless broadband, and Value-added services.

Project Objectives:

The primary goal is to accelerate the client’s Data & AI initiatives via a secure, hybrid cloud foundation on AWS while systematically modernizing the IT estate as part of the cloud migration.

Key Project Objectives include:

• Cloud Foundation & Landing Zone: Deploy target hybrid network architectures, establishing a secure Landing Zone and hybrid Data/AI platforms on AWS.
• Security, Compliance & Governance: Operationalize on-prem tokenization (achieving zero raw PII in the cloud), resolve policy blockers to include AWS in the ISMS, and establish a Cloud Center of Excellence (CCoE) to govern Cloud adoption.
• AI Chatbot & Voicebot Design & Implementation: Develop and operationalize a flagship Customer Care Chatbot and Voicebot as the first hybrid-setup consumer.

Key Responsibilities:

• Design, implement, and maintain the AWS Data Platform foundation, including Amazon S3 lake layout, Apache Iceberg table format standardization, and AWS Glue Data Catalog integration.
• Build and optimize scalable batch and streaming data pipelines using Amazon EMR (Serverless and EMR-on-EKS), Apache Spark/PySpark, Apache Flink, and Amazon Athena with workgroup cost caps.
• Implement data tokenization and de-identification pipelines on the data side (PA-T) using Protegrity/Spark UDFs for batch, Spark Streaming for micro-batch, and Kafka Connect SMT for streaming PII masking prior to cloud transit.
• Build and manage real-time streaming architectures with Amazon MSK (Managed Streaming for Apache Kafka), MSK Replicator/MirrorMaker2, and Schema Registry integration.
• Implement Medallion Architecture (Bronze, Silver, Gold layers) for lakehouse data modeling, automating Iceberg table registration and backfill frameworks.
• Execute data migration waves across non-PII and PII datasets using AWS DataSync, automated register steps, and streaming migration configs.
• Configure data access control and governance models using AWS Lake Formation, cross-engine authorization, AWS Macie for PII detection, and Informatica Data Catalog (Axon, EDC, BDQ) integration.
• Build automated schema-drift identification components, GitLab CI/CD pipeline integration for dataset synchronization, and automated data contracts/circuit breakers.
• Implement DataOps observability, centralizing logging via Amazon CloudWatch, configuring FinOps cost & anomaly monitoring, and setting up automated alerts.
• Author technical documentation, operational runbooks, disaster recovery (DR) procedures, and cutover/rollback playbooks for data platform hardening.

Requirements:

Mandatory Technical Skills:

• 4+ years of hands-on experience as a Data Engineer or DataOps Engineer building enterprise-grade data platforms and pipelines.
• Strong expertise with AWS Data Analytics stack: Amazon S3, AWS Glue Data Catalog, AWS Lake Formation, Amazon EMR (EMR-on-EKS / Serverless), Amazon Athena, AWS DataSync, and Amazon MSK.
• Deep experience with Apache Iceberg table format, cataloging, compaction, and schema evolution.
• Proficient in Apache Spark / PySpark and Spark Streaming for batch, micro-batch, and real-time data processing.
• Strong experience in Medallion Lakehouse Architecture design and implementation (Bronze, Silver, Gold layers).
• Practical experience in implementing data tokenization and encryption at scale (e.g., Protegrity, Thales, FPE, or Spark UDF-based de-identification pipelines).
• Solid knowledge of event-driven architectures & streaming: Apache Kafka / Amazon MSK, Kafka Connect (SMT), Schema Registry, and MirrorMaker2 / MSK Replicator.
• Expertise with Relational Databases (Amazon RDS, PostgreSQL, Oracle) and data synchronization techniques.
• Hands-on experience with DataOps CI/CD & Automation: GitLab CI/CD, Infrastructure-as-Code (Terraform / AWS CDK), schema-drift detection, and data contract validation.
• Familiarity with data governance tools and enterprise data catalogs (e.g., Informatica Axon/EDC, AWS Lake Formation).

Strong Plus (Nice-to-Have Skills):

• AWS Certified Data Analytics – Specialty or AWS Certified Data Engineer – Associate.
• Experience in telecom domain data models, CDR processing, and high-throughput real-time telemetry.
• Experience with cloud-side tokenization/detokenization via Athena UDFs / AWS Lambda.
• Familiarity with containerization (Docker, EKS, Kubernetes) for big data runtimes.
• Experience with AWS Macie and FinOps cost-allocation/anomaly-detection frameworks.

Soft Skills & Team Fit:

• Strong critical thinking, problem-solving, and analytical skills.
• Excellent communication and collaboration skills to work closely with cross-functional teams (Data Science, Cloud/Platform, Security, Governance).
• Results-oriented, proactive mindset with strong ownership of deliverables within an Agile / Scrum framework.
• Upper-Intermediate+ English level (written and spoken).

We offer*:

• Flexible working format - remote, office-based or flexible
• A competitive salary and good compensation package
• Personalized career growth
• Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
• Active tech communities with regular knowledge sharing
• Education reimbursement
• Memorable anniversary presents
• Corporate events and team buildings
• Other location-specific benefits

*benefits are not applicable for freelancers

OUR COMPANY  

HBM is a European company building exciting new products from scratch for startups and helping mature companies in their journey towards data-driven innovation and AI based solutions. Our expertise refers to EnergyTech, FinTech, ClimateTech, SocialTech, PropTech, etc. 

Founded in Ukraine and developed based on Scandinavian culture, HBM is hiring both in Ukraine and the EU for our customers located in Europe and USA.  

 Our values include skills, passion, excellence, equality, openness, mutual respect, and trust. 

 At HBM, you can become a part of growing company, work with creative colleagues, and enjoy modern technologies and creating AI-based solutions. You’ll be part of a strong corporate culture combined with the agility and flexibility of a start-up backed by proven outsourcing and development practices, a human-oriented leadership team, an entrepreneurial mindset, and an approach to work-life balance. 

 PROJECT 

Our customer is an Icelandic climate technology company company focused on permanent carbon storage solutions. The organization develops and operates innovative technology that captures and stores CO₂ by converting it into stable minerals underground, helping industrial partners reduce emissions and advance sustainability goals. The company combines scientific innovation, large-scale operational expertise, and a mission-driven culture focused on addressing climate change. 

We are looking for a Senior Data Engineer who will be responsible for developing, enhancing, and maintaining enterprise data platform and analytical data flows, as well as supporting ongoing operational work. The role contributes to creating maximum value from data for internal stakeholders by automating routine processes and prepare the platform for a self-service use by customer’s employees. 

The qualified candidate will work together with an on-site Head of Data & SW Development and will handle long-term improvements, data ingestion automation, and new data platform related development. 

Tech stack: Databricks & Lakehouse setup, SQL, Spark, Python, Terraform, Grafana. 

WE PROVIDE YOU WITH THE FOLLOWING EXCITING CHALLENGES 

• Design, build, maintain, and improve data pipelines and data platform components in Azure Databricks  
• Automate manual data collection, processing, and reporting workflows to improve efficiency and data quality  
• Develop scalable data ingestion, transformation, and monitoring solutions using Python, Spark, Terraform, and cloud-native services  
• Own day-to-day operational activities related to the data platform, including pipeline troubleshooting and optimization  
• Support scientists and technical users by enabling self-service data access and exploration capabilities  
• Build reliable data processing solutions that transform laboratory, field, and operational data into trusted datasets  
• Improve platform observability and monitoring using Grafana and related tooling  
• Collaborate closely with the Head of Data on planning, prioritization, architecture decisions, and implementation approaches  
• Identify opportunities to standardize data processes, eliminate manual work, and increase adoption of data-driven practices across the organization  
• Contribute to the foundation for future AI-enabled and self-service data capabilities within the Databricks platform  
• Document solutions, operational procedures, and architectural decisions 

WE EXPECT FROM YOU 

• Degree (bachelor or master) in computer science or a comparable field of study  
• 6+ years of experience working with data engineering solutions and data pipelines 
• Strong Python development skills  
• Hands-on experience with Apache Spark and Databricks  
• Experience building, maintaining, and troubleshooting production-grade data pipelines  
• Experience with Infrastructure as Code, preferably Terraform  
• Experience with Azure cloud services  
• Experience with Git-based development workflows and CI/CD practices  
• Understanding of software engineering principles, including testing, maintainability, and operational support  
• Experience with Grafana for monitoring, observability, and data visualization  
• Ability to work independently, take ownership, and drive initiatives from planning through implementation  
• Experience automating manual business processes using data and software solutions  
• Strong analytical and problem-solving skills 
• Excellent communication skills, ability to communicate with technical and non-technical stakeholders  
• Action and quality-oriented mindset 
• Experience of work the distributed, cross-culture Agile environment 
• English: upper-intermediate / advanced 

 WOULD BE A PLUS 

• Experience building self-service data platforms  
• Experience with streaming or near real-time data processing  
• Experience with AI/ML capabilities in Databricks  
• Experience optimizing Spark workloads and data processing pipelines  
• Familiarity with shell scripting 

WE OFFER YOU 

• Modern technologies, new products development, different business domains. 
• Start-up agility combined with mature delivery practices and management team. 
• Strong focus on your technical and personal growth. 
• Transparent career development and individual development plan. 
• Flexible working mode (remote/work from office), full remote possibility. 
• Competitive compensation and social package 
• Focus on the well-being and human touch. 
• Flat organization where everyone is heard and is invited to contribute. 
• Work-life balance approach to work. 
• Passion and Fun in everything we do. 

We are looking for a Senior Data Engineer to join a strategic marketing technology (Martech) transformation initiative for a large enterprise organization. The project focuses on modernizing the client's marketing data platform, rebuilding data pipelines, and improving marketing attribution across multiple business brands.

As part of a dedicated data engineering team, you will help build a scalable, cloud-native data platform on Google Cloud Platform (GCP), enabling better visibility into marketing performance and supporting data-driven decision making. This is engagement through the end of 2026, working closely with the client's Data Engineering, Product, and Martech teams.

Responsibilities:

• Design, develop, and maintain scalable data pipelines on Google Cloud Platform (GCP).
• Build and optimize data models in BigQuery to support analytics and marketing reporting.
• Rebuild legacy marketing data pipelines following modern engineering best practices and enterprise standards.
• Contribute to the development of a canonical enterprise data model and semantic data layer.
• Support the implementation and adoption of DataHub as the enterprise data catalog.
• Develop automated CI/CD pipelines for data engineering workflows and enforce coding and data quality standards.
• Ensure proper data isolation between multiple business brands while enabling consolidated reporting where appropriate.
• Collaborate closely with Product Managers, Martech specialists, Analytics teams, and other data engineers to deliver business value.
• Optimize data processing performance, reliability, monitoring, and cost efficiency.

Required Qualifications:

• 5+ years of experience as a Data Engineer.
• Strong hands-on experience with Google Cloud Platform (GCP).
• Advanced experience with BigQuery.
• Strong SQL and Python development skills.
• Experience designing and implementing modern ELT/ETL pipelines.
• Solid understanding of dimensional modeling, semantic modeling, and enterprise data architecture.
• Experience building production-grade CI/CD pipelines for data platforms.
• Familiarity with Git and modern software engineering best practices.
• Strong communication skills and ability to work within cross-functional teams.
• Upper-Intermediate or higher English.

Nice to Have:

• Experience with marketing technology (Martech) platforms and marketing attribution.
• Hands-on experience integrating: Google Ads, Google Analytics (GA4), Search Ads 360 (SA360), Campaign Manager 360, DV360
• Experience with DataHub or similar data catalog solutions.
• Experience implementing semantic layers for analytics platforms.
• Understanding of multi-brand or multi-tenant data architectures.
• Experience modernizing legacy data platforms and rebuilding pipelines from scratch.

The Data/Ingestion Engineer will design and implement scalable ingestion pipelines for high-volume structured and unstructured insurance documents, including PDFs, scans, emails, Word, Excel and PowerPoint files. 

The role includes OCR/document extraction, text cleaning and normalization, semantic chunking, metadata tagging, vector storage/RAG preparation, validation, monitoring and integration with downstream AI solutions.

• At least 7 years of experience as Data Engineer with Python and SQL
• At least 5 with AWS, the rest should be more or less but should be real experience that the candidate is able to talk about and give a concrete example/
• Must to have : 
  • Python
  • SQL
  • AWS
  • S3
  • Step Functions
  • CloudWatch
  • Data/document ingestion pipelines
  • Processing of unstructured documents:
  • PDF
  • scanned documents
  • Word
  • Excel
  • PowerPoint
  • emails
  • OCR / document extraction technologies
  • AWS Textract or equivalent
  • Connectors/integrations with sources such as:
  • SharePoint
  • email
  • Text cleaning and normalization
  • Metadata extraction/tagging
  • Automated error monitoring
  • Validation of OCR/extraction results
  • Git
  • CI/CD
  • Testing
  • Public-cloud data processing/document ingestion experience
  • Good to have :
  • Vector databases
  • RAG
  • Semantic chunking
  • Vector-storage schema design
  • Retrieval mechanisms
  • Azure
  • Databricks
  • AWS Textract specifically
  • Experience within insurance / financial services
  • Experience handling enterprise security and low-latency SLA requirements
• English Level C1
• Duration of the mission :  1 year

Client

Our client is one of the leading Europe’s alternative asset managers that provide effective capital solutions to meet needs of wide range of corporate and real asset owners.

Project overview

The project focuses on building a cloud based data platform that centralizes, transforms, and delivers high quality data across multiple business domains. The environment supports advanced analytics, data governance, and scalable data processing using modern Azure and Snowflake technologies.

Position overview

We are looking for a Senior Data Engineer with strong expertise in Snowflake and Azure data technologies to contribute to the development and enhancement of a large scale enterprise data platform. You will work on designing, building, and optimizing modern data solutions that support analytics, reporting, and business decision making. This role is ideal for a data professional who enjoys working with cloud data platforms, complex data pipelines, and modern data warehousing architectures.
 

Technology stack

Snowflake, Snowflake SQL, Snowflake Stored Procedures, Snowflake UDFs, Snowpark, Python, Apache Airflow, Azure SQL Database, Microsoft SQL Server, T SQL, dbt, Airflow Cosmos, Snowflake CLI, Azure Cloud Services, Data Warehousing, ETL, ELT, Medallion Architecture, Layered Architecture, Dimensional Modeling, Git

Responsibilities

• Develop and maintain scalable data solutions on the Snowflake platform
• Design and implement Snowflake databases, schemas, tables, views, and materialized views
• Build, optimize, and support ETL and ELT data pipelines
• Create and maintain stored procedures and user defined functions using Python and SQL
• Design and develop data warehouse solutions using layered and Medallion architecture principles
• Optimize Snowflake and SQL Server performance through indexing, partitioning, query tuning, and workload optimization
• Develop complex SQL queries utilizing analytical and window functions
• Collaborate with architects, business stakeholders, and engineering teams to deliver high quality data solutions
• Support data quality validation, reconciliation, and testing activities
• Prepare and maintain technical documentation related to data solutions and architecture
• Contribute to continuous improvement initiatives and engineering best practices

Requirements

• Strong experience with Snowflake data platform engineering
• Advanced knowledge of Snowflake SQL
• Experience with Snowflake databases, schemas, tables, views, and materialized views
• Experience developing Snowflake stored procedures and UDFs using Python and SQL
• Strong experience with Apache Airflow
• Strong Python development skills
• Hands on experience with Azure SQL Database, Microsoft SQL Server, and T SQL
• Experience creating and optimizing tables, views, stored procedures, functions, indexes, and partitions
• Strong knowledge of SQL performance tuning and query optimization
• Experience developing ETL and ELT pipelines
• Experience building enterprise data warehouse solutions
• Knowledge of Layered Architecture and Medallion Architecture
• Experience with dimensional modeling
• Good verbal and written English communication skills

Nice to have

• Experience with dbt
• Experience with Airflow Cosmos
• Knowledge of Snowpark
• Experience working with Snowflake CLI
• Experience with Master Data Management design and integration
• Experience with QlikView or Qlik Sense
• Experience with data quality testing, validation, and reconciliation
• Experience creating technical documentation
• Experience within investment banking or asset management environments
• Experience using Claude AI within the Software Development Life Cycle

Шукаємо у команду Senior DevOps / SRE Engineer (Remote).

Якщо ви маєте бажання працювати в невеликому колективі разом із сильною командою, радо розглянемо ваш відгук! Ми шукаємо не лише сильного DevOps Engineer, а й інженера із досвідом Site Reliability Engineering (SRE), який готовий брати на себе технічне лідерство, ділитися знаннями та допомагати розвиватися іншим членам команди. Націлені на довгострокову співпрацю з компенсацією, що базуватиметься на вашому досвіді роботи.
 

Завантаженість: повна зайнятість — 40 год/тиждень. Робота віддалена. Оплата погодинна.
 

Стек технологій:

- Scripting: Bash | Python | Makefile

- Containers and Orchestration: Kubernetes | Docker | GKE

- Infrastructure as Code (IaC): Terraform | Ansible

- CI/CD & Automation: GitHub Actions | CircleCI | TravisCI

- Cloud Providers & IaaS: AWS | GCP | Azure | Hetzner

- Databases: SQL | NoSQL (PostgreSQL, Redis…)

- Authorization and Security: OpenSSL

- Observability and Monitoring: Prometheus | Grafana | ELK Stack (Elasticsearch, Logstash, Kibana)

- Operating Systems: Linux
 

Site Reliability Engineering (SRE):
- SLI / SLO / SLA | Incident Response | Root Cause Analysis (RCA) | - High Availability | Capacity Planning | Disaster Recovery | 
- Performance Optimization | Reliability Engineering | 
 

Буде плюсом:

- Infrastructure Management: Developing and implementing infrastructure, configuration, and code branching strategies.

- Quality Assurance & Testing: Conducting integration, regression, UI/UX, API, load/stress, and other types of testing.

- Bug Tracking & Issue Management: Identifying, documenting, and tracking software defects and bugs.

- Performance & Optimization: Monitoring and analyzing quality metrics to identify areas for improvement.

- Automated Testing: Experience with automated testing frameworks such as Selenium, Appium, JMeter, etc.

Mentoring & Leadership:
- Досвід менторингу Junior/Middle інженерів, проведення code review, технічних обговорень, knowledge sharing та допомоги команді у професійному розвитку.
 

Вимоги до кандидата:

- Досвід роботи DevOps Engineer та практичний досвід Site Reliability Engineering (SRE)
- Досвід побудови та підтримки високодоступних production-систем із високими вимогами до reliability.

- Досвід роботи з Kubernetes у production-середовищі

- Розуміння принципів високої доступності, масштабування та відмовостійкості систем

- Досвід побудови та підтримки CI/CD процесів

- Англійська мова — Intermediate+

- Уміння слідувати code conventions

- Здатність навчатися

- Здатність наполегливо працювати

- Критичне мислення

- Витривалість

- Спроможність лаконічно та чітко пояснювати свою думку

- Бажання ділитися знаннями, проводити менторинг та допомагати розвиватися іншим членам команди

- Відданість нашим проєктам

- Прозорість
 

Що отримаєте:

- Віддалені умови роботи

- Удосконалення навичок планування та продуктивності

- Робота у вимогливому, але дружньому середовищі серед A-гравців

- Регулярний зворотний зв'язок і допомога в плануванні вашого навчального процесу та кар'єрного зростання

- Незалежність від місцевого ринку та долучення до глобальної економіки

- Підвищення рівня володіння англійською мовою

- Корпоративні навчання, технічні воркшопи та тренінги для постійного підвищення кваліфікації

We are looking for a senior data engineer with deep Databricks experience to own the data platform behind one of our largest enterprise accounts and lead its migration from AWS to Databricks.

About the role

Data infrastructure is the foundation of everything we deliver. Our models, dashboards, and the decisions our clients make on top of them are only as reliable as the pipelines underneath, which is why we put the platform first.

You will join a small senior team working inside the client's environment. Through the rest of 2026 you will take ownership of the existing pipelines and the architecture around them. From 2027 you will lead the migration to Databricks: designing the target architecture, planning the cutover, and doing the implementation. This is a hands-on engineering role with real architectural authority. You decide how the platform gets built, and then you build it.

You will be the person the team turns to on anything Databricks, working alongside senior machine learning engineers and data scientists who depend on what you deliver.

What you will do:

• Design and build the target Databricks architecture, and plan the migration from the current AWS stack
• Build and maintain production data pipelines: ingestion, transformation, and orchestration
• Own data quality, including the checks and alerts that catch problems before the client does
• Design the data models the analytics and machine learning work sits on
• Support the pipelines you build: monitoring, alerting, and the on-call reality of production data
• Work directly with the client's engineers, and explain technical decisions to people who are not engineers

Requirements:

• Five or more years in data engineering or software development
• At least three years of production Databricks work
• Deep experience with Spark, Delta Lake, and Unity Catalog
• Has planned and executed at least one migration onto Databricks from another platform, and can walk through the challenges
• Expert Python: clean modular code, comfortable with optimization and profiling
• Advanced SQL and relational database work: raw SQL, data modeling, scripted migrations
• Systems design: how storage and compute fit together, end to end pipelines, containerized applications
• Cloud operations on AWS or equivalent: networking, identity and access management, monitoring and alerting, and the domain knowledge for why things break
• Comfortable working inside a client's environment, with their data and their constraints
• Strong written English, and spoken English good enough to present your work
• Available for some overlap with United States working hours

Nice to have:

• Databricks certification
• Deep AWS experience specifically, since that is the platform we are migrating from
• dbt, Airflow, or a similar transformation and orchestration framework
• Machine learning pipeline experience, since models run on top of what you build
• Background in utilities, energy, telecoms, or another regulated industry
• Experience being the senior engineer a client relies on directly

You are:

• Careful with data quality, and you ask when something looks wrong instead of guessing
• A self starter who can turn a general requirement into a specific estimate and a deliverable
• Someone who documents. We write things down, and every number we deliver carries its source
• Comfortable in a small remote team where nobody is checking your hours

About Bennett Data Science

We have spent ten years building predictive analytics and data science for enterprise clients, from some of the largest companies in the world to startups in Silicon Valley. Our engineers sit inside the client's systems and build critical systems. The work ships to production and lands at the bottom line.

What we offer:

• Competitive compensation, reviewed as you grow
• Work from anywhere, with a flexible schedule
• A small senior team, so your work is visible and your decisions are important
• Direct exposure to enterprise clients and the problems they actually have
• Mentorship from senior staff, and the space to grow and mentor others

Our client is looking for a Senior Data Engineer to design, scale, and maintain high-performance ingestion, Change Data Capture (CDC), and delivery pipelines. In this role, you will architect enterprise-grade data platforms to process large-scale metadata, algorithmic feature extraction, and licensable asset datasets.

This team sits at the foundation of the data processing and feature enrichment pipelines. Driving consistency and performance across an ever-growing data lake, we provide guardrails and sound system design patterns to inform the creation and fine-tuning of the industry's cutting-edge foundational large language models.

You will leverage Google Cloud Platform (GCP) as the primary compute foundation of a multi-cloud data processing ecosystem, alongside hybrid data warehouses and query engines including Snowflake and ClickHouse to provide low-latency data exposure, incremental delta tracking, and seamless integration endpoints for downstream AI/ML consumption.

Primary Tech Stack: GCP, ClickHouse, Snowflake, dbt Core, Apache Airflow, Python, SQL.

Responsibilities

• Ingestion Architecture & GCP Foundation:
• Architect, build, and optimize scalable job creation, batching, and high-throughput ingestion pipelines hosted on GCP.
• Implement serverless and containerized data pipelines using GCP infrastructure (BigQuery, Dataflow, Pub/Sub, Cloud Storage, GKE).
• Enterprise ETL, CDC & Incremental Processing:
• Design robust Change Data Capture (CDC) and incremental update workflows to enable fast delta synchronization across platforms.
• Build dbt transformation models and Airflow orchestrations following Data Vault / dimensional modeling patterns.
• Implement delta tracking methods (e.g., hash keys, hash diffs) to track row-level asset modifications and metadata events.
• Multi-Engine Analytics & Query Optimization:
• Maintain and optimize Snowflake integration layers, handling deduplication logic, canonical content filtering, and policy restrictions.
• Deploy and manage ClickHouse as a high-concurrency analytical engine to deliver sub-second metadata retrieval and indexing.
• Construct dedicated integration views and API query layers allowing internal and external clients to consume real-time delta updates.
• Data Quality, Governance & Scalability:
• Enforce metadata schema standardization, canonical asset mapping, and automated validation layers across diverse asset types (images, video, 3D models, audio).
• Collaborate closely with cross-functional teams to integrate metadata pipelines with feature extraction tools, search indexes, and licensing platforms.
   

Requirements

• Google Cloud Platform (GCP): 4+ years of production experience with core GCP data technologies (BigQuery, Dataflow, Cloud Pub/Sub, GKE, Google Cloud Storage).
• Database & Query Engines: Deep technical proficiency with both Snowflake (warehousing, semi-structured queries, integration layers) and ClickHouse (real-time aggregation, low-latency columnar storage, indexing).
• Enterprise ETL / CDC: Proven experience architecting CDC pipelines and incremental delta loading for multi-terabyte dataset scale.
• Data Transformation & Orchestration: Hands-on mastery of dbt Core/Cloud and Apache Airflow in an enterprise CI/CD setting (e.g., GitHub Actions).

• Programming Skills: Advanced skills in Python and SQL; familiarity with Go, Java, or Scala is a strong plus.

   

Nice-to-Have Qualifications:

• Experience with Data Vault 2.0 architecture, hash-key generation, and raw/staging/integration layer design.
• Background handling AI/ML metadata, computer vision feature vectors, C2PA provenance headers, or semantic search configurations.
• Familiarity with media metadata schemas (images, 3D model stems, video derivatives).
• Experience with other major cloud providers such as AWS, Azure, or Oracle Cloud.
   

Key Success Metrics (First 90 Days)

• GCP Pipeline Optimization: Standardize ingestion batching and pipeline compute infrastructure on GCP.
• ClickHouse & Snowflake Integration: Establish streamlined integration views exposing low-latency metadata delta changes.
• Automated CDC Workflows: Finalize automated dbt delta transformations with hash tracking across primary warehouse layers.
   

We offer*:

• Flexible working format - remote, office-based or flexible
• A competitive salary and good compensation package
• Personalized career growth
• Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
• Active tech communities with regular knowledge sharing
• Education reimbursement
• Memorable anniversary presents
• Corporate events and team buildings

• Other location-specific benefits

  *not applicable for freelancers

Our client is looking for a Senior Data Engineer to design, scale, and maintain high-performance ingestion, Change Data Capture (CDC), and delivery pipelines. In this role, you will architect enterprise-grade data platforms to process large-scale metadata, algorithmic feature extraction, and licensable asset datasets.

This team sits at the foundation of the data processing and feature enrichment pipelines. Driving consistency and performance across an ever-growing data lake, we provide guardrails and sound system design patterns to inform the creation and fine-tuning of the industry's cutting-edge foundational large language models.

You will leverage Google Cloud Platform (GCP) as the primary compute foundation of a multi-cloud data processing ecosystem, alongside hybrid data warehouses and query engines including Snowflake and ClickHouse to provide low-latency data exposure, incremental delta tracking, and seamless integration endpoints for downstream AI/ML consumption.

Primary Tech Stack: GCP, ClickHouse, Snowflake, dbt Core, Apache Airflow, Python, SQL.

Responsibilities

• Ingestion Architecture & GCP Foundation:
• Architect, build, and optimize scalable job creation, batching, and high-throughput ingestion pipelines hosted on GCP.
• Implement serverless and containerized data pipelines using GCP infrastructure (BigQuery, Dataflow, Pub/Sub, Cloud Storage, GKE).
• Enterprise ETL, CDC & Incremental Processing:
• Design robust Change Data Capture (CDC) and incremental update workflows to enable fast delta synchronization across platforms.
• Build dbt transformation models and Airflow orchestrations following Data Vault / dimensional modeling patterns.
• Implement delta tracking methods (e.g., hash keys, hash diffs) to track row-level asset modifications and metadata events.
• Multi-Engine Analytics & Query Optimization:
• Maintain and optimize Snowflake integration layers, handling deduplication logic, canonical content filtering, and policy restrictions.
• Deploy and manage ClickHouse as a high-concurrency analytical engine to deliver sub-second metadata retrieval and indexing.
• Construct dedicated integration views and API query layers allowing internal and external clients to consume real-time delta updates.
• Data Quality, Governance & Scalability:
• Enforce metadata schema standardization, canonical asset mapping, and automated validation layers across diverse asset types (images, video, 3D models, audio).
• Collaborate closely with cross-functional teams to integrate metadata pipelines with feature extraction tools, search indexes, and licensing platforms.
   

Requirements

• Google Cloud Platform (GCP): 4+ years of production experience with core GCP data technologies (BigQuery, Dataflow, Cloud Pub/Sub, GKE, Google Cloud Storage).
• Database & Query Engines: Deep technical proficiency with both Snowflake (warehousing, semi-structured queries, integration layers) and ClickHouse (real-time aggregation, low-latency columnar storage, indexing).
• Enterprise ETL / CDC: Proven experience architecting CDC pipelines and incremental delta loading for multi-terabyte dataset scale.
• Data Transformation & Orchestration: Hands-on mastery of dbt Core/Cloud and Apache Airflow in an enterprise CI/CD setting (e.g., GitHub Actions).

• Programming Skills: Advanced skills in Python and SQL; familiarity with Go, Java, or Scala is a strong plus.

   

Nice-to-Have Qualifications:

• Experience with Data Vault 2.0 architecture, hash-key generation, and raw/staging/integration layer design.
• Background handling AI/ML metadata, computer vision feature vectors, C2PA provenance headers, or semantic search configurations.
• Familiarity with media metadata schemas (images, 3D model stems, video derivatives).
• Experience with other major cloud providers such as AWS, Azure, or Oracle Cloud.
   

Key Success Metrics (First 90 Days)

• GCP Pipeline Optimization: Standardize ingestion batching and pipeline compute infrastructure on GCP.
• ClickHouse & Snowflake Integration: Establish streamlined integration views exposing low-latency metadata delta changes.
• Automated CDC Workflows: Finalize automated dbt delta transformations with hash tracking across primary warehouse layers.
   

We offer*:

• Flexible working format - remote, office-based or flexible
• A competitive salary and good compensation package
• Personalized career growth
• Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
• Active tech communities with regular knowledge sharing
• Education reimbursement
• Memorable anniversary presents
• Corporate events and team buildings

• Other location-specific benefits

  *not applicable for freelancers

Build what happens behind the query.

Work on the core of a high-performance data platform — query planning, optimization, distributed execution and performance at scale.

About Our Client

Our client is a leading enterprise data platform company building an open, high-performance data lakehouse for AI and analytical workloads. The platform combines an intelligent SQL query engine, an AI-ready semantic layer, and an open catalog built on Apache Iceberg — enabling Fortune 500 companies across finance, energy, manufacturing, and logistics to unify, query, and govern data at massive scale across cloud and on-premise sources.

About the Role

We are looking for a Senior Software Engineer to work on the core query engine of a large-scale distributed data platform. You will own significant components across query planning, optimization, and execution — both driving improvements and acting as a trusted owner of production quality: diagnosing complex customer-reported issues and delivering safe, well-tested fixes across supported release lines. Engineers with staff-level scope and ambitions are very welcome.

This is a systems-level, backend engineering role focused on distributed data processing internals — not application development or CRUD services.

Responsibilities

• Architect, design, and implement query engine components — planner, optimizer, execution operators, memory management, and data access layers
• Diagnose and resolve complex production and customer-reported issues across the full query path — correctness, performance, memory, concurrency, resiliency, and compatibility
• Develop, review, and backport fixes across multiple supported release lines; strengthen regression prevention
• Write and review high-quality, performance-critical code in Java and/or C++
• Analyze query profiles, execution plans, logs, and telemetry to drive performance optimization — profiling, benchmarking, and eliminating bottlenecks in latency- and throughput-sensitive paths
• Integrate with columnar formats, open table formats, and connectivity drivers
• Improve regression tests, diagnostics, and observability; contribute to CI/CD architecture and quality processes in Jenkins
• Mentor middle engineers, lead code reviews, and help set technical standards

• Debug complex cross-layer issues spanning query planning, distributed execution, memory management, and I/O, partnering with US-based engineering leads

   

Required Qualifications

• B.S. or M.S. in Computer Science, Computer Engineering, or a related field
• 5+ years in backend / systems software engineering with a track record of owning significant components end-to-end
• Strong proficiency in Java or C++ (comfort with both is especially valued), with deep understanding of OOP, systems design, memory management, concurrency, and asynchronous programming
• Advanced SQL and solid understanding of query execution internals — planning, optimization, operator implementation
• Experience building or extending data processing systems: query engines, distributed databases, ETL/ELT, or analytical platforms
• Ability to assess change risk in a mature production system; experience supporting software across multiple versions, branches, or customer environments
• Hands-on experience with Jenkins and modern CI/CD workflows; Docker and Kubernetes
• Hands-on experience with at least one major cloud (GCP, AWS, or Azure)
• Confident Git/GitHub workflows and rigorous code-review habits
• Comfortable with AI-assisted development workflows — using modern AI tools for code comprehension, debugging, and test generation, and critically validating their output
• Prior experience mentoring engineers, leading technical designs, or owning components end-to-end
• English Upper-Intermediate or higher (B2+) — daily written and verbal communication with a US-based engineering team

• Availability to work EU business hours shifted 2–3 hours later for daily overlap with US West Coast mornings

   

Desired Skills

• Apache Arrow (columnar in-memory format) and SQL planner/optimizer frameworks such as Apache Calcite
• LLVM-based runtime expression compilation / code generation
• Open table formats — Apache Iceberg, Delta Lake, or Hudi — and columnar file formats such as Parquet, Avro, or ORC
• MPP query engines (Presto, Trino, or similar); background with distributed data platforms such as Spark, Snowflake, or Databricks
• Deep understanding of query optimization (cost-based, rule-based), vectorized execution, memory management, spilling, and caching
• Connectivity drivers: JDBC, ODBC, Arrow Flight
• Managed Kubernetes (GKE/EKS/AKS) and multi-cloud exposure; Terraform; cloud object storage (AWS S3, Azure Data Lake Storage, Google Cloud Storage)
• SaaS operations experience: incident response, profiling, observability, release and backport processes

• Systems-level performance work and open-source contributions to data infrastructure projects

   

Details

• Engagement: Long-term contract
• Location: Europe (EU / EEA / UK), remote
• Working hours: EU business hours, shifted 2–3 hours later for daily overlap with the US West Coast team

Responsibilities 

• Design, develop, and maintain backend application logic for invoice retrieval, validation, processing, and routing.
• Integrate with external systems through REST APIs using JSON and OAuth 2.0, including content-based routing scenarios.
• Build, deploy, and operate the solution as a Databricks App using Databricks Workflows, Delta Lake, and Unity Catalog.
• Develop and maintain an operational user interface for monitoring processing status, exceptions, and error handling.
• Implement robust logging, monitoring, observability, and error-handling mechanisms to ensure production reliability.
• Apply strong software engineering practices, including version control, CI/CD, automated unit and integration testing, and controlled deployments.
• Troubleshoot production issues and continuously improve application stability, performance, and maintainability.
• Collaborate with business and technical stakeholders to clarify requirements and deliver reliable application functionality.

     Requirements

• Strong hands-on experience with Databricks Apps, Databricks Workflows, Delta Lake, and Unity Catalog.
• Experience with e-invoicing solutions or invoice-processing workflows.
• Strong Python backend development skills, including validation logic, routing, API integrations, and application-level processing.
• Solid experience working with REST APIs, JSON, OAuth 2.0, and content-based routing patterns.
• Hands-on experience developing user interfaces with Streamlit or React within Databricks Apps.
• Good practical experience with Git, CI/CD pipelines, automated unit and integration testing, logging, monitoring, and production support.
• Strong understanding of software engineering principles and production-grade application development.
• Proven experience delivering and supporting stable production applications, rather than working exclusively with data pipelines.
• Strong troubleshooting and problem-solving skills, with the ability to diagnose application and integration issues.

Nice to have

• Experience integrating with SAP Ariba, ERP, procurement, or related enterprise platforms.
• Knowledge of UBL, PEPPOL, or similar electronic invoicing standards.
• Previous experience delivering and operating Databricks Apps in production environments.

About the team
We are a focused engineering team of 5 to 9 people at a multi-product tech SaaS company. We work across personalization, recommendation, fraud detection, and some GenAI, through real-time, near-real-time, and batch pipelines. The team is a blend of Python backend engineers, ML engineers, QA (manual and automation), a data engineer, and data scientists. Our shared mandate is to take experimentation from our data scientists and turn it into
robust, production-grade systems, either as standalone services or integrated into wider platforms.

The role
You will build the data pipelines that power our experimentation and production systems, the backbone behind fraud detection, recommendations, and personalization. You will work primarily with Kafka (sometimes Flink), Spark on Databricks, and Airflow, modelling and optimizing data for both experimentation and production serving, and partnering closely with data scientists and backend engineers on data contracts and reliability.

Responsibilities
• Build and operate batch and streaming data pipelines (structured & unstructured data)
• Work primarily with Kafka, Spark on Databricks, and Airflow
• Model and optimize data for both experimentation and production serving
• Partner with data scientists and backend engineers on data contracts and reliability
• Own pipeline quality: testing, monitoring, and performance
• Take part in the teams on-call rotation for the pipelines you build and operate

Requirements
• 3+ years of data engineering or software engineering for data

• Strong Python, including PySpark
• Kafka and stream processing
• Databricks experience
• Airflow or similar orchestration tools
• Strong SQL and data modeling, plus solid relational database skills
• Software engineering mindset
• Fluent spoken English, as you’ll work with an international team and all interviews will be conducted in English.

Nice to have
• Flink

• ML Feature stores
• MLOps and ML pipeline exposure
• Docker and Kubernetes
• SaaS or high-scale product domain experience

Why join us
• Work on high-scale, high-throughput systems
• Broad range of technologies, stacks, and problems
• Contribute to turning experimentation into production systems
• A strong engineering culture that values technical depth and ownership
• Flexibility through hybrid and remote working
• A distributed team across Dubai and Europe
 

You will get

• A position in a product development company with a dynamic environment and several concurrent projects;
• Continuous self-improvement and growth, including budget for certifications and courses;
• Competitive salary plus financial bonuses;
• Medical insurance coverage;
• Company prepaid AI agent;
• Wellbeing package: online-yoga classes, Yakaboo, BetterMe App: Health Coaching, BetterMe App: Mental Health;
• Corporate events and fun team-building activities.
• Remote-first culture
   

Interview Stages

1. HR Interview (30-45 minutes) — Initial conversation to discuss your experience, career goals, and cultural fit.
2. Technical Interview (1 hour) — In-depth technical interview covering relevant skills.
3. Final Interview (1,5 hour) — A comprehensive discussion with the team, focusing on role-specific competencies and alignment with company values.
4. Reference check & Job Offer

We are looking for an experienced Senior Data Engineer to join a growing Data Engineering and Analytics department. In this role, you will design and develop advanced data solutions for complex customer environments, with a strong focus on Google Cloud Platform and modern GCP data technologies. You will work across the full data lifecycle- from architecture and data modeling to pipeline development, data platforms, analytics, and production deployment.

Key Responsibilities:

• Design and develop scalable data solutions on GCP.
• Lead the technical design and implementation of customer data projects.
• Understand business and technical requirements and translate them into effective data architectures.
• Build and maintain ETL/ELT pipelines, Data Lakes, Lakehouses, and cloud-based Data Warehouses.
• Design data models and integration processes for Batch and real-time workloads.
• Work with structured, semi-structured, and unstructured data.
• Select the appropriate technologies based on performance, scalability, security, and cost requirements.
• Implement data quality, monitoring, governance, and orchestration processes.
• Work closely with Data Architects, Data Engineers, DevOps teams, analysts, and customer stakeholders.
• Participate in the development of analytics, AI, and ML solutions where relevant.
  
  Requirements:
• At least 5 years of professional experience as a Data Engineer – mandatory.
• Proven hands-on experience developing data solutions on GCP – mandatory.
• Experience with data visualization tools such as Looker, Power BI, Tableau, or QuickSight.
• Experience with data modeling, orchestration, performance optimization, and large-scale data processing.
• Strong Python development skills, including building data pipelines and ETL/ELT processes.
• High proficiency in SQL – mandatory.
• Experience designing and developing cloud-based Data Warehouses and Lakehouse solutions.
• Experience with ETL/ELT and transformation tools such as dbt, Dataform, Rivery, or similar platforms.
• Familiarity with CI/CD, Git, Infrastructure as Code, and production deployment practices.
• Strong analytical and problem-solving skills with excellent attention to detail.
• Ability to learn new technologies independently and work across multiple projects.
• Strong experience with several of the following GCP services: BigQuery, Cloud Storage, Dataflow, Dataproc, Cloud Composer, Cloud Run or Cloud Functions
• Fluent English.
  
  Advantages
• Hands-on experience with AWS or Microsoft Azure data services.
• Experience with services such as AWS Glue, Redshift, EMR, Kinesis, Azure Data Factory, Synapse, or Databricks.
• Experience with real-time data processing and streaming architectures.
• Experience with Kafka or other event-driven platforms.
• Knowledge of AI and ML services such as Vertex AI, Gemini, BigQuery ML, SageMaker, Bedrock, or Azure Machine Learning.
• Relevant GCP professional certifications.
• Previous experience working in consulting or customer-facing technology projects.

Our client is looking for a Data Engineer.

Role Overview:Our client is looking for a Data Engineer to help them develop and maintain their data platform. Their infrastructure is primarily AWS-based, with Airflow, S3 and Redshift at its core, alongside a legacy stack based on MS SQL Server, SSIS and custom Python scripts. You will work on building reliable data pipelines, improving data infrastructure, and gradually modernising legacy processes.Key Responsibilities:

• Design, develop and maintain scalable and reliable data pipelines and ETL/ELT processes;
• Develop and optimise data warehouse solutions;
• Develop and maintain data infrastructure, primarily on AWS, while also supporting parts of our legacy infrastructure;
• Develop APIs and data integrations;
• ontribute to the development and maintenance of real-time data pipelines using Kafka;
• Automate and optimise data processes and workflows;
• Monitor, troubleshoot and improve the reliability, performance, security and data quality of pipelines and infrastructure;
• Work closely with the analytics team and other stakeholders to understand their data needs and deliver effective solutions;
• Maintain clear and up-to-date documentation for data pipelines, infrastructure and workflows;
• Contribute to the improvement and modernisation of existing data processes and architecture.

Required Skills & Experience:

• 2+ years of professional experience in data engineering;
• Strong SQL and Python skills;
• Hands-on experience with AWS, including S3, Redshift, Glue and Lambda;
• Hands-on experience with Apache Airflow;
• Strong understanding of data warehousing concepts and data modelling;
• Hands-on experience designing, developing and maintaining production data pipelines, with a focus on reliability, monitoring and data quality;
• Solid understanding of relational and non-relational databases;
• Familiarity with Git and CI/CD practices, preferably GitLab;
• Experience developing APIs and data integrations;
• Kafka and real-time data processing experience is a plus;
• MS SQL Server and SSIS experience is a plus;
• Good written and verbal communication skills;
• Upper-intermediate English or higher.

The company guarantees you the following benefits:

• A positive workplace atmosphere that creates a culture of collaboration and support, making it a place you'll love working in;
• Competitive compensation and regular career development reviews;
• Flexible working hours and remote working options, you'll enjoy the freedom that the company provides;
• A generous vacation and sick leave policy, allowing you to take time off and enjoy a work-life balance;
• Financial assistance for professional development, helping you stay ahead of the curve and love your career path;
• Educational Allowances that give you the opportunity to expand your knowledge and experience;
• You'll have a monthly allowance for personal activities, giving you the opportunity to pursue your interests and hobbies outside of work;
• A comprehensive health insurance plan, depending on your current location;
• Referral program with financial rewards for bringing top talent to the company;
• Engaging in team-building activities and corporate parties.

Interview process:

• HR Interview with the Recruiter;
• Techincal inetrview;
• Final interview with the team.

If you find this opportunity right for you, don't hesitate to apply or get in touch with us if you have any questions!

We are looking for a Senior Data Engineer to join a project in the tech capital of the world – Silicon Valley.
 

17:00–02:00 Kyiv time = 7:00–16:00 Pacific Time (PST/PDT).
 

Project Idea

The project was founded back in 2014 with the goal of connecting private and government universities with regular people like we are. You have a variety of auditoriums, gyms, classrooms, and other venue options available for community use –
schedule facility uses and manage requests from the community all in one place.

Just imagine that you’re a football player and you can rent a football field at Harvard to play with your friends. Amazing, right?

Must have:

• 5 + years of experience in Data Engineering or related roles;
• Strong SQL skills and hands-on experience with PostgreSQL (including Aurora Serverless);
• Solid knowledge of Python for data processing and automation;
• Experience building and maintaining ETL/ELT pipelines using cloud-native tools (e.g., AWS Lambda, S3, SQS);
• Proven experience working with MongoDB and MongoDB Atlas, including event-driven architectures using Atlas Triggers, Stream Processing;
• Proficiency with dbt for building modular, testable, and well-documented data transformation workflows;
• Good understanding of data modeling principles for OLAP/OLTP systems, including normalization and dimensional modeling;
• Demonstrated experience designing and implementing data warehouses and data marts;
• Working knowledge of Node.js, particularly in backend logic tied to data ingestion or transformation workflows;
• Familiarity with cloud data platforms (e.g., AWS) and serverless computing patterns;
• A technical degree (e.g., Computer Science, Engineering, Math) is a plus;
• Upper-Intermediate English or higher for effective communication and documentation.

Soft skills:

• Proactive – you take ownership and act without waiting for direction;
• Detail-oriented – you deliver accurate, high-quality work;
• Initiative-driven – you’re eager to improve processes and take action.

Responsibilities:

• Design, implement, and maintain scalable and reliable data pipelines using Python, dbt, and AWS Lambda;
• Build and optimize data architectures to support analytics, reporting, and machine learning use cases, including data warehouse and data mart modeling on PostgreSQL (Aurora Serverless);
• Develop and manage ELT workflows that extract data from MongoDB (using Atlas triggers) and load into staging and production layers in PostgreSQL;
• Ensure data consistency and lineage by applying robust data quality checks, auditing, and reconciliation logic;
• Collaborate with cross-functional teams to gather data requirements, understand business logic, and translate them into efficient data models and transformations;
• Monitor and troubleshoot SQS/Lambda-based ingestion pipelines, addressing issues related to concurrency, message processing, and data duplication;
• Contribute to the semantic layer design used by BI and reporting tools to ensure consistency and accessibility of business metrics;
• Maintain and evolve dbt models (staging, intermediate, and marts) aligned with software engineering and analytics best practices;
• Drive continuous improvement in data engineering processes and data governance standards, ensuring scalability, maintainability, and security.