Jobs Data Engineer

We are seeking a Middle Data Engineer to join our AI-Native Delivery Pod. In this role, you will move beyond traditional ETL development to become an "Architect of Data Flows," leveraging AI agents to design, build, and test complex data pipelines.

Your focus will be on data-intensive, No-UI projects where the primary value lies in the integrity, speed, and structure of the data. You will use Claude Code, Cursor, and agentic workflows to automate routine SQL and Python development, allowing you to concentrate on logic validation, data quality assurance, and performance optimization within cloud environments. As part of an AI-Native pod, you will operate at a velocity significantly higher than traditional teams by treating AI as your primary executor.

Required Qualifications

• Degree: Bachelor’s degree in Computer Science, Applied Mathematics, Data Engineering, or a related technical field.
• Professional Experience: 3+ years of experience in Data Engineering, specifically building and maintaining production-grade data pipelines.
• Tech Stack Mastery: Strong proficiency in Python (Pandas, PySpark, Boto3) and advanced SQL (window functions, optimization, procedural SQL).
• Data Tooling: Hands-on experience with orchestration tools (e.g., Airflow, Prefect, or Dagster) and data transformation frameworks like dbt.
• Cloud Ecosystem: Practical experience with cloud data warehouses and services, specifically within the Azure ecosystem (Azure Data Factory, Synapse, Databricks, or ADLS).
• AI-Native Proficiency: Proven experience using Generative AI tools (Claude Code, GitHub Copilot) to accelerate coding, migrations, and documentation.
• Data Quality & Testing: Solid understanding of data testing principles and experience implementing automated Data Quality Gates.
• Methodology: Familiarity with Spec-Driven Development, where code is generated based on formalized requirements and pre-defined test cases.

Preferred Qualifications

• Agentic Workflows: Experience building or using agentic systems to automate data processing or metadata management.
• Big Data Frameworks: Experience with distributed processing at scale (e.g., Spark, Flink, or Kafka).
• DataOps: Experience with containerization (Docker, Kubernetes) and building CI/CD pipelines specifically for data workloads.
• Systematic Debugging: Ability to apply a structured approach to debugging non-linear errors in massive datasets.

Job Responsibilities

• Pipeline Orchestration: Design and implement scalable ETL/ELT processes, piloting AI agents to generate boilerplate code and optimize complex queries.
• Agentic Execution: Use Claude Code CLI to implement data transformations, automate schema migrations, and generate pipeline documentation.
• Deep Review: Conduct rigorous analysis of AI-generated code, focusing on data lineage, cost-efficiency of cloud resources, and security.
• Data Quality Assurance: Define and implement automated "Quality Gates" to ensure data integrity and correctness at every stage of the pipeline.
• Context Hygiene: Maintain the project’s operational memory (the CLAUDE.md file and /docs folder) to ensure AI agents remain effective and aligned with project goals.
• Skill Library Contribution: Help build and refine the pod’s "Skills" library for Claude Code, automating recurring data engineering patterns.
• Stakeholder Collaboration: Work closely with the AI Solution Architect to implement data models and with the AI Reliability Engineer to integrate data into the broader project ecosystem.

Department/Project Description

You will join a specialized AI-Native Delivery Pod focused on complex, high-stakes Data Engineering projects. Our mission is to build robust data infrastructures for enterprise clients using a Fixed Price model, delivering results with unprecedented speed through the synergy of human expertise and agentic execution. Projects typically involve building modern Data Warehouses (DWH), Analytics Lakes, and AI-ready data foundations within the Azure cloud. We operate in an environment of high autonomy where every engineer is an operator of cutting-edge AI technology.

Skill Category

AI/ML

Keyskills - Must Have

• Python
• claude
• Data Engineer

We are seeking a skilled, hands-on Data Engineer with demonstrated experience in Databricks, Apache Spark, Python, and real-time data streaming solutions.

• Responsibilities:

  - Implement resource composition logic provided by the Product team into scalable and efficient PySpark code.
  - Develop, optimize, and maintain PySpark applications within the Databricks environment.
  - Utilize and maintain existing CI/CD pipelines in Databricks for automated build, testing, and deployment.
  - Write and maintain unit tests to ensure high code quality and reliability.
  - Perform data validation and quality checks to ensure compliance with defined business rules
  - Support and manage production deployments, ensuring system stability and performance.
  - Troubleshoot and resolve issues across development, testing, and production environments.
  - Collaborate with Product, QA, DevOps, and healthcare domain teams to ensure accurate implementation.
  - Maintain technical documentation for code, workflows, and deployment processes.
  - Pull from and push code to GitHub repositories, following version control and branching best practices.

• Mandatory Skills Description:

  - 5+ years of experience in data engineering roles
  - Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).
  - 4+ years of experience with Databricks data pipelines
  - Strong hands-on experience with PySpark and/or Apache Spark.
  - Experience working with CI/CD pipelines and DevOps best practices.
  - Proficiency with CI/CD, GitHub (pull requests, branching, code reviews).
  - Experience writing unit tests and performing data validation
  - Experience supporting production systems and deployments
  - Strong analytical and troubleshooting skills.

• Nice-to-Have Skills Description:

  - Knowledge of FHIR (Fast Healthcare Interoperability Resources) standards.
  - Experience with healthcare data integration projects.
  - Familiarity with REST APIs and FHIR server integrations.
  - Experience in Agile/Scrum development environments.

At TechBiz Global, we are providing recruitment service to our TOP clients from our portfolio. We are currently seeking an ETL / Data Pipeline Specialist to join one of our clients' teams. If you're looking for an exciting opportunity to grow in a innovative environment, this could be the perfect fit for you.

Location: Remote
Reports To: Senior ETL Specialist
Employment Type: Full-Time Employee
Working Hours: Overnight shift aligned with US Central Time
 

Role Overview
We are looking for an ETL / Data Pipeline Engineer to join our data infrastructure team. This role operates on an overnight schedule aligned with US Central Time, providing critical coverage for automated scraping and ingestion systems that run during US business hours and overnight windows. You will help manage and monitor our existing fleet of web scrapers, build new data collection pipelines for regulatory and energy market sources, and contribute to the broader evolution of our ETL architecture.
This is a hands-on engineering role. You will work closely with our Houston-based data and analytics teams to ensure data flows reliably from source to production databases, and that our clients receive accurate, timely intelligence every morning.

Key Responsibilities
Overnight Operations & Data Quality
• Monitor and manage overnight scraper and ingestion runs, triaging failures and applying fixes in real time to minimize data gaps before US market open
• Verify data completeness and quality across all automated feeds, flagging anomalies and coordinating with the Houston team on persistent issues
• Maintain run logs, error documentation, and escalation notes for seamless async handoffs
New & Expanded Data Collection
Build and maintain scrapers, parsers, and ingestion pipelines across a growing set of energy market data domains, including but not limited to:
• Pipeline operator portals, electronic bulletin boards, and related filings (notices, maintenance, capacity, gas quality, customer indices)
• Government and regulatory agency databases at the federal, state, and provincial level across North America
• International energy data sources covering European, Canadian, and Mexican supply, demand, and power markets
• Emissions and environmental reporting systems
• Financial and corporate filings, including public company disclosures and production reporting
• Geospatial and mapping data related to production, infrastructure, and market geography
ETL Architecture & Enhancement
• Contribute to the design and build-out of our broader ETL infrastructure, including scheduling, orchestration, and error handling
• Write transformation logic to clean, normalize, and load raw data into PostgreSQL staging and production tables
• Optimize existing pipelines for performance, reliability, and cost efficiency
• Help build monitoring dashboards and alerting for pipeline health and data freshness
• Document data lineage, schema changes, and pipeline dependencies
Technical Environment
• Languages: Python (primary), SQL, Bash scripting
• Database: PostgreSQL, Mongo and Snowflake
• Infrastructure: AWS (EC2, S3, Lambda), Docker
• Scraping: Selenium, Playwright, BeautifulSoup, Scrapy, or similar frameworks
• Orchestration: cron, Airflow, or equivalent scheduling tools
• Version Control: Git / GitHub
• Communication: Slack, with async handoffs to US-based team

Required Qualifications
• 3+ years of experience building and maintaining ETL pipelines or data engineering systems
• Strong Python skills with experience in web scraping, data parsing, and automation
• Proficiency in SQL and experience working with relational databases (PostgreSQL preferred)
• Experience with headless browsers, anti-bot mitigation, and scraping resilience patterns
• Comfortable working independently during overnight hours with minimal real-time supervision
• Strong debugging instincts and ability to triage pipeline failures quickly
• Clear written communication in English for async collaboration and documentation
 

Preferred Qualifications
• Experience with energy, commodities, or financial data pipelines
• Familiarity with FERC-regulated pipeline data, EIA reporting, or utility/regulatory filings
• Experience with PostgreSQL-specific features (partitioning, materialized views, JSONB, pg_cron, logical replication)
• Familiarity with infrastructure-as-code (Terraform, CloudFormation) or containerized deployments
• Prior experience working on a distributed team across time zones
 

What We Offer
• Competitive compensation and benefits with room to grow as the team scales
• Direct impact on the data infrastructure behind a leading natural gas intelligence platform
• A small, senior team where your work is visible and valued from day one
• Flexible remote work with clear async workflows
• Exposure to the North American energy markets and commodity data at scale

N-iX is looking a proactive Data Engineer to join our vibrant team.

As a Data Engineer, you will play a critical role in designing, developing, and maintaining sophisticated data pipelines, Ontology Objects, and Foundry Functions within Palantir Foundry. The ideal candidate will possess a robust background in cloud technologies, data architecture, and a passion for solving complex data challenges.

Key Responsibilities:

• Collaborate with cross-functional teams to understand data requirements, and design, implement and maintain scalable data pipelines in Palantir Foundry, ensuring end-to-end data integrity and optimizing workflows.
• Gather and translate data requirements into robust and efficient solutions, leveraging your expertise in cloud-based data engineering. Create data models, schemas, and flow diagrams to guide development.
• Develop, implement, optimize and maintain efficient and reliable data pipelines and ETL/ELT processes to collect, process, and integrate data to ensure timely and accurate data delivery to various business applications, while implementing data governance and security best practices to safeguard sensitive information.
• Monitor data pipeline performance, identify bottlenecks, and implement improvements to optimize data processing speed and reduce latency. 
• Troubleshoot and resolve issues related to data pipelines, ensuring continuous data availability and reliability to support data-driven decision-making processes.
• Stay current with emerging technologies and industry trends, incorporating innovative solutions into data engineering practices, and effectively document and communicate technical solutions and processes.

Tools and skills you will use in this role:

• Palantir Foundry
• Python
• PySpark
• SQL
• TypeScript

Requirements:

• 5+ years of experience in data engineering, preferably within the pharmaceutical or life sciences industry;
• Strong proficiency in Python and PySpark;
• Proficiency with big data technologies (e.g., Apache Hadoop, Spark, Kafka, BigQuery, etc.);
• Hands-on experience with cloud services (e.g., AWS Glue, Azure Data Factory, Google Cloud Dataflow);
• Expertise in data modeling, data warehousing, and ETL/ELT concepts;
• Hands-on experience with database systems (e.g., PostgreSQL, MySQL, NoSQL, etc.);
• Proficiency in containerization technologies (e.g., Docker, Kubernetes);
• Effective problem-solving and analytical skills, coupled with excellent communication and collaboration abilities;
• Strong communication and teamwork abilities;
• Understanding of data security and privacy best practices;
• Strong mathematical, statistical, and algorithmic skills.

Nice to have:

• Certification in Cloud platforms, or related areas;
• Experience with search engine Apache Lucene, Webservice Rest API;
• Familiarity with Veeva CRM, Reltio, SAP, and/or Palantir Foundry;
• Knowledge of pharmaceutical industry regulations, such as data privacy laws, is advantageous;
• Previous experience working with JavaScript and TypeScript.

We offer*:

• Flexible working format - remote, office-based or flexible
• A competitive salary and good compensation package
• Personalized career growth
• Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
• Active tech communities with regular knowledge sharing
• Education reimbursement
• Memorable anniversary presents
• Corporate events and team buildings
• Other location-specific benefits

*not applicable for freelancers

Quantum is a global technology partner delivering high-end software products that address real-world problems. 

We advance emerging technologies for outside-the-box solutions. We focus on Machine Learning, Computer Vision, Deep Learning, GIS, MLOps, Blockchain, and more.

Here at Quantum, we are dedicated to creating state-of-the-art solutions that effectively address the pressing issues faced by businesses and the world. To date, our team of exceptional people has already helped many organizations globally attain technological leadership.

We constantly discover new ways to solve never-ending business challenges by adopting new technologies, even when there isn’t yet a best practice. If you share our passion for problem-solving and making an impact, join us and enjoy getting to know our wealth of experience!

About the position

Quantum is expanding the team and has brilliant opportunities for a proactive Senior Data Engineer who can design, implement, and evolve scalable data systems in AWS. We are building a greenfield analytics platform supporting both batch and real-time data processing. This role is a strategic blend of hands-on development, high-level architectural decision-making, and end-to-end platform ownership.

The client is a technological research company that utilizes proprietary AI-based analysis and language models to provide comprehensive insights into global stocks in all languages. Our mission is to bridge the knowledge gap in the investment world and empower investors of all types to become “super-investors.”

Through our generative AI technology implemented into brokerage platforms and other financial institutions’ infrastructures, we offer instant fundamental analyses of global stocks alongside bespoke investment strategies, enabling informed investment decisions for millions of investors worldwide.

Must have skills:

• 5+ years of experience in Data Engineering
• Strong hands-on experience with Apache Spark (including Structured Streaming)
• Experience building both batch and streaming pipelines in production environments
• Proven experience designing AWS-based data lake architectures (S3, EMR, Glue, Athena)
• Experience with event streaming platforms such as Apache Kafka or Amazon Kinesis
• Experience implementing lakehouse formats such as Delta Lake
• Strong understanding of partitioning strategies and schema evolution
• Experience using SparkUI and AWS CloudWatch for profiling and optimization
• Strong understanding of Spark performance tuning (shuffle, skew, memory, partitioning)
• Proven track record of cost optimization in AWS environments
• Experience with Docker and CI/CD pipelines
• Experience with Infrastructure as Code (Terraform, AWS CDK, or similar)
• Familiarity with monitoring and observability practices
• Upper-Intermediate or higher English proficiency (spoken and written)

Would be a plus:

• Experience in the financial domain
• Experience running Spark workloads on Kubernetes
• Exposure to or interest in Large Language Models (LLMs) and AI integration
• Experience implementing data quality frameworks or metadata/lineage systems

Your tasks will include:

• Design and implement batch and streaming data pipelines using Apache Spark
• Build and evolve a scalable AWS-based data lake architecture
• Develop and maintain real-time data processing systems (event-driven pipelines)
• Own performance tuning and cost optimization of Spark workloads
• Define best practices for data modeling, partitioning, and schema evolution
• Implement monitoring, observability, and data quality controls
• Contribute to infrastructure automation and CI/CD for data workflows

• Participate in architectural decisions and mentor other engineers

   

We offer:

• Delivering high-end software projects that address real-world problems
• Surrounding experts who are ready to move forward professionally
• Professional growth plan and team leader support
• Taking ownership of R&D and socially significant projects
• Participation in worldwide tech conferences and competitions
• Taking part in regular educational activities
• Being a part of a multicultural company with a fun and lighthearted atmosphere
• Working from anywhere with flexible working hours

• Paid vacation and sick leave days

   

Join Quantum and take a step toward your data-driven future.

We are seeking a skilled, hands-on Data Engineer with demonstrated experience in Databricks, Apache Spark, Python, and real-time data streaming solutions.

Responsibilities

Implement resource composition logic provided by the Product team into scalable and efficient PySpark code.

Develop, optimize, and maintain PySpark applications within the Databricks environment.

Utilize and maintain existing CI/CD pipelines in Databricks for automated build, testing, and deployment.

Write and maintain unit tests to ensure high code quality and reliability.

Perform data validation and quality checks to ensure compliance with defined business rules

Support and manage production deployments, ensuring system stability and performance.

Troubleshoot and resolve issues across development, testing, and production environments.

Collaborate with Product, QA, DevOps, and healthcare domain teams to ensure accurate implementation.

Maintain technical documentation for code, workflows, and deployment processes.

Pull from and push code to GitHub repositories, following version control and branching best practices.

Skills

Must have

5+ years of experience in data engineering roles

Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).

4+ years of experience with Databricks data pipelines

Strong hands-on experience with PySpark and/or Apache Spark.

Experience working with CI/CD pipelines and DevOps best practices.

Proficiency with CI/CD, GitHub (pull requests, branching, code reviews).

Experience writing unit tests and performing data validation

Experience supporting production systems and deployments

Strong analytical and troubleshooting skills.

Nice to have

Knowledge of FHIR (Fast Healthcare Interoperability Resources) standards.

Experience with healthcare data integration projects.

Familiarity with REST APIs and FHIR server integrations.

Experience in Agile/Scrum development environments.

We are looking for a Senior Data Engineer who is passionate about building scalable, cloud-native data solutions and eager to grow into a broader architecture role. You will lead the technical delivery of modern data platform solutions while engaging with clients and internal stakeholders to design, implement, and optimize enterprise-grade data systems.

Responsibilities

• Lead end-to-end development and implementation of scalable data pipelines, data lakes, and cloud-native data platforms
• Ensure performance, security, and reliability of data solutions through CI/CD, testing, and automation best practices
• Participate in solution design with architects and senior stakeholders - contribute to data architecture, system integrations, and technology selection
• Translate business needs into data models, transformation logic, and orchestration workflows
• Work closely with business and technical stakeholders to clarify requirements, demonstrate value, and refine deliverables
• Participate in pre-sales activities: help shape proposals, provide technical insights, and create solution roadmaps
• Stay on top of modern data platform trends (e.g., Data Mesh, Lakehouse, real-time analytics, observability)

• Explore new tools and techniques - contribute to N-iX’s internal Data & Analytics accelerators and reference architectures

   

Requirements

• 5-7 years of hands-on experience in data engineering
• Strong SQL, Python, and Spark skills
• Strong expertise in Azure and Databricks modern data platforms
• Knowledge of Airflow, dbt, Kafka, or similar tools preferred
• Proven experience building data lakes, lakehouses, data warehouses, and streaming pipelines
• Familiarity with architectural concepts like Data Mesh, Data Fabric, and cloud-native data architectures
• Experience leading delivery teams or mentoring other engineers
• Excellent verbal and written communication in English
• Willingness to travel for business (when needed) and to engage directly with clients
   

We offer*:

• Flexible working format - remote, office-based or flexible
• A competitive salary and good compensation package
• Personalized career growth
• Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
• Active tech communities with regular knowledge sharing
• Education reimbursement
• Memorable anniversary presents
• Corporate events and team buildings
• Other location-specific benefits

*not applicable for freelancers

Digis is looking for a Senior Data Engineer to join our long-term partnership with a US-based global leader in hospitality analytics.

About the Project

Our client is a premier SaaS provider that transforms front-desk staff into revenue generators for the world’s most iconic hotel brands, including Hilton and Marriott. Their platform operates in over 2,000 properties across 124 countries, managing billions in revenue.

The project is in an active development stage, focusing on processing massive data volumes to deliver real-time business value.
You will join a large, distributed team (approx. 50 people) and collaborate directly with the US-based CTO.

Your Responsibilities
 

• Manage data pipelines from ingestion to ready-to-use analytical models.
• Help scale case detection processes and improve existing services by splitting them into smaller microservices.
• Build, debug, and optimize ETL jobs using AWS Glue, EMR, and Spark.
• Write and improve SQL queries for PostgreSQL/Redshift, and work with DynamoDB where needed.
• Support workflows in AWS ECS Fargate and set up local environments for testing and deployment.
   

Requirements
 

• 5+ years of experience as a Data Engineer.
• 2+ years of hands-on production experience with Spark or PySpark.
• 3+ years of experience with the AWS ecosystem (specifically Glue, EMR, S3, and Redshift).
• English: High level (Upper-Intermediate/Advanced) for frequent technical meetings and direct collaboration with the US CTO.

We Offer
 

• 20 paid vacation days per year
• 5 paid sick leaves per year (no medical documents required)
• Personalized development plan with training compensation
• English courses compensation
• Work equipment if needed (PC / laptop / monitor)
• Opportunity to switch between projects and technologies within Digis
• Full accounting and legal support

Profisea is an Israeli boutique DevOps and Cloud company with a full cycle of services. For more than nine years, we have been implementing best practices of GitOps, DevSecOps, and FinOps, and providing Kubernetes-based infrastructure services to help businesses of all sizes —SMB, SME, or large enterprise clients to stay innovative and effective.     

We are looking for a Data Engineer with 3+ years of experience building and maintaining scalable data platforms in cloud environments. The ideal candidate will design, implement, and optimize data pipelines that support analytics and data-driven applications across AWS and GCP ecosystems.
Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.
• Build and manage data processing solutions using AWS Athena, AWS Glue, Glue Data Catalog, EMR, and S3.
• Implement data ingestion and replication pipelines using AWS DMS and other integration tools.
• Develop and orchestrate workflows using Apache Airflow and GCP Cloud Composer.
• Manage and optimize analytical workloads in Google BigQuery.
• Perform database migrations using AWS DMS and GCP Database Migration Service.
• Work with relational and NoSQL databases including PostgreSQL, MongoDB, Amazon RDS, DocumentDB, OpenSearch/Elasticsearch, and CloudSQL.
• Ensure data quality, reliability, and performance across data platforms.
• Collaborate with data analysts, data scientists, and software engineers to enable efficient data access and analytics.

Requirements

• 3+ years of experience in Data Engineering or related roles.
• Hands-on experience with AWS data services including Athena, Glue, Glue Data Catalog, EMR, DMS, and S3.
• Experience working with GCP data platforms, especially BigQuery and Cloud Composer.
• Experience with database migration tools such as AWS DMS or GCP Database Migration Service.
• Strong knowledge of SQL and data modeling.
• Experience working with PostgreSQL, MongoDB, and search platforms such as OpenSearch or Elasticsearch.
• Experience managing relational databases like RDS and CloudSQL.
• Familiarity with workflow orchestration tools such as Airflow.
• Understanding of data warehousing, data lakes, and distributed data processing concepts.

Nice to Have

• Experience with large-scale data lake architectures.
• Experience optimizing BigQuery or Athena query performance and cost.
• Knowledge of infrastructure-as-code and CI/CD for data platforms.
• Experience working in multi-cloud environments (AWS + GCP).

What we Offer: 

• Competitive salary   
• Remote work 
• Flexible schedule 
• Career growth 
• Sport compensations 
• Professional working environment, where you’d be an essential member of our company 
• Corporate culture, mutual support. 

Looking for Backend with #Python #Machine learning #AWS stack

We are looking for a Senior Data Engineer to build and scale a modern, AWS-based data platform powered by serverless pipelines, DuckDB, and Delta Lake architecture. You will work on integrating multiple SaaS and operational data sources into a robust data warehouse used for analytics and Tableau reporting. This role requires a hands-on engineer comfortable with Python-based data processing, AWS services, and non-traditional data stack components (DuckDB, Polars, etc.).

Seniority: Senior
English: Upper
Start date: April 4, 2026
Duration: 1-3 month

Project Description:

With more than 15 years of experience, the company blends human expertise with artificial intelligence to unlock new opportunities for individuals, organizations, and society. Its solutions turn complex data into clear insights — helping to understand past events and guide what can, could, or should be done next to transform industries and better serve people.

As a diversified leader in data science and AI, the company operates across various sectors and continues to grow at a rapid pace, creating new opportunities along the way. The team values individuals who are curious, driven, and passionate about solving complex challenges.

The organization brings together a multidisciplinary team of professionals, including data scientists, actuaries, statisticians, business analysts, strategy consultants, engineers, and product developers — all working together to leverage data for meaningful and transformative impact.

There is a strong focus on fostering a culture of growth, ownership, and continuous development. At the same time, the company emphasizes the importance of balance, offering a flexible and supportive environment where individuals feel valued and empowered.

Required Skills:

We are looking for candidates with strong hands-on experience in the following areas:

Cloud

• Google Cloud Platform (GCP)
• BigQuery
• Vertex AI
• Cloud Composer
• Cloud Functions

Programming

• Python
• SQL
• dbt

DevOps & Infrastructure

• Docker
• Kubernetes
• GitHub
• CI/CD pipelines

Additional

• Experience with AI / ML

Important:
A strong GCP background is critical for this role. Candidates without proven GCP experience are unlikely to be a strong fit.

We are looking for a Data Engineer to design and implement scalable data pipelines that support Generative AI applications. You will work with large-scale datasets, optimize performance, and ensure seamless data integration across AI workflows.

Responsibilities:

• Design and develop scalable data pipelines to support AI/ML models.
• Work with structured and unstructured data for training and inference.
• Optimize ETL workflows for real-time and batch processing.
• Ensure data quality, governance, and security in cloud environments.
• Collaborate with AI/ML engineers to streamline data workflows.

• Deploy and manage data infrastructure on AWS, GCP, or Azure.

   

Requirements:

• 3+ years of experience in Data Engineering and Big Data technologies.
• Expertise in ETL pipelines, data warehousing, and distributed computing.
• Strong knowledge of SQL, NoSQL, and data lakes.
• Experience with cloud-based data services (AWS Glue, BigQuery, Azure Data Factory).
• Proficiency in Python, Spark, Kafka, Airflow.

• Experience with CI/CD, DevOps, and Agile methodologies.

   

  What We Offer: 

  🚀 A role with real ownership and influence on delivery

  🌍 International eCommerce projects with global clients

  📈 Transparent growth path toward Middle+ → Senior PM

  🎓 Company-supported learning & certification programs

  🏠 Fully remote work with a flexible schedule

  🏝 Paid vacation & sick leave

  💰 Competitive compensation

  🔎 Mature processes and open team culture

We are looking for an experienced Data Engineer to design and implement scalable data pipelines, own core storage layers, and help build the foundations of a modern DataOps lifecycle.
 

Job requirements

• 3+ years of professional experience in Data Engineering / DWH development.
• Strong SQL expertise and solid Python skills (pandas, pyarrow).
• Production experience with Apache Airflow (production DAGs, scheduling, operators).
• Hands‑on work with PostgreSQL (schema design, migrations, partitioning) and ClickHouse (MergeTree engines, materialized views, performance tuning). 
• Experience with Data warehouse design — layered architecture, data modelling, denormalisation.
• Experience integrating REST APIs and third-party data sources
• Experience with Docker / Docker Compose — containerised services and multi-service environments
• Fluency in English.

Nice to Have:

• Experience with Apache Superset (RLS, dashboard layer).
• Familiarity with AI/LLM tooling for data (NL-to-SQL, vector databases). 
• Experience in the Fintech field: FX, brokerage, CRM/BO systems.
• Experience with multi-tenant data architectures.

Job responsibilities

• Design, build, and maintain ETL/ELT pipelines rom multiple product datasources.
• Model and optimise data structures in ClickHouse (views, engines, schema evolution).
• Orchestrate workflows using Apache AirflowCollaborate with BI teams (Superset) and backend teams (FastAPI/Python) to build reliable data products.
• Contribute to architectural decisions.
• Collaborate closely with product managers, BI developers, and DevOps to deliver data products end‑to‑end.

Job benefits

● Competitive and Attractive Pay.
● Flexible hours for a better work-life balance.
● Work your way: Hybrid flexibility to fit your life.
● 21 vacation + 7 no-questions-asked sick days per year.
● Career growth: Continuous Development & Performance Reviews.
 

The Role

We are looking for a Senior Data Engineer.

This is not a standard ETL role. We are building a dynamic data ecosystem where AI is deeply integrated—both as a productivity multiplier and as a core component of our data processing logic for identity data enrichment and data scoring.

You will own the infrastructure that handles over 10,000 executions per minute, ensuring stability, scalability, and data integrity. You will work with a modern stack on Google Cloud Platform, utilizing Cloud Functions and Kubernetes.

We are looking for an engineer who improves infrastructure, automates everything, and is eager to implement LLM-based logic directly into high-load data flows.

Responsibilities

• AI-Driven Data Scoring: Design and implement pipelines that utilize LLMs to analyze and score identity data in real-time. You will integrate AI models directly into the decision-making loop, balancing accuracy with latency and cost.
• Own the Data Architecture: Architect scalable data solutions using GCP and Python. You will manage data storage and retrieval using BigQuery and Apache Iceberg to support querying of TBs of data.
• Heavy Data Processing: Utilize Apache Spark for data transformations and batch processing when lightweight cloud functions are not enough.
• Manage High-Load Orchestration: Maintain and optimize our system instances. It involves complex dataflows, custom Python nodes, and performance tuning for 10k+ execs/minute.
• Release Lifecycle (CI/CD): Take ownership of the deployment process, ensuring that updates to pipelines and infrastructure are released safely with proper testing and rollback strategies.
• Database Optimization: Manage PostgreSQL performance under heavy load, optimizing complex queries and indexing strategies.
• Active AI Usage: Use Claude Code and other AI engineering tools to accelerate your own development, refactoring, and testing processes.
• Incident Resolution: Proactively monitor the system. When alerts fire, you investigate the root cause—whether it’s a database lock or an LLM hallucination—and fix it permanently.

Requirements

• 4-5+ years of experience in Data Engineering or Backend Engineering with a strong data focus.
• Production AI Integration: Experience integrating LLMs (OpenAI, Anthropic, Gemini) into production applications via API. You understand latency, token limits, and how to structure data for AI scoring.
• Expertise in GCP: Understanding of Google Cloud Platform (Cloud Functions, IAM, Networking).
• Strong Python: You write clean, efficient, and testable code. You are comfortable building custom logic where standard tools fall short.
• Big Data Stack: Experience with BigQuery, Apache Spark, and modern table formats like Apache Iceberg.
• Kubernetes (K8s): Experience deploying and scaling services in containerized environments.
• Workflow Automation: Understanding of workflow orchestration tools at a deep technical level. N8N is a big part of our domain, so familiarity with it is highly valuable.
• PostgreSQL Mastery: Proven ability to handle heavy write/read loads and optimize schemas.
• English: B2+ (Upper-Intermediate) or higher.

Culture & Mindset

• Self-improvement: You are a fast learner. You don't fear AI replacing you; you master it to replace your manual tasks.
• Ownership: You treat the Identity domain as your own business. If a scoring model drifts or a pipeline slows down, you notice it and fix it without being asked.
• Internal Locus of Control: You take responsibility for outcomes. If an external API fails, you build a fallback mechanism instead of just blaming the provider.
• Get It Done: You prioritize shipping value. You know when to use a simple script and when to build a complex architecture.

Openness: You share knowledge freely. If you find a better way to prompt the AI for scoring, you share it with the team.

Are you passionate about building large-scale cloud data infrastructure that makes a real difference? We are looking for a Senior Data Engineer to join our team and work on an impactful healthcare technology project. This role offers a remote work format with the flexibility to collaborate across international teams.

At Sigma Software, we deliver innovative IT solutions to global clients in multiple industries, and we take pride in projects that improve lives. Joining us means working with cutting-edge technologies, contributing to meaningful initiatives, and growing in a supportive environment.

CUSTOMER
Our client is a leading medical technology company. Its portfolio of products, services, and solutions is at the center of clinical decision-making and treatment pathways. Patient-centered innovation has always been, and will always be, at the core of the company. The client is committed to improving patient outcomes and experiences, regardless of where patients live or what they face. The Customer is innovating sustainably to provide healthcare for everyone, everywhere. 

PROJECT
The project focuses on building and maintaining large-scale cloud-based data infrastructure for healthcare applications. It involves designing efficient data pipelines, creating self-service tools, and implementing microservices to simplify complex processes. The work will directly impact how healthcare providers access, process, and analyze critical medical data, ultimately improving patient care.

Responsibilities:

• Collaborate with the Product Owner and team leads to define and design efficient pipelines and data schemas
• Build and maintain infrastructure using Terraform for cloud platforms
• Design and implement large-scale cloud data infrastructure, self-service tooling, and microservices
• Work with large datasets to optimize performance and ensure seamless data integration
• Develop and maintain squad-specific data architectures and pipelines following ETL and Data Lake principles
• Discover, analyze, and organize disparate data sources into clean, understandable schemas

Requirements:

• Hands-on experience with cloud computing services in data and analytics
• Experience with data modeling, reporting tools, data governance, and data warehousing
• Proficiency in Python and PySpark for distributed data processing
• Experience with Azure, Snowflake, and Databricks
• Experience with Docker and Kubernetes
• Knowledge of infrastructure as code (Terraform)
• Advanced SQL skills and familiarity with big data databases such as Snowflake, Redshift, etc.
• Experience with stream processing technologies such as Kafka, Spark Structured Streaming
• At least an Upper-Intermediate level of English