Senior Data Engineer (Databricks / AWS)

TechBar is a software services company working with clients across the US and Europe. We are investor-backed, growing steadily, and known for putting senior, thoughtful engineers in front of our clients.

We are looking for a Senior Data Engineer to join our client’s team at a wealth management firm modernizing its data platform using Databricks and AWS.
 

About the Project

You will design, build, and maintain reliable data pipelines and lakehouse models that power analytics, reporting, and data products across the business.

The platform processes financial data, including client portfolios, transactions, positions, and performance. Some initiatives may involve migrating existing workloads from Snowflake to Databricks.
 

Responsibilities

• Build the lakehouse: Design and develop scalable batch and streaming pipelines on Databricks using Spark and Delta Lake, following a medallion architecture (Bronze / Silver / Gold).
• Own data on AWS: Work with S3, IAM, and related AWS services to ingest, store, and secure data from internal systems and third-party financial data providers.
• Model for the business: Build clean, scalable, and well-documented data models for analytics, reporting, and downstream consumers.
• Ensure data quality and trust: Implement data quality checks, monitoring, and lineage. Manage access and governance using Unity Catalog.
• Optimize performance and cost: Tune Spark jobs, cluster configurations, and storage layouts to improve performance and control cloud costs.
• Automate delivery: Apply CI/CD and infrastructure-as-code practices to Databricks workflows and environments.
• Support migration: Assess and migrate data pipelines, models, and workloads from Snowflake to Databricks where needed.
• Collaborate: Work closely with analysts, data scientists, and business stakeholders to translate requirements into reliable data solutions.
   

Requirements

• 6+ years of experience in data engineering, including significant hands-on experience with Databricks in production.
• Strong knowledge of Apache Spark / PySpark and advanced SQL.
• Solid Python skills for data engineering.
• Hands-on experience with AWS as the underlying cloud platform for data systems, including S3, IAM, and related data services.
• Experience with Delta Lake and lakehouse / medallion architecture.
• Experience building data models for analytics and reporting.
• Experience with CI/CD, Git, and production data operations, including monitoring, troubleshooting, and performance tuning.
• Ability to own data pipelines end-to-end, from design and implementation to production support.
• Strong understanding of data quality, security, governance, and cost optimization.
• Excellent communication skills, with the ability to collaborate with both technical and non-technical stakeholders.
• Ability to work independently and proactively in a distributed team.
• Fluent English.
• Currently based in the United States or the United Kingdom.

Tech Stack
 

Data Platform

• Databricks
• Apache Spark / PySpark
• Delta Lake
• Unity Catalog
• Databricks Workflows
• Delta Live Tables
   

Cloud Infrastructure

• AWS S3
• IAM
• AWS Glue
• AWS Lambda
• Kinesis / MSK
• CloudWatch
   

Languages

• Python
• SQL
• Scala (a plus)
   

Engineering Practices and Tools

• Git
• CI/CD
• Terraform or Databricks Asset Bundles
• Data quality frameworks
• Orchestration with Databricks Workflows or Airflow
   

Adjacent Technologies

• Snowflake (migration context)
• dbt
• BI tools used by downstream consumers
   

Nice to Have

• Experience migrating from Snowflake to Databricks or working on other large-scale data platform migrations.
• Experience with Unity Catalog governance and access management.
• Experience building streaming pipelines using Structured Streaming, Kafka, or Kinesis.
• Experience with Terraform or Databricks Asset Bundles.
• Experience with dbt.
• Databricks certifications: Data Engineer Associate or Professional.
• Experience in wealth management, asset management, or broader financial services, particularly with portfolios, positions, transactions, market data, and reference data.
• Experience working with regulated or sensitive data, including PII, compliance, and audit requirements.
   

What We Offer

• 20 paid vacation days per year.
• 10 paid sick days per year.
• 6 additional special days off per year.
• Company-provided equipment to ensure you have everything you need to work effectively.
• The opportunity to contribute to a modern Databricks and AWS data platform in the financial services domain.
• A long-term opportunity within an international, distributed team.
   

Location

We are considering candidates based in the United States and the United Kingdom only.

If you are a Senior Data Engineer with strong Databricks, Spark / PySpark, Python, SQL, and AWS experience who enjoys taking ownership of data solutions end-to-end, we look forward to hearing from you!