Senior Data Engineer (Databricks and Python)

About the Opportunity

We are seeking a highly motivated Senior Data Engineer to join an enterprise technology team working on modern, AI-enabled data platforms. This role is ideal for an experienced engineer who enjoys building scalable data solutions, collaborating with cross-functional teams, and driving engineering best practices in a cloud-native environment.

You will work on enterprise-grade applications and data processing solutions using modern technologies including Azure, Databricks, Python, and microservices.

Key Responsibilities

• Define and promote engineering best practices, coding standards, and development processes.
• Conduct code reviews to maintain high-quality, maintainable software.
• Design, develop, and maintain scalable Apache Spark applications.
• Build efficient and reliable data processing and transformation pipelines.
• Write clean, well-documented, and maintainable Python code.
• Optimize applications for performance, scalability, and efficient data handling.
• Collaborate with engineering, product, and business stakeholders to deliver high-quality solutions.
• Troubleshoot and resolve technical issues to ensure system reliability.
• Create and maintain technical documentation, workflows, and development standards.
• Communicate effectively with internal teams and external stakeholders while supporting project delivery.

Required Qualifications

• 5+ years of professional software engineering or data engineering experience.
• Strong experience with Apache Spark using Databricks and/or Azure Synapse/Fabric.
• Advanced Python programming skills, including data manipulation with Pandas, Polars, or similar libraries.
• Experience working with Parquet, Delta Tables, and other columnar storage formats.
• Solid understanding of large-scale data processing, transformation, and analytics workflows.
• Experience designing scalable applications using microservices architecture.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent organizational, communication, and time management abilities.
• Ability to work independently while collaborating effectively within distributed teams.

Preferred Qualifications

• Experience with Microsoft Azure cloud services, including Data Lake, Blob Storage, Service Bus, Redis, and related services.
• Familiarity with FastAPI.
• Experience with Docker and Kubernetes.
• Exposure to modern AI technologies, RAG pipelines, or large language model (LLM) applications is a plus.

Technology Stack

• Azure Cloud
• Apache Spark / Databricks
• Python
• Azure SQL
• MongoDB
• .NET 8 / ASP.NET Core
• Microservices Architecture
• GitHub Enterprise
• Angular
• Docker & Kubernetes
• AI/LLM technologies (LangChain, LangGraph, RAG)