Senior Data Engineer $$$$

Project description

We are seeking a skilled, hands-on Data Engineer with demonstrated experience in Databricks, Apache Spark, Python, and real-time data streaming solutions.

Responsibilities

Implement resource composition logic provided by the Product team into scalable and efficient PySpark code.

Develop, optimize, and maintain PySpark applications within the Databricks environment.

Utilize and maintain existing CI/CD pipelines in Databricks for automated build, testing, and deployment.

Write and maintain unit tests to ensure high code quality and reliability.

Perform data validation and quality checks to ensure compliance with defined business rules

Support and manage production deployments, ensuring system stability and performance.

Troubleshoot and resolve issues across development, testing, and production environments.

Collaborate with Product, QA, DevOps, and healthcare domain teams to ensure accurate implementation.

Maintain technical documentation for code, workflows, and deployment processes.

Pull from and push code to GitHub repositories, following version control and branching best practices.

Skills

Must have

5+ years of experience in data engineering roles

Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).

4+ years of experience with Databricks data pipelines

Strong hands-on experience with PySpark and/or Apache Spark.

Experience working with CI/CD pipelines and DevOps best practices.

Proficiency with CI/CD, GitHub (pull requests, branching, code reviews).

Experience writing unit tests and performing data validation

Experience supporting production systems and deployments

Strong analytical and troubleshooting skills.

Nice to have

Knowledge of FHIR (Fast Healthcare Interoperability Resources) standards.

Experience with healthcare data integration projects.

Familiarity with REST APIs and FHIR server integrations.

Experience in Agile/Scrum development environments.

Languages

English: C1 Advanced