Senior Data Engineer

Our client is the leader in the Early Childhood Education (ECE) sector, providing the most widely used developmental assessment and curriculum system services in the US.

They are looking for a highly skilled Data Engineer contractor to join their team and contribute to the development and maintenance of their Data Pipelines. You will be responsible for executing tasks independently, supporting efforts in Databricks leveraging your expertise in Python, Spark, Airflow and S3. Your primary focus will be on enhancing and optimizing data pipelines, and crafting and executing SQL transformations to update delta lake tables.

As a Data Engineer, you will:

• Develop and optimize data transformations using Python, PySpark, Airflow, S3, and Redshift;
• Craft SQL queries to efficiently manipulate data for storage in delta lake tables;
• Work and support efforts in Databricks;
• Ensure the reliability, scalability, and performance of our data processing solutions;
• Collaborate with our Senior Data Engineer to implement and enhance data pipelines;
• Independently execute tasks related to building, maintaining, and improving data pipelines.

Qualifications:

• 5+ years of hands-on experience in a data engineering or data system development role;
• Proven expertise in building and optimizing 'big data' pipelines and architectures;
• Proficiency in Python for data processing;
• Strong experience with Spark, with a focus on PySpark;
• Deep experience working in Databricks;
• Familiarity with AWS services such as EMR, S3, and MWAA (Airflow);
• Experience with Snowflake is strongly preferable;
• Solid understanding of data modeling, data warehousing, and ETL/ELT techniques;
• Proficiency in working with data in various formats, including parquet, structured, and unstructured data, stored in S3, Redshift, or other columnar databases;
• Exceptional collaboration skills with excellent verbal and written communication abilities;
• Keen attention to detail and a track record of solving complex problems.

We offer:

• Competitive market compensation;
• Fully remote work;
• Working on a very socially valuable product, alongside a highly professional, US-based, international team;
• Interesting challenges and the ability to grow and learn.