Lead Data Engineer

Role Overview:
As a Lead Data Engineer at QuintaGroup, you will lead the design, optimization, and implementation of data pipelines within the AWS ecosystem for a US-based B2B marketplace platform. The platform simplifies and accelerates business operations by providing seamless data solutions and advanced analytics. You’ll take ownership of collaboration with data scientists, analysts, and cross-functional teams to deliver innovative and impactful results.

Key Responsibilities:

• Lead and oversee the development, implementation, and optimization of data pipelines using PySpark in AWS environments.
• Utilize AWS services such as S3, Glue, Lambda to create scalable and efficient data solutions.
• Provide technical leadership to enhance PySpark workflows for performance, reliability, and cost-effectiveness.
• Maintain data quality through rigorous testing and monitoring processes.
• Apply data governance, security, and compliance best practices.
• Mentor and guide team members while documenting workflows, processes, and designs to support collaboration and maintenance.

Tech Stack:

• Programming Languages: Python (with Pandas and PySpark).
• AWS Services: S3, Glue (Glue Data Catalog, Glue Crawler, Glue Jobs with PySpark), Lambda, ECS, Athena, Aurora (RDS), AppConfig, API Gateway, Step Functions, Quicksight, EventBridge, AppSync, DynamoDB.
• Tools: Auth0.
• Infrastructure: Terraform for Infrastructure-as-Code.

Requirements:

• 5+ years of experience in data engineering, with a focus on PySpark.
• Strong experience with AWS services.
• Proficiency in Python and related frameworks or libraries.
• Experience with infrastructure-as-code tools (e.g., Terraform, CloudFormation) is a plus.
• Proven ability to mentor and guide team members, ensuring best practices are followed.
• Strong analytical and problem-solving skills with attention to detail.
• Excellent communication skills and ability to work in dynamic, team-oriented environments.
• Upper-Intermediate level of English.

We Offer:

• Flexible working format: remote, office-based, or hybrid.
• Competitive salary and compensation package.
• Personalized career growth and mentorship programs.
• Professional development tools, including tech talks and training sessions.
• Access to active tech communities and regular knowledge sharing.
• Education reimbursement opportunities.
• Memorable anniversary gifts.
• Corporate events and team-building activities.
• Location-specific benefits.