SouthRivers Data

About the Role

We are looking for a  Data Engineer to design, build, and operate large-scale data pipelines and lakehouse platforms in an enterprise environment. You will work hands-on with AWS data services, Apache Spark, and Python to deliver reliable, performant, and well-modeled data products that power analytics and downstream applications.

Key Responsibilities

• Design, implement, and maintain robust, reliable, and scalable data pipelines for batch and large-scale processing workloads.
• Build and evolve a Data Lakehouse on AWS using cloud object storage (S3), open table formats, and distributed processing frameworks.
• Develop ETL/ELT workflows in Python and Apache Spark, ensuring performance, cost-efficiency, and maintainability.
• Model data for analytical and reporting use cases, applying dimensional and analytical modeling best practices.
• Write advanced SQL for transformation, optimization, and ad-hoc analysis across large datasets.
• Operate and optimize AWS data services such as EMR, Glue, Athena, and S3-based data lakes.
• Troubleshoot pipeline and platform issues end-to-end, applying system-level thinking to identify root causes and durable fixes.
• Collaborate with analysts, data scientists, and platform engineers to translate business requirements into technical solutions.
• Contribute to data quality, observability, governance, and CI/CD practices for data workloads.

Required Qualifications

• Strong, proven hands-on experience in data engineering within enterprise environments.
• Top-notch advanced SQL skills and solid understanding of analytical and dimensional data modeling.
• Strong hands-on experience with Data Lakehouse or modern data platform concepts: cloud object storage, open table formats (e.g., Delta, Iceberg, Hudi), and distributed processing.
• Strong hands-on experience with AWS data services: EMR, Glue, Athena, and S3-based data lakes.
• Strong hands-on experience with Apache Spark for large-scale data processing.
• Strong Python skills for ETL development, data processing, and automation.
• Demonstrated experience designing, implementing, and maintaining robust and reliable data pipelines in production.
• Very strong analytical, problem-solving, and system-level thinking skills.

Nice to Have

• Experience with workflow orchestration tools (e.g., Airflow, Step Functions).
• Familiarity with infrastructure-as-code (Terraform, CloudFormation) and CI/CD for data.
• Exposure to data governance, cataloging (e.g., AWS Glue Data Catalog, Lake Formation), and data quality frameworks.
• Streaming experience (Kafka, Kinesis, Spark Structured Streaming).

WHO WE ARE

SouthRivers Data, we're an engineering-led company - built by engineers, run by engineers. That means no explaining why tech debt matters, no translating "we need refactoring time" into business speak, and no decisions made by people who've never written a line of code. We get it, because we've been there.

We believe that a great engineer is defined by two things equally: technical mastery and the ability to work with people. One without the other isn't enough for us - and probably isn't enough for you either.

REQUIREMENTS

• Experience in Python programming (at least 3 years of practical experience in Python programming)
• Practical experience with GenAI and agentic AI in Python (ideally using
• pydantic-ai as a framework), at least one year of practical experience, more is an advantage)
• Hands-on experience in developing complex end-to-end (Gen) AI applications (not necessarily alone, but as part of a cross-functional team), but in the role of Solution Engineer or AI Engineer
• Team player who can also work independently and proactively take on and execute tasks

• Excellent written and spoken English skills

   

NICE-TO-HAVE

• • Theoretical knowledge of the different concepts in the field of agent-based AI (models, tools, inferences, memory and related approaches such as RAG, protocols such as MCP)
• Experience in data science and machine learning
• Experience setting up CI/CD pipelines and steps (preferably Github Actions)
• Experience working in the Azure cloud
• Experience with Kubernetes and setting up deployments, jobs, etc.