Senior Data Engineer

Key Responsibilities  

• Design, build, and maintain scalable, reliable data pipelines (batch and streaming).  
• Develop and optimise data models, warehouses, and lakehouse architectures.  
• Build and automate CI/CD pipelines for data workflows and deployments.  
• Ensure data quality, integrity, lineage, and governance across environments.  
• Implement proactive monitoring and alerting for pipelines and data infrastructure (CloudWatch, Grafana).  
• Manage IaC for consistent, reliable data infrastructure.  
• Troubleshoot and optimise pipeline and query performance across environments.  
• Lead incident resolution and root cause analysis for data systems.  
• Collaborate with analysts, data scientists, and engineering teams, ensuring security, scalability, and operational excellence to understand data requirements and translate them into technical solutions  
• Document data architecture, lineage, and workflows  

Must have  

• 5+ years in data engineering, with solid DevOps/platform engineering exposure.  
• Strong SQL and data modelling skills (dimensional, normalised, lakehouse).  
• Proficient in Python; working knowledge of Scala, Java, or Go.  
• Hands-on experience with data pipeline/orchestration tools (Airflow, DBT).  
• Experience with distributed processing frameworks (Spark, Flink, or similar).  
• Advanced AWS expertise (Glue, EMR, Redshift, Athena, S3, Lambda, Kinesis).  
• CI/CD tools (GitHub Actions, GitLab CI, Jenkins, CircleCI).  
• Infrastructure as Code (CDK, CloudFormation).  
• Experience with streaming/event-driven architecture (Kafka, Kinesis, Apache Flink, or Pub/Sub).  
• Observability tools: Grafana, CloudWatch.  
• Expertise with data warehouse platforms (Redshift, Snowflake).  

Desirable  

• Experience with containerisation and orchestration (Docker, ECS, Kubernetes).  
• Understanding of data governance, cataloguing, and security/compliance (GDPR, lineage tools).  
• Experience with GitHub Enterprise, Atlassian APIs, and Slack bots.  
• Understanding of cost optimisation strategies in multi-cloud and data warehouse environments.