Azure Data Engineer

Client

The client is a premier institution offering world-class postgraduate business education, including MBA, Executive MBA, and specialised finance and management programs. Its mission is to transform global business practices. The client is globally recognized for its rigorous academics, exceptional faculty, and cutting-edge research. It consistently ranks among the world’s top business schools, securing high positions in global MBA rankings. The client fosters leadership and innovation, equipping students for impactful careers in the international business landscape. Join a great company, not merely an individual project.
 

Project overview

The Client recently completed their Data & AI Strategy and Roadmap, establishing a foundation for a data-driven future. The assessment phase reviewed the current state of Data and AI, analyzing technology, processes, resources, and structure, and provided strategic recommendations aligned with the School’s 5-year transformation plan. Building on this, the discovery phase focused on data governance, use cases, business drivers, service offerings, technology, and roles, delivering further strategic insights. With this groundwork complete, The Client is now entering the delivery phase, which includes implementing a Data Platform, a first use case, and establishing Data Governance roles, processes, and technologies.

Responsibilities
 

• Design, build, and optimize complex ETL/ELT pipelines using Azure Data Factory, including mapping data flows and orchestration.
• Develop scalable data processing solutions using Azure Synapse Analytics (dedicated SQL pools, Spark pools) for enterprise-grade analytics.
• Implement and maintain Medallion architecture (Bronze/Silver/Gold) on Azure Data Lake Storage (ADLS Gen2) with proper data organization, security, and governance.
• Build large-scale data transformation workflows using Azure Databricks/Synapse Spark with PySpark/Python.
• Develop and integrate Azure Functions (Python/C#) to enable event-driven processing and custom pipeline logic.
• Implement CI/CD pipelines for data platforms using Azure DevOps Pipelines or GitHub Actions.
• Automate cross-environment deployments using IaC tools (ARM, Bicep, Terraform).
• Optimize SQL queries, database objects, and Spark jobs for performance and reliability.
• Design dimensional models (Star/Snowflake schemas) and develop production-grade data models.
• Collaborate with cross-functional teams to clarify requirements, communicate architecture decisions, and resolve data issues.
   

Requirements
 

• Proven expertise with core Azure data engineering services:
• Azure Data Factory
• Azure Synapse Analytics (SQL & Spark)
• Azure Data Lake Storage Gen2
• Azure Databricks / Synapse Spark
• Azure Functions
• Strong command of Python and PySpark for data processing and automation.
• Advanced SQL skills: complex queries, stored procedures, optimization, and performance tuning.
• Solid understanding of data modeling, including design of facts, dimensions, and analytical structures.
• Hands-on experience building and maintaining CI/CD pipelines (Azure DevOps or GitHub Actions).
• Practical experience with Infrastructure-as-Code for multi-environment deployment (ARM, Bicep, Terraform).
• Excellent Git knowledge: branching strategies, pull requests, code reviews.
• Strong problem-solving abilities for debugging pipelines, resolving deployment issues, and optimizing performance.
• Strong communication skills for effectively explaining technical solutions to technical and non-technical stakeholders.
   

Nice to have
 

• Experience with Microsoft Fabric ecosystem, including:
• OneLake & Lakehouse
• Fabric Data Engineering / Warehouse
• Notebooks & Spark jobs
• Power BI development experience:
• Building dashboards and reports
• Advanced DAX
• Optimized data modeling
• Row-Level Security (RLS) setup
• Microsoft certifications:
• DP-203: Azure Data Engineer Associate
• DP-600: Fabric Data Engineer Associate