Azure Data Engineer – Metadata and Pipeline Development to $5600

Azure Data Engineer – Metadata & Pipeline Development

Partner: MDW Group – https://mdw-group.com

Project Goal: Enhance Azure Purview-based Data Catalog for improved metadata governance, traceability, and automation using modern data engineering and Generative AI practices.

Engagement Details

Rate: $35/hour

Start Date: Mid-June 2025

Duration: 6 months, renewable

Location: Remote

Language: English

About the Role

You will work on enhancing metadata cataloging and data governance capabilities by implementing advanced data pipelines and automation on Azure.

Key Responsibilities

• Develop Azure Functions to inject certified metadata into Azure Purview using APIs.
• Create Databricks notebooks to integrate metadata from external platforms such as ECS and Treasure Data.
• Support PowerApps-based data quality modules with structured metadata enrichment.
• Design and manage ADF pipelines for ingestion, codification, segmentation workflows.
• Use Azure SQL to manage transactional data related to validation and codification status.
• Collaborate with front-end teams to define API contracts and ensure clean data access.
• Implement error detection, monitoring, and alerting logic for failed jobs and invalid data.
• Contribute to CI/CD pipelines for data workflows using Azure DevOps.

Technical Skills Required

• Advanced Python programming.
• Strong experience with Azure Databricks (Delta Lake, Jobs, Cluster Management).
• Proficient in Azure Data Factory: pipeline design, parameterization, scheduling.
• Experience with Azure SQL Database and T-SQL.
• Familiarity with Azure Data Lake Storage Gen2.
• CI/CD pipeline implementation using Azure DevOps.
• Usage of Git for version control.

Bonus / Preferred

• Microsoft Graph API or Azure Logic Apps for email ingestion.
• Power BI or Synapse Analytics exposure.
• Familiarity with MDM concepts and enterprise data governance.