Principal Data Engineer

Description
The Client is one of the major banks in the UK. The project focuses on migrating and modernizing data assets from legacy platforms to a Databricks-based data environment. The team works closely with the client to transform existing Teradata scripts into dbt models, implement scalable data pipelines and support data migration using internal tools and cloud services.

Requirements
Hands-on experience with dbt (data build tool) for data transformation and pipeline development.
Strong Python programming skills for data processing and scripting.
Practical experience working with Databricks in a data engineering or analytics environment.
Experience with Cloud Platforms (AWS or Azure).
Experience working with SQL-based systems, including migration from legacy platforms such as Teradata.
Understanding of data migration, ETL/ELT pipelines, and data transformation frameworks.
Ability to work collaboratively with technical leads and external stakeholders.
Good written and verbal English communication skills.
 

Job responsibilities
Develop, maintain, and execute dbt and Python scripts to migrate database tables and views and integrate them with Databricks.
Participate in data migration activities, ensuring reliable and efficient transformation of legacy data assets.
Work closely with Barclays stakeholders to operate and support DIH and other internal data migration tools.
Assist in the design and implementation of data transformation pipelines within the Databricks ecosystem.
Validate migrated datasets and ensure data quality, consistency, and completeness throughout the migration process.
Troubleshoot and resolve issues related to data transformation, pipelines, and integrations.
Contribute to documentation related to data migration processes, dbt models, and pipeline architecture.
Collaborate with cross-functional teams including data engineers, architects, and business stakeholders to deliver migration tasks.