We are looking for a data engineer with experience in developing data pipelines and join our team working on a new transaction monitoring application used across multiple businesses.
As a Data Engineer, you will be responsible for the ingestion of data into the system. You will start by extracting the existing ETL layer from legacy R code to a robust, modern platform capable of serving multiple data models being developed. You will then look to scale this up to support multiple different data sources and pipelines. You will work with the data scientists as end consumers of the data to ensure we are meeting their needs. You will contribute to the team's strategy around deployment best practices.
This is an exciting opportunity to work on an important project, which will have a huge impact on our future architecture.
• Working closely with a data-centric application, hosting algorithms to detect possible market abuse.
• Designing the ETL architecture, as we look to extract it from an existing legacy application. After that, building out additional ETL layers to support the onboarding of additional data sources.
• Working closely with quants/data scientists to ensure that they have the data necessary to add new algorithms, and that the data is of the necessary quality and timeliness to support these.
• Act as the subject matter expert regarding data pipelines to the DevOps focused team and to external stakeholders
• Building a close relationship with clients and stakeholders to understand the use case for the platform, and prioritise work accordingly
• Working well in a multidisciplinary DevOps-focused team, building a close relationship with other developers, Quants/Data Scientists and production support teams
• You have experience building data pipelines with Python. You understand how these should be hosted and how to take them into production in a supportable way.
• You have experience working with message queues, traditional databases (SQL) and NoSQL databases.
• You have worked closely with data scientists before and may have experience creating pipelines that can serve ML/statistical algorithms.
• You have high development standards, especially for code quality, code reviews, unit testing, continuous integration and deployment
• You have proven capability to interact with clients and deliver results, taking ideas to production
• You have experience working in fast paced development environments
• You agree that verbal and written communication skills are vital
Experience with Spark or Scala
Experience with Kafka or Solace
Experience with KDB
Experience working with R code.
English: B2 Upper Intermediate
Luxoft is a high-end application outsourcing provider of choice and a trusted technology advisor to Global 2000 and medium-sized growth companies that apply compelling technologies to obtain leadership positions in their respective markets.
Luxoft today finds the Best talents, proposes career growth & employment benefits. Our teams are involved in high complicity & innovative projects for the Top leaders companies around the Globe.
DOU company page:
Job posted on
18 January 2021