Senior Data Engineer (offline)

About the Customer:
The сustomer is a leading provider of vehicle lifecycle solutions, enabling the companies that build, insure, repair, and replace vehicles to power the next generation of transportation. The company delivers advanced mobile, artificial intelligence, and connected car technologies through its platform, connecting a vibrant network of 350+ insurance companies, 24,000+ repair facilities, OEMs, hundreds of parts suppliers, and dozens of third-party data and service providers. The customer's collective set of solutions inform decision-making, enhance productivity, and help clients deliver faster and better experiences for end consumers. 

The сustomer’s company was ranked #17 in the Top 100 Digital Companies in Chicago in 2020 by Built in Chicago, an online community for digital technology entrepreneurs in Chicago, and was named one of Forbes best mid-sized companies to work for in 2019 – an important accolade and retention tool for the 2,600+ full-time company employees (alongside 350 dedicated contractors).
The сompany’s corporate headquarters is in downtown Chicago in the historic Merchandise Mart—a certified LEED (Leadership in Energy and Environmental Design) building that is also known to be a technology hub within the broader metro.

About the Project:
Since 2018 the Customer has been working on the Analytics Platform. It is on Hadoop and on the Hortonworks Data Platform. The Customer is planning to move it to Amazon EMR in 2021.

The Customer has different products. This platform is where all these data come into one data lake and he can then do the next generation analytics.

Project Advantages:
Cross product analytics
Analytics for every new product customer has. Analytics team products is how the customer sells the products value to clients
Quarterly Business Review meetings use data to explain how customer’s product is helping clients in their business
You'll get to work with a cross-functional team
You will learn the customer’s company business

Project Tech Stack:
Technologies used are all open source Hadoop, Hive, PySpark, Airflow, Kafka to name a few

Project Stage:
Active Development

Must Have Qualifications:
4 + years’ experience building, maintaining, and supporting complex data flows with structural and unstructural data
Proficiency in Python and PySpark
Hands-on experience with HDFS, HIVE, and SQOOP
Experience building data-pipelines/microservices with Apache Kafka
Experience in Apache Airflow to orchestrate and schedule complex data flows
Ability to use SQL for data profiling and data validation
Experience in Unix commands and scripting
Practical knowledge on AWS components such as EMR and S3
Master’s or Bachelor’s degree

Nice to have
Experience and understanding of Continuous Integration and Continuous Delivery (CI/CD)
Understanding in performance tuning in distributed computing environment (such as Hadoop cluster or EMR)
Familiarity with BI tools (such as Tableau and MicroStrategy) and high comfort level using data modeling techniques.

Responsibilities:
Build end-to-end data flows from sources to fully curated and enhanced data sets. This can include the effort to locate and analyze source data, create data flows to extract, profile, and store ingested data, define and build data cleansing and imputation, map to a common data model, transform to satisfy business rules and statistical computations, and validate data content
Modify, maintain, and support existing data pipelines to provide business continuity and fulfill product enhancement requests
Provide technical expertise to diagnose errors from production support teams
Coordinate within on-site teams as well as work seamlessly with the US team

About Olsys

OLSYS Ltd provides full-service solutions for mid-market and enterprise organizations.

With 15+ years of experience, 100+ projects and 50+ strong technical experts in the team, we continue to grow by expanding our development team in Europe, as well as expanding the base of new clients and projects.

As an enterprise software development company, we are building long term partnerships helping our clients accelerate their digital experiences with reasonable IT investments.
Our tailored approach, e-commerce focus, and flexible solutions allow us to design, develop, and deliver scalable, integrated commerce platforms that drive profits and boost the business.

Our industry focus:
— Banking and Finance (business continuity management, planning system, banking app development);
— Retail and E-Commerce (B2B Commerce, B2C Commerce, Digital/E-Commerce customer experience);
— Healthcare (website development for Healthcare organizations).

Our expertise includes:
— Commerce Solutions
— Development Services
— Creative Services
— Quality Assurance and Testing
— Maintenance and Support
— E-Commerce Consulting

Company website:
https://olsysltd.com/

DOU company page:
https://jobs.dou.ua/companies/olsys/