Big Data Developer (offline)

Skills:
● have 4-5 years experience in software development
● Unix/Linux, bash, python, virtualisation (docker, Kubernetes),
● deployment orchestration (chef, puppet, ansible), configuration servers ZooKeeper, monitoring and distributed logging (Prometheus, Grafana, GrayLog) experience
● have strong knowledge of Scala /Java
● have knowledge of distributed data processing frameworks Hadoop, Spark, Kafka.
● have proven experience of building production-ready ETL processing pipelines
● have familiarity with Big Data persistent distributed storages Impala, Elasticsearch, Aerospike, HBase
● have proven working record with relational databases (e.g. PostgreSQL

As a plus:
● Experience in AWS S3 environment (S3 storage)

Your responsibilities will include:
● Carrying out efficient integration with our data providers via various API endpoints and data representation formats.
● Building and deploying an in-house distributed ETL pipeline for processing petabytes of data per day
● Enable an accurate, comprehensive and reliable data storage in our distributed data warehouses based on the needs of other teams
● Providing continuous improvements in the way data is being processed and stored based on the feedback and needs of the business or other teams
● Setting up monitoring for key performance metrics and overall systems’ behaviour to promptly react in case any anomaly detected
● You will be responsible for optimisation of ETL pipelines, maintaining over 60 Spark jobs. Building a data lake for data scientists and analysts.
● Experimenting with new tools and technologies to produce cutting-edge solutions to business problems
● Be a part of a self-organising, results-oriented agile team using Kanban to complete new product launches.

We offer:
• Location — Kyiv
• Good compensation;
• Cozy office (on of the best coworking in Kyiv);
• Benefit package

About Data Science UA

We understand that you need more than just a search engine to find IT and technical professionals or to progress your own career, that is why we established Data Science UA.
One place connecting business and developers.

Company website:
https://data-science-ua.com/

DOU company page:
https://jobs.dou.ua/companies/data-science-ua/