Middle Data Engineer

Middle Data Engineer
Remote | Full-time

We are seeking an experienced Data Engineer to join our team. In this role, you will utilize AWS tools to build and maintain components of a modern data warehouse. You will tackle challenging problems, manage large-scale datasets, and enhance ETL/ELT workflows to deliver impactful solutions.

Key Responsibilities:

• Gather and integrate data from various platforms such as BigQuery, ClickHouse, Hadoop (via HDFS or Hive), and AWS S3.
• Work with modern table formats like Apache Iceberg for efficient management of large datasets.
• Develop, monitor, and improve ETL/ELT workflows.
• Design and implement a centralized data catalog with clean, validated, and well-documented data marts.
• Automate checks for data quality and consistency across systems.
• Write and maintain technical and project documentation.

Requirements:

• Minimum of 2 years of hands-on experience in Python and data engineering roles.
• Practical experience with AWS cloud services, including S3, Athena, and Redshift.
• Proven experience working on ETL, ELT, and data warehouse-related tasks.
• Experience tuning SQL queries and optimizing ETL performance (e.g., partitioning, indexing).
• Familiarity with task scheduling systems and distributed job queues.
• Understanding of database design and trade-offs between different database types.
• Experience working with Hadoop-based systems, HDFS, and query engines like Hive.
• Familiarity with Iceberg tables or similar table formats (e.g., Delta Lake, Hudi).
• Basic understanding of computer science principles.
• Strong analytical thinking and a proactive approach to problem-solving.
• Comfort working in Linux environments and with containerized applications using Docker.
• Proficiency with Git and familiarity with Git-flow collaboration practices.

Nice to Have:

• Experience with workflow orchestration tools like Apache Airflow.
• Familiarity with DBT for data transformation.
• Knowledge of Infrastructure as Code tools (Terraform, Pulumi).
• Experience with NoSQL databases such as MongoDB, Elasticsearch, or Redis.
• Familiarity with the Kafka ecosystem (Kafka Streams, Kafka Connect).
• Experience with BI platforms and visualization tools like Tableau, PowerBI, or Apache Superset.

Benefits:

• Education budget of $600 per year.
• Professional English courses.
• Medical insurance.