Senior Data Engineer

Dataforest is seeking an experienced Senior Data Engineer to join our dynamic team. You will be responsible for developing and maintaining data-processing architecture, as well as optimizing and monitoring our internal systems.
Requirements:
- 3+ years of commercial experience with Python.
- Solid foundational knowledge of ElasticSearch, including:

• Ability to perform batch updates using bulk operations.
• Understanding index mapping and how to adapt it for your project’s needs.

• (Nice to Have) Some exposure to vector search concepts.

  - Experience working with PostgreSQL databases.
  - Proven experience in setting up and managing monitoring systems with CloudWatch, Prometheus, and Grafana.
  - Profound understanding of algorithms and their complexities, with the ability to analyze and optimize them effectively.
  - Excellent programming skills in Python with a strong emphasis on optimization and code structuring.
  - Solid understanding of ETL principles and best practices.
  - Excellent collaborative and communication skills, with demonstrated ability to mentor and support team members.
  - Experience working with Linux environments, cloud services (AWS), and Docker.
  - Strong decision-making capabilities with the ability to work independently and proactively.
  
  Will be a plus:
  - Experience in web scraping, data extraction, cleaning, and visualization.
  - Understanding of multiprocessing and multithreading, including process and thread management.
  - Familiarity with Redis.
  - Experience with Flask / Flask-RESTful for API development.
  - Knowledge and experience with Kafka.
  
  Key Responsibilities:
  - Develop and maintain a robust data processing architecture using Python.
  - Effectively utilize ElasticSearch and PostgreSQL for efficient data management.
  - Design and manage data pipelines using Kafka and SQS.
  - Optimize code structure and performance for maximum efficiency.
  - Design and implement efficient ETL processes.
  - Analyze and optimize algorithmic solutions for better performance and scalability.
  - Collaborate within the AWS stack to ensure flexible and reliable data processing systems.
  - Provide mentorship and guidance to colleagues, fostering a collaborative and supportive team environment.
  - Independently make decisions related to software architecture and development processes to drive the project forward.
  
  We offer:

- Great networking opportunities with international clients, challenging tasks;

- Building interesting projects from scratch using new technologies;

- Personal and professional development opportunities;

- Competitive salary fixed in USD;

- Paid vacation and sick leaves;

- Flexible work schedule;

- Friendly working environment with minimal hierarchy;

- Team building activities and corporate events.