Consultant Data Engineer (Python/Databricks)

Softermii is looking for a part-time Data Engineering Consultant / Tech Lead who will do technical interviews, assist with upcoming projects, and occasionally be hands-on with complex development tasks — including data pipeline design and solution optimization on Databricks.

Type of cooperation: Part-time

 

⚡️Your responsibilities on the project will be:

• Interview and hire Data Engineers
• Supervise work of other Engineers and have hands on for the most complicated tasks from backlog, focus on unblocking other data Engineers in case of technical difficulties
• Develop and maintain scalable data pipelines using Databricks (Apache Spark) for batch and streaming use cases.
• Work with data scientists and analysts to provide reliable, performant, and well-modeled data sets for analytics and machine learning.
• Optimize and manage data workflows using Databricks Workflows and orchestrate jobs for complex data transformation tasks.
• Design and implement data ingestion frameworks to bring data from various sources (files, APIs, databases) into Delta Lake.
• Ensure data quality, lineage, and governance using tools such as Unity Catalog, Delta Live Tables, and built-in monitoring features.
• Collaborate with cross-functional teams to understand data needs and support production-grade machine learning workflows.
• Apply data engineering best practices: versioning, testing (e.g., with pytest or dbx), documentation, and CI/CD pipelines

🕹Tools we use: Jira, Confluence, Git, Figma

🗞Our requirements to you:

• 5+ years of experience in data engineering or big data development, with production-level work.
• Architect and develop scalable data solutions on the Databricks platform, leveraging Apache Spark, Delta Lake, and the lakehouse architecture to support advanced analytics and machine learning initiatives.
• Design, build, and maintain production-grade data pipelines using Python (or Scala) and SQL, ensuring efficient data ingestion, transformation, and delivery across distributed systems.
• Lead the implementation of Databricks features such as Delta Live Tables, Unity Catalog, and Workflows to ensure secure, reliable, and automated data operations.
• Optimize Spark performance and resource utilization, applying best practices in distributed computing, caching, and tuning for large-scale data processing.
• Integrate data from cloud-based sources (e.g., AWS S3), ensuring data quality, lineage, and consistency throughout the pipeline lifecycle.
• Manage orchestration and automation of data workflows using tools like Airflow or Databricks Jobs, while implementing robust CI/CD pipelines for code deployment and testing.
• Collaborate cross-functionally with data scientists, analysts, and business stakeholders to understand data needs and deliver actionable insights through robust data infrastructure.
• Mentor and guide junior engineers, promoting engineering best practices, code quality, and continuous learning within the team.
• Ensure adherence to data governance and security policies, utilizing tools such as Unity Catalog for access control and compliance.
• Continuously evaluate new technologies and practices, driving innovation and improvements in data engineering strategy and execution.
• Experience in designing, building, and maintaining data pipelines using Apache Airflow, including DAG creation, task orchestration, and workflow optimization for scalable data processing.
• Upper-Intermediate English level.

👨‍💻Who will you have the opportunity to meet during the hiring process (stages):
Call, HR, Tech interview, PM interview.

🥯What we can offer you:

• We have stable and highly-functioning processes – everyone has their own role and clear responsibilities, so decisions are made quickly and without unnecessary approvals. 
• You will have enough independence to make decisions that can affect not only the project but also the work of the company.
• We are a team of like-minded experts who create interesting products during working hours and enjoy spending free time together.
• Do you like to learn something new in your profession or do you want to improve your English? We will be happy to pay 50% of the cost of courses/conferences/speaking clubs.
• Do you want an individual development plan? We will form one especially for you + you can count on mentoring from our seniors and leaders.
• Do you have a friend who is currently looking for new job opportunities? Recommend them to us and get a bonus.
• And what if you want to relax? Then we have 21 working days off.
• What if you are feeling bad? You can take 5 sick leaves a year.
• Do you want to volunteer? We will add you to a chat, where we can get a bulletproof vest, buy a pickup truck or send children's drawings to the front.
• And we have the most empathetic HRs (who also volunteers!). So we are ready to support your well-being in various ways.

👨‍🏫A little more information that you may find useful:

- our adaptation period lasts 3 months, this period of time is enough for us to understand  each other better;

- there is a performance review after each year of our collaboration where we use a skills map to track your growth;

- we really have no boundaries in the truest sense of the word – we have flexible working day is up to you.

Of course, we have a referral bonus syst