Lead Big Data Engineer (with Python experience)

We are looking for an experienced data engineer with design and

development experience in automating scalable and high-performance data processing systems (batch and/or streaming) on the cloud (AWS preferably).

What You’ll Do

● Design our data models for optimal storage and retrieval on the cloud and to meet critical product and business requirements

● Build scalable and highly-performant distributed data processing systems as we migrate to the cloud

● Work closely with our business stakeholders to flesh out and deliver on requirements in an agile manner

● Set and evolve data standards and best practices

● Contribute to the data architecture and align it with the business and technology

● Adhere to and enforce software development best practices on the cloud in areas including but not limited to CI/CD, code reviews, automated testing, operational excellence, data quality etc.

What You Should Have

● 5+ years of experience with Big Data

● 4+ years programming experience in Java Python

● 3+ years experience using an enterprise cloud-based solution

● 3+ years implementing data processing pipelines on the cloud (batch and/or streaming)

● 3+ years of experience with Spark, Kafka

● Advanced understanding of SQL, relational and NoSQL databases required

● Experience with various data access patterns, streaming technology, data quality, data modeling, data performance, and cost optimization