Data Engineer (PostgreSQL, Snowflake, Google BigQuery, MongoDB, Elasticsearch)
We are looking for a Data Engineer with a diverse background in data integration to join the Data Management team. Some data are small, some are very large (1 trillion+ rows), some data is structured, some data is not. Our data comes in all kinds of sizes, shapes and formats. Traditional RDBMS like PostgreSQL, Oracle, SQL Server, MPPs like StarRocks, Vertica, Snowflake, Google BigQuery, and unstructured, key-value like MongoDB, Elasticsearch, to name a few.
We are looking for individuals who can design and solve any data problems using different types of databases and technology supported within our team. We use MPP databases to analyze billions of rows in seconds. We use Spark and Iceberg, batch or streaming to process whatever the data needs are. We also use Trino to connect all different types of data without moving them around.
Besides a competitive compensation package, you’ll be working with a great group of technologists interested in finding the right database to use, the right technology for the job in a culture that encourages innovation. If you’re ready to step up and take on some new technical challenges at a well-respected company, this is a unique opportunity for you.
Responsibilities:
- Work within our on-prem Hadoop ecosystem to develop and maintain ETL jobs
- Design and develop data projects against RDBMS such as PostgreSQL
- Implement ETL/ELT processes using various tools (Pentaho) or programming languages (Java, Python) at our disposal
- Analyze business requirements, design and implement required data models
- Lead data architecture and engineering decision making/planning.
- Translate complex technical subjects into terms that can be understood by both technical and non-technical audiences.
Qualifications: (must have)
- BA/BS in Computer Science or in related field
- 5+ years of experience with RDBMS databases such as Oracle, MSSQL or PostgreSQL
- 2+ years of experience managing or developing in the Hadoop ecosystem
- Programming background with either Python, Scala, Java or C/C++
- Experience with Spark. PySpark, SparkSQL, Spark Streaming, etc…
- Strong in any of the Linux distributions, RHEL,CentOS or Fedora
- Working knowledge of orchestration tools such Oozie and Airflow
- Experience working in both OLAP and OLTP environments
- Experience working on-prem, not just cloud environments
- Experience working with teams outside of IT (i.e. Application Developers, Business Intelligence, Finance, Marketing, Sales)
Desired: (nice to have)
- Experience with Pentaho Data Integration or any ETL tools such as Talend, Informatica, DataStage or HOP.
- Deep knowledge shell scripting, scheduling, and monitoring processes on Linux
- Experience using reporting and Data Visualization platforms (Tableau, Pentaho BI)
- Working knowledge of data unification and setup using Presto/Trino
- Web analytics or Business Intelligence a plus
- Understanding of Ad stack and data (Ad Servers, DSM, Programmatic, DMP, etc)