Full-Stack Systems Engineer (Go / Data Pipelines / Distributed Systems)

PROJECT OVERVIEW

Our client has engaged us to write a high-volume, high-throughput data mastering system capable of processing hundreds of millions of records per day while achieving data mastering accuracy on par with similar commercial systems. 

For this, we plan to launch a greenfield project that will combine REST API design, concurrent processing, larger-scale datasets (tens of TB), and rules engine design. We are looking for a full-stack, senior software developer who is excited to implement, and enhance such a system. 

This position is truly full-stack, which means that you will:

• Design and implement processing pipelines, including:
  • Message brokering engines (based on NATS and Kafka)
  • Pipeline design (state machine, exception handling, etc.)
  • Design and implement a data mastering rules engine including matching, merging, and history retention/representation.
  • Design and implement storage mechanisms for high-speed and high-volume storage and retrieval, using Clickhouse, PostgreSQL, and other SQL/NoSQL technologies as appropriate.
• Design and implement REST API interfaces to the system, including:
  • Data representation
  • Authentication and authorization
  • Asynchronous request handling
  • Error and exception handling
  • Design and implement data management UI for admin users

  • Implement all of the above using CI/CD, unit and integration testing, containerization, and Kubernetes deployment orchestration

     

REQUIREMENTS

English: Fluent (C1 or above).

Programming: Strong experience in Go.

Web APIs: Solid experience building REST APIs using frameworks like Gin, with authentication, schema validation (e.g., JSON schema), etc.

Concurrency: Some experience with high-volume, concurrent processing pipelines and message brokers like NATS, Kafka, or RabbitMQ.

Databases:

• Strong skills in designing and scaling RDBMS schemas (100+ GB, sharding, etc.).
• Some experience with search engines like Elasticsearch or Solr, including scaling, faceted search, and phonetic matching.
• Some knowledge of multi-tenancy in databases and search engines.

DevOps:

• Solid experience with Docker in development.
• Some experience with Docker/Kubernetes for CI/CD and deployment.

Testing & Monitoring:

• Experience writing unit, functional, and integration tests for web services.
• Some experience with APM tools (e.g., DataDog, New Relic).