Senior Production Security Engineer

About the Client

Our client is a neocloud building purpose-built GPU infrastructure for AI workloads. We operate large-scale clusters powering training and inference for some of the most demanding AI customers in the market, and we’re rapidly expanding. Our infrastructure runs on NVIDIA and AMD accelerators, InfiniBand and high-speed Ethernet fabrics, and a production stack spanning bare-metal provisioning, Kubernetes, and OpenStack.

We’re small, senior, and moving fast. The people who do well here own problems end-to-end and make decisions with incomplete information.
 

The role

Security is a core part of who we are and how we operate. We’re hiring a Senior Production Security Engineer to help strengthen and scale our security capabilities.

This is not a compliance-focused role, and it is not purely advisory. We are looking for a hands-on senior engineer who can identify high-risk gaps and opportunities, partner with engineering and leadership teams to design effective solutions, and implement those solutions through code, automation, and tooling. In addition to security-focused initiatives, this person will contribute meaningfully to broader infrastructure, platform, and production engineering efforts.
Beyond the technical responsibilities, this role plays a key part in fostering a security-first engineering culture. You will advocate for best practices, educate teams, and help raise the security bar across the organization. This is an opportunity to shape the future of our security function and become a foundational leader as it continues to grow.
 

What this role is, and is not

This role is:

• A senior engineer who drives the core of a security focus.
• A self-directed practitioner who identifies security weaknesses, opportunities to level up, prioritizes them, and builds a credible roadmap.
• A cross-functional partner who works with infrastructure, platform, cloud, and leadership to recommend action and architect solutions.
• A builder who implements solutions in software, tooling, and infrastructure, not just slides and policies.
• A reviewer and advisor who partners with developers and teams on proposed projects from a security perspective.
• A versatile contributor who still gets meaningful work done in other engineering domains.

This role is not:

• A regulatory or legal compliance role. Compliance evidence will be a byproduct of good engineering, not the job itself.
• An audit-and-recommend-only role. We need someone who builds what they recommend.
   

What you’ll work on

The scope below outlines where this role will harden, automate, and validate Client’s security posture across our infrastructure, platform and customer facing surfaces.  We expect the person in this role to refine priorities based on what they learn in their first weeks.
 

Internal tools, services, and applications that harden our security posture

• Proper network segmentation best practices across tenants, management, and customer planes.
• SSH and system access management architecture
• Multi-tier secret storage for both internal users and customers.
• User account management and lifecycle policy for core systems (joiners, movers, leavers done right).
• Secret rotation pathways for core internal systems, automated, auditable, and not reliant on tribal memory.
• x509 certificate issuing architecture, an internal PKI we can actually operate and rotate.
   

Security-focused monitoring, alerting, and auditing

• Multi-redundancy Vault admin logging.
• GCP access and action logging with real retention and alerting.
• Automatic alerting and reporting from log aggregation pipelines.
• Network traffic analysis and reporting, feeding off our existing fabric telemetry.
• Security-focused dashboards that are actually used, not just built.
   

Customer-facing secret and identity management

As our cloud offering matures, this role helps define the customer-side story:

• KMS architecture and options.
• Encryption strategy, at rest, in transit, and tenant-scoped.
• Security reporting surfaces for customers.
• Customer login, identity, and account management for the cloud platform.
   

Review of what we already have

• Own and drive third-party penetration tests and red team engagements.
• Lead a full design and architecture review of our current stack and document the risk landscape.
• Translate findings into a prioritized, resourced remediation plan.
   

Clear policies and partnerships around third-party code

• Own the liability model, or make the deliberate decision to pay someone else to.
• Processes and tooling for third-party software review, and dependency risk.
• A “happy path” for engineers, the easy, supported way to pull in third-party code safely.
   

What we’re looking for

Required

• 7+ years of hands-on engineering experience, with a substantial portion focused on security in production environments.
• Demonstrated experience designing and operating in multi-tenant, tenant-isolated environments, cloud, neocloud, hosting, or regulated enterprise.
• Deep, practical familiarity with secrets management (HashiCorp Vault or equivalent), PKI, and identity systems (Okta, Google Workspace, or similar).
• Strong Linux and OS fundamentals, you understand what is actually happening on the box, not just what the tool reports.
• Working fluency in networking and network security: firewalls, segmentation, VPN, BGP at a conceptual level, and modern L3/L4 controls.
• Experience with at least one of: Kubernetes security (admission control, network policy, workload identity), cloud IAM at depth (GCP, AWS, or Azure), or infrastructure-as-code security review.
• CI/CD automation design and implementation experience, enough to build the paved road yourself when needed.
• Experience driving or responding to a third-party audit, penetration test, or customer-led security review.

• Clear, direct written communication, much of our team and work is async.

   

Strongly preferred

• Experience being the first or one of the first dedicated security hires at a company, and building a function from there.
• Experience in GPU cluster, HPC, or AI infrastructure environments.
• Familiarity with InfiniBand, RoCE, or other high-performance fabrics from a security and segmentation perspective.
• Experience with SOC 2, ISO 27001, HIPAA, or similar frameworks, enough to know what auditors actually care about, without being a compliance specialist.
• Experience supporting sensitive customer workloads (regulated industries, government, frontier AI labs).

How we work

• We’re remote-first across US, LATAM, and EU time zones.
• We write things down, decisions, architecture, runbooks, post-mortems, threat models.
• We use AI tooling heavily in day-to-day operations and expect everyone on the team to be fluent with it and to help us get more leverage from it.
• We ship, we debug, we iterate. We don’t process-engineer our way around problems that need to be solved.

AI as a force multiplier

We’re making a deliberate bet that AI changes the shape of infrastructure and security teams. Our plan is to scale Client's platform faster than we scale headcount, using coding agents to write and refactor automation, AI-assisted testing and validation to harden our changes, and LLM-driven tooling to compress the work of investigation, documentation, and operational review. We’re already doing this in production: Claude Code for SRE operations, MCP servers exposing Jira, NetBox, and Checkmk, and an active effort to turn tribal infrastructure knowledge into AI-consumable formats.

For security specifically, we want someone who is excited about this, not someone who tolerates AI tooling because it’s in the JD. That means:

• Using AI-assisted workflows as the default for tasks like log review, config audit, threat modeling support, and policy drafting.
• Identifying where agents can own meaningful slices of security work, triaging alerts, drafting RCAs, generating and validating hardening configurations, reviewing pull requests for security-relevant changes.
• Building the substrate the team needs to make AI actually useful, good documentation, clear schemas, MCP integrations, and structured knowledge about our environment.
• Being honest about where AI is adding value and where it isn’t, with the team and with leadership.

If your reaction to this is “finally, a team that’s serious about this,” we should talk.

What success looks like

First 30 days: You’ve built working relationships with the infrastructure, platform, network, and cloud teams, shadowed enough of our production environment to understand where real risk sits, and produced a draft of our top security issues ranked by impact and effort.

First 90 days: You own a credible, prioritized security risk register that leadership and engineering actually use in planning. You’ve made at least one meaningful architectural call, on access, secrets, or identity, that the team agrees was the right one. Shared admin-scoped SSH keys and shared or weak passwords on critical systems have a concrete plan and timeline to be eliminated.

First 12 months: A functioning internal PKI and secret rotation story is in place for core systems. Security-relevant logging, alerting, and dashboards cover our highest-value systems with an on-call path that works. A third-party penetration test has been executed and findings have a remediation plan with owners and dates. Engineering teams have a clear, low-friction path for third-party code and for new service design reviews. You and leadership have a shared point of view on what the next hire into the security function looks like, and whether this becomes a team, a CSO search, or both.

Benefits:

• Unlimited Paid Time Off (PTO)
• Zero-cost employer-covered health insurance
• Company-funded 401k contributions
• Transparent, candid culture with 1:1 coaching, performance reviews, and a consistent feedback loop
• Diverse, respectful, high-performing coworkers you’ll want to achieve greatness with!