Staff Software Engineer- Reliability

Responsibilities

• Collaborate with researchers and engineers to specify the availability, performance, correctness, and efficiency requirements of the current and future versions of our GPU infrastructure.
• Work with multiple GPU cloud providers to scale up, scale down, maintain and monitor our 000's GPUs in many clusters.
• Design and implement solutions to ensure the scalability of our infrastructure to meet rapidly increasing demands.
• Implement and manage monitoring systems to proactively identify issues and anomalies in our production environment.
• Implement fault-tolerant and resilient design patterns to minimize service disruptions.
• Build and maintain automation tools to streamline repetitive tasks and improve system reliability.
• Participate in an on-call rotation to respond to critical incidents and ensure 24/7 system availability alongside other infrastructure developers.
• Develop and maintain service level objectives (SLOs) and service level indicators (SLIs) to measure and ensure system reliability.

Experience

• Proven work experience 10+ yrs as an reliability engineer, production engineer, infrastructure software engineer or a similar role in a fast-paced, rapidly scaling company.
• Strong proficiency in GPU cloud infrastructure, including the underlying concepts of scheduling, scaling, cloud storage, networking and security.
• Proficiency in programming/scripting languages.
• Experience with containerization technologies and container orchestration platforms like Kubernetes or equivalent.
• Knowledge of IaC tools such as Terraform or CloudFormation or equivalent.
• Excellent problem-solving and troubleshooting skills.
• Strong communication and collaboration skills.
• Experience with observability tools; examples include DataDog, Prometheus, Grafana, Splunk and ELK stack or similar.
• Knowledge of security best practices in cloud environments.
• Good to have experience as an SRE within the AI/ML space is strongly preferred.

Compensation

• The pay range for this position in California is $200,000 - $250,000yr; however, base pay offered may vary depending on job-related knowledge, skills, candidate location, and experience. We also offer competitive equity packages in the form of stock options and a comprehensive benefits plan.