Tech Lead Manager, Inference

Location: SF Bay Area, CA

Department: Systems Research & Engineering

Location Type: HYBRID

Employment Type: FULL_TIME

The Role

What You’ll Do

• Spend at least half your time hands-on in the serving stack: architect and build core platform components, own the hardest design decisions, and debug the toughest production incidents yourself
• Lead, grow, and develop the inference engineering team: own hiring, coaching, and career growth, and build the team’s operational culture — on-call, incident response, capacity planning, and postmortems
• Set the technical roadmap for the serving platform: model serving engines, request routing and scheduling, autoscaling, caching, observability, and deployment pipelines
• Own the platform’s SLOs and economics: latency and availability targets, GPU utilization, and cost per generation across every model we serve
• Partner closely with research to ship new model architectures into production on day zero, and to integrate serving into online RL and evaluation loops
• Manage and optimize inference workloads across heterogeneous fleets — multiple clusters, clouds, and GPU vendors — including capacity planning and hardware bring-up
• Build sophisticated scheduling and queueing systems that optimally leverage expensive GPU resources against live traffic patterns, cluster availability, and user priority

Representative Projects

• Design intelligent routing and scheduling that optimizes request distribution across thousands of GPUs in multiple regions and clouds
• Stand up disaggregated prefill/decode serving with tiered KV-cache reuse across GPU memory, DRAM, NVMe, and network storage
• Autoscale and hot-swap models across the fleet to dynamically match GPU supply with live demand across production, research, and experimental workloads
• Take a new multimodal architecture from research checkpoint to a production deployment serving millions of users, including quantization, speculative decoding, and precision/regression validation across hardware platforms
• Build end-to-end tracing that follows any inference request through its full lifetime — queueing, routing, prefill, decode, and delivery
• Integrate the inference stack into an online reinforcement learning pipeline where serving throughput directly gates training progress

Background

• 8+ years of engineering experience in large-scale distributed systems or ML infrastructure, with several years building and operating model-serving or inference platforms in production
• Experience running inference platforms at scale — you have operated fleets on the order of thousands of GPUs across multiple clusters or clouds, and you understand what breaks at that scale
• Technical leadership experience, including managing or leading engineers through periods of rapid growth — and a genuine desire to keep at least half your time in hands-on technical work rather than move into pure management
• Deep, practical expertise in LLM and foundation-model serving engines (vLLM, SGLang, TensorRT-LLM, or equivalent) — ideally you’ve modified engine internals, debugged edge cases under load, and contributed improvements back
• Strong command of the serving-performance toolkit: continuous batching, KV-cache management, quantization, speculative decoding, and parallelism strategies (TP/EP/pipeline)
• Strong Python and PyTorch; experience operating services on Kubernetes at scale
• Experience with queues, scheduling, traffic control, and fleet management at scale

Bonus Points

• Experience serving diffusion, video, or other multimodal generative models (not just text), and with FFmpeg/multimedia processing
• Experience with modern networking stacks — RDMA (RoCE, InfiniBand), NVLink — including KV-cache transfer and multi-node serving topologies
• Experience across heterogeneous accelerator platforms (NVIDIA, AMD, TPU, Trainium) and the porting/validation work that comes with them
• Contributions to open-source serving infrastructure (vLLM, SGLang, Ray, Kubernetes ecosystem)
• Systems-language depth (Rust, C++, CUDA/HIP) for kernel- and runtime-level optimization