• Build and operate telemetry storage systems optimized for large-scale time-series and event data.

• Contribute to observability standards across services, including metrics, tracing instrumentation, logging, and SLO implementation.

Infrastructure and Platform Observability

• Build visibility across compute, storage, and networking layers of the platform.

• Instrument GPU clusters, inference workloads, and distributed training environments.

• Detect infrastructure degradation such as:

• GPU throttling,

• Network congestion,

• Storage latency,

• Hardware degradation.

• Implement telemetry pipelines for GPU, CPU, network, and storage performance metrics.

Customer-Facing Observability

• Build dashboards and monitoring tools that expose system health and performance to both internal teams and customers.

• Provide insights into workload performance including:

• GPU utilization,

• Storage throughput,

• Network latency,

• Distributed inference performance.

• Develop performance analysis tools that help customers understand system bottlenecks.

Network and Infrastructure Telemetry

• Develop and maintain network observability platforms.

• Build telemetry collectors and exporters using Python or Go.

• Ingest telemetry from infrastructure components including:

• NVIDIA Cumulus Linux,

• VyOS routers,

• Citrix NetScaler / WAF.

• Design telemetry ingestion pipelines using protocols such as:

• gNMI,

• SNMP,

• Streaming telemetry.

Reliability Engineering

• Design advanced alerting and anomaly detection systems.

• Contribute to platform SLOs, SLIs, and reliability metrics.

• Build automated detection of infrastructure anomalies.

• Integrate observability signals with operational workflows and incident management systems.

• Participate in on-call rotations supporting platform observability and telemetry infrastructure.

Cross-Team Collaboration

• Partner with platform, networking, storage, and compute teams to instrument services.

• Work closely with operations teams to improve monitoring and incident response.

• Provide guidance and mentorship to engineers on observability best practices.

• Promote good observability practices across teams and help engineers adopt effective instrumentation and monitoring patterns.

Technical Stack: Observability and telemetry technologies used across the platform include:

Observability Framework

• Prometheus.

• OpenTelemetry.

• Grafana.

• Distributed logging systems.

• High-scale telemetry databases, such as ClickHouse or similar.

Hardware and Infrastructure Telemetry

• Redfish / BMC telemetry.

• IPMI.

• Linux system metrics.

• Hardware health monitoring and node lifecycle telemetry.

NVIDIA GPU Telemetry

• NVIDIA DCGM.

• DCGM Exporter.

• NVML.

• NVIDIA GPU Operator telemetry stack.

• NVSwitch / NVLink telemetry.

AI Workload Telemetry

• Distributed training telemetry.

• Inference latency and throughput metrics.

• NCCL communication health.

• GPU synchronization latency.

• KV-cache access latency for inference workloads.

• Dataset loading and storage I/O performance.

Networking Telemetry

• NVIDIA NetQ.

• gNMI streaming telemetry.

• SNMP.

• Network flow telemetry.

• RDMA / RoCE performance monitoring.

What you’ll need

Required Experience

• Proven experience operating large distributed infrastructure platforms.

• Strong background in observability systems and telemetry pipelines.

• Experience building metrics, logging, tracing, alerting, and dashboards at production scale.

• Strong programming skills in Go, Python, or Rust.

• Experience with large-scale time-series data platforms.

• Experience with large-scale GPU cloud platforms, HPC environments, or AI infrastructure.

• Experience monitoring AI workloads such as training or inference clusters.

Infrastructure Knowledge

• Deep understanding of distributed systems observability.

• Familiarity with cloud-native infrastructure such as Kubernetes, automation, and CI/CD.

• Experience operating observability systems for high-performance or large-scale environments.

Networking and Infrastructure Telemetry

• Experience monitoring complex networking environments.

• Familiarity with telemetry protocols such as gNMI, SNMP, and streaming telemetry.

• Experience integrating network and system telemetry into centralized monitoring platforms.

Analytical Skills

• Strong data analysis capabilities.

• Ability to interpret complex telemetry signals and translate them into actionable insights.

• Ability to diagnose performance issues across distributed systems.