Distributed Systems Engineer - High-Availability Dispatch

Location: Remote

Department: Engineering

About the Role:

Glydways’ Dispatch system is the centralized brain that coordinates our autonomous vehicle fleet. We’re looking for a senior distributed systems / backend engineer to design and implement state sharing between Dispatch instances, hot failover mechanisms, and robustness testing for this safety-critical, real-time service. This is an application-layer role: you will work primarily on the Dispatch codebase (C++), making stateful services correct and resilient under failure, not on generic DevOps, cloud account management, or CI/CD pipelines. You’ll partner closely with autonomy and ops teams to harden behavior, design recovery flows, and drive down flaky and unsafe production states over time.

Candidates whose experience is limited to Kubernetes administration, CI/CD tooling, or cloud configuration without owning stateful application behavior are not a fit for this role.

Responsibilities:

• Design and implement state sharing and replication between multiple Dispatch instances (tickets, journeys, vehicle state, restrictions).
• Build leader election and failover mechanisms (active/standby, hot/warm backup) that guarantee a single authoritative Dispatch at a time and clean handoff on failures.
• Harden Dispatch behavior for restart-safety and idempotency, ensuring retries, replays, and partial failures do not cause double assignment, inconsistent state, or unsafe conditions.
• Design and run stress, load, and fault-injection tests (including chaos experiments) to validate Dispatch behavior under high load, network issues, and process crashes.
• Improve system hardening and recovery flows, defining how Dispatch enters safe modes, recovers from faults, and resumes normal operation in a controlled way.
• Extend and tune observability for Dispatch (logs, metrics, traces, SLOs) so state divergence, failover events, and backlog issues are visible and diagnosable.
• Collaborate with autonomy, product, and ops teams to translate algorithmic and operational requirements into concrete guarantees around state, failover, and robustness.
• Participate in on-call and incident response for Dispatch, lead root-cause analysis for reliability issues, and drive long-term fixes into the application code and architecture.

Knowledge, Skills and Abilities:

• Proven experience designing and shipping stateful distributed services that stay correct under failures.
• Strong programming background in a systems language (C++ strongly preferred) and comfort working at the application layer (routing, tickets, vehicle state, safety envelopes).
• Hands-on experience with leader election / primary–secondary patterns, active/standby or similar, and state replication / recovery (snapshots, event logs, replay, or equivalent).
• Deep understanding of idempotent operations and message semantics (retries, duplicates, out-of-order messages) in networked, message-driven systems (TCP/UDP, gRPC, pub/sub, etc.).
• Experience designing and running stress, load, soak, and fault-injection/chaos tests for distributed systems, and using their results to drive system hardening.
• Strong observability and incident-response skills: defining SLOs, instrumenting metrics/traces, debugging complex failure modes, and leading postmortems for stateful services.
• Safety-critical or mission-critical mindset: familiarity with failure-mode analysis and designing for fail-safe / fail-operational behavior is a plus.
• Experience with cloud platforms is a plus, but this is not a pure DevOps or CI/CD role; candidates must have meaningful ownership of application-level behavior and state.