Staff Software Engineer, ML Infrastructure

Why Attentive needs you

• Define and lead cross-functional ML infrastructure and ML platform projects
• Demonstrate the ability to analyze, troubleshoot, coordinate, and resolve complex ML infrastructure issues
• Orchestrate Kubernetes and ML training / inference infrastructure exposed as an ML platform
• Expose and manage environments, interfaces, and workflows to enable ML engineers to develop, build, and test ML models and services
• Manage and expand our feature store implementation that allows ML teams to self-service data labeling, feature engineering, and batch inferencing
• Close the latency gap on model inference to online, real-time model serving
• Develop automation workflows to improve team efficiency and ML stability
• Analyze and improve efficiency, scalability, and stability of various system resources
• Partner with other teams and business stakeholders to deliver business initiatives
• Help onboard new team members, provide mentorship and enable successful ramp up on your team's code bases

About you

• You have been working in the areas of MLOps / ML Platform / Data Platform / Site Reliability Engineering / DevOps / Infrastructure for 8+ years, and have an understanding of best practices for DevOps applied to ML
• You have successfully led major cross-functional, cross-team ML infrastructure or ML platform projects
• Your passion is infrastructure and exposing platform capabilities through interfaces that enable high performance ML practices, rather than designing ML experiments (this team does not directly develop ML models)
• You have deep experience in Kubernetes applied to ML use cases such as CPU & GPU training, hosting and exposing ML tools, and managing ML endpoints as web services
• You understand the key differences between online and offline ML inferences and can voice the critical elements to be successful with each
• You have a background in software development and are passionate about bringing that experience to bear on the world of ML infrastructure
• You have experience with Infrastructure as Code using Terraform and can’t imagine a world without it
• You understand the importance of CI/CD in building high-performing teams and have worked with tools like Jenkins, CircleCI, Argo Workflows, and ArgoCD
• You are passionate about observability and worked with tools such as Splunk, Nagios, Sensu, Datadog, New Relic
• You are very familiar with containers and container orchestration and have direct experience with vanilla Docker as well as Kubernetes as both a user and as an administrator. 

Some sample projects

• Design and lead implementation of an online inference pipeline with champion/challenger model testing
• Unite existing pipelines across data, ML, and platform teams to handle low-latency, high volume real-time streaming use cases in production inference workflows
• Define golden path build and release pipelines for better reliability and Python package management
• Identify opportunities to improve scalability, resiliency, and cost efficacy of GPU training and inference workflows
• Design and lead implementation of a low-touch, automated model CI/CD pipeline

Our scale

• 8,000 brands powered by Attentive sent over 2.2 billion text messages over Cyber Week 2023 (Black Friday/Cyber Monday) representing a growth of 31% from 2022
• We sent 32 billion SMS messages in 2023, up 32% YoY. That’s an average of 87 million per day
• Our production cluster contains over 18,000 containers which serve 200+ services
• Our streaming services process over 80 billion events per month

What we use

• Our infrastructure runs primarily in Kubernetes hosted in AWS’s EKS
• Infrastructure tooling includes Istio, Datadog, Terraform, CloudFlare, and Helm
• Our backend is Java / Spring Boot microservices, built with Gradle, coupled with things like DynamoDB, Pulsar, AirFlow, Postgres, Planetscale, and Redis, hosted via AWS
• Our frontend is built with React and TypeScript, and uses best practices like GraphQL, Storybook, Radix UI, Vite, esbuild, and Playwright
• Our automation is driven by custom and open source machine learning models, lots of data and built with Python, Metaflow, HuggingFace 🤗, PyTorch, TensorFlow, and Pandas