Responsibilities

As a Storage Infrastructure Engineer, you will be responsible for leading, designing, building, operating, scaling, maintaining, and evolving Workato enterprise-grade storage infrastructure. You will work closely with all engineering & infrastructure teams in Workato leading our global storage story for different use cases, scenarios, enterprise-grade real-time latency-critical requirements, and global scale. You will work closely with the engineering leadership team and will have a direct long-term strategic impact on the modernization and evolution of the Workato architecture. You are the expert in different distributed storage systems, know how to use them effectively, and understand the internals and deep principles behind them.

At the same time, it is a hands-on role so there is the expectation of a significant contribution to storage infrastructure as a code, monitoring, continuous analysis of slow requests and trends, insights, reliability, and upgrades. We expect from all Workato engineering roles deep knowledge & understanding of computer systems, desire & love to go to the lowest level of the details, do troubleshooting using modern techniques and tools.

It is required to have practical experience with modern infrastructure building tools and principles such as IaC, knowledge and experience of running distributed storage systems in cloud (AWS preferred) and container orchestration systems (kubernetes is mandatory) in highly reliable, highly scalable manner. 

One expectation is great understanding of reliability and availability in terms of nines for different use cases, knowledge of architecture & design patterns and procedures for different number of nines. Knowledge and experience of storage security, auditability and compliance requirements are highly desired. 

Conduct research and emulation work to help to select future storage solutions. Be it just schema optimization of a 100TB database, or move from PostgreSQL to TiDB or any other storage.

Workato storage layer is the most mission-critical highly loaded distributed infrastructure. It based on industry leading mature technologies such as PostgreSQL, Redis, Clickhouse, MySQL/MariaDB deployed in self-hosted or vendor managed (e.g. AWS Aurora) version.

We are currently working on scaling, upgrading, automating, securing, modernizing the storage layer to meet new requirements of rapidly growing business such as:

1. Automate storage maintenance to fully eliminate manual work
2. Support zero-downtime DB upgrades
3. Support massive compute scale: thousands of network clients (up to hundred applications)
4. Bring-in proper observability & monitoring 
5. Scale storage up to 5x-10x in short term
6. Re-design storage architecture for longer term scalability up to 100x times
7. Keep cost under control in predictable manner
8. Take ownership of distributed storage story and be center of distributed storage expertise

Requirements

Qualifications / Experience / Technical Skills

• 8+ years of trackable work experience with deploying and supporting highly scalable distributed enterprise-grade storage.

• Production experience of operating, maintaining, troubleshooting, scaling clustered sharded highly-available systems based on RDBMS like PostgreSQL, MariaDB.

• Great knowledge and understanding of ecosystem of OSS, auxiliary tools, plugins, extensions, clustering, solutions around RDBMS like PgBouncer, Vitess, Galera.

• Experience of zero-downtime major upgrades and migrations of highly loaded RDBMS and other storages with complicated transactional workload.

• Understanding of compliance certification, auditability, security, access controls for storage systems.

• Experience of building cross-regional Disaster Recovery storage solutions with low RTO/RPO numbers according to modern industry standard (from hours to minutes). Experience of designing and implementing cross-regional distributed master-master storage systems.

• Experience of increasing of number of nines in terms of reliability and availability for storages, architecture re-design, justification, requirements collection, base storage technology evaluation and comparison according to requirements.

• Experience of building proper observability, monitoring, alerting, logging and analyzing storage health. Ability to troubleshoot bottlenecks of distributed storage solutions.

• Experience of managing complex storage infrastructure in the cloud (in Kubernetes clusters, AWS cloud) using Infrastructure as Code tools (Terraform is highly preferred).

• Experience deploying stateful and stateless application into Kubernetes with modern tools like, e.g Kustomize, Helm, ArgoCD, etc.

• Experience with AWS cloud computing (EC2, EBS, RDS, EKS, Route53, VPCs, Subnets, Route Tables).

• Basic knowledge of one or more high-level programming languages, such as Python, Go. Basic knowledge of Ruby, readiness to read Ruby monolith.

Soft Skills / Personal Characteristics

• Proper balance between being hands-on and deeper analytical approaches.

• Interest in modern big distributed storage technologies, architectures.

• Readiness to work remotely with teams distributed across the world and timezones.

• Spoken English (at the level enough to pass technical interviews and later work with colleagues).