Site Reliability Engineer - TikTok Machine Learning Engine

TikTok is the leading destination for short-form mobile video. Our mission is to inspire creativity and bring joy. TikTok has global offices including Los Angeles, New York, London, Paris, Berlin, Dubai, Singapore, Jakarta, Seoul and Tokyo.

Why Join Us
Creation is the core of TikTok's purpose. Our platform is built to help imaginations thrive. This is doubly true of the teams that make TikTok possible.
Together, we inspire creativity and bring joy - a mission we all believe in and aim towards achieving every day.
To us, every challenge, no matter how difficult, is an opportunity; to learn, to innovate, and to grow as one team. Status quo? Never. Courage? Always.
At TikTok, we create together and grow together. That's how we drive impact - for ourselves, our company, and the communities we serve.
Join us.

About The Team
AML-Engine is committed to solving full stack problems in large-scale Machine Learning, including training sample production and storage, feature engineering , distributed training, distributed services, OnlineLearning, resource scheduling, FL , heterogeneous hardware and computer architecture, ensuring high reliability of deploying Machine Learning infrastructure, supporting and accelerating research and iteration of related Machine Learning algorithms; CMS operation mode: support byte monetization/recommendation/LIVE/electricity supplier/search and Volcano Engine business; Closed loop of business form: a complete saas form, covering the entire life cycle of algorithm engineers. From writing the first line of code to the business launch, the complete coverage is in the engine.

The team has always aimed to build an industry SOTA -oriented recommendation/advertising/e-commerce/LIVE/search ranking Machine Learning system, and currently has a large number of industry-leading implementations in training/scheduling/inference. Team atmosphere is open/free.

Responsibilities
- Resource management: Service Capacity and Budget / Server Arrival Tracking, Putaway and Quota Allocation / Resource Transfer Efficiency Workflow build-up / Cluster utilization governance.
- Service delivery: Adaptation between different cloud service vendors and self-built datacenters / localization docking of Global/US/European service environments / construction of new region and datacenter / Continuous Delivery of module updates.
- Quality direction: Undertake the monitoring of System / continuous Construction of operation and maintenance capabilities and tools / Formulation and Management of SLA / Improving the efficiency of Oncall / Dealing with accidents and Tracking the implementation of improvement measures.
- Research, design, and develop computer and network software or specialised utility programs.
- Analyse user needs and develop software solutions, applying principles and techniques of computer science, engineering, and mathematical analysis.
- Update software, enhances existing software capabilities, and develops and direct software testing and validation procedures.
- Work with computer hardware engineers to integrate hardware and software systems and develop specifications and performance requirements.- Bachelor’s degree in Computer Science or equivalent with 3+ years of relevant experience
- Experience in analyzing, troubleshooting, deploying, maintaining or improving distributed/large-scale systems.
- Understand basic Python syntax; ability to read functional code.
- Experience using open source components, at least two of the following:
1. Sql Storage (e.g. RDS, Mysql, Oracle, TiDB/TiKV)
2. Nosql (e.g. Redis, Memcache, Mongo, leveldDB, RocksDB)
3. Big Data Frameworks (e.g. HDFS, Hadoop, Yarn, Flink, Kafka, Spark, Storm, K8s)
4. Distributed Coordination Service (e.g. ETCD, Zookeeper)
5. Application Performance Management(e.g. Grafana, ClickHouse, Hive, Falcon, Zabbix, Prometheus)
6. Cloud Native Tech(e.g. Kubernetes/K8S, Docker)
7. and some others, like ZeroMQ/RabbitMQ, GraphDB, Celery, MapReduce, Jenkins, etc
- Ability to thrive in a fast-paced environment.
- Strong understanding of code optimizing and routine tasks automation.
- Good communication and teamwork skills to clearly communicate technical concepts with other teammates.

Preferred Qualifications
- Ability to program simple logic in Python/HTML/Javascript.
- Experience in resource management and task scheduling for large scale distributed systems.
- Proficiency in at least one machine learning framework: Hardware-Software Co-Design, High Performance Computing, ML Hardware Acceleration (e.g. GPU/TPU/RDMA) or ML Framework (e.g. TensorFlow/PyTorch)

TikTok is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At TikTok, our mission is to inspire creativity and bring joy. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.