P-932

The last 10 years was the decade of the cloud.  The next 10 years will be the decade of Data and AI.  This market will be over $3T by 2030.  In a market this big, the winner(s) will win on the strength of a rich product portfolio combined with strong platform play.

Databricks is uniquely positioned in this market as both a powerful platform built on open source and open standards, and boasting of a rich product portfolio spanning verticals such as SQL, model lifecycle, data engineering, analytics, and observability for data and AI.  Databricks has always been a visionary in this space, embracing AI before it was cool.  Our platform story is the result of years of investment in a general purpose platform that is driving the hockey stick growth in our user community and customers.

The Compute Fabric team in Databricks powers this rocketship.  Compute Fabric is organized into serverful and serverless.  Serverful Compute is Databricks’ original Compute offering, still powering most of Databricks revenue at EOY 2024.  It’s the most sophisticated, powerful, and scalable platform for orchestrating data-centric compute on the planet.

What We Do

The charter of the Serverful Compute team, aka Clusters Compute Team, or Clusters Team, is to create the most reliable, cost-competitive, and easy to use compute platform for the modern data cloud, which “just works”.  Towards that goal, we are scaling our platform for the next order of magnitude of growth, while raising the bar significantly on reliability, TCO, capabilities, and performance.  We work on complex problems at the forefront of distributed systems, cloud computing, ML infrastructure, and compute.

In serverful, the end user is typically doing two things: (1) using a Databricks product (e.g. jobs, notebooks), and (2) managing compute (clusters) for the product.  In serverless, compute is completely transparent to the user and is automatically managed through the user-facing product (e.g. SQL).  Although this distinction is orthogonal to hosting model, for historic and business reasons we offer serverful in customer account (customer-hosted) and serverless in Databricks account (self-hosted).  Serverful has thus gotten coupled with the customer-hosted offering.

Serverful Compute Team in BLR will fully own the following areas from the above table by end of FY26:

• Compute Plane (aka Data Plane).
• Stability

The impact you will have:

• Hire great engineers to build an outstanding team
• Support engineers in their career development by providing clear feedback and developing engineering leaders
• Ensure high technical standards by instituting processes (architecture reviews, testing) and culture (engineering excellence)
• Work with engineering and product leadership to build a long-term roadmap
• Coordinate execution and collaborate across teams to unblock cross-cutting projects

What we look for:

• Great at hiring and developing talent, especially leadership
• Great at creating efficient processes that increase velocity and quality
• Managed teams that build and operate high-volume distributed systems in a SaaS environment
• 12+ years of extensive experience with large-scale distributed systems and building services on top of containers
• 7+ years of proven managerial experience in managing and mentoring engineering teams
• Have experience scaling engineering teams from 5 to 30+
• Team player that will work with other departments (PM, Sales, CS)
• BS or higher in Computer Science, or a related field