Senior Systems Software Engineer, Spark Service - Accelerated Spark

We are seeking expert Senior System Software Engineers adept at Apache Spark to join our team. Data scientists spend a considerable amount of time exploring data and iterating over machine learning (ML) experiments. Every hour of compute required to sort through datasets, extract features and fit ML algorithms impedes an efficient business workflow. NVIDIA believes that data science workflows can benefit tremendously from being accelerated, to enable data scientists to explore many more and larger datasets to drive towards their business goals, faster and more efficiently.

At NVIDIA, we are passionate about working on hard problems that have an impact. You will need to have previous experience working with Apache Spark applications, implementing big data applications for a variety of customers, programming skills, and familiarity with open source distributed processing frameworks. You should be comfortable working with interdisciplinary teams. You will work with an engineering team accelerating Apache Spark with GPUs using CUDA and open source libraries. This is an opportunity to work with a team that is developing the Spark RAPIDS open source library to accelerate Spark applications. This is a strategic investment for NVIDIA! The code is being adopted by multiple cloud service providers and Apache Spark distributions.

What you'll be doing:

• Design and develop a world-class GPU accelerated Apache Spark service.

• Enable a collection of micro-services to provide the ability to run Spark applications on Kubernetes or other platforms.

• Implement the REST API and its client libraries to simplify customer adoptions.

• Customize the open-source projects to meet the project requirements.

• Deploy and verify the solution on CSPs or on-prem Kubernetes environments.

• Engage open source communities, including Apache Spark and RAPIDS, for technical discussions and contributions

What we need to see:

• 8+ years of experience in software development.

• 5+ years hands on experience with web service design and development

• BS/MS/PhD in computer science or a related field or equivalent experience

• Experience with REST service frameworks like Spring Boot

• Familiarity with the modern data open source ecosystem (Apache Spark, Apache Kyuubi, Apache Zookeeper, gRPC, etc)

• Experience with Kubernetes and Helm charts and Experience building performance and reliable service APIs

• Experience working on public and private cloud platforms and Experience with object storage and distributed file systems

• Prior experience supporting enterprise customers

• Solid understanding of Python and Scala/Java

• Excellence at communicating, presenting and explaining technical topics

Ways to stand out from the crowd:

• Working experience with Spark distributions: Databricks, AMS EMR, GCP Dataproc, Azure Synapse Analytics.

• Contributions to major open source projects such as Apache Spark, Apache Kyuubi, Apache Ranger, Apache Iceberg, and Delta Lake.

• Development experience of Apache Spark Data Sources and connectors.

• Development experience with Spark Client interfaces/tools like Jupyter, Zeppelin, Spark Connect, and BI tools.

• Working knowledge of secrets and encryption management systems and Basic ML/DL experience with PyTorch, TensorFlow, Spark ML and XGBoost.

NVIDIA is widely considered to be one of the technology world’s most desirable employers. We have some of the most forward-thinking and hardworking people on the planet working for us. If you're creative, passionate and self-motivated, we want to hear from you! NVIDIA is leading the way in groundbreaking developments in Artificial Intelligence, High-Performance Computing and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services. 

The base salary range is 220,000 USD - 339,250 USD. Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.