Databricks SME (R-00117)

Team: Security Engineering Team

Location: 100% Remote

Commitment: Full-Time

Workplace Type: remote

Position Responsibilities

• Performance & Optimization: Optimize data pipelines, configure clusters, and implement best practices for cost and performance.
• Governance & Security: Implement security protocols, such as Unity Catalog, IAM, and data access controls.
• Technical Leadership: Act as a trusted advisor to engineering teams, mentor staff, and define technical roadmaps.
• Collaboration: Work with data engineers and stakeholders to align technical solutions with business requirements. 
• Technical Expertise: Strong proficiency in Databricks, Apache Spark, Python/PySpark, and SQL.
• Cloud Proficiency: Experience with cloud platforms, specifically AWS (S3, Glue) or Azure.
• Data Modeling: Expertise in data modeling, data profiling, and ETL/ELT pipeline design
• Architecture & Development: Design and implement scalable data pipelines, data models, and Lakehouse architectures.
• Implement CI/CD pipelines, use tools like Azure DevOps or Git for version control, and automate routine tasks.
• Integrate data from various cloud sources and services (AWS, Azure, GCP) such as Blob Storage, Data Lake Storage, and other databases.
• Optimize Spark jobs & monitor system metrics to quickly troubleshoot issues related to the Databricks environment.

Position Qualifications

• Requires 3-5 years of experience in data engineering, data warehousing, or data architecture roles. 
• US citizenship and ability to obtain and maintain a federal security clearance may be required 
• Bachelors degree is highly preferred