Senior Databricks Engineer (Platform & Data Engineering)

Location: Danville, Pennsylvania, United States

Employment Type: Full time

Key Responsibilities

• Databricks Administration: Manage workspaces, clusters, jobs, security configurations, libraries, and overall platform health.
• Data Engineering: Build and optimize data pipelines and transformations using PySpark and T‑SQL.
• Automation & Development: Create automation and tooling using Python, Bash, and PowerShell.
•  Governance & Security: Implement and automate data governance, access controls, auditability, and compliance best practices.
• Note: A significant portion of our development focuses on governance and security automation.
• Solution Architecture: Translate conversational ideas into clear solution architectures; break work into tasks in Planner or Azure DevOps for team execution.
• Effort Estimation: Estimate level of effort and forecast delivery dates with reasonable accuracy.
• Documentation: Produce high-quality documentation including workflows, gap analyses, design specs, build docs, test plans, and transition documentation.
• Testing Oversight: Oversee unit, scenario-based, integrated, capacity, and parallel testing.
• On-Call Support: Participate in a 24×7×365 on-call rotation to support Databricks, SQL Server, and other relevant applications supported by the team.
• Upgrades & Monitoring: Support system upgrades, patches, performance tuning, observability, and proactive monitoring.
• Data Integrity: Maintain accountability for data integrity across applications and pipelines.
• Change Management: Complete system build and documentation in adherence to departmental guidelines and change management policy.
• Peer Review: Review teammates’ work and provide constructive feedback.
• Communication & Escalation: Communicate effectively with peers, stakeholders, and leadership; provide regular status updates and escalate issues appropriately.
• Facilitation & Leadership: Drive discussions, mentor accountable leaders, and coordinate resources across initiatives.
• Training & Mentorship: Provide feedback on training materials; conduct training sessions; develop and mentor other team member.
• Policies & Vendor Engagement: Help develop department guidelines, standards, procedures, and policies; facilitate vendor reviews and selection processes.
• Integrations (as applicable): Demonstrate understanding of integration and technical requirements for peripherals/devices that interface with assigned applications.
• Support Coordination: Coordinate remote and/or on-site support as needed for end-users.

Optional Contributions

• Create complex AI/BI dashboards.
• Contribute to CI/CD pipelines and integrations using Azure DevOps or GitHub.
• Apply Terraform for infrastructure as code.
• Provide MS SQL Server administrative support.

Work is typically performed in an on-campus or home office environment. Accountable for satisfying all job specific obligations and complying with all organization policies and procedures. The specific statements in this profile are not intended to be all-inclusive. They represent typical elements considered necessary to successfully perform the job.

*Relevant experience may be a combination of related work experience and degree obtained (Associate’s Degree = 2 years; Bachelor’s Degree = 4 years).