Responsibilities:

- Build ingestion pipelines for Databricks using a medallion architecture (bronze, silver, and gold layers), leveraging Auto Loader and Delta Lake.
- Implement "before-and-after" reconciliation pipelines, including record counts, field-by-field comparisons, key/table mapping between versions, and large-scale validation rule application.
- Model the gold layer for reporting, covering approval/rejection rates, reasons for outcomes, and views segmented by cluster and object.
- Expose results via an API for a web application (using Databricks SQL Warehouse with the Statement Execution API, or a custom-built service) and document the API contract.
- Configure governance within Unity Catalog—including permissions, data lineage, and auditing—which are critical for client production data.
- Orchestrate processes using Workflows/Jobs, incorporate data quality checks (such as DLT expectations or similar), and manage version control via Databricks Asset Bundles.

Requirements:

- Experience in data engineering, with at least 2 years specifically using Databricks (PySpark, Delta Lake, Unity Catalog, Workflows).
- Advanced proficiency in SQL and Python.
- Experience with data reconciliation or validation within migration projects.
- Knowledge of SAP data structures.
- Experience in developing reports based on system data.

- Advanced English for conversation.