Databricks Tech Lead / Architect

Location: Raritan, NJ

Databricks Tech Lead / Architect

Role OverviewWe are looking for an experienced Databricks Tech Lead / Architect with 10–15 years ofexperience in data engineering, data transformation, ETL/ELT, and modern cloud dataplatforms. The candidate will be responsible for designing and leading scalable datasolutions using Databricks, Apache Spark, Delta Lake, SQL, Python and cloud dataservices.Key Responsibilities Design and architect end-to-end data engineering and data transformation solutionsusing Databricks. Lead the development of scalable ETL/ELT pipelines for batch and near-real-timedata processing. Develop robust data transformation frameworks using Databricks, Apache Spark,PySpark and SQL. Design and implement Delta Lake / Lakehouse architectures, including dataingestion, transformation, optimization and consumption layers. Define architecture patterns for bronze, silver and gold data layers and implementappropriate data modelling strategies. Design scalable ingestion frameworks for structured and unstructured data fromdatabases, APIs, files, cloud storage and enterprise applications. Develop reusable frameworks for data cleansing, transformation, enrichment,validation and reconciliation. Optimize Spark workloads, Databricks clusters, SQL queries and data pipelines forperformance and cost efficiency. Implement incremental processing, CDC, SCD Type 1/2, partitioning, caching anddata optimization strategies. Define and implement data quality, data validation, exception handling andmonitoring mechanisms. Establish engineering standards around coding, CI/CD, version control, testing,deployment and operational support. Work with cloud platforms such as AWS/Azure/GCP and integrate Databricks withcloud-native data services. Lead technical design discussions, architecture reviews and proof-of-concepts. Mentor and guide data engineers and provide technical leadership across projects. Collaborate with enterprise/data architects to ensure alignment with broader dataplatform and integration architecture.Required Technical Skills Databricks & Spark- Strong hands-on experience with Databricks and Apache Spark.- Excellent knowledge of PySpark and Spark SQL.- Strong understanding of Delta Lake and Lakehouse architecture.- Experience with Databricks Workflows, Jobs, notebooks, clusters andperformance optimization. Data Engineering / ETL / ELT- Strong understanding of ETL and ELT architectures and design patterns.

- Experience designing complex data transformation pipelines.- Strong SQL skills and experience with large-scale data processing.- Experience with:- Incremental data processing- CDC- SCD Type 1 / Type 2- Data reconciliation- Data quality and validation- Error handling and restartability- Metadata-driven pipelines- Batch and near-real-time processing Data Architecture- Strong understanding of Data Lake, Data Warehouse and Lakehousearchitectures.- Experience with dimensional modelling, star/snowflake schemas and datamarts.- Understanding of data partitioning, file formats and storage optimization.- Experience working with Parquet, JSON, CSV, Avro and other data formats. Experience with one or more major cloud platforms:- AWS: S3, Glue, Lambda, EMR, Redshift, IAM- Azure: ADLS, Data Factory, Synapse, Event Hub- GCP: GCS, BigQuery, Dataflow, Pub/Sub Experience integrating Databricks with enterprise applications, databases, APIs andcloud storage platforms. DevOps / Engineering Practices- Git-based development and branching strategies.- CI/CD using tools such as Azure DevOps, GitHub Actions, Jenkins or GitLab.- Experience with automated testing and deployment of Databricksnotebooks/jobs.- Infrastructure-as-Code exposure using Terraform is desirable.- Strong understanding of SDLC, Agile/Scrum and production supportprocesses.