Databricks Engineer
Location: Brazil
Department: Data Engineer / BI
Workplace: remote
Description
We’re looking for a hands-on Databricks Engineer to help design, build, and scale a modern data platform running on Apache Spark and Delta Lake. This role sits at the intersection of data engineering, platform architecture, and performance optimization. You’ll work closely with data scientists, analysts, and backend teams to ensure reliable, high-performance data pipelines and well-governed datasets.
Responsibilities
- Design and implement end-to-end data pipelines using Databricks (Jobs, Workflows, Delta Live Tables)
- Build and maintain scalable ETL/ELT processes leveraging Apache Spark (PySpark / Scala)
- Develop data models using Delta Lake, including schema design, partitioning strategies, Z-ordering, and optimization techniques
- Manage and optimize Databricks clusters (autoscaling, spot instances, instance pools, cluster policies)
- Implement CI/CD pipelines for Databricks deployments (e.g., using Databricks Repos, Terraform, Azure DevOps / GitHub Actions)
- Work with structured and semi-structured data (JSON, Parquet, Avro) at scale
- Ensure data quality and reliability through validation frameworks, unit/integration testing, and monitoring
- Implement data governance practices (Unity Catalog, access controls, lineage tracking, auditing)
- Troubleshoot performance issues (job failures, skew, shuffle bottlenecks, memory pressure) and optimize Spark workloads
- Integrate Databricks with cloud-native services (AWS S3, Azure Data Lake Storage, GCP BigQuery)
- Collaborate with data consumers to define SLAs, data contracts, and service interfaces
Requirements
- Strong experience with Databricks (production workloads, not just notebooks)
- Deep understanding of Apache Spark internals (execution plan, Catalyst optimizer, Tungsten engine)
- Proficiency in PySpark (preferred) or Scala
- Solid knowledge of Delta Lake (ACID transactions, time travel, compaction, OPTIMIZE, VACUUM)
- Experience with distributed data processing and large-scale datasets (TB+ scale)
- Familiarity with orchestration tools (Databricks Workflows, Airflow, or similar)
- Experience with version control and CI/CD pipelines
- Knowledge of cloud platforms (AWS / Azure / GCP), including IAM and storage services
- Strong SQL skills and understanding of data warehousing concepts
- Experience with data modeling techniques (star schema, medallion architecture)
Nice to Have
- Experience with streaming pipelines (Structured Streaming, Auto Loader)
- Knowledge of ML workflows on Databricks (MLflow, feature stores)
- Infrastructure-as-Code experience (Terraform, ARM, CloudFormation)
- Exposure to Unity Catalog and data governance frameworks
- Experience with cost optimization strategies in Databricks environments
- Familiarity with DBT or similar transformation tools
There are more than 50,000 engineering jobs:
Subscribe to membership and unlock all jobs
Engineering Jobs
60,000+ jobs from 4,500+ well-funded companies
Updated Daily
New jobs are added every day as companies post them
Refined Search
Use filters like skill, location, etc to narrow results
Become a member
🥳🥳🥳 452 happy customers and counting...
Overall, over 80% of customers chose to renew their subscriptions after the initial sign-up.
To try it out
For active job seekers
For those who are passive looking
Cancel anytime
Frequently Asked Questions
- We prioritize job seekers as our customers, unlike bigger job sites, by charging a small fee to provide them with curated access to the best companies and up-to-date jobs. This focus allows us to deliver a more personalized and effective job search experience.
- We've got over 200,000 jobs from 15,000+ vetted companies. No fake or sleazy jobs here!
- We aggregate jobs from 15,000+ companies' career pages, so you can be sure that you're getting the most up-to-date and relevant jobs.
- We're the only job board *for* software engineers, *by* software engineers… in case you needed a reminder! We add thousands of new jobs daily and offer powerful search filters just for you. 🛠️
- Every single hour! We add 2,000-3,000 new jobs daily, so you'll always have fresh opportunities. 🚀
- Typically, job searches take 3-6 months. EchoJobs helps you spend more time applying and less time hunting. 🎯
- Check daily! We're always updating with new jobs. Set up job alerts for even quicker access. 📅
What Fellow Engineers Say