Data Engineer (Azure/Databricks) | SR

Team: IT

Location: Brazil

Commitment: Full-time

Workplace Type: remote

Accountabilities:

• Build and maintain scalable, reliable data pipelines using modern distributed processing technologies, ensuring efficient data ingestion, transformation, and delivery.
• Design and implement data models using Databricks, SAP Datasphere, Azure Data Factory, Azure Data Lake Storage (ADLS), Python, and relational databases.
• Extract, transform, and organize data to create and maintain Data Lakes, Data Lakehouses, and analytical tables.
• Structure and manage data tables within Databricks, ensuring performance, consistency, and accessibility.
• Develop ETL pipelines for both structured and unstructured data, using APIs and specialized tools for file analysis and data validation.
• Integrate data from APIs, including CXL sources, and transform and process the resulting datasets within Databricks.
• Structure RAG (Retrieval-Augmented Generation) solutions using contracts and other business documents as data sources.
• Apply data engineering best practices to ensure data quality, scalability, reliability, and performance.
• Work with multiple data sources and technologies to support analytical, operational, and AI-driven use cases.
• Contribute to the evolution of cloud-based data architecture and help identify opportunities to improve data processing and integration workflows.

Requirements:

• Advanced professional experience and strong technical knowledge of Databricks.
• Solid experience with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS).
• Proven experience implementing Data Lake and Data Lakehouse architectures.
• Strong programming skills in Python and experience with Apache Spark and distributed data processing.
• Professional experience with Apache Airflow for data pipeline orchestration.
• Experience working with SQL and NoSQL databases, including technologies such as PostgreSQL, MongoDB, and Cassandra.
• Strong understanding of data modeling, ETL/ELT processes, data integration, and large-scale data processing.
• Ability to work with both structured and unstructured datasets and transform complex sources into reliable analytical assets.
• Analytical mindset and strong problem-solving skills, with attention to data quality and technical details.
• Ability to work independently on complex data engineering challenges while collaborating effectively with technical and business stakeholders.
• Desirable: Knowledge of Kafka and event-driven data architectures.
• Desirable: Experience with dbt and modern data transformation practices.
• Desirable: Knowledge of AWS Glue and BigQuery.
• Desirable: Experience with AI, RAG, or document-based data solutions.
• Desirable: Familiarity with SAP Datasphere.

Benefits:

• Opportunity to work with modern Azure and Databricks data technologies.
• Exposure to advanced AI, Generative AI, and RAG use cases.
• Opportunity to work with cloud-native Data Lake and Data Lakehouse architectures.
• Professional development in a technology-focused and innovation-driven environment.
• Access to modern AI-powered tools and technologies.
• Opportunities to work with distributed and borderless teams.
• Exposure to emerging technologies and current industry trends.
• Career development opportunities and continuous learning.
• Opportunity to contribute to projects focused on digital transformation and AI-driven innovation.