Azure Data Engineer (Azure Databricks & Pyspark)

12 month contract Azure Data Engineer (Azure Databricks & PySpark)

Job Description

We’re hiring a talented Azure Data Engineer enthusiast to work in our platform to help

ensure that our data quality is flawless. As a company, we have millions of new data

points every day that come into our system. You will be working with a passionate team

of engineers to solve challenging problems and ensure that we can deliver the best data

to our customers, on-time. You will be using the latest cloud data lake technology to

build robust and reliable data pipelines.

Job Responsibilities

● Develop expertise in the different upstream data stores and systems across the

company.

● Design, develop and maintain data integration pipelines for the organization growing

data sets and product offerings.

● Build unit testing and QA plans for data processes.

● Build data validation testing frameworks to ensure high data quality and integrity.

● Write and maintain documentation on data processes.

● Developing and maintaining data models and schemas.

● Strong analytical experience with database in writing complex queries, query

optimization, debugging, user defined functions, views, indexes etc.

● Write code that adheres to coding standards, procedures, and techniques. Maintain

the integrity of existing program logic according to specifications.

● Actively participate in the code review process to ensure development work adheres

to standards and specifications (including peer review and code review external to

team).

● Respond to all inquiries and issues in a timely manner as developed code/program

moves through the testing process.

● Participate in scrum, sprints, and backlog grooming meetings.

● Evaluate interrelationships between applications to determine whether a change in

one part of a project would impact or cause undesirable results in related applications

and design for effective interfaces between interrelated applications.

● Improve the health of system assets by identifying enhancements to improve

performance through tuning and monitoring, reliability, and resource consumption.

● Evaluate and troubleshoot root-cause analysis for production issues and

system failures; determine corrective action(s) and propose improvements to

prevent their recurrence.

● Maintain up-to-date business domain knowledge and technical skills in software

development technologies and methodologies.

● Provide input in the selection, implementation and use of development tools and best

practices.

Technical:

● BS or MS in Computer Science or equivalent experience.

● 4 + years of experience in Databricks/Apache Spark with Azure data storage

solutions handling large datasets.

● Expert in SQL and Spark SQL, including advanced analytical queries.

● Proficiency in Python (data structures, algorithms, object-oriented programming,

using API’s) and familiarity with PySpark.

● Experience working with Databricks Delta tables, unity catalog, Dataframe API, read

and write from/to various data sources and data formats.

● Experience with both Batch and streaming data pipeline.

● Knowledge of Azure Data Factory, Azure Data Lake, Azure SQL DW, Azure SQL is a

plus.

Nice to haves

● Understanding of PostgreSQL and MS SQL

● Experience working in a fast-paced environment.

● Experience in an Agile software development environment.

● Ability to work with large datasets and perform data analysis

● Worked on migration project to build Unified data platform.

● Experience working with JIRA, Azure DevOps CI/CD pipelines.