Data Engineer, Advanced

Remote Work: Yes

A Data Engineer will be responsible for understanding the client's technical requirements, design and
build data pipelines to support the requirements. In this role, the Data Engineer, besides developing the
solution, will also oversee other Engineers' development. This role requires strong verbal and written
communication skills and effectively communicate with the client and internal team. A strong
understanding of databases, SQL, cloud technologies, and modern data integration and orchestration
tools like Azure Data Factory (ADF), Informatica, and Airflow are required to succeed in this role.

• Play a critical role in the design and implementation of data platforms for the AI products
• Develop productized and parameterized data pipelines that feed AI products leveraging GPUs and
CPUs.
• Develop efficient data transformation code in spark (in Python and Scala) and Dask.
• Build workflows to automate data pipeline using python and Argo.
• Develop data validation tests to assess the quality of the input data.
• Conduct performance testing and profiling of the code using a variety of tools and techniques.
• Guide Data Engineers in delivery teams to follow the best practices in deploying the data pipeline
workflows.
• Lead design sessions to develop ETL logic that meets business and product requirements.
• Implement data pipelines that meet the design and are efficient, scalable, and maintainable.
• Provide technical guidance and meet best practices during the data pipeline development.
• Work with clients to understand and document their data, technology, and how it would integrate
with Antuit's cloud solutions.
• Act as a technical escalation lead for customer and development team issues.
• Build data pipeline frameworks to automate high-volume and real-time data delivery for our data
hub.
• Operationalize scalable data pipelines to support data science and advanced analytics.
• Optimize customer data science workloads and manage cloud services costs/utilization.
• Developing sustainable data driven solutions with current new generation data technologies to drive
our business and technology strategies.

Required Qualifications:

• Bachelors,Master's or Ph.D. Degree in Computer Science or Engineering.
  7-10 years experience in Data engineering field
• Strong understanding of data models that feed advanced AI-driven applications
• Fluent level of English (written and verbal) as well as local language as applicable

• Preferred Qualifications:
  At least 3 years of experience with Azure Data Factory (ADF), Informatica and similar, is
  critical.
  At least 1 year of Spark experience (preferably PySpark)
  Experience with relational databases, SQL, and Python is preferred.
• Experience working with Retail, Consumer Goods, and Manufacturing data models is a
  plus.