Data Engineer

- - - - - - - - - - - -

Design and impact analysis

·        Contribute to define feasibility (technical options...), functional design and solution validation.

·        Performs impact analysis related to data captation (system performance in production, security, etc.)

·        Identifies the source of physical data

·        Enforces security and confidentiality rules for data on its perimeter

Implementation and Deployment

·        Completes and executes data collection procedures

·        Builds data collection infrastructure (IoT: Internet of Things, sensors, database, files, etc.)

·        Deploys data collection infrastructure and procedures

·        Measures the impact on the technical chain implemented (from the connected object to the storage system)

·        Completes the necessary tests to validate the solution

·        Documents the implemented solutions

Support and troubleshooting

·        Analyzes and understands the origin of a complex malfunction, incident or bug.

·        Adopts a proactive approach to avoid or identify root causes of problems.

·        Provides technical assistance to users.

---------------------------------------------------------------------------------------------

 Experience Level : 9-12 years

Description :
Expertise in Data Engineering Domain. Strong technical skills, coupled with excellent communication and strategic thinking
Strong experience in designing solutions and implementing medallion Datalake architecture.
Strong handson experience working on Pyspark, Databricks
Good Planning and execution capability
Ability to guide team through technical roadblocks
Experience working on integration of AI model with Pyspark and Databricks will be a huge plus
Must have ability to optimize the Data pipelines written in PySpark, Databricks
Must have experience working on Azure cloud platform , Azure Data Factory, Azure Data bricks
Well versed with ETL and ELT patterns used for Data Integration
Know how of integrating the data from different sources both structured and unstructured.
Strong experience in cloud architecture
In Depth knowhow about different databases{Postgre, Synapse, snowflake , Azure Storage } .
Collaborate with data scientists, engineers, and IT teams to ensure smooth model operations.      

Technical :
    Languages : Python, SQL, Pyspark
    Azure Cloud Platform
    Database : Synapse, Snowflake, Azure Data Storage, NoSql
    Integration : Kafka
    Cloud Computing: Knowledge of cloud platforms Azure and its AI services is increasingly important.

Excellent communication and collaboration skills
Ability to drive initiatives for complex technical concepts to both technical and non-technical audiences, and collaborate effectively with data scientists, software engineers, and business stakeholders.
Problem-solving and critical thinking
Strategic thinking:  ability to understand business goals and translate them into a data and AI strategy is essential.
Leadership: lead or co-lead projects on Data/AI, requiring the ability to guide and motivate teams.
Stay up-to-date on the latest trends and technologies in data engineering and cloud computing.