Data Engineer

Responsibilities:

• A Data Engineer should be able to build, operationalize and monitor data processing systems
• Create robust and automated pipelines to ingest and process structured and unstructured data from various source systems into analytical platforms using batch and streaming mechanisms leveraging cloud native toolset
• Implement custom applications using tools such as EventHub’s, ADF and other cloud native tools as required to address streaming use cases
• Engineers and maintain ELT processes for loading data lake (Cloud Storage, data lake gen2)
• Leverages the right tools for the right job to deliver testable, maintainable, and modern data solutions
• Respond to customer/team inquiries and escalations and assist in troubleshooting and resolving challenges
• Works with other scrum team members to estimate and deliver work inside of a sprint
• Research data questions, identifies root causes, and interacts closely with business users and technical resources
• Should possess ownership and leadership skills to collaborate effectively with Level 1 teams.
• Must have experience in raising tickets with Microsoft and engaging with them to address any service or tool outages in production.

Qualifications:

• 3+ years of professional technical experience
• 3+ years of hands-on Data Architecture and Data Modelling – SME level
• 3+ years of experience building highly scalable data solutions using Azure data factory, Spark, Databricks, Python
• 3+ years of experience working in cloud environments (AWS and/or Azure)
• 2+ years of programming languages such as Python, Spark and Spark SQL.
• Should have strong knowledge on architecture of ADF and Databricks.
• Able to work with Level1 teams to resolve platform outages in production environments.
• Strong client-facing communication and facilitation skills
• Strong sense of urgency, ability to set priorities and perform the job with little guidance
• Excellent written and verbal interpersonal skills and the ability to build and maintain collaborative and positive working relationships at all levels
• Strong interpersonal and communication skills (Written and oral) required
• Should be able to work in shifts
• Should have knowledge on azure Dev Ops process.

Key Skills:

• Azure Data Factory, Azure Data bricks, Python, ETL/ELT, Spark, Data Lake, Data Engineering, EventHub’s, Azure delta, Spark streaming