Data Engineer III

Team: Product Development

Location: Chennai, Hyderabad

Commitment: Full-time

Workplace Type: onsite

Job Responsibilities:

• Utilize architectural and design skills for a variety of data-oriented tasks.
• Establish and maintain strong relationships with business stakeholders.
• Build highly reliable & scalable data systems with good data governance that provide complete, secure, accurate and timely data for analytics. 
• Expertise in data & dimensional modeling with good business acumen in supporting new product features and services - including financial reporting. 
• Assist development and analytics organization to meet its delivery goals with regards to data design, flow and infrastructure. 
• Perform Data Modeling, Big data architecture design, support and implementation of OLTP, real time OLAP/data warehouse systems and ETL pipelines. 
• Track/Build & monitor data/system KPI’s like storage, performance & usage 
• Hands on administration of big data databases/compute engines (Redshift, EMR, Spark or similar) and optimization/tuning techniques 
• Prepare required documentation as outlined by departmental standards. 
• Identify, analyze and back track data problems as needed. 
• Provide recommendations for application and system improvements. 
• Mentor other team members 
• Work with business users to understand business requirements, issues and business and/or client processes. 
• Develop, test, and maintain high-performance of our data systems to meet the requirements of the business and/clients while adhering to departmental standards. 
• Perform quality assurance testing for all work performed. 
• Meet with agile teams as required to define and document application requirements. 
• Follow project development, deployment process & maintain industry standards and best coding practices. 
• Maintain security and organization of the company’s data. 
• Provide off-hour support as assigned. 
• Gather business requirements/features, estimate the tasks, design & develop scalable solutions. 
• Work with manager to ascertain the company’s data requirements. 
• Plan work to meet project deadlines, accommodate demands by development teams, set priorities and escalate issues appropriately. 
•  Provide recommendations to development teams for data restructuring and performing complex maintenance.
• Deploy and manage changes in development, staging and production.
• Assist development and dev ops teams on SQL queries and tuning. 

Required Skills & Experience:

• At least 7 years’ experience in data engineering, Data modeling, SQL, ETL, Big database, Data warehousing & basic-level administration of data infrastructure
• At least 7 years coding experience in Java, Python, R or other equivalent programming language
• At least 4 years big data experience 
• At least 2 years’ experience as technical lead architecting data lake/data warehouse/lake house systems & mentoring team of Data Engineers / DBAs 
• Proficient in a variety of big-data tools and technologies, including Hadoop, Spark, etc., handling different storage formats like Parquet, Delta/Apache Hudi/Apache Iceberg, & managing/working with different OLTP systems & data ingestion mechanisms. 
• Experience building large scale data lakes & warehouse systems leveraging cloud environments including AWS technologies –S3, Redshift, Aurora/RDS, DMS, EC2, Glue spark, Cloud watch, & EMR 
• Experience with data orchestration tools like Airflow 
• Experience in building pipelines, ingesting data from heterogeneous systems onto data lake environments in real time 
• Proficient in Python, SQL and PL-SQL, query tuning, optimization, ETL, ELT, and Data Warehousing 
• Proficient in Business Intelligence (BI), analytic database and products, able to take business requirements and translate into database tables, pipeline design and tasks. 
• General data-specific concepts, frameworks and processes 
• Agile development practices 
• Working within an SDLC 
• Designing data warehouses including definition, structure, documentation, maintenance, long-range requirements, operational guidelines, and protection 
• Data governance, meta data management, data security measures, retention & archival across data lake layers 

Other Skills and Experience

• Cloud experience, such as AWS, GCP or Azure
• Building/Managing data lake & lake house architectures. 
• Experience working on different data lake storage formats. 
• Excellent oral and written communication 
• Multi-tasking and managing multiple priorities. 
• Working well in a team-oriented, collaborative environment with people from different disciplines and varying degrees of technical experience 
• Working in an Agile team environment. 

Behavioral Competencies:

Cultivates Innovation
Decision Quality
Manages Complexity
Drives Results
Business Insight