Senior Data Engineer

Location: Bangalore - Manyata Tech Park Road

Time Type: Full time

Job Description

Organization: At CommBank, we never lose sight of the role we play in other people’s financial wellbeing. Our focus is to help people and businesses move forward to progress. To make the right financial decisions and achieve their dreams, targets, and aspirations. Regardless of where you work within our organization, your initiative, talent, ideas, and energy all contribute to the impact that we can make with our work. Together we can achieve great things.

Job Title: Senior Data Engineer
Location: Bangalore - Manyata Tech Park

Business & Team: Home Buying

Impact & contribution:

• You will play a pivotal role in driving business outcomes through effective data management and analysis.

• Enhancing Decision Making with Data

• Ensuring Data Security and Compliance

• Future Trends in Data Engineering

Roles & Responsibilities:

• Design and develop highly reliable and scalable data pipelines and data platforms with comprehensive test coverage.

• Collaborate with stakeholders to analyze and translate requirements to technical implementation.

• Identify and drive opportunities for continuous improvement within the team and in delivery of products.

• Provide mentoring and technical assistance to other members of the team, including more junior Data Engineers.

• Produce high quality, sustainable solutions to meet business requirements, leveraging approved delivery frameworks and by applying industry best practice.

• Provide technical governance of product delivery to ensure successful delivery and adoption. Deliver data engineering solutions aligned to core concepts of data design, preparation, transformation, and load.

• Build and implement data pipelines in distributed data platforms including warehouses, databases, data lakes and cloud lake houses to enable data predictions and models, and reporting and visualisation analysis via data integration tools and frameworks.

• Contribute to domain planning, providing guidance to ensure that technical deliveries are aligned to engineering direction and strategy.

Essential Skills:

• 8+ years of experience in Data Engineering within large-scale, data-intensive environments.

• Hadoop, Spark, Scala/Python, Hive, SQL, AWS(Cloud), HDFS.

• Proficiency in building large/complex data pipelines and hands on GenAI models.

• Strong expertise in Data Warehousing concepts and implementation, with hands-on experience in Hadoop technologies (Spark, Scala) and migration initiatives to AWS Cloud.

• Proficiency in AWS services including EC2, S3, Lambda, Athena, Kinesis, Redshift, Glue, EMR, DynamoDB, and CloudWatch.

• Advanced SQL skills with proven experience in writing efficient queries, database management, performance tuning, and handling complex data transformations.

• Strong experience in designing and managing data orchestration workflows using Apache Airflow.

• Solid understanding of streaming data processing and event-driven architectures.

• Experience in CI/CD pipeline development, automation, and deployment of production-grade code.

• Deep understanding of Data Modelling, Data Warehousing, ETL/ELT processes, and Slowly Changing Dimensions (SCD) concepts.

• Hands-on experience with GitHub or other version control platforms.

• Proven expertise in developing and optimizing data pipelines across diverse data formats and sources.

• Strong working knowledge of Linux/Unix environments for development, testing, and troubleshooting.

• Relevant AWS certifications such as AWS Certified Data Engineer, AWS Certified Data Analytics – Specialty, AWS Certified Developer, or AWS Certified Solutions Architect are highly desirable.

• Hands-on experience with data modelling tools such as Erwin, with the ability to design and develop data models, data warehouses, data assets, and data products.

• Practical experience with AI tools, technologies, and automation solutions, with demonstrated success in driving business value and efficiency improvements through AI-enabled capabilities.

• Snowflake knowledge and implementation experience will be considered an added advantage.

Education Qualifications:

• Bachelor’s degree in engineering in Computer Science/Information Technology