Senior Data Engineer - (Real-Time Streaming)

Location: Chennai, Tamil Nadu, India

Workplace: on_site

Description

We are looking for an experienced Senior Data Engineer with strong expertise in Real-Time Streaming, Java, Apache Kafka, Apache Flink, and PySpark to build and operate high-performance, scalable streaming data platforms supporting enterprise Business Platforms and Risk & Compliance initiatives.

The ideal candidate will have extensive experience designing and implementing real-time data pipelines, event-driven architectures, and distributed data processing systems capable of handling high-volume, low-latency data workloads. This role requires strong engineering expertise in modern data platforms, stream processing, and Big Data technologies while collaborating closely with architects, platform teams, and business stakeholders.

Requirements

Key Responsibilities

• Design, develop, and maintain scalable real-time data pipelines using Apache Kafka, Apache Flink, Java, and PySpark.
• Build high-throughput, low-latency streaming applications supporting enterprise-scale business and regulatory workloads.
• Develop event-driven data processing solutions using Kafka and modern streaming technologies.
• Design and implement distributed data processing pipelines for real-time analytics and operational reporting.
• Collaborate with Data Architects, Platform Engineering teams, Analytics teams, and Business stakeholders to understand requirements and deliver scalable data solutions.
• Develop reusable, secure, and maintainable data engineering components following enterprise engineering standards.
• Optimize streaming jobs for performance, scalability, fault tolerance, and reliability.
• Build and maintain batch and streaming ETL/ELT pipelines using PySpark.
• Monitor, troubleshoot, and resolve production issues within streaming data platforms.
• Participate in architecture discussions and contribute to the design of modern enterprise data platforms.
• Ensure data quality, governance, monitoring, logging, and operational excellence across streaming pipelines.
• Participate in Agile/Scrum ceremonies including sprint planning, backlog refinement, stand-ups, and retrospectives.
• Create technical documentation, implementation guides, and operational runbooks.

Required Technical Skills

Programming

• Strong hands-on experience with Java (Mandatory)
• Strong proficiency in Python
• Excellent understanding of Object-Oriented Programming (OOP) principles

Real-Time Streaming

• Apache Kafka
• Apache Flink
• Kafka Streams
• Event-Driven Architecture
• Stream Processing
• Real-Time Data Engineering

Big Data

• PySpark
• Apache Spark
• Distributed Data Processing
• Data Pipeline Development

Databases

• SQL
• Relational Databases
• NoSQL databases (preferred)

Data Engineering

• ETL/ELT Development
• Data Ingestion
• Data Transformation
• Data Integration
• Pipeline Orchestration
• Data Quality
• Performance Optimization

Development & DevOps

• Git
• CI/CD pipelines
• Linux environment
• API Integration
• Monitoring & Logging tools

Required Competencies

• Strong experience designing scalable distributed data systems.
• Excellent understanding of real-time streaming architectures.
• Ability to build secure, resilient, and fault-tolerant data pipelines.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management skills.
• Ability to work effectively in Agile delivery environments.
• Strong ownership mindset with focus on delivery quality and operational excellence.
• Ability to collaborate across cross-functional engineering and business teams.

Preferred Experience

• Experience working on enterprise-scale data platforms.
• Exposure to Risk & Compliance or Business Platform data ecosystems.
• Experience supporting mission-critical, high-volume production environments.
• Knowledge of cloud-based data engineering platforms is an advantage.
• Experience with modern data architecture and distributed systems.

Education

Bachelor's or Master's Degree in Computer Science, Information Technology, Data Engineering, or a related field (or equivalent industry experience).

Boolean Keywords (for better profile quality on job portals)

Mandatory Skills:

• Java
• Apache Kafka
• Apache Flink
• PySpark
• Real-Time Streaming
• Data Engineering
• Event-Driven Architecture