Senior Data Engineer - (Flink/Kafka/PySpark)

Location: Bengaluru, Karnataka, India

Workplace: on_site

Description

We are looking for an experienced Data Engineer with strong expertise in real-time data streaming and distributed data processing technologies. The ideal candidate should have hands-on experience in building scalable, event-driven data platforms using Apache Flink, Kafka, Java/Scala, and PySpark. The candidate will be responsible for designing, developing, and maintaining high-performance data engineering solutions within large-scale enterprise environments.

Requirements

Key Responsibilities:

• Design, develop, and maintain real-time streaming data pipelines and event-driven architectures.
• Build scalable and fault-tolerant data engineering solutions using Apache Flink, Kafka, and Java/Scala.
• Develop and optimize distributed data processing applications for enterprise-grade systems.
• Implement best practices for performance tuning, resiliency, security, and compliance requirements.
• Collaborate with cross-functional teams to design technical and application architectures.
• Ensure production-grade monitoring, troubleshooting, and operational excellence of distributed services.
• Contribute to CI/CD implementation and DevOps best practices.
• Work closely with engineering teams to drive technical improvements and architectural decisions.

Mandatory Technical Skills:

• Minimum 4+ years of development and design experience in:
• Apache Flink
• Java or Scala
• Apache Kafka
• PySpark
• Real-time data streaming technologies
• Event-driven architectures
• Hands-on experience with:
• Apache Flink (Beam or Spark Streaming experience is also valuable)
• Kafka ecosystem and streaming platforms
• JVM tuning and performance optimization
• Distributed systems design and implementation
• Docker and Kubernetes containerization
• Linux OS administration and Shell scripting
• SQL and NoSQL databases
• CI/CD tools such as GitHub and Jenkins
• Design patterns and their implementation
• Production monitoring and troubleshooting of distributed services

Nice to Have Skills:

• Redis or other caching technologies.
• Experience with Spark Streaming or Apache Beam.
• Experience working with banking or fintech platforms.

Data Engineering & Security Requirements:

• Strong understanding of data security principles and controls.
• Experience implementing secure data transfer mechanisms including:
• CRON jobs
• ETL processes
• JDBC and ODBC scripts
• Knowledge of encryption, anonymization, data integrity, and policy controls in large-scale infrastructures.
• Understanding of data sensitivity requirements, including:
• Protection of PII data
• Secure logging practices
• Secure in-memory data handling
• Ability to identify security design gaps and recommend enhancements.
• Experience implementing wrapper solutions for legacy or third-party components to ensure compliance requirements are met.

Infrastructure & Networking Knowledge:

• Working knowledge of:
• DNS
• Proxy servers
• ACLs
• Networking policies
• Troubleshooting network-related issues

Functional Requirements:

• Experience working in Agile environments.
• Ability to design and implement scalable technical architectures.
• Conduct technology research and benchmarking activities.
• Ability to influence engineering teams on technical best practices.
• Experience working in enterprise-scale environments.
• Banking, Financial Services, or FinTech domain experience is preferred.

Soft Skills:

• Excellent communication and interpersonal skills.
• Strong problem-solving and analytical abilities.
• Self-driven and capable of working independently.
• Ability to collaborate effectively with cross-functional teams and stakeholders.
• Strong presentation and listening skills.
• Positive attitude and ability to foster a collaborative team environment.
• Passionate about engineering excellence and continuous improvement.