Lead Data Engineer

Location: Bengaluru, India

Department: Data & AI

Experience: 6-8 Years

• Real-Time Data Engineering
• Design, develop, and maintain real-time data pipelines using Apache Flink and Apache Kafka.
• Develop production-grade streaming applications for high-volume and low-latency workloads.
• Implement data transformation, filtering, enrichment, aggregation, and event processing.
• Build reliable event-processing pipelines with appropriate error handling and recovery mechanisms.
• Consume and publish events across Kafka topics. • Implement appropriate partitioning, consumer groups, offsets, and delivery mechanisms.
• Troubleshoot streaming pipeline failures and performance issues.

Apache Flink

• Develop and maintain Apache Flink jobs forreal-time data processing.
• Implement:
  • Stream transformations
  • Filtering
  • Mapping
  • Aggregations
  • Joins
  • Windows
  • Event-time processing
  • Watermarks
  • State management
• Implement Flink checkpointing and recovery mechanisms.
• Optimize Flink jobs for performance, scalability, and resource utilization.
• Monitor Flink jobs for latency, throughput, failures, and resource consumption.
• Troubleshoot state, checkpointing, backpressure, and processing issues.

Kafka

• Develop Kafka-based ingestion and streaming pipelines.
• Create and manage Kafka topics and event streams.
• Workwith partitions, offsets,consumer groups, replication, and retention.
• Implement reliableproducer and consumer applications.
• Handle message ordering, retries,duplicate events, and replay scenarios.
• Monitor Kafka performance and troubleshoot consumerlag and throughput issues.
• Work with Kafka schemasand serialization formats.

CDC & Debezium

• BuildCDC-based ingestion pipelines using Debezium.
• Configure and maintain Debeziumconnectors.
• Capture source-system inserts, updates,and deletes.
• Publish CDC events into Kafka.
• Handle initial snapshots and incremental CDC processing.
• Manage schemaevolution and changesin source systems.
• Implement data reconciliation and consistency checks.
• Troubleshoot CDC failures and source-to-target data issues.

Data Orchestration

• Develop and maintain data workflows using Apache Airflow orequivalent orchestration frameworks.
• Buildreusable DAGs for:
  • Data ingestion
  • CDC workflows
  • Data validation
  • Flink job execution
  • Data transformation
  • ClickHouse loading
  • Downstream integrations
• Implement workflowdependencies, scheduling, retries,backfills, SLAs, and alerting.
• Integrate Airflow workflows with Kafka, Flink, Debezium, ClickHouse, APIs, and cloud services.
• Monitor workflow execution and troubleshoot failures.
• Develop reusable operators, sensors,and workflow components where appropriate.
• Useevent-driven triggers where real-time workflows require them.

ClickHouse & Analytical Data

• Integrate streaming data pipelines with ClickHouse.
• Design efficient analytical data models.
• Develop and optimize SQL queries.
• Implement appropriate partitioning, sorting, indexing,and retention strategies.
• Optimize data ingestion and query performance.
• Support downstream analytical use cases,dashboards, and reporting requirements.

Data Quality& Reliability

• Implement data validation and quality checksthroughout the pipeline.
• Buildreconciliation mechanisms betweensource and target systems.
• Monitor data freshness, completeness, accuracy, and consistency.
• Implement error handling, retry,replay, and recoverymechanisms.
• Establish loggingand observability for critical pipelines.
• Support incident investigation and root-cause analysis.

Integration & APIs

• Integrate streaming and analytical data with APIs, endpoints, dashboards, and downstream applications.
• Develop data interfaces and integration components.
• Workwith application teams to define data contractsand integration requirements.
• Support future integrations and additional data consumers.

Engineering Practices

• Follow modernsoftware engineering practices around:
  • Git
  • Code reviews
  • Unit testing
  • Integration testing
  • CI/CD
  • Logging
  • Monitoring
  • Documentation
• Develop reusable and maintainable data engineering components.
• Participate in technical designdiscussions and architecture reviews.

• Mentor other Data Engineers and contribute to engineering standards.

Required Skills& Experience

• 6–8 years ofexperience in Data Engineering.
• Strong hands-onexperience with Apache Flink — mandatory/core requirement.
• Strong hands-on experience with Apache Kafka.
• Hands-on experience with Debeziumand CDC.
• Strong programming experience in Java or Scala.
• Python experience is an advantage.
• Strong SQL skills.
• Experience with analytical databases; ClickHouse is highly preferred.
• Hands-on experience with Apache Airflowor another data orchestration framework.
• Strong understanding of distributed systemsand real-time data processing.
• Experience developing and supporting production-grade streaming pipelines.
• Strong understanding of Kafka:
  • Topics
  • Partitions
  • Offsets
  • Consumer groups
  • Replication
  • Retention
• Experience with data transformation, enrichment, filtering, aggregation, and event processing.
• Experience troubleshooting performance and reliability issues.
• Familiarity with cloud platforms and containerized environments.

Preferred Skills

• Advanced experience with Apache Flink, including:
  • State management
  • Checkpoints
  • Savepoints
  • Watermarks
  • Event time
  • Windows
  • Backpressure
  • State backends
• Experience with Apache Airflow, Dagster,Prefect, or Apache NiFi.
• Experience with Kafka Schema Registry.
• Experience with Avro, Protobuf, or JSON.
• Experience with Kubernetes.
• Experience with AWS / Azure / GCP.
• Experience with CI/CD pipelines.
• Experience with Docker and containerized applications.
• Experience with Terraform or other Infrastructure as Code tools.
• Experience with data observability and monitoring tools.
• Experience buildinghigh-volume, low-latency real-time data platforms.
• Experience with REST APIs and systemintegrations.