Staff Software Development Engineer - Data Platforms

Team: Data Platform

Location: Bengaluru

Commitment: Full Time

Workplace Type: onsite

Key Responsibilities:

• Core Platform Development
• Architecture & Design: Build POCs, create architecture designs, and evaluate trade-offs for new services and migrations
• Real-time Data Pipeline: Build and maintain highly scalable, accurate, clean and real-time events collection systems
• Data Foundation for AI & MCP: Specifically design and build data pipelines to support the development and training of AI models and autonomous AI Agents.
• This includes creating robust data infrastructure for our MCP, ensuring data quality and availability for personalized learning paths and skill validation.
• Data Quality Assurance: Continuously work towards detecting, alerting, and fixing quality issues to build trust in our data
• System Enhancement: Improve existing systems and services using latest tooling and engineering standards
• Cost Optimization: Improve unit economics across ingestion, storage, and processing components


• Technical Leadership & Ownership
• Complete Development Lifecycle: Own the entire development process from ideation to deployment
• Cross-functional Collaboration: Work closely with different teams to achieve common goals through efficient coordination
• Technical Documentation: Maintain up-to-date documentation with architecture diagrams for entire projects
• Service Ownership: Take complete ownership of services within your workstream and drive continuous improvement
• Operational Excellence: Establish and maintain robust monitoring, alerting, and operational procedures for mission-critical data systems.


• Team Leadership & Growth
• Knowledge Sharing: Write blogs and articles to help team members and the community
• Architecture Breakdown: Break down high-level architecture into manageable services and tasks
• Technology Trends: Learn and share new trends in technology, engineering, and productivity
• Team Productivity: Actively work towards improving team productivity and knowledge
• Mentorship: Guide and mentor junior team members


• Business Impact
• Data Governance: Govern data assets across the entire company and audit data posture
• Time to Insight: Reduce time to insight for data team stakeholders
• Business Decision Support: Assist and challenge business decisions using data insights
• Stakeholder Service: Make data available to all stakeholders in an efficient, quick, quarriable, and intuitive manner

Skills and attributes for success:

• Experience: 8+ years of progressive professional experience in data engineering, with a proven track record of designing, building, and operating large-scale, high-throughput data platforms, ideally in a consumer-facing or streaming industry.
• Extreme Scale Experience: Demonstrated experience working with data systems that handle hundreds of millions of events/requests per minute, billions of daily events, and petabytes of storage (e.g., S3, HDFS).
• Programming: Strong proficiency in at least one major programming language such as Python, Java, Scala, or Go, with a focus on building robust, production-grade data applications.
• Distributed Data Processing: Deep expertise and practical experience with distributed data processing frameworks like Apache Spark, Apache Flink, or equivalent technologies for batch and streaming workloads.
• AI/ML Data Experience: Specific experience in building data pipelines for Machine Learning model training, inference, and feature stores. Familiarity with data needs for AI Agents or similar intelligent systems.
• Messaging & Streaming: Extensive experience with high-throughput messaging queues and stream processing platforms like Apache Kafka, Kinesis, or equivalent.
• Cloud Platforms: Hands-on expertise with at least one major cloud platform (AWS, GCP, or Azure) and their data-related services, with a strong preference for AWS (S3, EMR, Kinesis, Glue, Redshift, etc.).
• Data Warehousing/Lakes: Strong understanding and hands-on experience with modern data warehousing concepts and data lake architectures (e.g., Snowflake, BigQuery, Redshift, Delta Lake, Iceberg, Hudi).
• ETL/Orchestration: Expertise in building, optimizing, and orchestrating complex ETL/ELT pipelines using tools like Apache Airflow, Prefect, or other workflow management systems.
• Databases: Strong SQL skills and experience with both relational (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., Cassandra, DynamoDB).
• Data Modeling: Advanced data modeling skills (dimensional, Kimball, data vault) for large, complex datasets.
• Problem-Solving: Exceptional analytical and problem-solving abilities, with a proven track record of debugging and optimizing complex distributed systems.
• Communication: Excellent communication, collaboration, and interpersonal skills, with the ability to articulate complex technical concepts to diverse audiences.
• Mentorship: Proven ability to mentor, guide, and technically lead senior and junior engineers.