Description
Data Engineering – Data Lead
Location: Mumbai, India
Department: Software Development
Experience: 6-8Yrs
Role: Data Engineering – Data Lead
Experience: 6-10Years
Notice Period: Immediate - 15 days
Work Location: Mumbai(Lower Parel)
Role Summary
The Data Lead – Data Engineering will be responsible for designing, building, and operating the organization’s enterprise data platform on AWS. The role will oversee real-time and batch ingestion of structured and unstructured data from internal systems and external data providers and ensure reliable data availability for analytics and business intelligence on Databricks.
This role requires a hands-on technical leader who actively participates in architecture design, pipeline development, and platform optimization, while guiding the engineering team. The candidate should be comfortable working in a fast-paced, agile development environment with rapid iteration and continuous delivery of data products.
Key Responsibilities
1. Enterprise Data Platform Architecture
∙ Design and implement the AWS-based enterprise data lake architecture.
∙ Build scalable frameworks to handle structured, semi-structured, and unstructured datasets.
∙ Define standards for data ingestion, transformation, storage, and access.
∙ Ensure seamless integration with the Databricks analytics platform.
2. Data Ingestion & Integration
Design and develop real-time and batch data ingestion pipelines for sources such as:
Internal systems
∙ Trading & order management systems
∙ Portfolio management platforms
∙ Client onboarding / KYC systems
∙ CRM platforms
∙ ERP / accounting systems
External sources
∙ Market data vendors
∙ Research and news feeds
∙ Documents and reports
∙ Audio or surveillance data
Technologies
AWS AppFlow, AWS Lambda, AWS Glue, Amazon S3, Amazon Athena
3. Real-Time Data Processing
∙ Develop event-driven data pipelines to support near-real-time data ingestion.
∙ Enable real-time use cases such as: trading analytics, operational monitoring, compliance and surveillance analytics.
4. Security & Governance
Ensure platform security and compliance through: AWS Key Management Service (KMS) for encryption, AWS Secrets Manager for credential management, AWS Security Hub for security monitoring, AWS Config for configuration governance, AWS CloudTrail for audit trails
5. Monitoring & Observability
∙ Implement monitoring frameworks using: AWS CloudWatch, Grafana dashboards
∙ Monitor: pipeline performance, infrastructure health, data freshness, ingestion failures.
6. DevOps & Platform Automation
∙ Implement CI/CD pipelines using GitLab.
∙ Automate deployment and testing of data pipelines.
∙ Establish standards for version control, code quality, and automated deployments.
7. Data Quality & Metadata
∙ Implement frameworks for data validation, reconciliation, and monitoring.
∙ Manage metadata and data lineage using AWS Glue Data Catalog.
8. Integration with Databricks
∙ Deliver curated and optimized datasets for analytics on Databricks.
∙ Collaborate with analytics teams to enable BI, advanced analytics, and ML workloads.
9. Hands-on Technical Leadership
∙ Actively participate in pipeline development, architecture design, and technical problem solving.
∙ Provide technical guidance and code reviews to the data engineering team.
∙ Drive adoption of engineering best practices and reusable data frameworks.
10. Agile Delivery & Collaboration
∙ Operate in an Agile / iterative development environment.
∙ Work closely with analytics teams, business stakeholders, and platform engineers.
∙ Deliver incremental data products and platform capabilities with rapid turnaround.
Key Technology Stack
- Cloud Platform: AWS
- Data Platform: Amazon S3, AWS Glue, Amazon Athena
- Data Ingestion: AWS AppFlow, AWS Lambda
- Security & Governance: AWS KMS, AWS Secrets Manager, AWS Security Hub, AWS Config, AWS CloudTrail
- Monitoring: AWS CloudWatch, Grafana
- DevOps: GitLab
- Analytics Platform: Databricks
- Programming: Python, PySpark, SQL
Experience
∙ 8–10 years in data engineering / platform engineering
∙ Proven experience building cloud-native data platforms
∙ Strong hands-on experience developing data pipelines
∙ Experience working with real-time and batch data processing
∙ Experience handling structured and unstructured data
∙ Experience in wealth management / capital markets / financial services preferred
There are more than 50,000 engineering jobs:
Subscribe to membership and unlock all jobs
Engineering Jobs
60,000+ jobs from 4,500+ well-funded companies
Updated Daily
New jobs are added every day as companies post them
Refined Search
Use filters like skill, location, etc to narrow results
Become a member
🥳🥳🥳 452 happy customers and counting...
Overall, over 80% of customers chose to renew their subscriptions after the initial sign-up.
To try it out
For active job seekers
For those who are passive looking
Cancel anytime
Frequently Asked Questions
- We prioritize job seekers as our customers, unlike bigger job sites, by charging a small fee to provide them with curated access to the best companies and up-to-date jobs. This focus allows us to deliver a more personalized and effective job search experience.
- We've got over 200,000 jobs from 15,000+ vetted companies. No fake or sleazy jobs here!
- We aggregate jobs from 15,000+ companies' career pages, so you can be sure that you're getting the most up-to-date and relevant jobs.
- We're the only job board *for* software engineers, *by* software engineers… in case you needed a reminder! We add thousands of new jobs daily and offer powerful search filters just for you. 🛠️
- Every single hour! We add 2,000-3,000 new jobs daily, so you'll always have fresh opportunities. 🚀
- Typically, job searches take 3-6 months. EchoJobs helps you spend more time applying and less time hunting. 🎯
- Check daily! We're always updating with new jobs. Set up job alerts for even quicker access. 📅
What Fellow Engineers Say