Senior Data Engineer
Location: Pune, Maharashtra, India
Department: Engineering
About Us
HG Insights is the global leader in technology intelligence, delivering actionable AI driven insights through advanced data science and scalable big data solutions. Our platform informs go-to-market decisions, and we influence how businesses spend millions of marketing and sales budgets.
What You’ll Do:
- Design, build, and optimize large-scale distributed data pipelines for processing billions of events from and to multiple sources of data.
- Architect and scale enterprise-grade big-data systems, including data lakes, ETL/ELT workflows, and syndication platforms for customer-facing products.
- Orchestrate pipelines and workflows with Airflow.
- Write optimized data into analytical databases such as ClickHouse, DuckDB, and Redshift
- Ensure data quality, consistency, and reliability across the pipeline.
- Monitor pipeline performance and troubleshoot data issues.
- Collaborate with product teams to develop features across databases, and backend services.
- Implement cutting-edge solutions for data ingestion, transformation, and analytics.
- Drive system reliability through automation, CI/CD pipelines (Docker, Kubernetes, Terraform), and infrastructure-as-code practices.
What You’ll Be Responsible For
- Development of the Data side of our Platform, ensuring scalability, performance, and cost-efficiency across distributed systems.
- Collaborating in agile workflows (daily stand-ups, sprint planning) to deliver features rapidly while maintaining system stability.
- Ensuring security and compliance across data workflows, including access controls, encryption, and governance policies.
What You’ll Need
- BS/MS/Ph.D. in Computer Science or related field, with 7+ years of experience building production-grade big data systems.
- Strong SQL skills and solid data modeling fundamentals with hands-on experience building ETL / ELT pipelines.
- Proficiency in Python for data processing and integrations.
- Familiarity with dbt, Airflow, Databricks and modern analytics engineering practices
Nice-to-Haves
- Knowledge of data governance frameworks and compliance standards (GDPR, CCPA).
- Contributions to open-source big data projects or published technical blogs/papers.
- DevOps proficiency in monitoring tools (Prometheus, Grafana) and serverless architectures.
There are more than 50,000 engineering jobs:
Subscribe to membership and unlock all jobs
Engineering Jobs
60,000+ jobs from 4,500+ well-funded companies
Updated Daily
New jobs are added every day as companies post them
Refined Search
Use filters like skill, location, etc to narrow results
Become a member
🥳🥳🥳 452 happy customers and counting...
Overall, over 80% of customers chose to renew their subscriptions after the initial sign-up.
To try it out
For active job seekers
For those who are passive looking
Cancel anytime
Frequently Asked Questions
- We prioritize job seekers as our customers, unlike bigger job sites, by charging a small fee to provide them with curated access to the best companies and up-to-date jobs. This focus allows us to deliver a more personalized and effective job search experience.
- We've got over 200,000 jobs from 15,000+ vetted companies. No fake or sleazy jobs here!
- We aggregate jobs from 15,000+ companies' career pages, so you can be sure that you're getting the most up-to-date and relevant jobs.
- We're the only job board *for* software engineers, *by* software engineers… in case you needed a reminder! We add thousands of new jobs daily and offer powerful search filters just for you. 🛠️
- Every single hour! We add 2,000-3,000 new jobs daily, so you'll always have fresh opportunities. 🚀
- Typically, job searches take 3-6 months. EchoJobs helps you spend more time applying and less time hunting. 🎯
- Check daily! We're always updating with new jobs. Set up job alerts for even quicker access. 📅
What Fellow Engineers Say