Data Engineer IV - IN
Location: IND-Work from Home-Remote
Time Type: Full time
Job Description
Key Responsibilities
Experience - 9+yrs
- Data Platforms & Pipeline Engineering
- Lead end-to-end development of scalable data pipelines and orchestration frameworks using Azure Data Factory (ADF), Azure Synapse Analytics, Azure Databricks, and Microsoft Fabric.
- Build robust real-time and batch data pipelines, including integration with streaming sources (e.g., Event Hubs, Kafka) and structured streaming engines.
- Design and implement Structured Streaming applications in Spark for near-real-time processing of streaming data.
- Develop and maintain ETL/ELT pipelines and transformations leveraging Spark, PySpark, SQL, and fabric orchestration capabilities.
- Architect and implement data solutions using Microsoft Fabric, including OneLake, Dataflows, warehouses, and Fabric capacity planning to support enterprise analytics.
- Collaborate on data governance, cataloging, and asset organization using Unity Catalog within Databricks and Fabric environments.
- Manage Microsoft Fabric capacity and resource utilization to optimize performance and cost efficiency for analytics workloads.
- Design, deploy, and optimize Databricks dashboards and reporting artifacts for business stakeholders.
- Apply best practices for data modelling, caching, file sizing, and performance tuning of Spark and Delta Lake jobs (e.g., Z-ORDER, broadcast joins, adaptive query execution).
- Oversee governance, access controls, metadata management, and lineage using Unity Catalog.
- Lead and mentor a team of data engineers, fostering best practices in development, operations, documentation, and quality.
- Work with cross-functional teams (architecture, BI, data science, DevOps) to translate business requirements into scalable data solutions.
- Partner with stakeholders to define data strategy, standards, and architectural roadmaps.
- Establish and enforce standards for data quality, testing, monitoring, operational observability, and governance.
- Implement secure, compliant data access and lineage frameworks across cloud data platforms.
- Implement CI/CD pipelines, infrastructure-as-code for data platform artifacts, and automated testing frameworks for data jobs and workflows.
Required Skills & Qualifications
- 10+ years of hands-on experience in data engineering on Azure with deep expertise in ADF, Synapse, Databricks, and Microsoft Fabric.
- Proven experience with real-time data processing, streaming architectures, and Spark Structured Streaming.
- Strong proficiency in Azure Data Factory, Spark (PySpark), SQL, Azure Synapse Analytics, Databricks Runtime, and cloud storage.
- Solid knowledge of Unity Catalog for data governance, security, and access management.
- Experience designing and managing Databricks Dashboards, performance optimization, cost controls, and data platform resource tuning.
- Expertise in building scalable, fault-tolerant, and high-throughput batch & streaming data solutions.
- Excellent leadership, cross-team collaboration, and communication skills.
About Rackspace Technology
We are the multicloud solutions experts. We combine our expertise with the world’s leading technologies — across applications, data and security — to deliver end-to-end solutions. We have a proven record of advising customers based on their business challenges, designing solutions that scale, building and managing those solutions, and optimizing returns into the future. Named a best place to work, year after year according to Fortune, Forbes and Glassdoor, we attract and develop world-class talent. Join us on our mission to embrace technology, empower customers and deliver the future.
More on Rackspace Technology
Though we’re all different, Rackers thrive through our connection to a central goal: to be a valued member of a winning team on an inspiring mission. We bring our whole selves to work every day. And we embrace the notion that unique perspectives fuel innovation and enable us to best serve our customers and communities around the globe. We welcome you to apply today and want you to know that we are committed to offering equal employment opportunity without regard to age, color, disability, gender reassignment or identity or expression, genetic information, marital or civil partner status, pregnancy or maternity status, military or veteran status, nationality, ethnic or national origin, race, religion or belief, sexual orientation, or any legally protected characteristic. If you have a disability or special need that requires accommodation, please let us know.
There are more than 50,000 engineering jobs:
Subscribe to membership and unlock all jobs
Engineering Jobs
60,000+ jobs from 4,500+ well-funded companies
Updated Daily
New jobs are added every day as companies post them
Refined Search
Use filters like skill, location, etc to narrow results
Become a member
🥳🥳🥳 452 happy customers and counting...
Overall, over 80% of customers chose to renew their subscriptions after the initial sign-up.
To try it out
For active job seekers
For those who are passive looking
Cancel anytime
Frequently Asked Questions
- We prioritize job seekers as our customers, unlike bigger job sites, by charging a small fee to provide them with curated access to the best companies and up-to-date jobs. This focus allows us to deliver a more personalized and effective job search experience.
- We've got over 200,000 jobs from 15,000+ vetted companies. No fake or sleazy jobs here!
- We aggregate jobs from 15,000+ companies' career pages, so you can be sure that you're getting the most up-to-date and relevant jobs.
- We're the only job board *for* software engineers, *by* software engineers… in case you needed a reminder! We add thousands of new jobs daily and offer powerful search filters just for you. 🛠️
- Every single hour! We add 2,000-3,000 new jobs daily, so you'll always have fresh opportunities. 🚀
- Typically, job searches take 3-6 months. EchoJobs helps you spend more time applying and less time hunting. 🎯
- Check daily! We're always updating with new jobs. Set up job alerts for even quicker access. 📅
What Fellow Engineers Say