Data Quality Engineer, Enterprise Data Platform - Data Quality & Data governance) , AWS , SQL & Python , Cloud, Power BI

Location: Bengaluru, KA, in

Company Description

At Western Digital, our vision is to power global innovation and push the boundaries of technology to make what you thought was once impossible, possible.

At our core, Western Digital is a company of problem solvers. People achieve extraordinary things given the right technology. For decades, we’ve been doing just that. Our technology helped people put a man on the moon.

We are a key partner to some of the largest and highest growth organizations in the world. From energizing the most competitive gaming platforms, to enabling systems to make cities safer and cars smarter and more connected, to powering the data centers behind many of the world’s biggest companies and public cloud, Western Digital is fueling a brighter, smarter future.

Binge-watch any shows, use social media or shop online lately? You’ll find Western Digital supporting the storage infrastructure behind many of these platforms. And, that flash memory card that captures and preserves your most precious moments? That’s us, too.

We offer an expansive portfolio of technologies, storage devices and platforms for business and consumers alike. Our data-centric solutions are comprised of the Western Digital®, G-Technology™, SanDisk® and WD® brands.

Today’s exceptional challenges require your unique skills. It’s You & Western Digital. Together, we’re the next BIG thing in data.

Job Description

About the Role 

We are seeking a skilled and forward-thinking Data Quality Engineer to advance the data trust, governance, and certification framework for our enterprise Data Lakehouse platform built on Databricks, Apache Iceberg, AWS (Glue, Glue Catalog, SageMaker Studio), Dremio, Atlan, and Power BI. 

This role is critical in ensuring that data across Bronze (raw), Silver (curated), and Gold (business-ready) layers is certified, discoverable, and AI/BI-ready. You will design data quality pipelines, semantic layers, and governance workflows, enabling both Power BI dashboards and Conversational Analytics leveraging LLMs (Large Language Models). 

Your work will ensure that all 9 dimensions of data quality (accuracy, completeness, consistency, timeliness, validity, uniqueness, integrity, conformity, reliability) are continuously met, so both humans and AI systems can trust and use the data effectively. 

ESSENTIAL DUTIES AND RESPONSIBILITIES 

Data Quality & Reliability 

• Build and maintain automated validation frameworks across Bronze → Silver → Gold pipelines. 

• Develop tests for schema drift, anomalies, reconciliation, timeliness, and referential integrity. 

• Integrate validation into Databricks (Delta Lake, Delta Live Tables, Unity Catalog) and Iceberg-based pipelines. 

Data Certification & Governance 

• Define data certification workflows ensuring only trusted data is promoted for BI/AI consumption. 

• Leverage Atlan and AWS Glue Catalog for metadata management, lineage, glossary, and access control. 

• Utilize Iceberg’s schema evolution & time travel to ensure reproducibility and auditability. 

Semantic Layer & Business Consumption 

• Build a governed semantic layer on gold data to support BI and AI-driven consumption. 

• Enable Power BI dashboards and self-service reporting with certified KPIs and metrics. 

• Partner with data stewards to align semantic models with business glossaries in Atlan. 

Conversational Analytics & LLM Enablement 

• Prepare and certify datasets that fuel conversational analytics experiences. 

• Collaborate with AI/ML teams to integrate LLM-based query interfaces (e.g., natural language to SQL) with Dremio, Databricks SQL, and Power BI. 

• Ensure LLM responses are grounded on high-quality, certified datasets, reducing hallucinations and maintaining trust. 

ML Readiness & SageMaker Studio 

• Provide certified, feature-ready datasets for ML training and inference in SageMaker Studio. 

• Collaborate with ML engineers to ensure input data meets all 9 quality dimensions. 

• Establish monitoring for data drift and model reliability. 

Holistic Data Quality Dimensions 

• Continuously enforce all 9 dimensions of data quality: 

• Accuracy, Completeness, Consistency, Timeliness, Validity, Uniqueness, Integrity, Conformity, Reliability. 

Qualifications

Required 

• Advance experience in data engineering, data quality, or data governance roles. 

• Strong skills in Python, PySpark, and SQL. 

• Hands-on with Databricks (Delta Lake, Unity Catalog, Delta Live Tables) and Apache Iceberg. 

• Expertise in AWS data stack (S3, Glue ETL, Glue Catalog, Athena, EMR, Redshift, SageMaker Studio). 

• Experience with Power BI semantic modeling, DAX, and dataset certification. 

• Familiarity with Dremio or query engines (Trino, Presto). 

• Knowledge of Atlan or equivalent catalog/governance tools. 

• Experience with data quality testing frameworks (Great Expectations, Deequ, Soda). 

Preferred 

• Exposure to Conversational Analytics platforms or LLM-powered BI (e.g., natural language query over Lakehouse/Power BI). 

• Experience integrating LLM pipelines (LangChain, OpenAI, AWS Bedrock, etc.) with enterprise data. 

• Familiarity with data observability tools (Monte Carlo, Bigeye, DataDogs, Grafana). 

• Knowledge of data compliance frameworks (GDPR, CCPA, HIPAA). 

• Cloud certifications: AWS Data Analytics Specialty, Databricks Certified Data Engineer. 

Additional Information

Western Digital is committed to providing equal opportunities to all applicants and employees and will not discriminate based on their race, color, ancestry, religion (including religious dress and grooming standards), sex (including pregnancy, childbirth or related medical conditions, breastfeeding or related medical conditions), gender (including a person’s gender identity, gender expression, and gender-related appearance and behavior, whether or not stereotypically associated with the person’s assigned sex at birth), age, national origin, sexual orientation, medical condition, marital status (including domestic partnership status), physical disability, mental disability, medical condition, genetic information, protected medical and family care leave, Civil Air Patrol status, military and veteran status, or other legally protected characteristics. We also prohibit harassment of any individual on any of the characteristics listed above. Our non-discrimination policy applies to all aspects of employment. We comply with the laws and regulations set forth in the Equal Employment Opportunity is the Law poster.

Western Digital thrives on the power and potential of diversity. As a global company, we believe the most effective way to embrace the diversity of our customers and communities is to mirror it from within. We believe the fusion of various perspectives results in the best outcomes for our employees, our company, our customers, and the world around us. We are committed to an inclusive environment where every individual can thrive through a sense of belonging, respect and contribution.

Western Digital is committed to offering opportunities to applicants with disabilities and ensuring all candidates can successfully navigate our careers website and our hiring process. Please contact us at [email protected] to advise us of your accommodation request. In your email, please include a description of the specific accommodation you are requesting as well as the job title and requisition number of the position for which you are applying.

#LI-RR1

Based on our experience, we anticipate that the application deadline will be 10/25/2024, although we reserve the right to close the application process sooner if we hire an applicant for this position before the application deadline. If we are not able to hire someone from this role before the application deadline, we will update this posting with a new anticipated application deadline.

WD thrives on the power and potential of diversity. As a global company, we believe the most effective way to embrace the diversity of our customers and communities is to mirror it from within. We believe the fusion of various perspectives results in the best outcomes for our employees, our company, our customers, and the world around us. We are committed to an inclusive environment where every individual can thrive through a sense of belonging, respect and contribution.

WD is committed to offering opportunities to applicants with disabilities and ensuring all candidates can successfully navigate our careers website and our hiring process. Please contact us at [email protected] to advise us of your accommodation request. In your email, please include a description of the specific accommodation you are requesting as well as the job title and requisition number of the position for which you are applying.

Notice To Candidates: Please be aware that WD and its subsidiaries will never request payment as a condition for applying for a position or receiving an offer of employment. Should you encounter any such requests, please report it immediately to WD Ethics Helpline or email [email protected].