Data Engineer for Multimodal Data Prep, Data Curation for Modeling

- 3+ years of data engineering experience
- Experience with data modeling, warehousing and building ETL pipelines
- Knowledge of batch and streaming data architectures like Kafka, Kinesis, Flink, Storm, Beam
- Knowledge of distributed systems as it pertains to data storage and computing
- Experience with SQL
- Experience with AWS technologies like Redshift, S3, AWS Glue, EMR, Kinesis, FireHose, Lambda, and IAM roles and permissions
- Experience with non-relational databases / data stores (object storage, document or key-value stores, graph databases, column-family databases)

Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.
Amazon strives to be the world’s most customer centric company with lot of high end innovation and product development. We have groundbreaking initiatives in the space of MLLM for Gen AI.

The DCM team is part of Amazon’s Gen AI Content Management and Protection organization that provides Data Curation capabilities for a variety of data formats.

Data Engineer is a newly created role to build world class data platform and deploy scalable data ingestion tools for AGI teams. The ideal candidate relishes working with large volumes of data, enjoys the challenge of highly complex technical contexts, and, above all else, is passionate about data. They are an expert with Petabyte scale data ingestion, processing multimodal data of all formats (videos, audio, images, vectors, texts), data modeling, ETL/ELT design and business intelligence tools and passionately partners with the business to identify strategic opportunities where improvements in data infrastructure creates outsized business impact. They are a self-starter, comfortable with ambiguity, able to think big (while paying careful attention to detail) and enjoys working in a fast-paced team. The ideal candidate need to possess exceptional technical expertise in very large scale lakehouses, Distributed computing at a scale of thousands of hosts on multiple clusters, Spark, BI systems with and AWS services.


Core Responsibilities


· Design, implement, and support a platform providing ad hoc access to large datasets for Videos, audio, images, text formats and mixed modalities
· Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using Spark or any other state of the art systems
· Implement data structures using best practices for lakehouses
· Model data and metadata for ad hoc and pre-built reporting, meeting read/write/summary optimized storages
· Interface with business customers, gathering requirements and delivering complete reporting solutions
· Build robust and scalable data integration (ETL) pipelines using Kotlin, Python, typescript and Spark.
· Build and deliver high quality datasets to support business analyst and customer reporting needs.
· Continually improve ongoingautomating or simplifying self-service Data ingestion at scale for customers
· Participate in strategic & tactical planning discussions