Data Engineer - Data Lakehouse Section, Analytics Data Engineering Department(ADED)

Job Description:

Business Overview

Rakuten group has almost 100 million customer base in Japan and 1 billion globally as well, providing more than 70 services in a variety such as ecommerce, payment services, financial services, telecommunication, media, sports, etc. 

AI Success Supervisory Department (AISSD) provides various solutions by leveraging Rakuten group's data. The department has international culture created by excellent employees joined around the world and provides the cutting-edge data science. Following the strategic vision "Rakuten as a data-driven membership company", AISSD is expanding our data activities across our multiple Rakuten group companies.

Department Overview

Data Lakehouse Section provides the platform to realize ’Digital Twin’. With hundreds of millions of members and trillions (Japanese Yen) in spending, Rakuten’s Membership enjoys an un-paralleled eco-system of benefits and is amongst the largest in the world. Our talented and driven team operates a portfolio of products and services that optimize Rakuten membership experiences using data.

You'll benefit from our network of global communities and collaborative culture that will help you build technical and functional skills and capabilities. And because we serve more than 28 countries industries globally, you'll have the opportunity to develop valuable industry-specific expertise.

The scale of our capabilities and client engagements and the unique way we innovate, operate and deliver value will give you the opportunity to deepen your existing skills even as you help create the latest technology trends. You'll have access to leading-edge technology.

Position:

Why We Hire

We are looking for a Senior Data Engineer to lead Digital twin in Data Lakehouse Section with 5+ years of experience, responsible for building reliable and scalable customer data platform.

Position Details

Responsibilities

- Utilize big data technologies to provide frameworks that appropriately replicates the stated data needs; hardware, software and cloud services included

- Responsible for the design and execution of abstractions and integration patterns (APIs) for data applications

- Engage with clients and stakeholders to understand their objectives, customer requirements, analyze complex problems and translate into technology solutions

- Research and properly evaluate sources of information to determine possible limitations in reliability, usability, and scalability

- Upskill and mentoring of team members

- Stay current on latest technology to ensure maximum ROI for clients

Mandatory Qualifications:

- Minimum of 5 years’ experience in building and operating big data platforms for analytical or operations use. At least 2 years’ experience in managing large scale unstructured and(or) real time data platform.

- Proficiency in Python programming and Libraries: Deep understanding of Python's data structures, algorithms, and best practices for writing scalable, efficient code. Proficiency in Python libraries such as Pandas, NumPy, PySpark, and Pydantic for efficient data manipulation, feature engineering, validation, and serialization.

- GCP Platform Experience: Hands-on experience GCP, including services like BigQuery, Cloud Storage, Cloud Functions, Dataflow, Cloud Pub/Sub, and GCP Identity & Access Management (IAM).

- Unstructured and real time data management & Optimization: Proven experience building and optimizing pipelines for managing unstructured and real time data, ensuring they can handle large-scale data loads across distributed cloud environments.

- Platform Engineering: Familiarity with platform engineering concepts, including Infrastructure-as-Code (IaC), CI/CD, cloud resource management, and designing systems for scalability and reliability in cloud environments.

- Containerization & Orchestration: Experience with containerization and orchestration using Docker and Kubernetes, managing cloud-native applications and services.

- In-depth knowledge of SQL and NoSQL databases, with experience managing large-scale, distributed data storage systems and writing complex queries for data extraction and transformation.

- Excellent problem-solving and debugging skills, with a proactive approach to optimizing data infrastructure and pipelines for reliability and performance.

- Strong communication and collaboration skills, with the ability to work effectively in agile teams, liaise with business stakeholders, and mentor junior team members.

Desired Qualifications:

- Analyze business requirements and design Python-based solutions that address specific data engineering tasks, ensuring scalability, performance, and maintainability.

- Develop, optimize, and maintain Python packages for data engineering tasks, ensuring modularity, reusability, and integration with existing data infrastructure (e.g., libraries, APIs, ETL frameworks).

- Proficient in using Python libraries like Pandas, NumPy, PySpark, Pydantic, and Django/Flask combined with robust exception handling and debugging to build reliable python-based solutions.

- Implement cloud-native solutions leveraging GCP services with focus on data pipeline efficiency and platform cost optimization.

- Work hands-on and directly with engineering solutions while ensuring on-time delivery of high-quality deliverables by cultivating culture of continuous learning, innovation, and collaboration.

- Provide technical guidance and industry best practices to team of talented engineers.

Other Information:

Additional information on Location

Rakuten Crimson House

#engineer  #applicationsengineer  #technologyservicediv