Description
As an IBM data engineer, you will be responsible for designing, constructing, installing, testing, and maintaining highly scalable data management systems. You will work closely with data architects, data scientists, and business analysts to understand data requirements, design data models, and implement solutions that optimize data ingestion, storage, and processing.
Key Responsibilities:
1. Data Pipeline Development: Develop and maintain data pipelines to ingest, transform, and load structured and unstructured data from various sources into data storage systems such as data lakes, data warehouses, and databases.
2. Data Modeling: Design and implement efficient data models to support analytical and operational needs, ensuring data integrity, accuracy, and consistency across different data sets.
3. Data Integration: Integrate data from multiple sources, including internal databases, external APIs, third-party data providers, and streaming data sources, using appropriate tools and technologies.
4. Data Processing: Implement data processing workflows using tools like Apache Spark, Hadoop, or IBM Cloud Pak for Data to perform ETL (Extract, Transform, Load) operations, data cleansing, and data enrichment.
5. Database Management: Manage and optimize databases, including relational databases (e.g., IBM Db2, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra), to ensure high performance, availability, and scalability.
6. Data Quality Assurance: Develop and implement data quality checks, validation rules, and monitoring processes to ensure data accuracy, completeness, and consistency over time.
7. Performance Tuning: Optimize data pipelines, queries, and processes for performance, scalability, and efficiency, leveraging techniques such as indexing, partitioning, and caching.
8. Collaboration: Collaborate with cross-functional teams, including data scientists, business analysts, software engineers, and stakeholders, to understand data requirements, prioritize tasks, and deliver data-driven solutions.
9. Documentation: Create and maintain documentation for data pipelines, data models, data dictionaries, and technical specifications to ensure transparency, repeatability, and knowledge sharing within the team.
Key Responsibilities:
1. Data Pipeline Development: Develop and maintain data pipelines to ingest, transform, and load structured and unstructured data from various sources into data storage systems such as data lakes, data warehouses, and databases.
2. Data Modeling: Design and implement efficient data models to support analytical and operational needs, ensuring data integrity, accuracy, and consistency across different data sets.
3. Data Integration: Integrate data from multiple sources, including internal databases, external APIs, third-party data providers, and streaming data sources, using appropriate tools and technologies.
4. Data Processing: Implement data processing workflows using tools like Apache Spark, Hadoop, or IBM Cloud Pak for Data to perform ETL (Extract, Transform, Load) operations, data cleansing, and data enrichment.
5. Database Management: Manage and optimize databases, including relational databases (e.g., IBM Db2, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra), to ensure high performance, availability, and scalability.
6. Data Quality Assurance: Develop and implement data quality checks, validation rules, and monitoring processes to ensure data accuracy, completeness, and consistency over time.
7. Performance Tuning: Optimize data pipelines, queries, and processes for performance, scalability, and efficiency, leveraging techniques such as indexing, partitioning, and caching.
8. Collaboration: Collaborate with cross-functional teams, including data scientists, business analysts, software engineers, and stakeholders, to understand data requirements, prioritize tasks, and deliver data-driven solutions.
9. Documentation: Create and maintain documentation for data pipelines, data models, data dictionaries, and technical specifications to ensure transparency, repeatability, and knowledge sharing within the team.
IBM
Business Development
Business Information Systems
CRM
Data Management
Software
0 applies
27 views
Jobs from our Partners
Software Engineer
Remote
US
Software Engineer - Frontend Engineer
San Francisco, CA
US
Camera Sr. Infrastructure Engineer I Storage – Clearance
Aurora, CO
US
Other Jobs from IBM
Application Developer: Cloud FullStack
Pune, India
Application Developer: Cloud FullStack
Pune, India
【IBM Japan】Mobile Developer
Tokyo, Japan
Data Engineer
Pune, India
There are more than 50,000 engineering jobs:
Subscribe to membership and unlock all jobs
Engineering Jobs
50,000+ jobs from 4,500+ well-funded companies
Updated Daily
New jobs are added every day as companies post them
Refined Search
Use filters like skill, location, etc to narrow results
Become a member
🥳🥳🥳 232 happy customers and counting...
Overall, over 80% of customers chose to renew their subscriptions after the initial sign-up.
Cancel anytime / Money-back guarantee