Staff Data Engineer

Job Summary:

We are looking for a highly skilled Staff Engineer for our Data and Cloud Engineering team with expertise in AWS and Azure. The ideal candidate will have a strong technical background in designing, building, developing, and implementing data pipelines and cloud solutions along with excellent technical guidance and communication skills. This role requires a strong technical background in cloud platforms, data architecture, and engineering best practices.

Key Responsibilities:

• Lead the design and implement robust, scalable, and secure cloud-based data pipelines and architectures in MS Azure.
• Provide technical direction and mentorship to a team of engineers
• Ensure best practices in code quality, architecture, and design.
• Design and implement secure, scalable, and high-performance cloud infrastructure.
• Manage cloud resources, optimize costs, ensure high availability and disaster recovery.
• Automate infrastructure provisioning and deployment processes using Infrastructure as Code (IaC) tools such as Terraform, CloudFormation, and ARM templates.
• Collaborate with cross-functional teams to understand data needs and deliver comprehensive cloud solutions.
• Oversee cloud infrastructure management, including monitoring, maintenance, and scaling of cloud resources.
• Ensure compliance with industry standards and regulatory requirements.
• Implement data governance policies and practices and ensure high data quality, integrity, and security across all cloud platforms.
• Develop and enforce best practices for cloud and data engineering, including documentation and code standards.
• Oversee the design and development of robust data pipelines and ETL processes.
• Identify and implement process improvements to enhance efficiency, quality, and scalability of data engineering and cloud operations.
• Troubleshoot and resolve complex technical issues related to data pipelines and cloud infrastructure.
• Stay current with emerging technologies and industry trends to drive innovation.

Tech-stack:

• Infrastructure: Glue, Lambda, Step Function, Batch, ECS, Quicksight, Machine Learning, Sagemaker, Dagster
• DevOps: Cloudformation, Terraform, Git, CodeBuild
• Database: Redshift, PostgreSQL, DynamoDB, Athena (Trino), Snowflake, Databricks
• Language: Bash, Python (PySpark, Pydantic, PyArrow), SQL

Qualifications:

• Min. 6 years of proven experience as a senior data and cloud engineer or similar role.
• Extensive experience with AWS and Azure cloud platforms and their data services (e.g., AWS Redshift, AWS Glue, AWS S3, Azure Data LakeAzure Synapse, Snowflake, Databricks).
• Strong understanding of ETL processes, data warehousing, and big data technologies.
• Proficiency in SQL, Python, and other relevant programming languages.
• Experience with infrastructure as code (IaC) tools such as Terraform, CloudFormation, or ARM templates.
• Knowledge of containerization and orchestration (e.g., Docker, Kubernetes).
• Understanding of cloud cost management and optimization strategies.
• Familiarity with CI/CD pipelines and DevOps practices.
• Excellent leadership, communication, and interpersonal skills.
• Ability to work in a fast-paced, dynamic environment.
• Strong problem-solving and analytical skills.
• Familiarity with data visualization tools (e.g., Power BI, QuickSight) is a plus.