Spark Engineer - (Spark using Scala, Spark SQL and Presto)

Responsibilities

• Discuss the Cost of Change (= code quality) with your team members continuously
• Write unit tests, integration tests and API tests
• Should be able to support the application 24/7 based on team on call rotations. 
• Collaborate a lot: Code Reviews and Pair Programming
• You would be expected to write code 100% of the time, with an emphasis on Test Drive Development
• Should be open to learn and adapt to new technology architectures and patterns
• You would be expected to code and also understand the impact of it on data
• Fair amount of experience with building and creating CI/CD pipelines
• Should be able to develop and maintain existing applications processing large datasets
• Follow best practices on application monitoring and orchestration and performance optimization
• Implement and Practice DevOps and SecOps for continuous incremental delivery and quality product with the guidance of senior engineers

Key Skills

• Required
• Bachelor or Master’s degree in computer science or related discipline (field), or equivalent work experience
• 2-4 years of Experience with Spark using Scala, Spark SQL and Presto. Upgrading, Maintaining and performance tuning large Spark/Scala applications.
• 2-4 years of advanced experience with Java and Relational databases.
• 0-3 years of experience working with AWS services (RDS, S3).
• Experience using monitoring and alert orchestration tools such as Prometheus, Grafana, OpsGenie/PagerDuty tools.
• Fluent in English, both spoken and written, with a large vocabulary (C1 english level)
• Understand basic object oriented principles and functional programming principles. Implements good coding practices with good unit and integration testing, with an emphasis on Test Driven Development.
• Ability to work independently and as part of a team in a fast-paced and dynamic environment.


• Preferred
• Strong understanding of Mathematics and Statistics
• Experience with Test-driven development
• Experience with building CICD pipelines in GitLab for applications running on Kubernetes (EKS) using Docker
• Follow Agile principles and feature related discussions and participate in grooming and planning sessions to effectively translate business requirements to Agile stories.
• Experience with Airflow DAGs and Python.