Capco

Principal Azure Data Engineer (Databricks)

London, UK
Unity Python Hadoop Scala Java Azure Spark Streaming Kafka SQL
Description

Principal Azure Data Engineer (Databricks)

Joining Capco means joining an organisation that is committed to an inclusive working environment where you’re encouraged to #BeYourselfAtWork. We celebrate individuality and recognize that diversity and inclusion, in all forms, is critical to success. It’s important to us that we recruit and develop as diverse a range of talent as we can and we believe that everyone brings something different to the table – so we’d love to know what makes you different. Such differences may mean we need to make changes to our process to allow you the best possible platform to succeed, and we are happy to cater to any reasonable adjustments you may require. You will find the section to let us know of these at the bottom of your application form or you can mention it directly to your recruiter at any stage and they will be happy to help.

Why Join Capco?

Capco is a global technology and business consultancy, focused on the financial services sector. We are passionate about helping our clients succeed in an ever-changing industry.

You will work on engaging projects with some of the largest banks in the world, on projects that will transform the financial services industry.

We are/have:

  • Experts in banking and payments, capital markets and wealth and asset management
  • Deep knowledge in financial services offering, including e.g. Finance, Risk and Compliance, Financial Crime, Core Banking etc.
  • Committed to growing our business and hiring the best talent to help us get there
  • Focused on maintaining our nimble, agile and entrepreneurial culture

As a Principal Data Engineer at Capco you will:

  • Work alongside clients to interpret requirements and define industry-leading solutions
  • Design and develop robust, well tested data pipelines
  • Demonstrate and help clients adhere to best practices in engineering and SDLC
  • Excellent knowledge of building event-driven, loosely coupled distributed applications
  • Experience in developing both on-premise and cloud-based solutions
  • Good understanding of key security technologies, protocols e.g. TLS, OAuth, Encryption
  • Support internal Capco capabilities by sharing insight, experience and credentials

Why Join Capco as a Principal Data Engineer?

  • You will work on engaging projects with some of the largest banks in the world, on projects that will transform the financial services industry.
  • You’ll be part of digital engineering team that develop new and enhance existing financial and data solutions, having the opportunity to work on exciting greenfield projects as well as on established Tier1 bank applications adopted by millions of users.
  • You’ll be involved in digital and data transformation processes through a continuous delivery model.
  • You will work on automating and optimising data engineering processes, develop robust and fault tolerant data solutions both on cloud and on-premise deployments.
  • You’ll be able to work across different data, cloud and messaging technology stacks.
  • You’ll have an opportunity to learn and work with specialised data and cloud technologies to widen the skill set.

Skills & Expertise: 

You will have experience working with some of the following Methodologies/Technologies;

Required Skills

  • Hands on working experience of the Databricks platform. Must have experience of delivering projects which use DeltaLake, Orchestration, Unity Catalog, Spark Structured Streaming on Databricks.
  • Extensive experience using Python, PySpark and the Python Ecosystem with good exposure to python libraries.
  • Experience with Big Data technologies and Distributed Systems such as Hadoop, HDFS, HIVE, Spark, Databricks, Cloudera.
  • Experience developing near real time event streaming pipelines with tools such as – Kafka, Spark Streaming, Azure Event Hubs.
  • Excellent experience in the Data Engineering Lifecycle, you will have created data pipelines which take data through all layers from generation, ingestion, transformation and serving.
  • Experience of modern Software Engineering principles and experience of creating well tested, clean and applications.
  • Experience with Data Lakehouse architecture and data warehousing principles, experience with Data Modelling, Schema design and using semi-structured and structured data.
  • Proficient in SQL & good understanding of the differences and trade-offs between SQL and NoSQL, ETL and ELT.
  • Proven experience DevOps and using building robust production data pipelines, CI/CD Pipelines on e.g. Azure DevOps, Jenkins, CircleCI, GitHub Actions etc.

Desirable Skills

  • Having a strong commercial focus and the ability to develop client relationships, spearhead sales opportunities and Data Engineering propositions.
  • An appetite to contribute to the wider Capco business outside of project assignments. This can be achieved through various means including thought leadership activities, supporting RFP's and the coaching/mentoring of more junior engineering team members.
  • Experience Developing in other languages e.g. Scala/Java.
  • Enthusiasm and ability to pick up new technologies as needed to solve problems.
  • Exposure to working with PII, Sensitive Data and understands data regulations such as GDPR.

We have been informed of several recruitment scams targeting the public. We strongly advise you to verify identities before engaging in recruitment related communication. All official Capco communication will be conducted via a Capco recruiter.

Capco
Capco
Consulting Finance Financial Services Insurance

0 applies

7 views

Other Jobs from Capco

Tech Lead - COM INGLÊS - Remoto

Remote Sao Paulo, Brazil

Web Developer

Bengaluru, India

Tech Lead

Bengaluru, India

Senior Web Developer

Bengaluru, India

There are more than 50,000 engineering jobs:

Subscribe to membership and unlock all jobs

Engineering Jobs

60,000+ jobs from 4,500+ well-funded companies

Updated Daily

New jobs are added every day as companies post them

Refined Search

Use filters like skill, location, etc to narrow results

Become a member

🥳🥳🥳 452 happy customers and counting...

Overall, over 80% of customers chose to renew their subscriptions after the initial sign-up.

To try it out

For active job seekers

For those who are passive looking

Cancel anytime

Frequently Asked Questions

  • We prioritize job seekers as our customers, unlike bigger job sites, by charging a small fee to provide them with curated access to the best companies and up-to-date jobs. This focus allows us to deliver a more personalized and effective job search experience.
  • We've got about 70,000 jobs from 5,000 vetted companies. No fake or sleazy jobs here!
  • We aggregate jobs from 5,000+ companies' career pages, so you can be sure that you're getting the most up-to-date and relevant jobs.
  • We're the only job board *for* software engineers, *by* software engineers… in case you needed a reminder! We add thousands of new jobs daily and offer powerful search filters just for you. 🛠️
  • Every single hour! We add 2,000-3,000 new jobs daily, so you'll always have fresh opportunities. 🚀
  • Typically, job searches take 3-6 months. EchoJobs helps you spend more time applying and less time hunting. 🎯
  • Check daily! We're always updating with new jobs. Set up job alerts for even quicker access. 📅

What Fellow Engineers Say

Sid avatar
Sid
Very nice portal for searching jobs in this rough market.
Mar 6, 2025
Michael Duran avatar
Michael Duran
Software Engineer
I've been using this job search site for a while now, and it’s honestly one of the best out there! The clean and easy-to-navigate UI makes the whole job-hunting process so much smoother. Plus, the job postings are always up-to-date, so I never feel like I’m wasting time. The cherry on top is the owner—super kind and always quick to respond. Definitely recommend checking it out if you're on the job hunt!
Aug 21, 2024
Sai avatar
Sai
It’s really great website for finding jobs based on skills it’s really helpful give a go
Aug 21, 2024
Adinadh avatar
Adinadh
What I like most about Echo Jobs is how easy it is to use. The platform helps me quickly find jobs that match my skills and interests, thanks to its great recommendations and filters. Yes, I would definitely recommend Echo Jobs to a friend. It makes job searching simple and efficient, making it a great tool for anyone looking for a new job.
Jul 23, 2024
As a student navigating the job market, I've found LinkedIn increasingly frustrating due to numerous fake postings by consultancies. In contrast, this job posting website has been a game-changer for me. It offers genuine opportunities and a straightforward application process, making it much easier to find and apply for real jobs. Highly recommend it to fellow students seeking reliable job listings!
Jul 16, 2024
Cliff Gor avatar
Echo Jobs has been exceptional in my job hunt where it provides one platform to job hunt and I don't have to open 10 websites just to look for a job. It has also helped me focus much on the job skill and the location filtering out the onsite jobs and remote ones. The only feature that I would request is to display fully remote jobs that are not restricted to a country since the one available shows ie, Remote, US yet. But if it could show remote only, that would be helpful not only to me but to other people applying for full remote and not tied to only US candidates
Apr 22, 2024
I found EchoJobs in 2022, and I love it. It has a lot of remote jobs. It's exclusive to software and technology jobs (helpful for devs like me). What I like the most are its filters and its API. If you're a tech professional seeking remote work, I highly recommend giving it a try to EchoJobs.
Mar 4, 2024
Would definitely recommend it! Excellent product, dedicated founder, Jobs are easier to find. Congrats 🎉 to the entire team!
Mar 3, 2024
Brandon Banks avatar
Brandon Banks
Echo Jobs is really impressive. It provides a great user experience with an ability to quickly search through the many job postings. There is an impressive amount of jobs here and it is quickly updated. The details in the each job posting is helpful when determining if it is worth pursuing. I would highly recommend using Echo Jobs to find the next step in your career.
Mar 2, 2024
Tyler Young avatar
Tyler Young
tylerayoung.com
Best wishes with EchoJobs—it's become my favorite job board overnight!
Dec 16, 2023
Simply put, it's the most up to date tech jobs aggregator I’ve found. I'm like... "I don't have to check 10+ jobs boards daily just to see if there's a new job listing? sign me up!" The filters are also quite helpful! The UI is very clean and straightforward. Love it!
Oct 5, 2023