BLACKBIRD.AI

Data Engineer

New York, NY US
USD 130k - 170k
Redis Machine Learning Java Streaming Kubernetes MongoDB Python Docker Elasticsearch Kafka
This job is closed! Check out or
Description

Blackbird.AI helps organizations discover emergent threats and stay one step ahead of real-world harm through our AI-powered Narrative and Risk Intelligence Platform. Our commitment is to prioritize safety and security, providing the tools to identify potential risks and ensure a safer environment proactively. No matter the job or where it’s located, we’re all connected by a shared vision: To lead and enhance the landscape of risk intelligence.

Reporting to the Head of Data Engineering, the Data Engineer will optimize data pipelines for our cutting-edge real-time streaming cloud-hosted analytics platform. You will be the key driver of our collection, analysis, and visualization processes and will play a pivotal role in shaping the future of our platform.

The Data Engineer will encompass writing ETL processes, designing database systems, and developing tools for query and analytic processing, all focusing on real-time streaming applications. Additionally, you will elevate performance tradeoffs, own the database architecture, and lead build automation, continuous integration, and deployment efforts while ensuring adherence to security requirements.

As the Data Engineer, you’ll have the chance to:

  • Write ETL processes to support the ingestion and normalization of various data formats from social media, news, and web sources.
  • Design robust database systems and developing tools for query and analytic processing, especially for real-time streaming applications.
  • Conduct performance analysis and empirical studies to evaluate tradeoffs (e.g., cost vs. throughput/latency).
  • Develop, manage, and own the database architecture for our real-time streaming cloud-hosted analytics platform.
  • Lead build automation, continuous integration, deployment, and performance optimization efforts while ensuring compliance with security requirements.
  • Lead design of test suites and inline instrumentation as needed to ensure data correctness

What you’ll bring:

  • Bachelor’s degree in Computer Science or related field.
  • Proven success in deploying products in the cloud and SaaS model, with expertise in building optimized processing pipelines for streaming analytics applications and cloud-agnostic solutions (e.g., Kubernetes, Docker).
  • Expertise in databases and query optimization, including PostgresSQL, ElasticSearch, MongoDB, Redis, and Druid, with additional experience in NoSQL and graph databases being advantageous.
  • Experience in horizontally scaling databases.
  • Proficiency in Kafka and Airflow, with a deep understanding of runtime profiling tools to optimize throughput and latency, and able to establish comparative performance benchmarks.
  • Strong skills in build automation, continuous integration, and deployment (CI/CD) tools like Webpack, Buddy, Jenkins, and Docker.
  • Expert-level Python coding skills.
  • Experience collaborating with distributed teams.

Helpful to have:

  • Technical background in Artificial Intelligence (AI) and Machine Learning (ML).
  • Experience designing and implementing interactive query-driven main-machine intelligence systems.
  • Solid skills in Java.

We’ve outlined specific skills, experience, and requirements for this position, but don’t stress if you don’t meet every single one. Our Talent Team is dedicated to discovering exceptional individuals, and they might identify a relevant aspect of your background that suits this role or another opportunity within Blackbird.AI.

If you have passion for the role, please still apply.

What’s in it for you:

Blackbird.AI is embarking on an exciting growth journey with numerous opportunities for career development within the company. You will join a nurturing, inclusive, and experienced team.

Join us as we soar to new heights!

Values:

At Blackbird.AI, our core values shape how we work and make decisions. Our values inspire us to be authentic and continue improving.

We embrace a strong sense of responsibility to society, recognizing the vital role our services play in empowering governments, communities, and individuals to foster critical thinking and empowerment. We believe in integrating personal and professional lives with societal needs, emphasizing the importance of creating an environment that attracts top talent and provides substantial growth opportunities. We are motivated by the potential of science and technology to impact humanity positively.

BLACKBIRD.AI
BLACKBIRD.AI
Artificial Intelligence Enterprise Software Homeland Security Journalism Machine Learning National Security Natural Language Processing Predictive Analytics Public Safety SaaS

3 applies

175 views

There are more than 50,000 engineering jobs:

Subscribe to membership and unlock all jobs

Engineering Jobs

50,000+ jobs from 4,500+ well-funded companies

Updated Daily

New jobs are added every day as companies post them

Refined Search

Use filters like skill, location, etc to narrow results

Become a member

🥳🥳🥳 166 happy customers and counting...

Overall, over 80% of customers chose to renew their subscriptions after the initial sign-up.

Cancel anytime / Money-back guarantee

Wall of love from fellow engineers