Software Engineer - Real Time Data Platform (Senior / Principal)

Blackbird.AI helps organizations discover emergent threats and stay one step ahead of real-world harm through our AI-powered Narrative and Risk Intelligence Platform. Our commitment is to prioritize safety and security, providing the tools to identify potential risks and ensure a safer environment proactively. No matter the job or where it’s located, we’re all connected by a shared vision: To lead and enhance the landscape of risk intelligence. 

As a Senior/Principal Software Engineer reporting to the Director of Engineering, Data Platform, you will not only optimize data pipelines for our advanced real-time streaming cloud-hosted analytics platform but also lead the development of new features and capabilities. Your role will be pivotal in shaping the future of our platform, offering the chance to solve some of the most challenging data problems in the industry.

As a Software Engineer, you’ll have the chance to:

• Design and implement real time distributed data processing systems analyzing public data and detecting emergent threats
• Lead and manage the development and optimization of ETL processes for various data formats from social media, news, and web sources.
• Design and implement robust database systems and develop tools for query and analytic processing, focusing on real-time streaming applications.
• Conduct performance analysis and empirical studies, making strategic decisions on tradeoffs (e.g., cost vs. throughput/latency).
• Develop, manage, and oversee the database architecture for our real-time streaming cloud-hosted analytics platform.
• Spearhead build automation, continuous integration, deployment, and performance optimization efforts, upholding our strict security requirements.
• Drive the project management, leading the development of new features and capabilities.
• Design test suites and implement inline instrumentation to ensure data correctness.

• Bachelor's degree in Computer Science or a related field.
• Minimum of 5 years of experience in data engineering and building data platforms.
• Demonstrated success in deploying cloud and SaaS model products, with expertise in building optimized processing pipelines for streaming analytics applications.
• Proficiency in databases and query optimization (PostgresSQL, ElasticSearch, MongoDB, Redis, Druid), including experience with NoSQL and graph databases.
• Experience in horizontally scaling databases.
• Expertise in Kafka and Airflow, with a strong understanding of runtime profiling tools.
• Experience in big data processing systems similar to Apache Spark, Flink, Beam or similar.
• Strong skills in build automation, continuous integration, and deployment (CI/CD) tools (Webpack, Buddy, Jenkins, Docker).
• Expert-level Python coding skills.
• Project management skills and the ability to lead development initiatives.

Desirable Skills:

• Technical background in AI and ML.
• Experience in designing and implementing interactive query-driven main-machine intelligence systems.
• Experience in working with distributed teams.

Visa Sponsorship: No visa sponsorship currently available. Candidates should be able to work without requiring visa sponsorship now or in the future in their preferred work location.

We’ve outlined specific skills, experience, and requirements for this position, but don’t stress if you don’t meet every single one. Our Talent Team is dedicated to discovering exceptional individuals, and they might identify a relevant aspect of your background that suits this role or another opportunity within Blackbird.AI.

This updated job description highlights the seniority and leadership expectations while maintaining the core responsibilities and qualifications of the Software Engineer role. It also emphasizes the opportunity to be part of a team focused on solving complex and significant data challenges.

If you have passion for the role, please still apply.