AI Operations Engineer

Location: Lisbon, Lisbon, pt

Job Description

Tieto Iberia is part of the Tieto Group and combines local expertise with the global strength of Tieto, a leading software and digital engineering services company with worldwide presence and capabilities.

We are looking for a Machine Learning Operations/ Language Model Operations Engineer. In this role, you will be responsible for designing, building, and operating the end-to-end infrastructure that supports production AI systems, ensuring reliability, observability, data quality, and continuous improvement of machine learning and LLM-based solutions.

Key Responsibilities

• Design and implement scalable, source-agnostic ingestion pipelines for production ML and LLM data.

• Build and maintain data processing workflows including ingestion, redaction, storage, classification, and slicing of production signals.

• Define and implement data storage strategies, including tiering, retention policies, and privacy-aware data handling.

• Develop observability and debugging tools, including dashboards and query systems for production AI systems.

• Implement evaluation and monitoring frameworks, including offline evaluation sets, online autoraters, and regression detection systems.

• Build automated triage systems to identify, classify, and surface production failures.

• Develop and maintain PII redaction mechanisms and enforce data governance and compliance policies at ingestion level.

• Design and operate LLM evaluation mining workflows to continuously improve model and prompt performance.

• Implement alerting systems to detect regressions across model and prompt deployments.

• Collaborate with AI, engineering, and product teams to ensure robustness and reliability of production AI systems.

• Evaluate and select tooling, infrastructure, and hosting strategies for ML/LLM platforms.

• Own the operational reliability of the entire ML/LLM data and evaluation pipeline.

Requirements

• Proven experience building and operating production-grade data platforms or ML systems, including ingestion, storage, access control, monitoring, and on-call responsibilities.

• Hands-on experience developing ML/LLM evaluation systems (e.g., regression test sets, autoraters, LLM-as-a-judge frameworks, or golden datasets).

• Strong understanding of LLM observability, tracing, and debugging tools.

• Experience implementing data privacy controls such as PII redaction in production environments.

• Deep understanding of failure modes in ML and LLM systems (hallucinations, retrieval failures, agent loops, ASR/TTS degradation, prompt/model regressions).

• Strong production-level Python engineering skills, with a hands-on mindset.

• Solid understanding of data pipelines, system reliability, and distributed data processing.

Nice-to-Have Skills

• Experience in multi-tenant or SaaS architectures with strict data isolation requirements.

• Familiarity with Azure and/or AWS cloud ecosystems.

• Experience making infrastructure trade-off decisions between managed services and self-hosted solutions.

• Knowledge of vector databases, embedding techniques, and clustering or unsupervised failure detection methods.

• Experience with data versioning tools such as LakeFS, DVC, or Delta Lake.

• Familiarity with GDPR, data deletion workflows, and compliance-driven data systems.

• Exposure to embedded, automotive, or constrained environments.

• Experience with non-English language model evaluation or multilingual datasets.

• Experience working with LLM APIs such as Anthropic Claude, OpenAI models, or open-source alternatives.

• Familiarity with CI/CD workflows, GitHub-based development, and modern DevOps practices.

• Experience with dashboard development using TypeScript or similar frontend technologies.

Additional Information

At Tieto, we believe in the power of diversity, equity, and inclusion. We encourage applicants of all backgrounds, genders (m/f/d), and walks of life to join our team, as we believe that this fosters an inspiring workplace and fuels innovation. Our commitment to openness, trust, and diversity is at the heart of our mission to create digital futures that benefit businesses, societies, and humanity.

Diversity, equity and inclusion | Sustainability | Tieto