Lead Specialty Software Engineer- Observability & Agentic AI Platforms

Location: PHOENIX, AZ

Time Type: Full time

Job Description

About this role:

Wells Fargo is seeking a Lead AI Ops Engineer to own and advance the Commercial Observability Platform. This role provides technical leadership across agentic AI systems, AI‑powered observability, advanced analytics, and enterprise telemetry platforms, enabling proactive monitoring, faster root cause analysis, and improved operational resilience across critical business applications.

This position is intended for a senior, hands‑on AI engineer who will serve as a technical role model and bar raiser, setting standards for engineering excellence in AI‑driven observability and operations

In this role, you will:

• Design, build, and maintain production‑grade AI and agentic systems that reason over observability data including logs, metrics, traces, events, and digital experience signals

• Develop LLM‑powered workflows to support automated incident analysis, intelligent alerting, operational insights, and root cause analysis (RCA) summaries

• Architect and implement agentic or multi‑agent AI workflows that decompose complex operational problems, analyze telemetry across multiple tools, and coordinate actionable recommendations

• Apply AIOps and machine learning techniques such as anomaly detection, correlation, pattern recognition, forecasting, noise reduction, and predictive insights

• Write and maintain Python‑based AI services, orchestration logic, and data pipelines deployed in production environments

• Establish best practices for AI system observability, governance, feedback loops, and continuous improvement

• Lead the design, implementation, and evolution of enterprise observability platforms supporting commercial applications

• Own and operate observability tools including Splunk Observability, Splunk (logs, metrics, traces), AppDynamics, and Glassbox

• Define and enforce standards for telemetry collection, including logging, metrics, distributed tracing, and real user monitoring

• Perform and lead complex root cause analysis by analyzing application code, logs, metrics, traces, infrastructure signals, and user experience data

• Act as a senior Splunk query developer, designing highly complex SPL queries that function as analytical programs to correlate large volumes of telemetry data

• Build and optimize advanced Splunk dashboards using multi‑stage SPL pipelines, statistical functions, joins, lookups, and enrichments

• Develop Splunk analytics that power real‑time operational insights, advanced alerting, historical analysis, and AI model inputs

• Design and develop Beacon / Telemetry APIs to collect custom application, platform, and business signals

• Build and maintain telemetry ingestion services that normalize, store, and enrich data for analytics and AI/ML solutions

• Partner closely with application engineering, SRE, and platform teams to improve reliability, performance, and operational maturity

• Provide technical leadership and mentoring, serving as a role model for strong AI, analytics, and observability engineering practices

• Influence engineering standards and contribute to long‑term observability and AI platform strategy

Required Qualifications:

• 5+ years of Specialty Software Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education

• 3+ years hands on experience in platform engineering, SRE, observability

• 3+ years hands‑on software engineering experience building production services, APIs, data pipelines, or AI systems

• 2+ years experience designing or implementing AI, AI Ops, or ML‑driven systems in production environments (LLMs, generative AI, or agentic AI systems)

• 2+ years experience with Splunk SPL, including writing advanced, multi‑stage queries equivalent to programmatic logic, building complex Splunk dashboards and analytics used for operational decision‑making, complex queries

• 5+ years experience in distributed systems, microservices, and cloud‑native architectures

• 2+ hands‑on experience with enterprise observability platforms such as Splunk, Splunk Observability, AppDynamics, or equivalent tools, Grafana or prometheus

Desired Qualifications:

• Proven ability to perform deep root cause analysis using application code and telemetry data

• Experience designing or implementing multi‑agent or autonomous AI workflows

• Familiarity with AI frameworks and tooling (for example: LangChain, LangGraph, AutoGen, CrewAI, or equivalent concepts)

• Experience designing and building custom telemetry ingestion pipelines or Beacon APIs

• Familiarity with OpenTelemetry and modern instrumentation standards

• Experience building internal observability, analytics, or AI platforms used by multiple engineering teams

• Ability to act as a technical bar raiser, influencing engineering standards across AI, analytics, and observability domains

Job Expectations:

• This position is not eligible for Visa sponsorship or transfer of visa

• Ability to work on-site at approved location

• Relocation assistance is not available for this position

Posting Locations:

• 2222 W Rose Garden Lane, Phoenix, AZ

Pay Range

Reflected is the base pay range offered for this position. Pay may vary depending on factors including but not limited to demonstrated examples of prior performance, skills, experience, or work location. Employees may also be eligible for incentive opportunities.

$159,000.00 - $254,000.00 USD Annual

Benefits:

• Information about Wells Fargo's US employee benefits

• Information about Wells Fargo's International employee benefits

Posting End Date:

24 Apr 2026

*Job posting may come down early due to volume of applicants.

We Value Equal Opportunity

Wells Fargo is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other legally protected characteristic.

Employees support our focus on building strong customer relationships balanced with a strong risk mitigating and compliance-driven culture which firmly establishes those disciplines as critical to the success of our customers and company. They are accountable for execution of all applicable risk programs (Credit, Market, Financial Crimes, Operational, Regulatory Compliance), which includes effectively following and adhering to applicable Wells Fargo policies and procedures, appropriately fulfilling risk and compliance obligations, timely and effective escalation and remediation of issues, and making sound risk decisions. There is emphasis on proactive monitoring, governance, risk identification and escalation, as well as making sound risk decisions commensurate with the business unit’s risk appetite and all risk and compliance program requirements.

Candidates applying to job openings posted in Canada: Applications for employment are encouraged from all qualified candidates, including women, persons with disabilities, aboriginal peoples and visible minorities. Accommodation for applicants with disabilities is available upon request in connection with the recruitment process.

Wells Fargo Recruitment and Hiring Requirements:

a. Third-Party recordings are prohibited unless authorized by Wells Fargo.

b. Wells Fargo requires you to directly represent your own experiences during the recruiting and hiring process.

Posting End Date: 

*Job posting may come down early due to volume of applicants.

We Value Equal Opportunity

Wells Fargo is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other legally protected characteristic.

Employees support our focus on building strong customer relationships balanced with a strong risk mitigating and compliance-driven culture which firmly establishes those disciplines as critical to the success of our customers and company. They are accountable for execution of all applicable risk programs (Credit, Market, Financial Crimes, Operational, Regulatory Compliance), which includes effectively following and adhering to applicable Wells Fargo policies and procedures, appropriately fulfilling risk and compliance obligations, timely and effective escalation and remediation of issues, and making sound risk decisions. There is emphasis on proactive monitoring, governance, risk identification and escalation, as well as making sound risk decisions commensurate with the business unit’s risk appetite and all risk and compliance program requirements.

Candidates applying to job openings posted in Canada: Applications for employment are encouraged from all qualified candidates, including women, persons with disabilities, aboriginal peoples and visible minorities. Accommodation for applicants with disabilities is available upon request in connection with the recruitment process.

Applicants with Disabilities

To request a medical accommodation during the application or interview process, visit Disability Inclusion at Wells Fargo.

Drug and Alcohol Policy

Wells Fargo maintains a drug free workplace.  Please see our Drug and Alcohol Policy to learn more.

Wells Fargo Recruitment and Hiring Requirements:

a. Third-Party recordings are prohibited unless authorized by Wells Fargo.

b. Wells Fargo requires you to directly represent your own experiences during the recruiting and hiring process.