Principal Engineer - Observability Solutions

At T-Mobile, we invest in YOU!  Our Total Rewards Package ensures that employees get the same big love we give our customers.  All team members receive a competitive base salary and compensation package - this is Total Rewards. Employees enjoy multiple wealth-building opportunities through our annual stock grant, employee stock purchase plan, 401(k), and access to free, year-round money coaches. That’s how we’re UNSTOPPABLE for our employees!

Are you ready for the next chapter in your Uncarrier journey?

We are seeking a Principal Engineer – Observability Solutions to drive the design, development, and delivery of end-to-end observability capabilities across T-Mobile’s technology ecosystem. This hands-on technical leadership role will focus on creating scalable, resilient, and intelligent observability solutions that empower teams to gain real-time visibility into application and infrastructure health, performance, and user experience.

The Principal Engineer – Observability Solutions will collaborate closely with software engineers, site reliability engineers, infrastructure teams, and business stakeholders to architect systems that capture, process, and analyze telemetry data (metrics, logs, traces, events). The ideal candidate is passionate about driving operational excellence, experienced in modern observability stacks, and comfortable influencing at both technical and strategic levels.

The Principal Engineer – Observability Solutions works on challenging technical assignments that require sound judgment in issue resolution and decision-making. They determine methods and procedures for new assignments and handle tasks that demand significant initiative and strategic thinking. They understand the broader implications of their work and provide recommendations for solutions. Typically, they receive minimal instruction on daily tasks and only general guidance for newly introduced assignments. In some cases, they may serve as an informal team leader.

We pride ourselves on encouraging a culture of innovation, advocating for agile methodologies, and promoting transparency in all that we do. Join us in embodying the spirit of the 'Un-carrier' and make a tangible impact!

Job Responsibilities:

• Architect and implement observability platforms and tools that support proactive detection, root cause analysis, and continuous improvement across distributed systems.
• Lead the design and development of core observability services, including metrics pipelines, log aggregation, tracing infrastructure, alerting systems, and data visualization.
• Leverage modern technologies such as Open Telemetry, cloud-native instrumentation, streaming data platforms, and AI/ML for anomaly detection and automated insights.
• Collaborate with engineering and SRE teams to define service-level objectives (SLOs), error budgets, and golden signals.
• Provide technical leadership and mentorship to engineering teams, ensuring best practices in system design, telemetry standards, and observability instrumentation.
• Champion observability as a first-class concern in the software development lifecycle.
• Influence platform strategy and roadmap through deep technical insight and alignment with business priorities.
• Technical Writing: Writes advanced documentation, architecture, capabilities, limitations, and advantages for technologies. Creates and modifies clear documentation on existing and new systems and services. Able to design and communicate architecture, design strategies, standards, technical system specifications and business requirements. Able to present complex ideas to non-technical audience. Creates and presents executive presentations that may translate technical issues into business impact. Participate in industry professional forums
• Technical Leadership: Monitor and influence relevant industry technologies and standards. Present highly technical concepts to both technical and non-technical decision-makers. Continuously learns, creates content, and teaches others specific subject areas. Informally coaches and contributes to the development of others through mentoring or in house workshops and learning sessions. Coach and develop engineers across functional teams on technology decisions. Influence technology and policy decisions made across the VP organization related to architecture, design strategies and standards. Understand financial decisions, including NPV and ROI, based on customer experience/business drivers. Owns and drives PoC and technology roadmaps. Influence technology and policy decisions made across VP organization.
• Technology Strategy: Evaluates new and existing technology options that support business goals. Proposes technology position for the company across the VP organization.

Education and Work Experience:

• Bachelor's Degree Engineering, Computer Science (Required)
• 7-10 years Technical engineering experience (Required)

Knowledge, Skills and Abilities:

• Focus: Able to develop and communicate goals in support of the business' mission. (Required)
• Drive: Self-motivated and able to work well under pressure with minimal supervision – including the ability to simultaneously support multiple strategically significant tasks under fixed/accelerated deadlines. (Required)
• Teamwork: Ability and desire to work cooperatively with others on a team. Develop, maintain, and strengthen partnerships that can provide information, assistance, and support. (Required)
• Communication: The ability to ensure that information is passed on to others who should be kept informed. Express self clearly in conversations and interactions with others. Clear business and technical writing skills. The ability to plan and deliver oral and written communications that make an impact and persuade their intended audiences. (Required)
• Customer Focus: The ability to demonstrate concern and meet external and/or internal customers’ needs. (Required)
• Analytical: The ability to tackle a problem by using a logical, systematic, sequential approach. (Required)
• Technical Expertise: The ability to demonstrate depth of knowledge and skill in a technical area. (Required)
• Forward Thinking: The ability to anticipate the implications and consequences of situations and take appropriate action to be prepared for possible contingencies. (Required)
• Influencing Others: The ability to gain others' support for ideas, proposals, projects, and solutions. (Required)
• Motivational Support: The ability to enhance others' commitment to their work. (Required)
• Inspiring Others: Ability to convey confidence in employees' ability to be successful, especially at (Required)
• Conceptual Thinking: The ability to find effective solutions by taking a holistic, abstract, or theoretical perspective. (Required)
• Strategic Thinking: The ability to analyze the organization's competitive position by considering market and industry trends, existing and potential customers (internal and external), and strengths and weaknesses as compared to competitors. (Required)
• Fostering Innovation: The ability to develop, sponsor, or support the introduction of new and improved method, products, procedures, or technologies. (Required)

• At least 18 years of age
• Legally authorized to work in the United States

Travel:
Travel Required (Yes/No):Yes

DOT Regulated:
DOT Regulated Position (Yes/No):No
Safety Sensitive Position (Yes/No):No

Base Pay Range: $126,800 - $228,900

Corporate Bonus Target: 20%

The pay range above is the general base pay range for a successful candidate in the role. The successful candidate’s actual pay will be based on various factors, such as work location, qualifications, and experience, so the actual starting pay will vary within this range.

At T-Mobile, employees in regular, non-temporary roles are eligible for an annual bonus or periodic sales incentive or bonus, based on their role. Most Corporate employees are eligible for a year-end bonus based on company and/or individual performance and which is set at a percentage of the employee’s eligible earnings in the prior year. Certain positions in Customer Care are eligible for monthly bonuses based on individual and/or team performance. To find the pay range for this role based on hiring location, https://paylookup.t-mobile.com/paylookup?reqID=REQ308395&paradox=1

At T-Mobile, our benefits exemplify the spirit of One Team, Together! A big part of how we care for one another is working to ensure our benefits evolve to meet the needs of our team members. Full and part-time employees have access to the same benefits when eligible. We cover all of the bases, offering medical, dental and vision insurance, a flexible spending account, 401(k), employee stock grants, employee stock purchase plan, paid time off and up to 12 paid holidays - which total about 4 weeks for new full-time employees and about 2.5 weeks for new part-time employees annually - paid parental and family leave, family building benefits, back-up care, enhanced family support, childcare subsidy, tuition assistance, college coaching, short- and long-term disability, voluntary AD&D coverage, voluntary accident coverage, voluntary life insurance, voluntary disability insurance, and voluntary long-term care insurance. We don't stop there - eligible employees can also receive mobile service & home internet discounts, pet insurance, and access to commuter and transit programs! To learn about T-Mobile’s amazing benefits, check out www.t-mobilebenefits.com.

Never stop growing!
As part of the T-Mobile team, you know the Un-carrier doesn’t have a corporate ladder–it’s more like a jungle gym of possibilities! We love helping our employees grow in their careers, because it’s that shared drive to aim high that drives our business and our culture forward. By applying for this career opportunity, you’re living our values while investing in your career growth–and we applaud it. You’re unstoppable!

T-Mobile USA, Inc. is an Equal Opportunity Employer. All decisions concerning the employment relationship will be made without regard to age, race, ethnicity, color, religion, creed, sex, sexual orientation, gender identity or expression, national origin, religious affiliation, marital status, citizenship status, veteran status, the presence of any physical or mental disability, or any other status or characteristic protected by federal, state, or local law. Discrimination, retaliation or harassment based upon any of these factors is wholly inconsistent with how we do business and will not be tolerated.

Talent comes in all forms at the Un-carrier. If you are an individual with a disability and need reasonable accommodation at any point in the application or interview process, please let us know by emailing ApplicantAccommodation@t-mobile.com or calling 1-844-873-9500. Please note, this contact channel is not a means to apply for or inquire about a position and we are unable to respond to non-accommodation related requests.