Senior System Engineering - Engineering Operations

Job Description:

Join AT&T and reimagine the communications and technologies that connect the world. Our Consumer Technology experience team is delivering innovative and reliable technology solutions to power differentiated, simplified customer experiences. Bring your bold ideas and fearless risk-taking to redefine connectivity and transform how the world shares stories and experiences that matter. When you step into a career with AT&T, you won’t just imagine the future-you’ll create it.

As a Senior System Engineer with the Engineering Operations team, the candidate will have an SRE mindset, have a strong sense of ownership, a relentless drive and passion to detect and resolve problems.  As functionality gets operationalized and adopted across the enterprise, he/she will be entrusted to help maintain a healthy production ecosystem that is responsive, user friendly and reliable to our user community. 

The role will partner with various stakeholders including product development teams, business leaders, end-user communities to ensure production applications are designed for Non-Functional Requirements. This is an exciting, hands-on, technical System Engineering position responsible for Site Reliability Engineering aspects such as developing a deep functional and technical knowledge-base of the application, creation of run books, developing observability features of the application in terms of alerts, monitoring and dashboards that enable proactive incident and problem detection, triaging of the incidents and conducting blameless post-mortems (after action reviews). 

Additionally, the person is responsible for Problem Management Engineering functions such as proactive problem detection, intake of problems from end users, functional  triage, customer impact assessments, preliminary technical analysis, logging the issues as defects and working with product/dev/quality teams in getting resolution.

Seeking a candidate with 5+ years of supporting large scale applications in production with an Engineering approach (SRE) – such as Java EE apps, ERP or CRM apps in an operations capacity. You will be responsible for operations support aspects of mission critical applications that are used by front office and retail agents for selling, service and support functions.

Candidates possess a strong Engineering background and are well-experienced in Observability tool-sets, and leverage them effectively to create alerts, dashboards. Additionally, the candidates are experienced in constantly looking out for automation opportunities and eliminate manual work.

Key Roles and Responsibilities:

•    Incident Management: Lead the response to production issues, ranging from identifying and troubleshooting problems to implementing immediate fixes. Ensure minimal downtime and adherence to service level agreements (SLAs).
•    Observability: Build alerting, monitoring and dashboards that identify problems proactively; 
•    Problem Solving: Utilize strong analytical, technical and functional skills to diagnose and resolve complex issues within production environments with a focus on immediate impact mitigation; work with dev teams to implement long-term solutions to prevent recurrence of incidents.
•    RunBooks and Documentation: Create and maintain comprehensive documentation for system architecture, configuration, deployment procedures, and troubleshooting guides
•    Automation: Develop and maintain scripts and automation tools to streamline operations, deployment processes, and repetitive tasks. Focus on automating recovery processes and routine maintenance tasks to improve system reliability and efficiency
•    Non Functional Requirements : Working with development teams, identify and provide the non-functional requirements and acceptance criteria during design and development, and ensure that these are met prior to moving the features to production
•    Performance Optimization: Monitor application performance using APM (Application Performance Management) tools such as Dynatrace, App Dynamics and ELK. Identify bottlenecks and work with dev teams to optimize the performance of applications through code improvements, configuration tuning, and resource optimization.
•    Adopt SRE best practices: Work with dev teams to define Non-Functional Requirements such as reliability, performance, scalability, application logging for observability, etc. Define SLI/SLOs, Error Budgets, Automation focus 
•    Conduct Blameless Postmortems / After Action Reviews: Work with dev/architect/quality engineering teams to identify and document patterns of failures as lessons learnt from incidents and follow up to implement the remediations to make the application resilient
•    Capacity Planning: Monitor system usage patterns and perform capacity planning to ensure scalability and reliability of  applications and services.
•    Security: Participate in security assessments and implement security best practices to safeguard applications and data. Respond promptly to security incidents and vulnerabilities
•    Release Management: Work with Release Management related to upcoming changes to production to identify risks and mitigate them. Collaborate with development teams to manage and support application releases and deployments. Ensure changes are rolled out in a controlled manner with minimal impact on production services.
•    Problem Management Engineering Expertise:  Proactive problem detection, trend and pattern analysis, assessment of impact of problems, functional analysis of problems. Management of Escalated issues, tracking and driving prompt resolution.
•    Communication: Provide metrics and status reports and review with leadership and stakeholder communities; establish processes surrounding metrics gathering, reporting and communication; Provide prompt visibility and status of escalated issues, incidents and outages to leadership, business partners and other key stakeholders; Strong verbal and written communication skills
•    Knowledge Transfer: Work closely with Product Development teams to ensure Knowledge Transfer related to changes to the system well in advance of change getting operationalized
•    On-call 24x7 support for agent facing applications – Home Grown J2EE apps as well as SaaS Platform apps -  Salesforce, Salesforce Marketing Cloud and Mulesoft

Qualifications:

• Bachelor’s degree in computer science, Information Technology, or related field.

• 5+ Years of technical engineering experience in production support, system monitoring, or SRE roles within large scale web application environments.

• Engineering experience – architecting and developing web applications

• Strong experience in Observability tools such as Dynatrace, App Dynamics, Splunk, ELK, Mulesoft AnyPoint, using them to create alerts, dashboards, reports, synthetic monitoring, etc.

• Strong problem-solving skills and experience troubleshooting complex issues.

• Understanding and working experience with integration technologies and API Gateways, MuleSoft, WebLogic, etc 

• Strong Proficiency in Object Oriented Programming Languages - Java, J2EE technologies, Javascript, and frameworks such as Spring

• Experience with automation tools and scripting languages (Python, Shell).

• Familiarity with containerization (Docker, Kubernetes) and cloud services (Azure).

• Knowledge of DevOps practices and tools (CI/CD pipelines, Git, Jenkins).

• Understanding of network protocols, load balancing, and security principles.

• Experience with database SQL queries.

• Ability to build Linux shell scripts on demand

• Excellent communication and teamwork skills.

• 5+ Years of technical hands-on, problem-solving experience in Engineering Operations / production support responsibilities with a Site Reliability Engineering (SRE) focus in a J2EE environment, ensuring the stability, performance, and continuous improvement of web applications and services.

Desirable Skills:
•    Certifications in Java, cloud technologies, or SRE methodologies.
•    Experience in Salesforce Sales,  Service and Marketing Clouds
•    Experience within high tech, software and/or wireless/telecom industry
•    A foundational understanding of Artificial Intelligence (AI) and Machine Learning (ML) principles and technologies is highly valued. 

This role requires a blend of technical expertise, agility, and a proactive approach to maintaining high levels of system performance and reliability. The ideal candidate will thrive in fast-paced environments and be committed to continuous learning and improvement.

Weekly Hours:

Time Type:

Location:

Salary Range:

It is the policy of AT&T to provide equal employment opportunity (EEO) to all persons regardless of age, color, national origin, citizenship status, physical or mental disability, race, religion, creed, gender, sex, sexual orientation, gender identity and/or expression, genetic information, marital status, status with regard to public assistance, veteran status, or any other characteristic protected by federal, state or local law. In addition, AT&T will provide reasonable accommodations for qualified individuals with disabilities.