Observability Engineer - Assistant Vice President

The Site Reliability Engineer (SRE) - Assistant Vice President is a technical professional responsible for the hands-on execution, technical implementation, and deployment of SRE and observability principles in a complex, critical, and large-scale multi-disciplinary environment. In this role, you will apply a deep understanding of multiple technology domains to enhance the reliability, performance, and efficiency of our Applications and Services. As a key technical contributor, you will drive the migration of applications from existing monitoring tools (Geneos ITRS, Prometheus, ELK, Splunk, AppDynamics, etc.) to Google Cloud Observability (GCO) and Grafana using OpenTelemetry (OTel) as the instrumentation standard. You will act as a hands-on technical authority, authoring reusable deployment solutions, configuring telemetry collectors, and providing direct technical onboarding support to application teams. We are seeking a passionate and experienced SRE to join our Production Management team. In this role, you will be instrumental in executing our strategy for end-to-end observability and resiliency, collaborating across the organization to ensure our services are stable, scalable, and fault-tolerant.

Culture & Strategy Foster Technical Excellence: Foster a culture of transparency, innovation, and technical excellence that encourages continuous improvement and automation. Collaborative Enablement: Partner with development and SRE teams to drive the adoption of OpenTelemetry (OTel) and Google Cloud Observability (GCO) and Grafana standards. Regulatory Compliance: Operate effectively within a highly regulated environment, ensuring all observability and deployment solutions comply with relevant enterprise standards and security requirements. Resiliency & Recovery Resiliency Patterns: Partner with development teams to leverage cloud-native services and implement established resiliency patterns to enhance application reliability. Recovery Testing Support: Support and participate in advanced recovery testing, including Production Swing Tests, Data Recovery Tests, and chaos engineering practices. Automation Drive: Drive the adoption and development of automation solutions (such as Ansible playbooks and Terraform) to minimize recovery time and streamline infrastructure deployments. Observability & Migration Execution Legacy Tool Migration: Actively execute the migration of applications from legacy monitoring tools (Geneos ITRS, GEM, ELK, Splunk, AppDynamics) to GCO and Grafana. Telemetry Configuration: Develop, test, and maintain OpenTelemetry Collector (or Grafana Alloy) configurations for metrics, logs, and traces. Reusable Deployment Solutions: Author and maintain reusable deployment solutions, including Helm charts and Ansible playbooks for deploying observability agents across OpenShift/Kubernetes and VM environments. Application Instrumentation: Partner with application development teams to effectively instrument applications using OpenTelemetry (OTel) standards, providing deep insights into system health and performance. GCO Feature Implementation: Aid with the design of GCO dashboards, log-based metrics, alerts, and SLO/SLI tracking to provide comprehensive visibility. Essential Skills 5+ Years of Experience: Deep understanding of SRE concepts, including SLOs, SLIs, error budgets, and toil reduction. Observability Instrumentation: Hands-on experience with OpenTelemetry (or Grafana Alloy) collectors. Container Orchestration: Proficiency in deploying, managing, and troubleshooting applications on OpenShift/Kubernetes. Helm Chart Management: Experience creating, modifying, and managing Helm charts for application deployment. Infrastructure as Code (IaC): Experience with IaC, configuration management, and automation tools (e.g., Ansible, Terraform). Modern Observability Tools: Hands-on experience with modern observability tools (e.g., Prometheus, Grafana, Loki, Mimir, Tempo, AppDynamics, Splunk, or Google