Lead Engineer - Observability Platform

Job description

We’re building a world of health around every individual — shaping a more connected, convenient and compassionate health experience. At CVS Health®, you’ll be surrounded by passionate colleagues who care deeply, innovate with purpose, hold ourselves accountable and prioritize safety and quality in everything we do. Join us and be part of something bigger – helping to simplify health care one person, one family and one community at a time. POSITION SUMMARY Join CVS Health Enterprise Technology and put your engineering skills to use shaping the future of developer platforms for software engineers at Fortune 6 scale. The CVS Observability Platform gives engineers at CVS frictionless access to application instrumentation no matter where their applications run. As a Lead Cloud Engineer on the Observability Platform team, you will be architecting and scaling the data pipelines for billions of logs, metrics, and traces produced by thousands of workloads spanning multiple public cloud providers and datacenters. You will leverage open standards, open source technologies, and in-house custom software to extend and create new platform features. You will work closely with the SRE team and engage with application teams across the company to inform the direction of the platform as it grows. Successful candidates will combine deep technical acumen with a willingness to collaborate and mentor teammates across teams. A keen understanding of distributed systems and operational thinking will help you excel in the role. KEY RESPONSIBILITIES • Design, build, and operate core observability platform services using Go, Python, Java (Spring Boot) • Lead enterprise-wide adoption of OpenTelemetry, including client libraries, semantic conventions, instrumentation patterns, and Collector/agent strategy • Architect and scale high‑throughput, fault‑tolerant telemetry pipelines (logs, metrics, traces) with a focus on performance, reliability, and cost efficiency • Develop self-service observability capabilities that simplify onboarding, troubleshooting, and adoption for application teams • Implement end-to-end monitoring of the observability platform itself, defining SLOs, health checks, and alerting • Collaborate with SRE, Platform, and Cloud teams to establish reliability standards, error budgets, and incident response practices • Participate in on‑call rotations and lead incident mitigation, root‑cause analysis, and post‑incident reviews • Automate operational workflows and eliminate manual toil through tooling, CI/CD enhancements, and platform automation • Ensure secure telemetry pipelines through mTLS, secrets management, and zero‑trust design patterns • Produce and maintain high-quality technical documentation, standards, and best practices • Engage with internal engineering teams to gather requirements, influence roadmap prioritization, and deliver platform improvements • Provide technical leadership through mentorship, design reviews, architectural guidance, and cross‑team collaboration with principal engineers and engineering leadership REQUIRED QUALIFICATIONS • 7+ years of experience in Software Engineering • 5+ years of experience with observability practices, including SLIs/SLOs/SLAs, alerting, and incident management • 5+ years building production-grade backend services in Go and/or Java • 5+ years implementing and operating OpenTelemetry, including OTLP, semantic conventions, and instrumentation patterns • 5+ years with cloud-native and containerized platforms (Docker, Kubernetes, Argo CD) • 5+ years working with public cloud platforms (AWS, GCP, or Azure) • 3+ years designing and scaling distributed, high‑volume data pipelines • 3+ years of experience with Helmcharts, Kustomize etc. • 3+ years working with Grafana OSS or comparable observability backends (e.g., Grafana, Loki, Tempo, Mimir) • 3+ years of experience with Infrastructure as Code tools such as Terraform or CloudFormation • 3+ years with relational databases (PostgreSQL, MySQL) PREFERRED QUALIFICATIONSExperience with service meshes and networking technologies such as Envoy and IstioExperience integrating or operating commercial observability platforms (Datadog, New Relic, AppDynamics, etc.)Experience with On-Call Scheduling tools such OpsGenie, PagerDuty, GoAlert etc.Experience with streaming and data platforms such as Kafka, Pulsar, or similar technologiesFamiliarity with time-series, NoSQL, or analytical databases (ClickHouse, Bigtable, Cassandra, etc.)Experience with cost optimization and capacity planning for large-scale telemetry systemsExperience with chaos engineering, resiliency testing, or fault injectionBackground in security‑aware platform design, including secure service‑to‑service communicationExperience mentoring senior engineers and influencing platform standards across organizationsStrong operational experience supporting 24x7 production systems, including on‑call responsibilitiesStrong technical communication and cross‑team collaboration skillsExperience operating in regulated or compliance‑heavy environments (e.g., healthcare, finance) EDUCATIONBachelor’s degree from accredited university or equivalent work experience (HS diploma + 4 years relevant experience)BUSINESS OVERVIEW Bring your heart to CVS Health Every one of us at CVS Health shares a single, clear purpose: Bringing our heart to every moment of your health. This purpose guides our commitment to deliver enhanced human-centric health care for a rapidly changing world. Anchored in our brand - with heart at its center - our purpose sends a personal message that how we deliver our services is just as important as what we deliver. Our Heart At Work Behaviors™ support this purpose. We want everyone who works at CVS Health to feel empowered by the role they play in transforming our culture and accelerating our ability to innovate and deliver solutions to make health care more personal, convenient and affordable. We strive to promote and sustain a culture of diversity, incl