Software Engineer - Data & Scalability Platform (8-12 Yrs)

Meet the Team The Splunk Agent Resilience team is defining the future of AI resilience. Our team provides scalable, cost-effective evaluation and guardrails that ensure AI agents behave as intended, improving reliability and reducing risks. This unified approach empowers our customers to confidently deploy and manage AI-powered applications with enhanced observability and control. As a Software Engineer on the Data & Scalability Platform team, you will own the data plane end to end: the stores that hold the data — relational, analytical/columnar, object storage, caches, and queues — the streaming and compute components that move and transform it, and the performance tooling that proves how all of it behaves under load. You will make that data plane correct, fast, cost-bounded, and tenant-isolated at production scale, own the capacity and performance model that says when and how it needs to grow, and partner closely with the product engineering teams that build on top of it.

Own and evolve the platform's data stores — relational, analytical/columnar, object storage, and caching — including schema design, migration safety, and retention. Design and operate the streaming and queueing backbone that carries data from ingest to queryable. Build and scale the compute and pipelines that move and transform data, including stream processors, writers, schedulers, and distributed worker fleets. Own end-to-end data performance: query and write latency, indexing and sharding strategy, and hot-path optimization. Own the capacity model for the data platform, along with quotas, rate limiting, and multi-tenant isolation. Drive cost efficiency across the data platform, including cost per unit of telemetry and cost attribution. Build and own performance tooling — load testing, profiling, and benchmarking — used to validate scale and guide optimization. Build observability instrumentation and telemetry pipelines so platform signals are correct, complete, and affordable. Own data operations: backup and restore, retention and deletion, replication, migration and backfill, and data-quality signals. Design, develop, test, and maintain production services and internal tooling in Python and/or Go, using secure coding practices and automated tests. Debug and resolve complex production issues across data stores, queues, and services, and contribute to monitoring, on-call support, and root-cause analysis. Contribute to platform architecture direction and act as a technical resource and mentor through design reviews, code reviews, and documentation.

Bachelor's degree with 7+ years of related experience, or Master's degree with 4+ years, or PhD with 1+ year of related experience, in Computer Science, Software Engineering, or a related field. Strong backend software engineering experience building, operating, and delivering highly scalable, reliable, production-grade data-intensive services and platforms. Proven experience with large-scale distributed systems, including designing and solving complex problems of scalability, availability, performance, reliability, and fault tolerance. Hands-on experience operating and tuning both an OLTP database (e.g., PostgreSQL, MySQL) and an analytical, columnar, or time-series store (e.g., ClickHouse, Druid, BigQuery, Snowflake) — schema design, indexing, query tuning, and migration safety. Production experience with streaming or queueing systems (e.g., Kafka, RabbitMQ, Pulsar, Kinesis) — partitioning, consumer-group semantics, delivery guarantees, and backlog and dead-letter handling. Experience scaling data-processing compute — queue consumers, async task workers, or streaming jobs — including concurrency tuning, batching, backpressure, and throughput behavior under sustained load. Strong proficiency in Python and solid experience with at least one additional backend programming language such as Go, Java, C++, or similar. Demonstrated