SRE Automation — Interview Q&A
Q: What's the precise definition of "toil," and why does the definition matter rather than just "tedious manual work"?
A: Toil specifically means work that's manual, repetitive, automatable, tactical (no lasting value once done), and scales linearly with service growth — all five together, not just "feels tedious." The precision matters because it distinguishes real automation candidates from work that just feels unpleasant but isn't actually toil — writing an incident postmortem is manual and can feel tedious, but it has lasting value and isn't repetitive in the same mechanical sense, so it's not a toil-reduction target the way "restart the same service the same way every time this alert fires" is. Misclassifying non-toil as toil leads teams to try automating things that don't actually benefit from it.
Q: How do you decide whether a specific failure mode is a good candidate for automated remediation versus something that should stay a human decision?
A: Three things need to be true: the failure is well-understood (not ambiguous or novel), it's safe to retry without side effects (no risk of something like double-processing a payment), and the correct fix is deterministic — same failure, same correct action, every time. Kubernetes' own OOMKill restart is the clean example: well-understood, safe to retry, deterministic fix. A failure with unclear scope or any risk of harm if retried incorrectly should instead get automated diagnostics (gathering context, notifying a human with everything they need already assembled) while the actual remediation action stays a human decision.
Q: What's the core risk of automated remediation, and how do you design against it?
A: Speed is automation's entire value proposition — reacting in seconds instead of the minutes a human takes to notice and act — but that same speed means a wrong automated action executes just as fast as a right one, before any human has a chance to catch it. The design discipline that addresses this: keep blast radius small and deliberately scoped for any automated action, add a circuit breaker that caps repeated attempts within a time window and escalates to a human once exceeded, and validate input from external triggers (an alert webhook) before using it to target a destructive action, the same way you'd validate user input in an application.
Q: Explain "automating away the signal" — why is this considered one of the more dangerous automation failure modes?
A: It's when automated remediation successfully suppresses the symptom of a real, worsening problem without anyone noticing the underlying issue is still there — an auto-restart that "fixes" a memory leak every time it nears the OOM threshold means the service never crashes and nobody ever gets paged, so the leak can grow worse for months completely invisibly. It's more dangerous than a loudly failing automation because there's no natural signal prompting investigation — the fix is making sure automated actions always emit their own visible metric every time they fire, so a rising remediation-trigger rate for one service becomes its own alertable trend rather than a silent background process.
Q: How would you validate that an automated remediation script actually still works, rather than just trusting it was correct when it was written?
A: The same discipline chaos engineering applies to testing system resilience applies directly to testing automation itself — deliberately trigger the exact failure condition the automation is meant to handle, in a controlled setting, and confirm it actually does the right thing. This matters because automation that was correct when written can silently stop working after an unrelated change — a deprecated API it calls, a renamed label it selects on, an expired credential — and without deliberate, regular validation, the first sign of that breakage is often a real incident where the automation was expected to help and didn't.
Q: Walk through the GitOps deployment pipeline and explain where the human checkpoint sits, and why it's kept there deliberately.
A: Code gets pushed, CI builds and tests it, a pipeline step updates the image reference in a Helm values file and opens a PR — that PR gets human review and merge, and only after merge does a GitOps controller (ArgoCD/Flux) detect the change and apply it to the cluster, continuously reconciling to revert any manual drift afterward. The PR review step is a deliberate, retained human checkpoint: full GitOps automation removes manual execution risk (a mistyped command, an undocumented manual change) but doesn't mean removing human judgment from the actual "should this deploy" decision — automating past that checkpoint too (auto-merge on green CI, no review) is a separate, higher-risk design choice that trades review-time safety for deploy speed, appropriate for some teams but not a default.
Q: A team is reluctant to trust automation after one bad automated action caused real harm. How do you rebuild that trust?
A: Reassurance alone rarely works — what actually rebuilds trust is a specific, rigorous postmortem on the failed action, applying the same standard used for any production incident: exactly what went wrong, and specifically which guardrail (a circuit breaker, input validation, a smaller blast radius) would have prevented or limited it. Then demonstrating that guardrail is actually in place — not just promised — before asking the team to trust automation again in a similar scenario. Trust comes from a verifiable, specific fix the team can check, not from a general assurance that "it won't happen again."