SRE Automation — Troubleshooting
Automation fired repeatedly and made an incident worse
If an automated remediation action executed many times in a short window without fixing the underlying problem, the missing piece is almost always a circuit breaker (covered on the Advanced tab) — automation without a hard limit on repeated attempts will keep re-triggering against a failure it fundamentally can't fix, and each attempt can itself add load or side effects on top of the original problem (a repeated restart storm causing the same thundering-herd pattern covered in Capacity Planning, for instance). The fix going forward is adding an attempt-count-within-a-time-window limit that escalates to a human once exceeded, rather than retrying indefinitely — but immediately, the priority is manually intervening to stop the automation loop first, then investigating why the underlying failure wasn't actually addressed by the automated action.
Auto-remediation is masking a real, worsening problem
If a metric (like automated-restart frequency for one service) has been quietly climbing for weeks with nobody noticing, the root cause is almost always that the automation's own actions weren't being tracked as a first-class signal — an auto-restart that "works" every time, from an alerting standpoint, produces no page and no visible symptom, which is exactly why a slowly-worsening underlying issue (a memory leak, a resource that isn't being released correctly) can go unnoticed far longer than it would without automation quietly compensating for it. The fix is retroactive and preventive both: audit current automation for actions with no visibility of their own trigger rate, and add that visibility (a dashboard panel, an alertable trend on remediation-action count) going forward — automated remediation should always be at least as observable as the failure it's remediating.
An automated action ran against the wrong target
If an automation script acted against a service, namespace, or resource it wasn't intended to touch, the most common cause is an overly broad selector or an unvalidated input from the triggering alert — a label match that's broader than intended (matching every service with a shared label, not just the specific one an alert referred to), or an alert payload field trusted without validation before being used to target an action. The fix is treating any input from an external trigger (an alert webhook, in particular) with the same validation discipline as user input in an application — confirm the target resource actually exists and matches the expected scope before taking any destructive or state-changing action against it, rather than trusting the alert payload's fields directly.
Automation stopped working after an unrelated change
If automation that was working correctly silently stops functioning — often discovered only when it should have helped during a real incident and didn't — the cause is typically that something it depended on changed without the automation being updated in step: an API it calls got deprecated or changed its response shape, a Kubernetes label/annotation convention it relies on for targeting got renamed, or a credential/permission it needs expired or was rotated without updating the automation's own access. This is the direct argument for the chaos-validation practice covered on the Advanced tab — regularly testing automation against its actual triggering condition, not just trusting it silently, is what catches this class of drift before a real incident is the first time anyone notices the automation quietly broke.
Team distrust after one bad automated action
A single automation mistake — even a well-designed system will eventually have one — can produce lasting reluctance to trust or expand automation further, which is a reasonable reaction but worth actively addressing rather than letting it quietly stall further automation work. The concrete fix is a genuine, specific postmortem on the failed action (same rigor as any other production incident postmortem): what specifically went wrong, what guardrail (circuit breaker, scope validation, escalation threshold) would have prevented or limited it, and demonstrating that guardrail is actually now in place before asking the team to trust automation again in a similar scenario. Trying to rebuild trust through reassurance alone, without a concrete fix the team can verify, rarely works as well as showing the specific gap has actually been closed.