SynfraCore
Synfracore
Start Learning
Navigation

Academies

Platform

RoadmapsLabsCertificationsInterviewPYQsAI AssistantCareer
Start Learning Free🗺️ Learning Roadmaps

SRE AutomationCheatsheets

Quick reference — commands, syntax, and patterns

✍️
Written by senior engineers. Reviewed for technical accuracy.· Updated 2025 · SynfraCore SRE Automation Team
Expert Content

SRE Automation — Quick Reference

Toil criteria — is this actually toil?

Manual         — requires a human to perform it
Repetitive     — done the same way, repeatedly
Automatable    — a script/system could do it deterministically
Tactical       — no lasting value once done (vs. e.g. a design doc)
Scales with growth — grows linearly with service/user count

All five present = toil, and a real automation candidate.
Missing one or more = probably not toil, automating it may not help.

Automation candidate checklist

[ ] Failure mode is well-understood (not ambiguous/novel)
[ ] Safe to retry without side effects (no risk of duplicate action,
    e.g. double-charging, double-sending)
[ ] Fix is deterministic — same input, same correct action every time
[ ] Blast radius of an automated action is small and well-scoped
[ ] A circuit breaker / max-attempts limit is in place
[ ] The action itself emits a visible signal (metric/log) every time it fires
[ ] An escalation path exists for when automation can't resolve it

If any of the first three are false, that failure mode is a candidate for better alerting/diagnostics automation (gather context, notify a human) rather than automated remediation.

Kubernetes self-healing quick reference

yaml
# Basic restart-on-failure
restartPolicy: Always

# Liveness probe — should check actual functional health, not just "is alive"
livenessProbe:
  httpGet: { path: /health/deep, port: 8080 }
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

# HPA — capacity auto-scaling as automation
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }

GitOps pipeline, at a glance

git push → CI build/test → CI updates Helm values → PR (human review) →
merge → ArgoCD/Flux detects change → applies to cluster →
continuous reconciliation (reverts manual drift automatically)

Key metrics to track

MetricTargetSignals a problem if...

|---|---|---|

Toil percentage< 50% of working hoursStays high despite automation investment
MTTR (auto-remediated)< 5 minutesAutomated path is slower than expected, or looping
Runbook automation coverage> 80% of P2+Large gap between "runbook exists" and "runbook is automated"
Manual intervention rate↓ ~20%/quarterFlat or rising — automation isn't reducing real manual work
Auto-remediation trigger rate (per service)Stable/lowRising trend — may indicate a worsening masked problem

Common tools by function

FunctionTool

|---|---|

System config / deployment automationAnsible
Custom auto-remediation scriptsPython + Kubernetes API client
Event-driven autoscaling beyond CPU/memKEDA
Alert-to-automation triggerPagerDuty/Opsgenie webhooks
Validating automation under real failureAWS FIS, LitmusChaos, Chaos Mesh
Share:
Join our Community
Daily tips, job alerts, interview help — join engineers learning together
Also Worth Exploring
← Back to all SRE Automation modules
CertificationNotes