Why Monitoring Cannot Be an Afterthought
Production systems fail without warning when you have no monitoring. You find out about the problem when a customer calls. With Prometheus and Grafana, you know about problems 30 minutes before users notice — and you have the data to fix them fast.
How Prometheus and Grafana Work Together
Your App (/metrics endpoint)
↓ (Prometheus scrapes every 30s)
Prometheus (stores time-series data)
↓ (Grafana queries with PromQL)
Grafana (dashboards and alerts)
↓ (Alertmanager routes alerts)
Slack / PagerDuty / Email
Install with Helm
bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
# Install Prometheus + Grafana + Alertmanager + node-exporter
helm install monitoring prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
-f values.yaml
# Access Grafana (default: admin / prom-operator)
kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring
Make Your App Observable
python
from prometheus_client import Counter, Histogram, start_http_server
REQUESTS = Counter('http_requests_total', 'Total HTTP requests', ['method', 'status'])
LATENCY = Histogram('http_request_duration_seconds', 'Request latency')
# Record metrics in your handlers
with LATENCY.time():
response = handle_request()
REQUESTS.labels(method='GET', status='200').inc()
# Expose /metrics on port 8001
start_http_server(8001)
Essential PromQL Queries
promql
# Request rate (per second, last 5 minutes)
rate(http_requests_total[5m])
# Error rate percentage
rate(http_requests_total{status=~"5.."}[5m])
/ rate(http_requests_total[5m]) * 100
# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
# Pod restart count (last 1 hour)
increase(kube_pod_container_status_restarts_total[1h]) > 0
Critical Alerts to Set Up on Day 1
yaml
groups:
- name: critical.rules
rules:
- alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
for: 5m
labels: { severity: critical }
annotations:
summary: "Error rate above 5%"
- alert: NodeDiskFull
expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.10
for: 5m
labels: { severity: critical }
annotations:
summary: "Node disk less than 10% free"
See Prometheus Academy for the complete observability path.
