If your dashboard shows the same green every day, it is not monitoring. It is decoration. Monitoring should tell you something new. It should surface anomalies, highlight trends, and alert on conditions that matter. A dashboard that never changes is a dashboard that is never looked at.
The philosophy of monitoring is simple: measure what matters, alert on what is actionable, and ignore everything else. Most monitoring systems fail not because they measure too little, but because they measure too much. The signal is buried in noise. The alert that fires every day is the alert that gets ignored.
THE DEEP DIVE
The Four Signals
There are only four signals that matter: latency (how long does it take?), traffic (how much work is there?), errors (what is failing?), and saturation (how full is the system?). These are the Google SRE four golden signals. Every metric you collect should map to one of these four.
# The four golden signals
#
# Latency: time to service a request
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
#
# Traffic: requests per second
rate(http_requests_total[5m])
#
# Errors: error rate
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])
#
# Saturation: how full is the resource
1 - (node_memory_available_bytes / node_memory_total_bytes)Alert Fatigue
Alert fatigue is the condition where engineers ignore alerts because there are too many. The fix is not "fewer alerts" — it is "better alerts." Every alert must be actionable. If the alert fires and the engineer cannot do anything about it, the alert should not exist.
PRINCIPLES
- Measure what matters. If you cannot explain why a metric is collected, stop collecting it.
- Alert on conditions, not symptoms. A CPU spike is a symptom. A latency increase is a condition.
- Every alert must be actionable. If the engineer cannot do anything, the alert is noise.
- Dashboards should tell a story. If the story is 'everything is green,' the dashboard is useless.
- Monitoring is not observability. Monitoring tells you what is broken. Observability tells you why.
IN PRACTICE
Useful Alert
P99 latency exceeds 500ms for 5 minutes. The engineer can investigate: is it a database slow query? A network issue? A capacity problem? The alert is actionable and specific.
Useless Alert
CPU usage exceeds 80% for 5 minutes. The engineer cannot do anything about this. CPU usage is a symptom, not a cause. The alert fires, the engineer ignores it, and the real problem goes undetected.
LIVE SIGNALS
These items surfaced from the intelligence pipeline at generation time.
- CVE-1999-0095 — The debug command in Sendmail is enabled, allowing attackers to execute commands as root. (NVD / CVE)
- CVE-1999-1471 — Buffer overflow in passwd in BSD based operating systems 4.3 and earlier allows local users to gain root privileges by specifying a long shell or GECOS field. (NVD / CVE)
- CVE-1999-1122 — Vulnerability in restore in SunOS 4.0.3 and earlier allows local users to gain privileges. (NVD / CVE)
- CVE-1999-1506 — Vulnerability in SMI Sendmail 4.0 and earlier, on SunOS up to 4.0.3, allows remote attackers to access user bin. (NVD / CVE)
- CVE-1999-0084 — Certain NFS servers allow users to use mknod to gain privileges by creating a writable kmem device and setting the UID to 0. (NVD / CVE)
ANTIPATTERNS
- Alerting on every metric that exceeds a threshold. Most thresholds are arbitrary.
- Using the same alert for all services. Each service has different failure modes.
- Ignoring the alert history. An alert that fires every day is not an alert — it is a log entry.
- Building dashboards that nobody looks at. If the dashboard is not used, delete it.
CHECKLIST
- All metrics map to one of the four golden signals
- Every alert is actionable
- Alert fatigue is measured (alert acknowledgment rate)
- Dashboards are reviewed monthly
- Monitoring covers the full stack (infrastructure, application, business)
YOUR MOVE
Audit your alerts. Count how many fire per week. For each one, ask: what did the engineer do when it fired? If the answer is 'nothing' or 'acknowledged and moved on,' delete the alert.