On Zenn Trending, the article The First Mistake a Former On-Premises Engineer Made in AWS Operations Monitoring Design — "Defining Normal" Came First made the rankings. It highlights the realization that starting off by setting threshold alerts on CPU, memory, and disk in fact puts the cart before the horse. This is not just about monitoring tools; it addresses a common pitfall in operational design: if you add alerts without defining what constitutes normal operation, alerts fire constantly, everyone starts ignoring them (alert fatigue), and you end up missing real incidents that demand attention.
In custom development, scenarios like "operations starting post-delivery without monitoring, where teams only discover failures when users report them" or "alerts firing continuously so that no one ever unmutes them" are all too common. As partners supporting system development for clients, we view this not as a matter of "whether a monitoring tool can be installed," but as an architectural challenge of "incorporating into design the ability to define normal, ensure only meaningful alerts fire, clarify who responds and how, and successfully hand over a system running stably in production." Connecting with our previous coverage of performance visibility in Lessons from Pinterest's "CPU Zombies" for Custom Development Performance SRE Audits (GH Media), processing reliability in Robust Backend Custom Development with Postgres and SQLite Workflows (GH Media), and resource optimization in Backend Memory and Cost Optimization for Custom Development (GH Media), this article structures our "Operational Handover Monitoring Design Support" into a custom development package.
Why "defining normal" comes first
| Dimension | Threshold-first (error-prone) | Normal-definition-first (2026) |
|---|---|---|
| Starting point | Trigger alert at 80% CPU | First articulate the normal state |
| Metrics | Focus solely on resources | Focus on user impact (SLIs) |
| Alerting | Ignored due to constant noise | Only meaningful alerts fire |
| Response | Unclear who handles it | Clear escalation paths |
| Missed incidents | Critical failures overlooked | Anomalies reliably detected |
| Deliverable | Alert fatigue | Quiet and dependable monitoring |
In short, "having monitoring installed" and "reliably detecting and resolving incidents" are two completely different things. Even in custom development, "defining normal via SLIs/SLOs, alerting only on deviations, establishing incident workflows, and handing over the system" has become a foundational quality baseline. This enables us to guarantee monitoring where alerts represent genuine calls to action as a core deliverable.
Three non-negotiable principles in monitoring design
Principle 1: Measure "user impact" before looking at resources
Even if CPU utilization is high, it is not an incident if users are unaffected. In custom development, we place SLIs such as availability, latency, and error rate at the core, and judge anomalies based on whether users are experiencing issues.
Principle 2: Establish a normal range (baseline) before alerting on deviations
Static thresholds fail across varying times of day and seasonal shifts. In custom development, we measure steady-state baselines and define anomalies as deviations from them, which reduces both false alarms and missed incidents.
Principle 3: Couple alerts with assigned responders and procedures
An alert without an assigned responder or runbook is merely noise. In custom development, we link response procedures and escalation targets to every alert, handing over systems in a state where teams can take immediate action when an alert fires.
The five phases of "Operational Handover Monitoring Design Support" for custom development
Phase 1: Current state audit (1 week)
- Inventorying existing alerts and evaluating noise ratios
- Identifying monitoring blind spots (unmeasured areas)
- Reviewing past incidents and missed detections
- Interviewing stakeholders regarding operational response structures
Phase 2: Normal definition and design (1–2 weeks)
- Defining SLIs/SLOs for critical user journeys
- Measuring steady-state baselines
- Designing alert conditions (deviations and duration)
- Designing escalation and notification channels
Phase 3: Implementation (2–3 weeks)
- Setting up metrics, logs, and trace collection
- Implementing dashboards and alerts
- Attaching runbooks to each alert
- Pruning and consolidating noisy alerts
Phase 4: Training and handover (1 week)
- Dry runs of alert triggering
- Walkthroughs of incident response workflows
- Establishing on-call rotations and duty rosters
- Handing over operational documentation
Phase 5: Ongoing operations (continuous)
- Periodic reviews of SLO achievement rates
- Remediating false positives and missed detections
- Expanding monitoring alongside new feature additions
Standard technology stack set for custom development
| Layer | Recommended technology | Alternative |
|---|---|---|
| Metrics | SLIs/SLOs (availability, latency, errors) | Resources only |
| Data collection | Metrics + logs + traces | Metrics only |
| Visualization | Dashboards per service | Graphs per server |
| Alerting | Deviation + duration conditions | Simple thresholds |
| Notifications | Routing by severity | All sent to the same destination |
| Response | Runbooks + on-call rotations | Verbal and person-dependent |
Which projects need this and which do not
| Projects requiring this | Low-priority projects |
|---|---|
| Mission-critical systems in production | Disposable environments for verification |
| Outages directly impact users | Downtime has negligible impact |
| Alert fatigue has set in | Moving past the stage of having no monitoring at all |
| Handing operations over to an internal team | Developers can monitor systems continuously |
| SLAs are promised to clients | No availability commitments exist |
Six clauses to include in client contracts
| Clause | Details | What the client should verify |
|---|---|---|
| Target scope | Monitored services | User flows targeted by SLIs |
| SLO | Target standards | Achievement criteria and exemptions |
| Alerting | Conditions and notification channels | Coverage hours |
| Response organization | On-call / runbooks | In-house vs. outsourced division of responsibilities |
| Handover | Dashboards / procedures | Maintenance framework |
| Ongoing maintenance | Reviews / remediation | Operating costs |
Client ROI estimate (assuming enterprise systems)
| Item | Threshold-first | Normal-definition-first | Difference |
|---|---|---|---|
| Detection | Discovered through user reports | Detected automatically in advance | Reduced downtime |
| Alert fatigue | Ignored | Addressed when fired | Suppression of missed incidents |
| Supported | Ad hoc and person-dependent | Immediate response via runbooks | Shortened recovery time |
| Trust | Eroded by frequent outages | Stable operation | Preserved customer satisfaction |
| Annual benefit | — | — | Reduced downtime + sustained trust |
Even factoring in upfront costs, accelerating the detection of critical outages and shortening recovery time fully justifies the investment by avoiding lost business opportunities and damaged trust.
Five common pitfalls
Pitfall 1: Setting alerts without defining normal operation
Alerts will fire constantly and everyone will ignore them. Define normal operation first using SLIs.
Pitfall 2: Looking solely at resource metrics
You will fail to notice when users encounter issues. Keep user impact at the core.
Pitfall 3: Relying on static thresholds
Diurnal fluctuations will trigger false alarms. Judge anomalies by deviations from baseline.
Pitfall 4: Operating without runbooks
When alerts trigger, no one will know what to do. Pair alerts with explicit procedures and responders.
Pitfall 5: Treating deployment as the finish line
False alarms and missed issues will accumulate. Continuously refine monitoring through scheduled reviews.
90-day action plan
| Week | Action |
|---|---|
| Week 1 | Alert inventory + noise ratio assessment |
| Week 2〜3 | SLI/SLO definition + baseline measurement |
| Week 4〜6 | Implementation + runbook linking + noise reduction |
| Week 7 | Alert drill + on-call structure setup |
| Week 8〜13 | SLO reviews + operational start of continuous remediation |
Summary — From "setting alerts for now" to "defining normal before handover"
Monitoring starts not by configuring static thresholds, but by "defining normal and turning only deviations from it into actionable alerts." In supporting system development for clients, our new core offering is "Operational Handover Monitoring Design Support," where we define normal operation through SLIs/SLOs, organize response workflows, and hand over systems running smoothly in operations to deliver monitoring where every alert warrants action. For performance visibility, please read Lessons from Pinterest's "CPU Zombies" for Custom Development Performance SRE Audits (GH Media), and for deployment workflow improvements, see CI/CD Improvements for Shortening Deployment Lead Time in Custom Development (GH Media).
If you are dealing with challenges such as "alerts firing so frequently that everyone ignores them," "learning about outages from user reports," or "wanting to hand over operations with total confidence," please feel free to reach out via our inquiry form.
Sources
- The First Mistake a Former On-Premises Engineer Made in AWS Operations Monitoring Design — "Defining Normal" Came First (Zenn Trending 2026-06-04)
- Lessons from Pinterest's "CPU Zombies" for Custom Development Performance SRE Audits (GH Media)
- Robust Workflow Backend Services with Postgres / SQLite for Clients (GH Media)
- Backend Memory and Cost Optimization for Client Projects (GH Media)
- CI/CD Improvements for Shortening Deployment Lead Time in Custom Development (GH Media)
