Your Alerts Are Training People to Ignore Them: Fixing

If every ping is urgent, nothing is: the conditioning loop you built

It is 2:17 AM. The on-call phone lights up: “CPU high on api pods.” You squint, open the chart, and by the time it loads the line is already drifting back down. No customer reports. No errors. No action. You acknowledge, roll over, and promise you’ll “fix alerting” later.

Tomorrow, it pings again. Next week, your brain has learned the lesson you taught it: most alerts are just loud weather reports.

That is alert fatigue in support teams in plain language. Repeated interrupts that don’t require meaningful human action train operators to delay, skim, or ignore future alerts—including the ones that actually matter. This isn’t a character flaw. It’s conditioning. If your system pages for things that self-resolve, pages without saying who owns it, or pages without telling someone what to do next, your team will adapt in the only sane way available.

A classic “teaches ignoring” alert is the threshold breach that reliably recovers on its own: “queue depth above 5,000 for 5 minutes” during every morning traffic ramp, or “error rate above 1%” during a known downstream deploy window. The alert fires, Slack fills up, nobody acts, and nothing happens. Technically, the monitoring “worked.” Practically, it trained everyone that the next alert is probably not worth standing up for.

The fix starts with a standard simple enough to enforce: an interrupt must map to an owner and a next action.

Owner means one accountable role or team, not “platform + app + whoever is awake.”
Next action means a real decision in the next few minutes: fail over, rollback, throttle, post a status update, run verification step A, escalate to team B.

If you can’t name the owner and the next action, you don’t have signal yet. You have data. Data can be valuable—but it should not wake people up.

Operators aren’t immune to noisy streams; they just build faster filters. They learn which services cry wolf, which alerts come with no context, and which channels are safe to mentally mute. Then the filter becomes culture: new hires get told “ignore those,” and support starts learning about outages from customer tickets (the least flattering monitoring strategy imaginable).

A fast self-check: when an alert fires, is it interrupting a human for a decision, or interrupting a human to provide data? If the honest answer is “data,” route it as a ticket or a log entry, and let someone review it during business hours with coffee and dignity.

Audit the last 2 weeks: which alerts caused action (and which just proved the system exists)?

Most teams try to reduce noise by tweaking a few thresholds and calling it done. That’s like trying to lose weight by switching the font on your nutrition label. Instead, do a short audit that shows what’s actually happening.

Pick the last two weeks of alert history and sample it.

If volume is huge, pull the last ~200 alerts.
If volume is modest, grab them all.

You’re not searching for perfection. You’re searching for the patterns that create alert fatigue in support teams.

Tag each alert with one outcome:

Action taken: a human changed something or initiated a clear escalation.
Acknowledged, no action: someone looked and chose to watch, wait, or ignore.
Auto-resolved: it cleared without human involvement.
Duplicate: same underlying event, different symptom.
Misrouted: wrong team/person/channel.

If you add one extra tag, make it Needs context—alerts where responders couldn’t tell impact or scope without hunting.

Now label each alert actionable or vanity. The rule is blunt on purpose: if nobody can name a next action beyond “look at a dashboard,” it’s vanity. It might be a useful metric. It’s not an actionable support/on-call signal.

Concrete examples:

“CPU above 80% for 5 minutes on service A” is usually vanity. CPU is not impact, and the next action is unclear.
“Checkout error rate above 2% for 10 minutes, top region us-east, started 14:02, ~120 customers affected in last 5 minutes” is actionable. Someone can decide whether to page the owner, shift traffic, or declare an incident.

Two actionability tests keep this honest:

Could someone act in 2 minutes? Not solve it—act. If you need 30 minutes of archaeology to understand the situation, it’s noise.
What decision did this force? If the answer is “none,” you interrupted someone for data.

Here’s what “context turns noise into signal” looks like.

Before: “Payment latency high.” No scope, no time window in human terms, no impact, no owner. The practical outcome is “ack and wait,” which is code for “I hope it goes away.”

After: “Payment p95 latency above 2s for 15 minutes, 3 regions affected, customer timeouts rising, recent deploy to payments at 10:40, owner payments on-call, first step verify rollback readiness.” Same underlying metric. Completely different triage behavior.

This is where teams get burned: they treat “acknowledged” as success. Acknowledged means “a human saw it.” The goal is meaningful action when action is required—and quiet when it isn’t.

Once you’ve tagged the set, sort by frequency. Your top noise sources are almost always boring:

Ambiguous triggers (“high CPU,” “pods restarting,” “error spike”) with no impact framing.
Missing context (no service, no owner, no known benign states, no recent change hint).
Threshold churn (flappy alerts during normal variance).
Duplicate symptom alerts (one root cause creates a storm).
Misrouting (ownership map is outdated, or the channel is wrong).

One practical tip: don’t start by muting the loudest. Start by eliminating the no-action alerts that happen most often. Those are the repeated “conditioning events” training people to ignore.

Track one metric immediately with no new tools:

Actionable rate = (alerts that led to action) / (alerts that reached a human) over the last two weeks.

If it’s under ~30%, you have a trust problem. Under ~10%, you’ve built a very efficient notification generator.

A tradeoff worth saying out loud: you can reduce volume by suppressing alerts until charts look pretty—and make outages harder to detect. The audit is meant to identify what to redesign, not what to hide.

For additional context on noisy alerting as a reliability risk, these are worth a read: [1] and [2]

Page, ticket, or log? The triage workflow that forces an owner and a next action

Assignment strategy	Best for	Advantages	Risks	Recommended when
Page (Critical Alert)	Customer-facing impact, immediate data loss, or security breach	Fastest response, dedicated owner, clear escalation path	High alert fatigue if overused, burnout, ignored pages	Service is down, critical data compromised, or major customer impact. Requires an explicit 'next action' (runbook step, verification step…
Ticket (High-Priority Incident)	Internal-only degradation, potential future customer impact, or non-critical data issues	Structured tracking, clear ownership, allows for planned work	Can be deprioritized, 'ticket rot' if not actively managed, no immediate action	Performance degradation, resource exhaustion, or non-critical system errors. Requires an explicit 'next action' (investigation, fix, or d…
Log (Informational/Audit)	Anomalies, trends, audit trails, or non-actionable observations	Low overhead, historical data for debugging, no interruption	Important signals can be missed, no immediate owner, no next action	Unusual but non-critical events, successful operations, or data for future analysis. No immediate action required, but clear criteria for…
Guardrail: No 'Informational' Pages	Protecting on-call from unnecessary interruptions	Ensures pages are only for actionable, urgent issues	Potentially missing early warning signs if not logged elsewhere	Any alert that does not require immediate human intervention should not page. These should be tickets or logs.
Default: Ticket (unless critical)	Reducing page noise, establishing a baseline for all alerts	Minimizes interruptions, forces clear criteria for paging, reduces alert fatigue	Slow response to truly critical issues if criteria are too strict	Starting to re-evaluate alert routing. All new alerts default to ticket unless explicitly justified as a page.
Ownership by Service/Team	Clear accountability, domain expertise for triage	Prevents 'everyone and no one' problem, faster resolution	Siloed knowledge, handoff issues between teams	Teams own specific services. Alerts are routed directly to the owning team's on-call or ticket queue.

Alert fatigue gets worse when every signal goes to the same place. When pages, tickets, and informational chatter all land in one stream, you force humans to do routing work under stress. That’s expensive—and it teaches people the channel can’t be trusted.

The cleanest way to reduce alert noise without losing safety is to decide what belongs in a page vs a ticket vs a log, and enforce one non-negotiable requirement: every interrupt has a named owner and a next action.

A simple triage decision that holds up in real ops conversations:

Time sensitivity: how quickly does inaction increase damage?
User impact: are customers failing right now, or is this internal risk?
Reversibility: can you safely wait and observe, or does delay make recovery harder?

Time-sensitive + customer-impacting + hard to reverse is a page. Important-but-not-urgent is a ticket. Neither urgent nor important is a log/dashboard.

What a page is for (and what it isn’t): a page is for “wake someone up because waiting until morning is irresponsible.” That’s it.

So no, “certificate expiring in 30 days” should not page at 3 AM. Neither should “a deployment happened,” “a thing changed,” or “a metric crossed a line we picked because it sounded responsible in a meeting.” Those are tickets or logs.

Common mistake: paging for internal discomfort. “Cache hit rate dropped” might matter, but it’s not automatically a wake-up event. Translate it into customer outcomes or an operational decision, or keep it out of paging.

Routing is where teams quietly hemorrhage time. The rule that prevents the “everyone and no one” problem is blunt: one alert, one accountable owner.

That doesn’t mean one person. It means one team/role is responsible to triage and either resolve or escalate. If an alert is “shared,” it’s ignored. If it’s “for everyone,” it’s for no one.

Re-routing should also be explicit. The initial owner either accepts and acts, or re-routes once to a clearly defined next owner. Multiple bounces are a smell: the alert is poorly defined, or the ownership map is stale.

Two scenarios that show how channel choice changes behavior:

Customer-impacting degradation: checkout is slow in two regions and timeouts are rising. Page the service owner. Next action: assess blast radius, decide on failover/rollback, post a status update. Support should be looped in, but engineering owns the fix.
Internal-only degradation with risk: batch jobs are falling behind, customers are still fine. This should be a high-priority ticket to the batch pipeline owner, with an escalation rule if it crosses a time-to-failure threshold.

Finally: stop sending mystery-meat alerts. A minimum payload that makes action possible includes impact, scope, start time + trend, owner, and a first step.

Example in human language: “Customers seeing elevated 500s on login in eu-west, started 09:12, error rate 4% and rising, likely tied to auth deploy at 09:05, owner auth on-call, first step check rollout status then consider rollback, ignore if maintenance flag is on.” That’s a page someone can act on.

For a solid perspective on “quiet by default” alerting that avoids spam conditions, see [3]

Fix signal at the source: thresholds, context, and severity people can understand

Once you have routing policies, an annoying truth shows up fast: most alert noise is created upstream, inside the alert definition. Operators can’t triage their way out of bad signals forever.

Fixing signal-to-noise at the source usually comes down to three moves:

Calm thresholds so they stop flapping.
Write severity so humans can understand it quickly.
Deduplicate symptom storms so you page on causes, not every downstream shiver.

Flappy thresholds are the fastest way to train “ignore it.” The usual pattern: alert on a naturally spiky metric with a short window. It fires, clears, fires again. The only consistent outcome is noise.

Rules of thumb that work (with a little humility):

Prefer sustained conditions over single spikes. If a spike lasts 90 seconds and has no customer impact, paging on it is theater.
Anchor thresholds to outcomes where possible. Latency, error rates, saturation, and backlog usually beat CPU alone.
If the metric has a daily pattern, don’t punish normal traffic ramps.

The caveat that matters: don’t “fix” flapping by widening the window until it never fires. That buys peace at the cost of detection time.

Here’s the tradeoff you should name explicitly in reviews: tighter thresholds reduce detection time but increase fatigue and missed pages because humans stop trusting the channel. Looser thresholds reduce fatigue but increase time to detection and mitigation. The balancing rule is risk plus detection objective: how fast do you need to know, given the business impact curve of this failure?

Concrete example: “CPU above 80% for 5 minutes pages.” In many systems, CPU bursts are normal during garbage collection or cache warmups. A better question is “is the service saturating in a way that hurts customers?” Keep CPU as context, but page on something that correlates with user pain (latency + errors + saturation), or on CPU only when it’s sustained and tied to an impact symptom.

Severity is where teams get vague in the worst possible way. “Sev 1 / Sev 2 / Sev 3” doesn’t help if nobody can remember what they mean at 3 AM.

Write severity in human impact terms, with an implied response:

Sev Critical: customers can’t complete a core action, or data integrity/safety is at risk. Response: immediate page, declare incident, coordinate.
Sev High: customers are degraded; revenue or trust is at risk if it persists. Response: page or urgent ticket depending on time sensitivity; start mitigation.
Sev Medium: internal degradation with a credible path to customer impact. Response: ticket with a clear SLA and an escalation rule.
Sev Low: informational/audit. Response: log, review in cadence.

The key is that severity is not “how bad the graph looks.” It’s “what the responder should do.” That’s how you make on-call alert triage faster and more consistent, which directly reduces alert fatigue in support teams.

Then there’s the storm problem: one root cause produces a blizzard of alerts. Database slows down, every service pages on latency, the database pages too, queues page, and support gets pinged. Everyone has data; nobody has clarity.

Strategy: pick a primary signal for paging and demote the rest.

Concrete example: ten “service error rate high” pages when the real issue is “database connection pool exhausted.” Promote the database saturation alert as the page. Route downstream symptoms to tickets or logs. Attach dependency context so responders know where to look first.

This is where teams get burned again: when they demote symptom alerts, they accidentally hide information. Don’t hide it—just protect the interrupt channel. Keep symptoms visible somewhere quieter so you can still debug and validate blast radius.

Finally, attach decision context early. Decide which fields must exist in the alert definition itself (not in someone’s head): service + environment/region, owner, severity, impact statement (even “internal only”), start time + trend, primary dashboard link, runbook/first step, and known benign states (maintenance mode, expected traffic events, batch windows).

Humans can add deeper diagnosis notes and timelines later. Don’t rely on humans to add basics like ownership and impact. That’s how you get “ack and hope.”

A practical reference on thresholds and grouping without noise: [4]

Two things that keep noise coming back: missing ownership and ‘we never close the loop’

Even teams that do a great cleanup regress. Not because they’re sloppy, but because change keeps happening: new services launch, teams reorganize, load patterns shift. Without a loop that maintains alert quality, you’ll be back where you started in a quarter.

Two failure modes cause most of the backsliding.

Failure mode #1: alerts with shared ownership become ignored ownership

Symptom: an alert lands in a shared channel and everyone assumes someone else is looking. The “owner” is a list of teams. The escalation path is unclear. Support pings engineering, engineering pings platform, and the only thing that moves quickly is frustration.

Countermeasure: every page has a single accountable owner. If multiple teams contribute, pick the team that can take the first meaningful action. Others become explicit escalation targets, not co-owners.

A very normal regression scenario: a reorg splits “platform” into “infra” and “runtime,” but alerts still route to “platform.” Nobody updates the ownership map. Within weeks you have misrouted pages and slower triage. Your loop should catch that within the month—not after the next major outage.

Failure mode #2: ‘acknowledge and move on’ without elimination work

Symptom: the same alert appears weekly, gets acknowledged, maybe gets a quick mitigation, then disappears from attention. No one files upstream work to redesign the alert or fix the underlying churn.

Countermeasure: every noisy page creates a closure item. Not “tune the threshold” by default, but “make the next alert more actionable, or remove it from the interrupt channel.” Tie it to a backlog and a named owner.

Another common regression scenario: seasonal load changes make “normal spikes” bigger. Suddenly old thresholds flap all day. If you have a weekly review cadence, you catch it early. If you don’t, your team quietly starts ignoring again.

Metrics are where teams either get serious or get performative. If your only metric is “alert volume,” people will game it by muting.

Use metrics that reflect quality and behavior:

Actionable rate (14 days): did alerts lead to meaningful action?
Auto-resolve rate (14 days): what cleared without human action? High is a smell for pages; acceptable for logs.
Duplicate rate (14 days): how much was the same event wearing different hats?
Time to triage (30 days): median minutes from fire to correct routing decision (page vs ticket vs log).
Page to meaningful action time (30 days): median minutes from page to first meaningful action, not just acknowledgment.

Guardrail against gaming: track detection and impact outcomes alongside volume. If volume drops but time-to-triage and time-to-action get worse, you didn’t improve. You just got quieter while staying fragile.

Alert quality metrics overview: [5]

Cadence is the unglamorous part that makes gains stick.

Weekly: 30 minutes reviewing the most recurring alerts by count; convert, demote, or fix at the source.
Monthly: severity calibration using a handful of real incidents; did severity match human impact and required response?
After incidents: alert cleanup while the memory is fresh; fix late/noisy/misrouted primary signals.

Support leadership can help here. Support sees customer pain patterns before metrics do. When support says “customers reported login failures but we didn’t get a page,” that’s a signal about your signal.

A 30-day reset plan to make alerts trustworthy again (without heroics)

You don’t need a quarter-long program and a new tool to reduce alert fatigue. You need a focused month where you stop the conditioning loop and rebuild trust.

Week 1: audit + triage policy + stop-the-bleeding changes

Classify the last 50 alerts with the outcome tags. Pick the top 10 by frequency and decide which ones should never have paged.

Publish a one-page triage policy for page vs ticket vs log, including the owner + next action rule. The point isn’t bureaucracy. It’s giving everyone the same decision language when they’re tired.

Then do two “stop-the-bleeding” changes only:

Convert obvious informational pages to tickets/logs.
Fix the most egregious misrouting.

Anything beyond that in week one tends to turn into a well-intentioned rewrite project that never ships.

Week 2–3: fix the top 3 sources and rewrite severities

Take the top three noise sources from the audit and fix them at the source. In practice, that usually means adding missing context, calming flappy thresholds, and choosing a primary signal to page on.

Rewrite severity definitions in human terms and align them with the routing policy so a tired operator can still make the correct call quickly.

This is also where you should remove “page because it’s scary” alerts. Replace them with “page because it’s customer-impacting and time-sensitive” alerts.

Week 4: measure, prune, and lock in ownership + cadence

Review actionable rate and duplicate rate for the month. If they didn’t move, don’t blame the team—recheck whether pages actually include owner, next action, and impact. Most “it didn’t improve” stories are really “we kept paging on data.”

Make one commitment device non-negotiable: a weekly 30-minute alert review with a named owner who can assign follow-ups. Put it on the calendar like it’s a production meeting, because it is.

Monday plan, concrete and realistic: first, classify your last 50 alerts. Then focus on three priorities. One, stop informational pages by converting them to tickets or logs. Two, ensure every remaining page names an owner and a next action. Three, fix at the source the top recurring flappy or duplicate alert. Your production bar for week one is simple: fewer interrupts that lead to no action, and faster triage for the ones that remain.

Sources

pagerly.io — pagerly.io
blog.ebbypeter.com — blog.ebbypeter.com
verid.dev — verid.dev
gse.kz — gse.kz
oneuptime.com — oneuptime.com

Your Alerts Are Training People to Ignore Them: Fixing Signal to Noise at the Source