Answer
Run the postmortem like an incident: contain the blast radius, agree on what the KPI is supposed to mean, then work backward to the first moment the number diverged. Quantify the gap, trace the metric lineage end to end, and test the five most common failure modes until you can reproduce the error and prove causality. Close by assessing decision impact, communicating a restatement plan, and putting lightweight controls in place so the same class of failure cannot quietly return.
You are not the first team to discover that the KPI you trusted for months was quietly wrong. The reflex is usually to hunt for a broken query or blame “the data.” A good postmortem does something more useful: it turns a painful surprise into a clear root cause, a bounded impact, and a set of controls that make the metric decision safe going forward.
Below is a practical, executive friendly way to run that postmortem without turning it into an endless archaeology project.
Immediate containment and stakeholder alignment (Day 0–2)
Treat this as a business incident, not a spreadsheet annoyance. Your goal in the first 48 hours is to stop accidental reuse of the wrong number, set expectations, and preserve evidence.
First, decide severity and scope. Ask: Was this KPI used for board reporting, external disclosures, compensation, or capital allocation? If yes, escalate immediately to Finance leadership and, if applicable, Legal or Compliance. If it was internal only, you still act quickly, but your communications can be simpler.
Second, freeze the current number and label it clearly as suspect wherever it appears. If the KPI is auto distributed in email digests or embedded in recurring decks, pause that distribution until you have an interim plan. Capture screenshots of dashboards, exports, the exact SQL or semantic layer definition, and the dataset versions used. Postmortems often fail because the team “fixes” things first and destroys the ability to prove what happened.
Third, align the humans. You need one accountable incident owner, one business owner for the metric, and one data owner for the source system. Create a single channel for updates and set a cadence, even if the update is “still investigating.” People tolerate uncertainty better than silence.
Here are the most common containment options and when to use them:
Freeze the number, stop distribution: use it when decisions are actively being made on the KPI. Document the metric contract: use it to stop definition drift and argument loops. Communicate interim guidance: use it when the business cannot pause and needs a caveated substitute. Create a timeline of changes: use it when you suspect a specific change window. Compare reported vs. recomputed KPI: use it to turn opinions into measurable deltas.
Practical tip: Assign one person to be “artifact librarian” for two days. Their job is to collect the exact links, screenshots, query text, and version identifiers so the investigation does not rely on memory.
Define the “correct” KPI: canonical definition, intent, and decision use
A KPI can be “wrong” in at least two ways: the computation is incorrect, or the metric no longer reflects the business intent. You cannot diagnose either without a canonical definition.
Write a one page metric contract that includes intent, formula, inclusion and exclusion rules, and the decision it is meant to support. Be explicit about:
Entity and event definitions. Are you counting users, accounts, orders, sessions, or something else? What is the event that qualifies?
Numerator and denominator contracts. For ratio metrics, define who is eligible versus who is merely observed. Silent denominator shifts are a classic way for a KPI to “improve” while reality gets worse.
Time logic. Is this based on event time or ingestion time? What is the time zone? What is the as of rule for late arriving data?
Attribution and status taxonomy. If you rely on lifecycle statuses, list the exact values and what they mean.
Known edge cases. Include examples and counterexamples, like a mini acceptance test suite in plain English.
Common mistake: Teams skip this and jump straight to debugging SQL. What to do instead is get the metric owner and a finance or operations partner to sign off on the contract first, even if it takes one intense meeting. Otherwise you risk “fixing” the KPI into a different KPI and restarting the whole argument next quarter.
Practical tip: Add one sentence that starts with “We use this KPI to decide…” If nobody can finish that sentence, you may be debugging a vanity metric.
Reconstruct the timeline: when did it become wrong and what changed?
Now you build a timeline that includes both technical and business changes. The goal is to find the first point of divergence and the set of plausible causes around it.
Collect the following artifacts and place them on a single chronological view:
Definition edits. Docs, metric catalogs, wiki pages.
Dashboard and semantic layer changes. BI calculated fields, filters, caching settings.
Data model and transformation deploys. New joins, dedupe logic changes, incremental load changes.
Source system releases and process changes. New statuses, redesigned forms, changes to what staff are instructed to enter.
Incentives. Any new targets, contests, compensation plans, or OKR changes.
Then identify the first date the KPI was wrong. If you have access to a “should be” recomputation, you can find the first week or day it diverged. If not, use proxies like sudden steps in the trend, unexplained variance shifts, or segment behavior changes.
A useful heuristic: Step change usually points to a definition, tracking, or pipeline change. Slow drift often points to coverage decay, process adaptation, or a denominator eligibility creep.
If you need a structured debugging flow, KPI Tree’s broken metric playbook is a solid reference: [1]
Quantify the discrepancy and map its shape
Do not argue in percentages in a meeting until you have mapped the discrepancy across time and segments.
Create two series: the reported KPI and the recomputed “correct” KPI based on the canonical definition. Then compute:
Absolute delta and percent error over time.
Bias direction. Always high, always low, or alternating.
Volatility. Did the error increase variance or dampen it?
Boundary effects. Month end, quarter end, daylight saving shifts, or end of business day cutoffs.
Segment shape. By region, product line, channel, device, plan tier, and any dimension that plausibly relates to tracking or process.
This shape often points you to the failure mode quickly. A constant offset suggests double counting or missing a fixed cohort. A scaling factor suggests a denominator change or a filter applied twice. Missing cohorts often show up as one region or one product disappearing from the metric.
If you want a practical framework for comparing trends versus collection artifacts, this repo is useful for thinking in “forensics” terms: [2]
One tasteful analogy: A KPI postmortem is like checking your bathroom scale. If it suddenly says you lost 20 pounds overnight, the first question is not your diet. It is whether the scale batteries are dying.
Trace lineage end-to-end: definition → source → transforms → semantic layer → dashboard/report
Now you do lineage. This is where most teams either get lost or get clarity. The trick is to keep it end to end and to reconcile counts at each step.
Start at the final output that stakeholders saw. Work backward through:
The dashboard or report logic. Filters, date grains, hidden joins, cached extracts, “include nulls” behaviors.
The semantic layer. Metric definitions, reusable measures, default filters, grouping logic, and naming collisions.
Transformation steps. Dedupe, identity stitching, currency conversions, status mapping, incremental loads.
Raw sources. Event logs, CRM objects, billing tables, operational systems.
At each boundary, reconcile simple invariants. Do row counts match expectations? Do sums by day reconcile? Do distinct entity counts jump in suspicious ways? A good lineage diagram highlights single points of failure and undocumented transformations, especially manual spreadsheet steps.
For a checklist of pre publication data quality checks, this is a helpful grounding reference: [3]
Root-cause checklist by failure mode (the “5 buckets”)
| Option | Best for | What you gain | What you risk | Choose if |
|---|---|---|---|---|
| Freeze the number, stop distribution | Immediate containment of a critical error | Prevent further bad decisions. centralize communication | Temporary data blackout. loss of trust if not communicated well | The error is significant and impacts critical decisions |
| Document the metric contract | Preventing future discrepancies. aligning stakeholders | Clear definition of the 'right' number. shared understanding | Time investment upfront. potential for disagreement on definitions | Metric definition is ambiguous or undocumented |
| Communicate interim guidance | Maintaining business operations during investigation | Stakeholder confidence. continued decision-making (with caveats) | Decisions made on potentially flawed data. requires clear disclaimers | Investigation will take time and decisions cannot be paused |
| Create a timeline of changes | Pinpointing the exact cause and timing of the error | Root cause identification. historical context | Can be time-consuming. relies on good version control/logging | The error appeared suddenly or after a known system change |
| Compare reported vs. recomputed KPI | Quantifying the error and understanding its characteristics | Precise error measurement. insight into error type (offset, scaling) | Requires re-running calculations. potential for new errors in recomputation | You need to understand the magnitude and pattern of the discrepancy |
| Inventory all data components | Comprehensive audit of the entire data pipeline | Identification of hidden dependencies or manual interventions | Very high effort. can be overwhelming for complex systems | The error source is elusive and not found by other methods |
Most KPI failures fall into five buckets. Use this as a structured checklist, not a guessing game.
1) Definition drift
Telltales include inconsistent filters across dashboards, renamed statuses, and “minor” clarifications that change who qualifies.
How to confirm: Diff the metric definition across time, dashboards, and teams. Recompute with each version and see which aligns with the discrepancy window.
2) Data pipeline or engineering issues
Common causes include schema changes, broken joins, dedupe regressions, timezone shifts, and incremental load bugs.
How to confirm: Reconcile row counts at each step, look for sudden null spikes in join keys, and rerun the pipeline for a small window with known good inputs.
3) Source system or process changes
This is where the business changed how it works, not the data team. New workflows, form defaults, policy changes, and training updates can all change what gets recorded.
How to confirm: Compare operational logs or CRM field population rates before and after. Interview the frontline team and ask what changed on the day the trend broke.
4) Late data, backfills, and as-of logic
If data arrives late or systems retroactively correct records, a KPI can be “right eventually” but wrong when reported.
How to confirm: Measure data completeness by age of data. Look for backfills and reprocessing jobs. Decide whether you need snapshots to stabilize reporting.
5) Incentives, gaming, and behavioral changes
If teams are measured on the KPI, they will optimize it. Sometimes that is good. Sometimes it creates edge cases that break the metric’s meaning.
How to confirm: Look for sudden shifts in distributions near thresholds, unusual spikes at period end, or changes concentrated in the team being measured.
For more on KPI drift and metric decay patterns, DataCult’s writeups are a helpful lens: [4] and [5]
Pinpoint the break: reproduce, isolate, and prove causality
A postmortem is not complete when you have a theory. It is complete when you can reproduce the wrong number, flip one factor, and get the right number.
Do this in a controlled environment:
Reproduce the reported KPI exactly as it was computed at the time. Same filters, same model version, same date logic, same cache behavior.
Bisect changes. If you have version control in your transforms or semantic layer, test older versions until you find the first bad one.
Toggle one change at a time. Change the timezone conversion, then the join key, then the status filter, rather than rewriting everything.
Create a minimal counterexample. Find one customer or one day where the KPI is provably wrong and trace that record through the pipeline.
Cross validate independently. Have Finance or another analyst compute the KPI from the raw source using the metric contract. The goal is to eliminate “same logic, different tool” false confidence.
If you are dealing with a sudden drop after tracking or definition changes, Calypso’s troubleshooting flow is aligned with this reproduce and isolate approach: [6]
Assess impact: decisions, forecasts, compensation, and external reporting
Once you know what broke, quantify what it changed.
Create an impact inventory:
Where it was used. OKRs, board decks, forecasting models, spend allocation, pricing decisions, headcount plans.
Whether decisions are reversible. A paused campaign is reversible. A compensation payout or external statement is not.
Restated history. Produce a corrected series for the affected window and show the delta in the same view stakeholders are used to.
Confidence bands. If the correction depends on uncertain backfills or incomplete sources, state a range and what drives it.
If this KPI influenced compensation, treat it as an employee trust issue, not just a data issue. The right move is usually to pay people based on the best available intent, then fix the measurement system so you never put them in that position again.
For symptoms and organizational causes of faulty KPIs, FP&A Trends provides a useful framing that bridges finance and analytics: [7]
Corrective actions: fixes, restatement, and communication plan
Corrective actions should match the failure mode and the audience.
First, fix the system at the right layer. If the definition was wrong, do not just patch the dashboard. Update the metric contract and the semantic definition so downstream uses inherit the fix. If the pipeline was wrong, patch the transform and add tests so it cannot regress.
Second, decide whether you are restating history. There are three common approaches:
Full restatement for the affected window, with the old series deprecated.
Versioned KPI, such as KPI v1 and KPI v2, with clear dates and a mapping.
Prospective fix only, when historical correction is impossible or not decision relevant.
Third, communicate like an adult. A good correction memo includes what changed, why it changed, when it changed, the quantified impact, and what to do now. If you bury the lead, people will assume the worst. If you overshare internals, they will tune out. Aim for clarity.
If you want a strong culture pattern, “ship KPIs like features” is a useful mental model: define, version, test, announce, deprecate. [8]
For postmortem structure patterns drawn from many pipeline incidents, Layline’s summary is a helpful reference: [9]
Prevent recurrence: governance, controls, and monitoring
Prevention is not a giant governance committee. It is a few small controls that make silent failure unlikely and loud failure fast.
Start with ownership and contracts. Every tier one KPI needs a named business owner and a named data owner, plus a living metric contract.
Add lightweight controls:
Data quality checks before publish. Completeness, freshness, duplicates, null join keys, and sanity bounds.
Change management. Any change to the metric definition, semantic layer, or upstream source requires a note in a changelog and a quick sign off.
Monitoring for drift. Alert when the KPI moves outside expected bands, when coverage drops, or when the denominator population changes sharply.
Snapshots for reporting stability. If late arriving data is common, decide the as of cutoff and keep stable snapshots for executive reporting.
Periodic reconciliation. Once a month, reconcile the KPI to an independent source of truth or a simpler calculation. Think of it as balancing your checkbook, but for metrics.
Common mistake: Teams only monitor the final KPI value. What to do instead is also monitor its ingredients, such as the denominator size, the percent of records with missing keys, and the share of events coming from each source. Many failures show up there days or weeks earlier.
If you do one thing first, do this: lock the metric contract, then build one reconciliation view that compares reported versus recomputed KPI by week and by key segment. That single chart will keep you honest, keep stakeholders aligned, and dramatically reduce the odds that you spend another year confidently reporting the wrong number.
Sources
- How to Debug a Broken Metric - KPI Tree
- Our core KPI suddenly dropped after a tracking or - Calypso
- KPI Mismatch: Why Numbers Don’t Match | DataCult
- KPI Drift: Why Metrics Decay and How to Stop It | DataCult
- From Symptoms to Cause: How to Identify Faulty KPIs | FP&A Trends
- What data quality checks should we run before publishing KPIs?
- Ship KPIs Like Features: The KPI Engineering Framework | May Mor
- AmirhosseinHonardoust/Trend-vs-Collection-Forensics
- What I Learned From Reading 50 Data Pipeline Postmortems - layline.io Blog
Last updated: 2026-06-21 | Calypso
Sources
- kpitree.co — kpitree.co
- github.com — github.com
- connect981.com — connect981.com
- datacult.ai — datacult.ai
- datacult.ai — datacult.ai
- calypso.ms — calypso.ms
- fpa-trends.com — fpa-trends.com
- scalewithmay.com — scalewithmay.com
- layline.io — layline.io