[{"data":1,"prerenderedAt":59},["ShallowReactive",2],{"/en/answer-library/before-rolling-out-a-new-dashboard-or-kpi-how-can-we-run-a-data-pre-mortem-to-pr":3,"answer-categories":36},{"id":4,"locale":5,"translationGroupId":6,"availableLocales":7,"alternates":8,"_path":9,"path":9,"question":10,"answer":11,"category":12,"tags":13,"date":15,"modified":15,"featured":16,"seo":17,"body":22,"_raw":27,"meta":29},"5fd1d449-2a47-47ec-ad2e-5c81bf1abb19","en","0ea1b336-2d0b-4888-abf6-ed0bb6066942",[5],{"en":9},"/en/answer-library/before-rolling-out-a-new-dashboard-or-kpi-how-can-we-run-a-data-pre-mortem-to-pr","Before rolling out a new dashboard or KPI, how can we run a data pre mortem to predict how the metric will be misread?","## Answer\n\nRun a data pre mortem as a short, structured workshop where the team assumes the new KPI failed in the real world and then works backward to explain how it got misread. You pre define the metric contract, list likely misinterpretations, map confounders, predict gaming, and then redesign the dashboard and decision rules so the most common mistakes are harder to make. Done well, it turns “we shipped a number” into “we shipped a number with guardrails, context, and a plan for what to do when it moves.”\n\nMost dashboards do not fail because the math is wrong. They fail because humans are busy, incentives are real, and context evaporates the moment the metric leaves the analyst’s screen. A data pre mortem is how you catch those predictable misreads before a KPI becomes a story people repeat in meetings.\n\n## What a “data pre mortem” is and when to use it\nA data pre mortem is a risk rehearsal. You gather the people who will build, use, and be judged by a metric and you assume that, three months after launch, the KPI caused confusion or bad decisions. Then you ask a simple question: “What went wrong, and how did we let the metric mislead us?” This is different from data QA. QA asks whether the data is accurate and pipelines run. A pre mortem asks whether the metric will be interpreted correctly and used responsibly, even when the data is technically correct.\n\nUse it when any of these are true.\n\n1) You are introducing a new KPI or redefining an old one.\n\n2) A new audience will consume the dashboard, especially executives or frontline managers who were not in the build.\n\n3) The metric will be linked to performance reviews, bonuses, targets, or budget decisions.\n\n4) You are making a major pipeline change such as a new event taxonomy, a source system migration, new identity stitching, or backfill logic.\n\nIf you only do one of these per quarter, do it for the KPI that could most easily become “analytics without context,” which is basically misinformation wearing a spreadsheet costume (see the context argument in https://building.theatlantic.com/analytics-without-context-is-misinformation-0ae4b017a090).\n\n## Pre work: assemble the metric contract and decision context\nThe pre mortem works when everyone starts from the same definitions. Your goal is not a long spec. It is a one page “metric contract” plus a clear decision context.\n\nThe metric contract should include the pieces that commonly get lost when a KPI is repeated secondhand. A KPI dictionary approach is a good model here (https://vladimirsiedykh.com/blog/kpi-dictionary-before-dashboard-build).\n\nInclude these items in plain language.\n\n1) Metric name and purpose. What problem is it meant to detect or manage?\n\n2) Operational definition. The exact calculation, including numerator and denominator.\n\n3) Time window and grain. Daily, weekly, trailing 28 days, cohort based, and so on.\n\n4) Inclusion and exclusion rules. What counts, what does not, and why.\n\n5) Default segmentation. Region, channel, product line, customer type, and the required “slices” that prevent misleading rollups.\n\n6) Data lineage. Source systems, key transformations, and where the “source of truth” lives.\n\n7) Refresh cadence and data latency. What “today” means.\n\n8) Known constraints. Missing data, tracking gaps, small sample volatility, or any period where comparisons are unsafe.\n\n9) Intended decisions and owners. Who is expected to act when it changes, and what decisions it is supposed to inform.\n\nPractical tip: add one sentence titled “Do not use for.” For example: “Do not use this KPI to evaluate individual rep performance,” or “Do not use day to day changes to decide pricing.” This single line prevents a lot of creative misuse.\n\nPractical tip: include a minimum sample threshold, even if it is a simple rule like “no segmentation view below 200 events in the period.” It saves you from arguing with noise later.\n\n## 90 minute workshop agenda (repeatable format)\nInvite a small, mixed group. Eight to twelve people is ideal.\n\nRoles to include.\n\n1) Metric owner. The business owner accountable for what happens when the KPI moves.\n\n2) Analytics and data. The people who know definitions, caveats, and pipeline behavior.\n\n3) Domain subject matter experts. The operators who know what is really happening in the field.\n\n4) Frontline users. The managers or teams who will actually use the dashboard under time pressure.\n\n5) Incentives partner if relevant. Finance, HR, sales ops, or whoever sets targets and compensation.\n\nArtifacts to bring.\n\n1) The one page metric contract.\n\n2) A rough dashboard mock, even if it is a sketch.\n\n3) A short glossary of terms and segments.\n\nTime boxed agenda.\n\n1) 0 to 10 minutes. Align on the decision the metric is supposed to improve and the top two risks.\n\n2) 10 to 25 minutes. Step 1 brainstorm: how this will be misread.\n\n3) 25 to 45 minutes. Step 2 confounders and alternative explanations.\n\n4) 45 to 60 minutes. Step 3 pre mortem Goodhart, predict gaming.\n\n5) 60 to 70 minutes. Step 4 edge cases and pipeline brittleness.\n\n6) 70 to 80 minutes. Step 5 build the metric bundle.\n\n7) 80 to 88 minutes. Step 6 dashboard design changes to make misreads harder.\n\n8) 88 to 90 minutes. Step 7 decision rules, escalation paths, and owners.\n\nThis “assume it failed, then explain why” structure borrows from classic pre mortem thinking (https://whennotesfly.com/concepts/decision-making/pre-mortem-analysis and https://www.theuncertaintyproject.org/tools/pre-mortem), adapted to KPI rollouts.\n\n## Step 1: Brainstorm “how this will be misread” (failure modes library)\nThis step is quantity over quality. Ask everyone to write down misreads silently for three minutes, then share and cluster. Silence first matters because it reduces groupthink and status effects, a common issue in brainstorms (https://www.problempop.io/blog-posts/how-to-conduct-a-pre-mortem-to-de-risk-your-next-big-feature-launch).\n\nHere is a failure modes library you can reuse, organized the way misreads show up in real organizations.\n\nContext loss.\n\nPeople forget the denominator, base rate, or volume. A percentage moves but the underlying count collapses.\n\nPeople ignore seasonality or calendar effects. The metric “drops” every holiday week and everyone panics anyway.\n\nPeople miss the time horizon mismatch. A weekly KPI is used to judge a quarterly initiative.\n\nDefinition and measurement ambiguity.\n\nTwo teams use slightly different definitions for the same word, like “active,” “qualified,” or “resolved.”\n\nRounding and formatting make small changes look meaningful.\n\nAggregation hides the story.\n\nA company wide number looks stable while one region is on fire and another is booming.\n\nSimpson’s paradox shows up. The overall rate improves while every segment worsens, or vice versa.\n\nBias and missingness.\n\nSelection bias. The metric only reflects customers who were measured, not all customers.\n\nSurvivorship bias. You only see the users who stuck around.\n\nMissing data is not random. Tracking fails more on older devices, certain geos, or certain workflows.\n\nTiming and latency.\n\nLate arriving data makes the last few days look worse than they are.\n\nOperational lag means the KPI responds weeks after the real world change.\n\nQualitative gaps.\n\nThe number moves but nobody knows “what changed on the ground.” The KPI becomes a debate about narratives.\n\nCommon mistake moment: teams often treat this step as a debate about who is right. Do not litigate the metric yet. Capture misreads first, then rank which ones are most damaging and most likely. You are building a risk register, not trying to win a meeting.\n\n## Step 2: Map confounders and alternative explanations (signal vs. noise)\n\n| Option | Best for | What you gain | What you risk | Choose if |\n| --- | --- | --- | --- | --- |\n| A / B Testing / Quasi-Experiments | Establishing causality with high confidence | Strong evidence for cause-and-effect relationships | High setup cost. ethical considerations. not always feasible | You can randomly assign users/groups and control variables |\n| Expert Review / Brainstorming | Leveraging collective knowledge and experience | Identification of domain-specific confounders. diverse perspectives | Groupthink. reliance on anecdotal evidence | You have access to subject matter experts and need qualitative insights |\n| Causal Sketching / DAG-lite | Understanding direct and indirect relationships | Visual clarity of assumptions. identification of potential confounders | Oversimplification of complex systems. time investment | You need to map out how variables influence each other before analysis |\n| Control Views / Stratification | Isolating the impact of a variable | More precise measurement of specific effects | Over-segmentation leading to small sample sizes | You suspect a specific factor is influencing your primary metric |\n| The 'What else changes?' Test | Quickly identifying obvious confounders | Fast, intuitive check for external factors | Missing subtle or non-obvious confounders | You're reviewing a metric or dashboard and need a rapid gut-check |\n| Defining 'Counter-Metrics' | Preventing unintended negative consequences | Holistic view of impact. early warning for adverse effects | Increased dashboard complexity. potential for analysis paralysis | Your primary metric has a clear potential downside or tradeoff |\n\nOnce you have misreads, you need a lightweight way to separate signal from noise. The core question is: “If this KPI moves, what else could be causing it besides the thing we care about?” This is where organizations most often overreact, because they mistake correlation plus urgency for causation.\n\nUse a simple method.\n\nFirst, do a causal sketch. Draw the KPI in the middle and add the top drivers around it: demand, supply constraints, product changes, pricing, marketing mix, customer composition, outages, policy changes, and competitor actions. Keep it “DAG lite” in spirit, not an academic exercise.\n\nSecond, run the “What else changes?” test. If the KPI rises, what other operational indicators typically move at the same time? If it falls, what else tends to fall?\n\nThird, define control views. Decide which segments you will always check before telling a story. Typical control views are by channel, by cohort age, by region, and by customer type.\n\nFourth, define counter metrics. For any KPI you want to push up, ask what you might accidentally push down.\n\nTo make this concrete, here is a decision oriented menu of controls you can use.\n\nA / B Testing / Quasi-Experiments helps when you truly need cause and effect.\n\nCausal Sketching / DAG-lite forces assumptions into the open so people can disagree productively.\n\nControl Views / Stratification prevents a single blended number from driving the wrong story.\n\nThe 'What else changes?' Test is the fastest way to catch obvious alternative explanations.\n\nPractical tip: choose two default “explainers” that always sit next to the KPI. Example: if conversion rate moves, always show traffic mix and page latency. It turns a vague story into a bounded investigation.\n\n## Step 3: Pre mortem Goodhart: predict gaming and perverse incentives\nIf the KPI will be used for targets or performance evaluation, assume it will be optimized. Not because people are bad, but because they are human and busy, and incentives are loud.\n\nUse three prompts.\n\nFirst: “If compensation or praise is tied to this number, how could someone win without creating real value?”\n\nSecond: “What are the easiest levers to pull that move the metric quickly?” Quick levers are where gaming tends to appear.\n\nThird: “What process loopholes does the metric create?” For example, reclassifying cases, delaying work until the reporting window resets, or routing easy customers to the measured channel.\n\nThen add guardrails. You can combine several without making the dashboard unreadable.\n\n1) Pair the KPI with a quality counter metric. If you push for faster resolution, also watch reopen rate or customer satisfaction.\n\n2) Add audit checks or random sampling. If a category can be manipulated, sample records and verify.\n\n3) Use caps and floors. If you know a metric is volatile at low volumes, suppress or gray out the view.\n\n4) Add a lightweight qualitative review. A short “what changed operationally?” note from the owner reduces confirmation bias, which is a real dashboard problem (https://atticusli.com/blog/posts/confirmation-bias-dashboard-design-teams-see-what-they-want).\n\n## Step 4: Stress test edge cases and data pipeline brittleness\nThis step is where you protect executives from the “we had an incident but the chart did not mention it” moment.\n\nWalk through these edge cases.\n\nLate arriving data and backfills. Decide whether the most recent days are labeled as preliminary.\n\nSchema and tracking changes. If event names or fields change, what breaks and how will you know?\n\nOutages and partial ingestion. What does the KPI look like during a logging outage, and does the dashboard warn users?\n\nReclassification. What happens when definitions change, like a new status taxonomy or a new fraud rule?\n\nSmall n volatility. If you slice too far, the metric becomes a coin flip with a nice font.\n\nVersioning and change logs. When the definition changes, can a user tell which version they are viewing?\n\nYou do not need a deep technical runbook here. You do need a clear agreement on what the dashboard should do during weird weeks, because weird weeks happen.\n\n## Step 5: Build the “metric bundle” (primary + counter metrics + context)\nA single KPI invites over interpretation. A metric bundle reduces that risk by making the most important context visible by default. This aligns with the broader point that missing context is often the most important part (https://answerhorizon.com/the-missing-context-is-often-the-most-important-part/).\n\nUse this template.\n\nPrimary KPI. The headline number.\n\nInput or leading indicators. The controllable drivers you expect to move first.\n\nOutcome or lagging indicators. The business result you actually care about.\n\nQuality and safety counter metrics. What you do not want to break while improving the KPI.\n\nGuardrail thresholds. Red lines that trigger investigation.\n\nDefault segments. The two or three cuts that prevent misleading averages.\n\nContext notes. Seasonality, known incidents, definition changes.\n\nExample. If your primary KPI is ecommerce conversion rate, the bundle might include traffic quality (new versus returning, channel mix), page performance, average order value, return rate, and customer support contact rate. If conversion rises while return rate spikes, you know you may be “selling” the wrong thing.\n\nCommon mistake moment: teams add ten companion charts “just in case” and call it context. What to do instead is pick a small bundle where each companion metric has a job, either explaining movement, protecting quality, or preventing gaming.\n\n## Step 6: Make misreads harder through dashboard design\nNow you translate the risk register into design choices. Many misreads are predictable UI problems: missing denominators, unclear time windows, and charts that encourage cherry picking. Dashboard design guidance consistently warns about these patterns (https://www.clicdata.com/blog/preventing-dashboard-misreads/ and https://bi-academy.org/business-intelligence/why-most-dashboards-fail-and-how-to-avoid-it/).\n\nDesign patterns that work in executive settings.\n\nProminent definitions. Put the one line definition next to the KPI, not hidden in a wiki.\n\nShow denominators and volumes. If you show a rate, show the count.\n\nBaseline comparisons. Always include a sensible comparison like prior period and same period last year when seasonality matters.\n\nSeasonality overlays and annotations. Mark holidays, launches, outages, and policy changes on the chart.\n\nDefault segmentation. If a rollup can mislead, do not make it the default view.\n\nWarning states. Gray out or label periods where data is incomplete or backfilled.\n\nTooltips with provenance. Where the data came from, when it refreshed, and what changed recently.\n\n“Do not use for” note. A small line that prevents the KPI from being used as a blunt instrument.\n\nLight humor, because it is true: a dashboard without context is like a smoke alarm that only beeps in Morse code, technically accurate, emotionally unhelpful.\n\n## Step 7: Define decision rules and escalation paths\nThe final step is where you stop the KPI from becoming a weekly argument. You define what action looks like.\n\nStart with decision rules written as if then statements.\n\nIf the KPI moves more than X percent and volume is above the minimum threshold, then the metric owner opens an investigation within one business day.\n\nIf the KPI moves and one of the guardrail counter metrics crosses its threshold, then escalate to the cross functional owner group.\n\nIf data freshness warnings are active, then the KPI is not used for performance decisions until cleared.\n\nThen assign ownership and timing.\n\nWho acts first. Name a role, not a department.\n\nInvestigation SLA. How fast someone has to respond.\n\nEscalation path. When to pull in data engineering, finance, ops, or product.\n\nPause conditions. When the dashboard should be labeled as unreliable, or the metric temporarily removed from performance conversations.\n\nDocumentation location. Put the metric contract, bundle definition, and decision rules where users will actually look, ideally linked from the dashboard itself.\n\nA data pre mortem is not bureaucracy. It is how experienced teams avoid predictable failures: context loss, confounding, Goodhart gaming, and brittle pipelines that turn normal variation into leadership drama. Do the ninety minutes, ship the metric bundle, and make the “what do we do when it moves?” decision before the number starts making decisions for you.\n\n### Sources\n\n- [Dashboard Design Mistakes That Lead to Data Misreads](https://www.clicdata.com/blog/preventing-dashboard-misreads/)\n- [How to Conduct a \"Pre-Mortem\" to De-Risk Your Next Big Feature Launch by Problem Pop](https://www.problempop.io/blog-posts/how-to-conduct-a-pre-mortem-to-de-risk-your-next-big-feature-launch)\n- [Analytics Without Context Is Misinformation | by Tobenna Oduah](https://building.theatlantic.com/analytics-without-context-is-misinformation-0ae4b017a090)\n- [Pre-Mortem Analysis: How to Find What Will Go Wrong Before It Does | When Notes Fly](https://whennotesfly.com/concepts/decision-making/pre-mortem-analysis)\n- [Pre-Mortem](https://www.theuncertaintyproject.org/tools/pre-mortem)\n- [Confirmation Bias in Dashboard Design: Why Teams Only See What They Want to See](https://atticusli.com/blog/posts/confirmation-bias-dashboard-design-teams-see-what-they-want)\n- [The Missing Context Is Often the Most Important Part](https://answerhorizon.com/the-missing-context-is-often-the-most-important-part/)\n- [Why Most Dashboards Fail (And How to Avoid It) – BI Academy](https://bi-academy.org/business-intelligence/why-most-dashboards-fail-and-how-to-avoid-it/)\n- [KPI dictionary before dashboard build: define metrics](https://vladimirsiedykh.com/blog/kpi-dictionary-before-dashboard-build)\n- [Pre-Mortem KPIs: Understanding What Could Go Wrong](https://trulyintelligent.business/methods/pre-mortem-kpis/)\n\n---\n\n*Last updated: 2026-04-30* | *Calypso*","decision_systems_researcher",[14],"signal-vs-noise-why-organizations-misread-data","2026-04-30T10:05:02.950Z",false,{"title":18,"description":19,"ogDescription":19,"twitterDescription":19,"canonicalPath":9,"robots":20,"schemaType":21},"Before rolling out a new dashboard or KPI, how can we run a","Most dashboards do not fail because the math is wrong.","index,follow","QAPage",{"toc":23,"children":25,"html":26},{"links":24},[],[],"\u003Ch2>Answer\u003C/h2>\n\u003Cp>Run a data pre mortem as a short, structured workshop where the team assumes the new KPI failed in the real world and then works backward to explain how it got misread. You pre define the metric contract, list likely misinterpretations, map confounders, predict gaming, and then redesign the dashboard and decision rules so the most common mistakes are harder to make. Done well, it turns “we shipped a number” into “we shipped a number with guardrails, context, and a plan for what to do when it moves.”\u003C/p>\n\u003Cp>Most dashboards do not fail because the math is wrong. They fail because humans are busy, incentives are real, and context evaporates the moment the metric leaves the analyst’s screen. A data pre mortem is how you catch those predictable misreads before a KPI becomes a story people repeat in meetings.\u003C/p>\n\u003Ch2>What a “data pre mortem” is and when to use it\u003C/h2>\n\u003Cp>A data pre mortem is a risk rehearsal. You gather the people who will build, use, and be judged by a metric and you assume that, three months after launch, the KPI caused confusion or bad decisions. Then you ask a simple question: “What went wrong, and how did we let the metric mislead us?” This is different from data QA. QA asks whether the data is accurate and pipelines run. A pre mortem asks whether the metric will be interpreted correctly and used responsibly, even when the data is technically correct.\u003C/p>\n\u003Cp>Use it when any of these are true.\u003C/p>\n\u003Col>\n\u003Cli>\u003Cp>You are introducing a new KPI or redefining an old one.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>A new audience will consume the dashboard, especially executives or frontline managers who were not in the build.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>The metric will be linked to performance reviews, bonuses, targets, or budget decisions.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>You are making a major pipeline change such as a new event taxonomy, a source system migration, new identity stitching, or backfill logic.\u003C/p>\n\u003C/li>\n\u003C/ol>\n\u003Cp>If you only do one of these per quarter, do it for the KPI that could most easily become “analytics without context,” which is basically misinformation wearing a spreadsheet costume (see the context argument in \u003Ca href=\"#ref-1\" title=\"building.theatlantic.com — building.theatlantic.com\">[1]\u003C/a>).\u003C/p>\n\u003Ch2>Pre work: assemble the metric contract and decision context\u003C/h2>\n\u003Cp>The pre mortem works when everyone starts from the same definitions. Your goal is not a long spec. It is a one page “metric contract” plus a clear decision context.\u003C/p>\n\u003Cp>The metric contract should include the pieces that commonly get lost when a KPI is repeated secondhand. A KPI dictionary approach is a good model here \u003Ca href=\"#ref-2\" title=\"vladimirsiedykh.com — vladimirsiedykh.com\">[2]\u003C/a>.\u003C/p>\n\u003Cp>Include these items in plain language.\u003C/p>\n\u003Col>\n\u003Cli>\u003Cp>Metric name and purpose. What problem is it meant to detect or manage?\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Operational definition. The exact calculation, including numerator and denominator.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Time window and grain. Daily, weekly, trailing 28 days, cohort based, and so on.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Inclusion and exclusion rules. What counts, what does not, and why.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Default segmentation. Region, channel, product line, customer type, and the required “slices” that prevent misleading rollups.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Data lineage. Source systems, key transformations, and where the “source of truth” lives.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Refresh cadence and data latency. What “today” means.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Known constraints. Missing data, tracking gaps, small sample volatility, or any period where comparisons are unsafe.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Intended decisions and owners. Who is expected to act when it changes, and what decisions it is supposed to inform.\u003C/p>\n\u003C/li>\n\u003C/ol>\n\u003Cp>Practical tip: add one sentence titled “Do not use for.” For example: “Do not use this KPI to evaluate individual rep performance,” or “Do not use day to day changes to decide pricing.” This single line prevents a lot of creative misuse.\u003C/p>\n\u003Cp>Practical tip: include a minimum sample threshold, even if it is a simple rule like “no segmentation view below 200 events in the period.” It saves you from arguing with noise later.\u003C/p>\n\u003Ch2>90 minute workshop agenda (repeatable format)\u003C/h2>\n\u003Cp>Invite a small, mixed group. Eight to twelve people is ideal.\u003C/p>\n\u003Cp>Roles to include.\u003C/p>\n\u003Col>\n\u003Cli>\u003Cp>Metric owner. The business owner accountable for what happens when the KPI moves.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Analytics and data. The people who know definitions, caveats, and pipeline behavior.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Domain subject matter experts. The operators who know what is really happening in the field.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Frontline users. The managers or teams who will actually use the dashboard under time pressure.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Incentives partner if relevant. Finance, HR, sales ops, or whoever sets targets and compensation.\u003C/p>\n\u003C/li>\n\u003C/ol>\n\u003Cp>Artifacts to bring.\u003C/p>\n\u003Col>\n\u003Cli>\u003Cp>The one page metric contract.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>A rough dashboard mock, even if it is a sketch.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>A short glossary of terms and segments.\u003C/p>\n\u003C/li>\n\u003C/ol>\n\u003Cp>Time boxed agenda.\u003C/p>\n\u003Col>\n\u003Cli>\u003Cp>0 to 10 minutes. Align on the decision the metric is supposed to improve and the top two risks.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>10 to 25 minutes. Step 1 brainstorm: how this will be misread.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>25 to 45 minutes. Step 2 confounders and alternative explanations.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>45 to 60 minutes. Step 3 pre mortem Goodhart, predict gaming.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>60 to 70 minutes. Step 4 edge cases and pipeline brittleness.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>70 to 80 minutes. Step 5 build the metric bundle.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>80 to 88 minutes. Step 6 dashboard design changes to make misreads harder.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>88 to 90 minutes. Step 7 decision rules, escalation paths, and owners.\u003C/p>\n\u003C/li>\n\u003C/ol>\n\u003Cp>This “assume it failed, then explain why” structure borrows from classic pre mortem thinking (\u003Ca href=\"#ref-3\" title=\"whennotesfly.com — whennotesfly.com\">[3]\u003C/a> and \u003Ca href=\"#ref-4\" title=\"theuncertaintyproject.org — theuncertaintyproject.org\">[4]\u003C/a>), adapted to KPI rollouts.\u003C/p>\n\u003Ch2>Step 1: Brainstorm “how this will be misread” (failure modes library)\u003C/h2>\n\u003Cp>This step is quantity over quality. Ask everyone to write down misreads silently for three minutes, then share and cluster. Silence first matters because it reduces groupthink and status effects, a common issue in brainstorms \u003Ca href=\"#ref-5\" title=\"problempop.io — problempop.io\">[5]\u003C/a>.\u003C/p>\n\u003Cp>Here is a failure modes library you can reuse, organized the way misreads show up in real organizations.\u003C/p>\n\u003Cp>Context loss.\u003C/p>\n\u003Cp>People forget the denominator, base rate, or volume. A percentage moves but the underlying count collapses.\u003C/p>\n\u003Cp>People ignore seasonality or calendar effects. The metric “drops” every holiday week and everyone panics anyway.\u003C/p>\n\u003Cp>People miss the time horizon mismatch. A weekly KPI is used to judge a quarterly initiative.\u003C/p>\n\u003Cp>Definition and measurement ambiguity.\u003C/p>\n\u003Cp>Two teams use slightly different definitions for the same word, like “active,” “qualified,” or “resolved.”\u003C/p>\n\u003Cp>Rounding and formatting make small changes look meaningful.\u003C/p>\n\u003Cp>Aggregation hides the story.\u003C/p>\n\u003Cp>A company wide number looks stable while one region is on fire and another is booming.\u003C/p>\n\u003Cp>Simpson’s paradox shows up. The overall rate improves while every segment worsens, or vice versa.\u003C/p>\n\u003Cp>Bias and missingness.\u003C/p>\n\u003Cp>Selection bias. The metric only reflects customers who were measured, not all customers.\u003C/p>\n\u003Cp>Survivorship bias. You only see the users who stuck around.\u003C/p>\n\u003Cp>Missing data is not random. Tracking fails more on older devices, certain geos, or certain workflows.\u003C/p>\n\u003Cp>Timing and latency.\u003C/p>\n\u003Cp>Late arriving data makes the last few days look worse than they are.\u003C/p>\n\u003Cp>Operational lag means the KPI responds weeks after the real world change.\u003C/p>\n\u003Cp>Qualitative gaps.\u003C/p>\n\u003Cp>The number moves but nobody knows “what changed on the ground.” The KPI becomes a debate about narratives.\u003C/p>\n\u003Cp>Common mistake moment: teams often treat this step as a debate about who is right. Do not litigate the metric yet. Capture misreads first, then rank which ones are most damaging and most likely. You are building a risk register, not trying to win a meeting.\u003C/p>\n\u003Ch2>Step 2: Map confounders and alternative explanations (signal vs. noise)\u003C/h2>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Option\u003C/th>\n\u003Cth>Best for\u003C/th>\n\u003Cth>What you gain\u003C/th>\n\u003Cth>What you risk\u003C/th>\n\u003Cth>Choose if\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>A / B Testing / Quasi-Experiments\u003C/td>\n\u003Ctd>Establishing causality with high confidence\u003C/td>\n\u003Ctd>Strong evidence for cause-and-effect relationships\u003C/td>\n\u003Ctd>High setup cost. ethical considerations. not always feasible\u003C/td>\n\u003Ctd>You can randomly assign users/groups and control variables\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Expert Review / Brainstorming\u003C/td>\n\u003Ctd>Leveraging collective knowledge and experience\u003C/td>\n\u003Ctd>Identification of domain-specific confounders. diverse perspectives\u003C/td>\n\u003Ctd>Groupthink. reliance on anecdotal evidence\u003C/td>\n\u003Ctd>You have access to subject matter experts and need qualitative insights\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Causal Sketching / DAG-lite\u003C/td>\n\u003Ctd>Understanding direct and indirect relationships\u003C/td>\n\u003Ctd>Visual clarity of assumptions. identification of potential confounders\u003C/td>\n\u003Ctd>Oversimplification of complex systems. time investment\u003C/td>\n\u003Ctd>You need to map out how variables influence each other before analysis\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Control Views / Stratification\u003C/td>\n\u003Ctd>Isolating the impact of a variable\u003C/td>\n\u003Ctd>More precise measurement of specific effects\u003C/td>\n\u003Ctd>Over-segmentation leading to small sample sizes\u003C/td>\n\u003Ctd>You suspect a specific factor is influencing your primary metric\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>The 'What else changes?' Test\u003C/td>\n\u003Ctd>Quickly identifying obvious confounders\u003C/td>\n\u003Ctd>Fast, intuitive check for external factors\u003C/td>\n\u003Ctd>Missing subtle or non-obvious confounders\u003C/td>\n\u003Ctd>You're reviewing a metric or dashboard and need a rapid gut-check\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Defining 'Counter-Metrics'\u003C/td>\n\u003Ctd>Preventing unintended negative consequences\u003C/td>\n\u003Ctd>Holistic view of impact. early warning for adverse effects\u003C/td>\n\u003Ctd>Increased dashboard complexity. potential for analysis paralysis\u003C/td>\n\u003Ctd>Your primary metric has a clear potential downside or tradeoff\u003C/td>\n\u003C/tr>\n\u003C/tbody>\u003C/table>\n\u003Cp>Once you have misreads, you need a lightweight way to separate signal from noise. The core question is: “If this KPI moves, what else could be causing it besides the thing we care about?” This is where organizations most often overreact, because they mistake correlation plus urgency for causation.\u003C/p>\n\u003Cp>Use a simple method.\u003C/p>\n\u003Cp>First, do a causal sketch. Draw the KPI in the middle and add the top drivers around it: demand, supply constraints, product changes, pricing, marketing mix, customer composition, outages, policy changes, and competitor actions. Keep it “DAG lite” in spirit, not an academic exercise.\u003C/p>\n\u003Cp>Second, run the “What else changes?” test. If the KPI rises, what other operational indicators typically move at the same time? If it falls, what else tends to fall?\u003C/p>\n\u003Cp>Third, define control views. Decide which segments you will always check before telling a story. Typical control views are by channel, by cohort age, by region, and by customer type.\u003C/p>\n\u003Cp>Fourth, define counter metrics. For any KPI you want to push up, ask what you might accidentally push down.\u003C/p>\n\u003Cp>To make this concrete, here is a decision oriented menu of controls you can use.\u003C/p>\n\u003Cp>A / B Testing / Quasi-Experiments helps when you truly need cause and effect.\u003C/p>\n\u003Cp>Causal Sketching / DAG-lite forces assumptions into the open so people can disagree productively.\u003C/p>\n\u003Cp>Control Views / Stratification prevents a single blended number from driving the wrong story.\u003C/p>\n\u003Cp>The 'What else changes?' Test is the fastest way to catch obvious alternative explanations.\u003C/p>\n\u003Cp>Practical tip: choose two default “explainers” that always sit next to the KPI. Example: if conversion rate moves, always show traffic mix and page latency. It turns a vague story into a bounded investigation.\u003C/p>\n\u003Ch2>Step 3: Pre mortem Goodhart: predict gaming and perverse incentives\u003C/h2>\n\u003Cp>If the KPI will be used for targets or performance evaluation, assume it will be optimized. Not because people are bad, but because they are human and busy, and incentives are loud.\u003C/p>\n\u003Cp>Use three prompts.\u003C/p>\n\u003Cp>First: “If compensation or praise is tied to this number, how could someone win without creating real value?”\u003C/p>\n\u003Cp>Second: “What are the easiest levers to pull that move the metric quickly?” Quick levers are where gaming tends to appear.\u003C/p>\n\u003Cp>Third: “What process loopholes does the metric create?” For example, reclassifying cases, delaying work until the reporting window resets, or routing easy customers to the measured channel.\u003C/p>\n\u003Cp>Then add guardrails. You can combine several without making the dashboard unreadable.\u003C/p>\n\u003Col>\n\u003Cli>\u003Cp>Pair the KPI with a quality counter metric. If you push for faster resolution, also watch reopen rate or customer satisfaction.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Add audit checks or random sampling. If a category can be manipulated, sample records and verify.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Use caps and floors. If you know a metric is volatile at low volumes, suppress or gray out the view.\u003C/p>\n\u003C/li>\n\u003Cli>\u003Cp>Add a lightweight qualitative review. A short “what changed operationally?” note from the owner reduces confirmation bias, which is a real dashboard problem \u003Ca href=\"#ref-6\" title=\"atticusli.com — atticusli.com\">[6]\u003C/a>.\u003C/p>\n\u003C/li>\n\u003C/ol>\n\u003Ch2>Step 4: Stress test edge cases and data pipeline brittleness\u003C/h2>\n\u003Cp>This step is where you protect executives from the “we had an incident but the chart did not mention it” moment.\u003C/p>\n\u003Cp>Walk through these edge cases.\u003C/p>\n\u003Cp>Late arriving data and backfills. Decide whether the most recent days are labeled as preliminary.\u003C/p>\n\u003Cp>Schema and tracking changes. If event names or fields change, what breaks and how will you know?\u003C/p>\n\u003Cp>Outages and partial ingestion. What does the KPI look like during a logging outage, and does the dashboard warn users?\u003C/p>\n\u003Cp>Reclassification. What happens when definitions change, like a new status taxonomy or a new fraud rule?\u003C/p>\n\u003Cp>Small n volatility. If you slice too far, the metric becomes a coin flip with a nice font.\u003C/p>\n\u003Cp>Versioning and change logs. When the definition changes, can a user tell which version they are viewing?\u003C/p>\n\u003Cp>You do not need a deep technical runbook here. You do need a clear agreement on what the dashboard should do during weird weeks, because weird weeks happen.\u003C/p>\n\u003Ch2>Step 5: Build the “metric bundle” (primary + counter metrics + context)\u003C/h2>\n\u003Cp>A single KPI invites over interpretation. A metric bundle reduces that risk by making the most important context visible by default. This aligns with the broader point that missing context is often the most important part \u003Ca href=\"#ref-7\" title=\"answerhorizon.com — answerhorizon.com\">[7]\u003C/a>.\u003C/p>\n\u003Cp>Use this template.\u003C/p>\n\u003Cp>Primary KPI. The headline number.\u003C/p>\n\u003Cp>Input or leading indicators. The controllable drivers you expect to move first.\u003C/p>\n\u003Cp>Outcome or lagging indicators. The business result you actually care about.\u003C/p>\n\u003Cp>Quality and safety counter metrics. What you do not want to break while improving the KPI.\u003C/p>\n\u003Cp>Guardrail thresholds. Red lines that trigger investigation.\u003C/p>\n\u003Cp>Default segments. The two or three cuts that prevent misleading averages.\u003C/p>\n\u003Cp>Context notes. Seasonality, known incidents, definition changes.\u003C/p>\n\u003Cp>Example. If your primary KPI is ecommerce conversion rate, the bundle might include traffic quality (new versus returning, channel mix), page performance, average order value, return rate, and customer support contact rate. If conversion rises while return rate spikes, you know you may be “selling” the wrong thing.\u003C/p>\n\u003Cp>Common mistake moment: teams add ten companion charts “just in case” and call it context. What to do instead is pick a small bundle where each companion metric has a job, either explaining movement, protecting quality, or preventing gaming.\u003C/p>\n\u003Ch2>Step 6: Make misreads harder through dashboard design\u003C/h2>\n\u003Cp>Now you translate the risk register into design choices. Many misreads are predictable UI problems: missing denominators, unclear time windows, and charts that encourage cherry picking. Dashboard design guidance consistently warns about these patterns (\u003Ca href=\"#ref-8\" title=\"clicdata.com — clicdata.com\">[8]\u003C/a> and \u003Ca href=\"#ref-9\" title=\"bi-academy.org — bi-academy.org\">[9]\u003C/a>).\u003C/p>\n\u003Cp>Design patterns that work in executive settings.\u003C/p>\n\u003Cp>Prominent definitions. Put the one line definition next to the KPI, not hidden in a wiki.\u003C/p>\n\u003Cp>Show denominators and volumes. If you show a rate, show the count.\u003C/p>\n\u003Cp>Baseline comparisons. Always include a sensible comparison like prior period and same period last year when seasonality matters.\u003C/p>\n\u003Cp>Seasonality overlays and annotations. Mark holidays, launches, outages, and policy changes on the chart.\u003C/p>\n\u003Cp>Default segmentation. If a rollup can mislead, do not make it the default view.\u003C/p>\n\u003Cp>Warning states. Gray out or label periods where data is incomplete or backfilled.\u003C/p>\n\u003Cp>Tooltips with provenance. Where the data came from, when it refreshed, and what changed recently.\u003C/p>\n\u003Cp>“Do not use for” note. A small line that prevents the KPI from being used as a blunt instrument.\u003C/p>\n\u003Cp>Light humor, because it is true: a dashboard without context is like a smoke alarm that only beeps in Morse code, technically accurate, emotionally unhelpful.\u003C/p>\n\u003Ch2>Step 7: Define decision rules and escalation paths\u003C/h2>\n\u003Cp>The final step is where you stop the KPI from becoming a weekly argument. You define what action looks like.\u003C/p>\n\u003Cp>Start with decision rules written as if then statements.\u003C/p>\n\u003Cp>If the KPI moves more than X percent and volume is above the minimum threshold, then the metric owner opens an investigation within one business day.\u003C/p>\n\u003Cp>If the KPI moves and one of the guardrail counter metrics crosses its threshold, then escalate to the cross functional owner group.\u003C/p>\n\u003Cp>If data freshness warnings are active, then the KPI is not used for performance decisions until cleared.\u003C/p>\n\u003Cp>Then assign ownership and timing.\u003C/p>\n\u003Cp>Who acts first. Name a role, not a department.\u003C/p>\n\u003Cp>Investigation SLA. How fast someone has to respond.\u003C/p>\n\u003Cp>Escalation path. When to pull in data engineering, finance, ops, or product.\u003C/p>\n\u003Cp>Pause conditions. When the dashboard should be labeled as unreliable, or the metric temporarily removed from performance conversations.\u003C/p>\n\u003Cp>Documentation location. Put the metric contract, bundle definition, and decision rules where users will actually look, ideally linked from the dashboard itself.\u003C/p>\n\u003Cp>A data pre mortem is not bureaucracy. It is how experienced teams avoid predictable failures: context loss, confounding, Goodhart gaming, and brittle pipelines that turn normal variation into leadership drama. Do the ninety minutes, ship the metric bundle, and make the “what do we do when it moves?” decision before the number starts making decisions for you.\u003C/p>\n\u003Ch3>Sources\u003C/h3>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https://www.clicdata.com/blog/preventing-dashboard-misreads/\">Dashboard Design Mistakes That Lead to Data Misreads\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.problempop.io/blog-posts/how-to-conduct-a-pre-mortem-to-de-risk-your-next-big-feature-launch\">How to Conduct a "Pre-Mortem" to De-Risk Your Next Big Feature Launch by Problem Pop\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://building.theatlantic.com/analytics-without-context-is-misinformation-0ae4b017a090\">Analytics Without Context Is Misinformation | by Tobenna Oduah\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://whennotesfly.com/concepts/decision-making/pre-mortem-analysis\">Pre-Mortem Analysis: How to Find What Will Go Wrong Before It Does | When Notes Fly\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.theuncertaintyproject.org/tools/pre-mortem\">Pre-Mortem\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://atticusli.com/blog/posts/confirmation-bias-dashboard-design-teams-see-what-they-want\">Confirmation Bias in Dashboard Design: Why Teams Only See What They Want to See\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://answerhorizon.com/the-missing-context-is-often-the-most-important-part/\">The Missing Context Is Often the Most Important Part\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://bi-academy.org/business-intelligence/why-most-dashboards-fail-and-how-to-avoid-it/\">Why Most Dashboards Fail (And How to Avoid It) – BI Academy\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://vladimirsiedykh.com/blog/kpi-dictionary-before-dashboard-build\">KPI dictionary before dashboard build: define metrics\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://trulyintelligent.business/methods/pre-mortem-kpis/\">Pre-Mortem KPIs: Understanding What Could Go Wrong\u003C/a>\u003C/li>\n\u003C/ul>\n\u003Chr>\n\u003Cp>\u003Cem>Last updated: 2026-04-30\u003C/em> | \u003Cem>Calypso\u003C/em>\u003C/p>\n\u003Ch2>Sources\u003C/h2>\n\u003Col>\n\u003Cli>\u003Ca href=\"https://building.theatlantic.com/analytics-without-context-is-misinformation-0ae4b017a090\">building.theatlantic.com\u003C/a> — building.theatlantic.com\u003C/li>\n\u003Cli>\u003Ca href=\"https://vladimirsiedykh.com/blog/kpi-dictionary-before-dashboard-build\">vladimirsiedykh.com\u003C/a> — vladimirsiedykh.com\u003C/li>\n\u003Cli>\u003Ca href=\"https://whennotesfly.com/concepts/decision-making/pre-mortem-analysis\">whennotesfly.com\u003C/a> — whennotesfly.com\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.theuncertaintyproject.org/tools/pre-mortem\">theuncertaintyproject.org\u003C/a> — theuncertaintyproject.org\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.problempop.io/blog-posts/how-to-conduct-a-pre-mortem-to-de-risk-your-next-big-feature-launch\">problempop.io\u003C/a> — problempop.io\u003C/li>\n\u003Cli>\u003Ca href=\"https://atticusli.com/blog/posts/confirmation-bias-dashboard-design-teams-see-what-they-want\">atticusli.com\u003C/a> — atticusli.com\u003C/li>\n\u003Cli>\u003Ca href=\"https://answerhorizon.com/the-missing-context-is-often-the-most-important-part\">answerhorizon.com\u003C/a> — answerhorizon.com\u003C/li>\n\u003Cli>\u003Ca href=\"https://www.clicdata.com/blog/preventing-dashboard-misreads\">clicdata.com\u003C/a> — clicdata.com\u003C/li>\n\u003Cli>\u003Ca href=\"https://bi-academy.org/business-intelligence/why-most-dashboards-fail-and-how-to-avoid-it\">bi-academy.org\u003C/a> — bi-academy.org\u003C/li>\n\u003C/ol>\n",{"body":28},"## Answer\n\nRun a data pre mortem as a short, structured workshop where the team assumes the new KPI failed in the real world and then works backward to explain how it got misread. You pre define the metric contract, list likely misinterpretations, map confounders, predict gaming, and then redesign the dashboard and decision rules so the most common mistakes are harder to make. Done well, it turns “we shipped a number” into “we shipped a number with guardrails, context, and a plan for what to do when it moves.”\n\nMost dashboards do not fail because the math is wrong. They fail because humans are busy, incentives are real, and context evaporates the moment the metric leaves the analyst’s screen. A data pre mortem is how you catch those predictable misreads before a KPI becomes a story people repeat in meetings.\n\n## What a “data pre mortem” is and when to use it\nA data pre mortem is a risk rehearsal. You gather the people who will build, use, and be judged by a metric and you assume that, three months after launch, the KPI caused confusion or bad decisions. Then you ask a simple question: “What went wrong, and how did we let the metric mislead us?” This is different from data QA. QA asks whether the data is accurate and pipelines run. A pre mortem asks whether the metric will be interpreted correctly and used responsibly, even when the data is technically correct.\n\nUse it when any of these are true.\n\n1) You are introducing a new KPI or redefining an old one.\n\n2) A new audience will consume the dashboard, especially executives or frontline managers who were not in the build.\n\n3) The metric will be linked to performance reviews, bonuses, targets, or budget decisions.\n\n4) You are making a major pipeline change such as a new event taxonomy, a source system migration, new identity stitching, or backfill logic.\n\nIf you only do one of these per quarter, do it for the KPI that could most easily become “analytics without context,” which is basically misinformation wearing a spreadsheet costume (see the context argument in [[1]](#ref-1 \"building.theatlantic.com — building.theatlantic.com\")).\n\n## Pre work: assemble the metric contract and decision context\nThe pre mortem works when everyone starts from the same definitions. Your goal is not a long spec. It is a one page “metric contract” plus a clear decision context.\n\nThe metric contract should include the pieces that commonly get lost when a KPI is repeated secondhand. A KPI dictionary approach is a good model here [[2]](#ref-2 \"vladimirsiedykh.com — vladimirsiedykh.com\").\n\nInclude these items in plain language.\n\n1) Metric name and purpose. What problem is it meant to detect or manage?\n\n2) Operational definition. The exact calculation, including numerator and denominator.\n\n3) Time window and grain. Daily, weekly, trailing 28 days, cohort based, and so on.\n\n4) Inclusion and exclusion rules. What counts, what does not, and why.\n\n5) Default segmentation. Region, channel, product line, customer type, and the required “slices” that prevent misleading rollups.\n\n6) Data lineage. Source systems, key transformations, and where the “source of truth” lives.\n\n7) Refresh cadence and data latency. What “today” means.\n\n8) Known constraints. Missing data, tracking gaps, small sample volatility, or any period where comparisons are unsafe.\n\n9) Intended decisions and owners. Who is expected to act when it changes, and what decisions it is supposed to inform.\n\nPractical tip: add one sentence titled “Do not use for.” For example: “Do not use this KPI to evaluate individual rep performance,” or “Do not use day to day changes to decide pricing.” This single line prevents a lot of creative misuse.\n\nPractical tip: include a minimum sample threshold, even if it is a simple rule like “no segmentation view below 200 events in the period.” It saves you from arguing with noise later.\n\n## 90 minute workshop agenda (repeatable format)\nInvite a small, mixed group. Eight to twelve people is ideal.\n\nRoles to include.\n\n1) Metric owner. The business owner accountable for what happens when the KPI moves.\n\n2) Analytics and data. The people who know definitions, caveats, and pipeline behavior.\n\n3) Domain subject matter experts. The operators who know what is really happening in the field.\n\n4) Frontline users. The managers or teams who will actually use the dashboard under time pressure.\n\n5) Incentives partner if relevant. Finance, HR, sales ops, or whoever sets targets and compensation.\n\nArtifacts to bring.\n\n1) The one page metric contract.\n\n2) A rough dashboard mock, even if it is a sketch.\n\n3) A short glossary of terms and segments.\n\nTime boxed agenda.\n\n1) 0 to 10 minutes. Align on the decision the metric is supposed to improve and the top two risks.\n\n2) 10 to 25 minutes. Step 1 brainstorm: how this will be misread.\n\n3) 25 to 45 minutes. Step 2 confounders and alternative explanations.\n\n4) 45 to 60 minutes. Step 3 pre mortem Goodhart, predict gaming.\n\n5) 60 to 70 minutes. Step 4 edge cases and pipeline brittleness.\n\n6) 70 to 80 minutes. Step 5 build the metric bundle.\n\n7) 80 to 88 minutes. Step 6 dashboard design changes to make misreads harder.\n\n8) 88 to 90 minutes. Step 7 decision rules, escalation paths, and owners.\n\nThis “assume it failed, then explain why” structure borrows from classic pre mortem thinking ([[3]](#ref-3 \"whennotesfly.com — whennotesfly.com\") and [[4]](#ref-4 \"theuncertaintyproject.org — theuncertaintyproject.org\")), adapted to KPI rollouts.\n\n## Step 1: Brainstorm “how this will be misread” (failure modes library)\nThis step is quantity over quality. Ask everyone to write down misreads silently for three minutes, then share and cluster. Silence first matters because it reduces groupthink and status effects, a common issue in brainstorms [[5]](#ref-5 \"problempop.io — problempop.io\").\n\nHere is a failure modes library you can reuse, organized the way misreads show up in real organizations.\n\nContext loss.\n\nPeople forget the denominator, base rate, or volume. A percentage moves but the underlying count collapses.\n\nPeople ignore seasonality or calendar effects. The metric “drops” every holiday week and everyone panics anyway.\n\nPeople miss the time horizon mismatch. A weekly KPI is used to judge a quarterly initiative.\n\nDefinition and measurement ambiguity.\n\nTwo teams use slightly different definitions for the same word, like “active,” “qualified,” or “resolved.”\n\nRounding and formatting make small changes look meaningful.\n\nAggregation hides the story.\n\nA company wide number looks stable while one region is on fire and another is booming.\n\nSimpson’s paradox shows up. The overall rate improves while every segment worsens, or vice versa.\n\nBias and missingness.\n\nSelection bias. The metric only reflects customers who were measured, not all customers.\n\nSurvivorship bias. You only see the users who stuck around.\n\nMissing data is not random. Tracking fails more on older devices, certain geos, or certain workflows.\n\nTiming and latency.\n\nLate arriving data makes the last few days look worse than they are.\n\nOperational lag means the KPI responds weeks after the real world change.\n\nQualitative gaps.\n\nThe number moves but nobody knows “what changed on the ground.” The KPI becomes a debate about narratives.\n\nCommon mistake moment: teams often treat this step as a debate about who is right. Do not litigate the metric yet. Capture misreads first, then rank which ones are most damaging and most likely. You are building a risk register, not trying to win a meeting.\n\n## Step 2: Map confounders and alternative explanations (signal vs. noise)\n\n| Option | Best for | What you gain | What you risk | Choose if |\n| --- | --- | --- | --- | --- |\n| A / B Testing / Quasi-Experiments | Establishing causality with high confidence | Strong evidence for cause-and-effect relationships | High setup cost. ethical considerations. not always feasible | You can randomly assign users/groups and control variables |\n| Expert Review / Brainstorming | Leveraging collective knowledge and experience | Identification of domain-specific confounders. diverse perspectives | Groupthink. reliance on anecdotal evidence | You have access to subject matter experts and need qualitative insights |\n| Causal Sketching / DAG-lite | Understanding direct and indirect relationships | Visual clarity of assumptions. identification of potential confounders | Oversimplification of complex systems. time investment | You need to map out how variables influence each other before analysis |\n| Control Views / Stratification | Isolating the impact of a variable | More precise measurement of specific effects | Over-segmentation leading to small sample sizes | You suspect a specific factor is influencing your primary metric |\n| The 'What else changes?' Test | Quickly identifying obvious confounders | Fast, intuitive check for external factors | Missing subtle or non-obvious confounders | You're reviewing a metric or dashboard and need a rapid gut-check |\n| Defining 'Counter-Metrics' | Preventing unintended negative consequences | Holistic view of impact. early warning for adverse effects | Increased dashboard complexity. potential for analysis paralysis | Your primary metric has a clear potential downside or tradeoff |\n\nOnce you have misreads, you need a lightweight way to separate signal from noise. The core question is: “If this KPI moves, what else could be causing it besides the thing we care about?” This is where organizations most often overreact, because they mistake correlation plus urgency for causation.\n\nUse a simple method.\n\nFirst, do a causal sketch. Draw the KPI in the middle and add the top drivers around it: demand, supply constraints, product changes, pricing, marketing mix, customer composition, outages, policy changes, and competitor actions. Keep it “DAG lite” in spirit, not an academic exercise.\n\nSecond, run the “What else changes?” test. If the KPI rises, what other operational indicators typically move at the same time? If it falls, what else tends to fall?\n\nThird, define control views. Decide which segments you will always check before telling a story. Typical control views are by channel, by cohort age, by region, and by customer type.\n\nFourth, define counter metrics. For any KPI you want to push up, ask what you might accidentally push down.\n\nTo make this concrete, here is a decision oriented menu of controls you can use.\n\nA / B Testing / Quasi-Experiments helps when you truly need cause and effect.\n\nCausal Sketching / DAG-lite forces assumptions into the open so people can disagree productively.\n\nControl Views / Stratification prevents a single blended number from driving the wrong story.\n\nThe 'What else changes?' Test is the fastest way to catch obvious alternative explanations.\n\nPractical tip: choose two default “explainers” that always sit next to the KPI. Example: if conversion rate moves, always show traffic mix and page latency. It turns a vague story into a bounded investigation.\n\n## Step 3: Pre mortem Goodhart: predict gaming and perverse incentives\nIf the KPI will be used for targets or performance evaluation, assume it will be optimized. Not because people are bad, but because they are human and busy, and incentives are loud.\n\nUse three prompts.\n\nFirst: “If compensation or praise is tied to this number, how could someone win without creating real value?”\n\nSecond: “What are the easiest levers to pull that move the metric quickly?” Quick levers are where gaming tends to appear.\n\nThird: “What process loopholes does the metric create?” For example, reclassifying cases, delaying work until the reporting window resets, or routing easy customers to the measured channel.\n\nThen add guardrails. You can combine several without making the dashboard unreadable.\n\n1) Pair the KPI with a quality counter metric. If you push for faster resolution, also watch reopen rate or customer satisfaction.\n\n2) Add audit checks or random sampling. If a category can be manipulated, sample records and verify.\n\n3) Use caps and floors. If you know a metric is volatile at low volumes, suppress or gray out the view.\n\n4) Add a lightweight qualitative review. A short “what changed operationally?” note from the owner reduces confirmation bias, which is a real dashboard problem [[6]](#ref-6 \"atticusli.com — atticusli.com\").\n\n## Step 4: Stress test edge cases and data pipeline brittleness\nThis step is where you protect executives from the “we had an incident but the chart did not mention it” moment.\n\nWalk through these edge cases.\n\nLate arriving data and backfills. Decide whether the most recent days are labeled as preliminary.\n\nSchema and tracking changes. If event names or fields change, what breaks and how will you know?\n\nOutages and partial ingestion. What does the KPI look like during a logging outage, and does the dashboard warn users?\n\nReclassification. What happens when definitions change, like a new status taxonomy or a new fraud rule?\n\nSmall n volatility. If you slice too far, the metric becomes a coin flip with a nice font.\n\nVersioning and change logs. When the definition changes, can a user tell which version they are viewing?\n\nYou do not need a deep technical runbook here. You do need a clear agreement on what the dashboard should do during weird weeks, because weird weeks happen.\n\n## Step 5: Build the “metric bundle” (primary + counter metrics + context)\nA single KPI invites over interpretation. A metric bundle reduces that risk by making the most important context visible by default. This aligns with the broader point that missing context is often the most important part [[7]](#ref-7 \"answerhorizon.com — answerhorizon.com\").\n\nUse this template.\n\nPrimary KPI. The headline number.\n\nInput or leading indicators. The controllable drivers you expect to move first.\n\nOutcome or lagging indicators. The business result you actually care about.\n\nQuality and safety counter metrics. What you do not want to break while improving the KPI.\n\nGuardrail thresholds. Red lines that trigger investigation.\n\nDefault segments. The two or three cuts that prevent misleading averages.\n\nContext notes. Seasonality, known incidents, definition changes.\n\nExample. If your primary KPI is ecommerce conversion rate, the bundle might include traffic quality (new versus returning, channel mix), page performance, average order value, return rate, and customer support contact rate. If conversion rises while return rate spikes, you know you may be “selling” the wrong thing.\n\nCommon mistake moment: teams add ten companion charts “just in case” and call it context. What to do instead is pick a small bundle where each companion metric has a job, either explaining movement, protecting quality, or preventing gaming.\n\n## Step 6: Make misreads harder through dashboard design\nNow you translate the risk register into design choices. Many misreads are predictable UI problems: missing denominators, unclear time windows, and charts that encourage cherry picking. Dashboard design guidance consistently warns about these patterns ([[8]](#ref-8 \"clicdata.com — clicdata.com\") and [[9]](#ref-9 \"bi-academy.org — bi-academy.org\")).\n\nDesign patterns that work in executive settings.\n\nProminent definitions. Put the one line definition next to the KPI, not hidden in a wiki.\n\nShow denominators and volumes. If you show a rate, show the count.\n\nBaseline comparisons. Always include a sensible comparison like prior period and same period last year when seasonality matters.\n\nSeasonality overlays and annotations. Mark holidays, launches, outages, and policy changes on the chart.\n\nDefault segmentation. If a rollup can mislead, do not make it the default view.\n\nWarning states. Gray out or label periods where data is incomplete or backfilled.\n\nTooltips with provenance. Where the data came from, when it refreshed, and what changed recently.\n\n“Do not use for” note. A small line that prevents the KPI from being used as a blunt instrument.\n\nLight humor, because it is true: a dashboard without context is like a smoke alarm that only beeps in Morse code, technically accurate, emotionally unhelpful.\n\n## Step 7: Define decision rules and escalation paths\nThe final step is where you stop the KPI from becoming a weekly argument. You define what action looks like.\n\nStart with decision rules written as if then statements.\n\nIf the KPI moves more than X percent and volume is above the minimum threshold, then the metric owner opens an investigation within one business day.\n\nIf the KPI moves and one of the guardrail counter metrics crosses its threshold, then escalate to the cross functional owner group.\n\nIf data freshness warnings are active, then the KPI is not used for performance decisions until cleared.\n\nThen assign ownership and timing.\n\nWho acts first. Name a role, not a department.\n\nInvestigation SLA. How fast someone has to respond.\n\nEscalation path. When to pull in data engineering, finance, ops, or product.\n\nPause conditions. When the dashboard should be labeled as unreliable, or the metric temporarily removed from performance conversations.\n\nDocumentation location. Put the metric contract, bundle definition, and decision rules where users will actually look, ideally linked from the dashboard itself.\n\nA data pre mortem is not bureaucracy. It is how experienced teams avoid predictable failures: context loss, confounding, Goodhart gaming, and brittle pipelines that turn normal variation into leadership drama. Do the ninety minutes, ship the metric bundle, and make the “what do we do when it moves?” decision before the number starts making decisions for you.\n\n### Sources\n\n- [Dashboard Design Mistakes That Lead to Data Misreads](https://www.clicdata.com/blog/preventing-dashboard-misreads/)\n- [How to Conduct a \"Pre-Mortem\" to De-Risk Your Next Big Feature Launch by Problem Pop](https://www.problempop.io/blog-posts/how-to-conduct-a-pre-mortem-to-de-risk-your-next-big-feature-launch)\n- [Analytics Without Context Is Misinformation | by Tobenna Oduah](https://building.theatlantic.com/analytics-without-context-is-misinformation-0ae4b017a090)\n- [Pre-Mortem Analysis: How to Find What Will Go Wrong Before It Does | When Notes Fly](https://whennotesfly.com/concepts/decision-making/pre-mortem-analysis)\n- [Pre-Mortem](https://www.theuncertaintyproject.org/tools/pre-mortem)\n- [Confirmation Bias in Dashboard Design: Why Teams Only See What They Want to See](https://atticusli.com/blog/posts/confirmation-bias-dashboard-design-teams-see-what-they-want)\n- [The Missing Context Is Often the Most Important Part](https://answerhorizon.com/the-missing-context-is-often-the-most-important-part/)\n- [Why Most Dashboards Fail (And How to Avoid It) – BI Academy](https://bi-academy.org/business-intelligence/why-most-dashboards-fail-and-how-to-avoid-it/)\n- [KPI dictionary before dashboard build: define metrics](https://vladimirsiedykh.com/blog/kpi-dictionary-before-dashboard-build)\n- [Pre-Mortem KPIs: Understanding What Could Go Wrong](https://trulyintelligent.business/methods/pre-mortem-kpis/)\n\n---\n\n*Last updated: 2026-04-30* | *Calypso*\n\n## Sources\n\n1. [building.theatlantic.com](https://building.theatlantic.com/analytics-without-context-is-misinformation-0ae4b017a090) — building.theatlantic.com\n2. [vladimirsiedykh.com](https://vladimirsiedykh.com/blog/kpi-dictionary-before-dashboard-build) — vladimirsiedykh.com\n3. [whennotesfly.com](https://whennotesfly.com/concepts/decision-making/pre-mortem-analysis) — whennotesfly.com\n4. [theuncertaintyproject.org](https://www.theuncertaintyproject.org/tools/pre-mortem) — theuncertaintyproject.org\n5. [problempop.io](https://www.problempop.io/blog-posts/how-to-conduct-a-pre-mortem-to-de-risk-your-next-big-feature-launch) — problempop.io\n6. [atticusli.com](https://atticusli.com/blog/posts/confirmation-bias-dashboard-design-teams-see-what-they-want) — atticusli.com\n7. [answerhorizon.com](https://answerhorizon.com/the-missing-context-is-often-the-most-important-part) — answerhorizon.com\n8. [clicdata.com](https://www.clicdata.com/blog/preventing-dashboard-misreads) — clicdata.com\n9. [bi-academy.org](https://bi-academy.org/business-intelligence/why-most-dashboards-fail-and-how-to-avoid-it) — bi-academy.org\n",{"date":15,"authors":30},[31],{"name":32,"description":33,"avatar":34},"Lucía Ferrer","Calypso AI · Clear, expert-led guides for operators and buyers",{"src":35},"https://api.dicebear.com/9.x/personas/svg?seed=calypso_expert_guide_v1&backgroundColor=b6e3f4,c0aede,d1d4f9,ffd5dc,ffdfbf",[37,40,44,48,52,55],{"slug":38,"name":38,"description":39},"support_systems_architect","These topics should stay grounded in real support workflow design, escalation logic, routing, SLAs, handoffs, and the messy reality of serving customers when volume spikes and patience drops.\n\nWrite like someone who has watched support automation fail at the escalation layer, seen teams confuse a chatbot with a support system, and knows exactly which shortcuts create rework later. Keep it useful and engaging: practical tips, failure-mode awareness, a touch of humor, and SEO angles tied to real operational questions support leaders actually search for.\n\nPriority storylines:\n- What support leaders should fix first when volume jumps and quality slips\n- When to route, resolve, escalate, or hand off without losing the thread\n- How to balance speed and quality when customers demand both at once\n- Where duplicate threads and fuzzy ownership start making support feel blind\n- What branch teams should watch besides ticket counts\n- Which warning signs show up before a support mess becomes obvious",{"slug":41,"name":42,"description":43},"revenue_workflow_strategist","Lead capture, qualification, and conversion systems","These topics should stay authoritative on lead capture, qualification, routing, scheduling, follow-up, and the awkward little leaks that quietly kill pipeline before sales blames marketing.\n\nWrite like a revenue operator who has seen junk leads flood inboxes, 'fast response' turn into low-quality chaos, and automations help only when the logic is brutally clear. The tone should be expert, practical, slightly opinionated, and engaging enough that readers feel guided instead of lectured. Strong SEO should come from high-intent workflow questions, not generic funnel chatter.\n\nPriority storylines:\n- Which inquiries deserve real energy and which ones need a graceful filter\n- What makes fast follow-up feel useful instead of chaotic\n- How teams route urgency, fit, and buying stage without turning ops into a maze\n- Where WhatsApp lead capture helps and where it quietly creates junk\n- What to automate first when the pipeline is leaking in five places at once\n- Why shared context often converts better than simply replying faster",{"slug":45,"name":46,"description":47},"conversational_infrastructure_operator","Messaging infrastructure and workflow reliability","These topics should sound grounded in real messaging operations that have already lived through retries, duplicates, broken handoffs, and the 2 a.m. dashboard panic nobody wants to repeat.\n\nWrite for operators and leaders who need reliability without being buried in infrastructure jargon. Keep the tone practical, confident, and human: tips that save time, common mistakes that quietly wreck reporting, and the occasional line that makes the pain feel familiar instead of robotic. Strong SEO angles should still be specific and high-intent.\n\nPriority storylines:\n- When branch numbers start looking better than the customer experience feels\n- How teams keep context intact when conversations move across people and channels\n- What leaders should fix first when messaging operations start feeling messy\n- Where duplicate activity quietly distorts dashboards and confidence\n- Which habits restore trust faster than another round of heroic firefighting\n- What 'ready for real volume' looks like when you strip away the swagger",{"slug":49,"name":50,"description":51},"growth_experimentation_architect","Growth systems, lifecycle messaging, and experimentation","These topics should show a sharp understanding of activation, retention, re-engagement, lifecycle messaging, and growth experimentation without slipping into generic personalization talk.\n\nWrite like someone who has seen onboarding flows underperform, win-back campaigns overstay their welcome, and A/B tests prove something useless with great confidence. Make it engaging, specific, and commercially smart: practical tips, what people get wrong, tasteful humor, and search-friendly angles that map to real buyer/operator intent.\n\nPriority storylines:\n- What an honest first-win moment in activation actually looks like\n- How re-engagement can feel timely instead of clingy\n- When trigger-first thinking helps and when segment-first wins\n- Which experiments deserve attention and which are just theater\n- How shared context changes retention more than one more campaign\n- What growth teams usually notice too late in lifecycle messaging",{"slug":12,"name":53,"description":54},"Research, signal design, and decision systems","These topics should turn messy signals, conversations, and branch-level events into trustworthy decisions without sounding academic or technical for the sake of it.\n\nWrite like an experienced advisor who knows that bad data usually looks fine right up until a team makes a confident wrong decision. Bring judgment, practical tips, and a little wit. The reader should leave with sharper instincts about what to trust, what to measure, and what usually goes wrong first. Keep the SEO intent strong by favoring concrete, decision-shaped subtopics over abstract thought leadership.\n\nPriority storylines:\n- Which branch numbers deserve trust and which are just polished noise\n- How to spot dirty signal before a confident meeting goes off the rails\n- When leaders should trust automation and when they still need human judgment\n- How to turn messy evidence into usable insight without cleaning away the truth\n- What teams repeatedly misread when comparing branches, conversations, and attribution\n- How to build a signal culture that helps decisions happen, not just slides",{"slug":56,"name":57,"description":58},"vertical_operations_strategist","Industry-specific authority topics","These topics should map cleanly to how each industry actually operates and feel unusually credible inside real operating environments, not generic across sectors.\n\nWrite like a strategist who understands that clinics, retail, real estate, education, logistics, professional services, and fintech each break in their own charming way. Keep the voice expert, practical, and engaging, with field-tested tips, sharp tradeoffs, and examples that feel rooted in how teams actually work. SEO should come from highly specific, industry-shaped searches with clear workflow intent.\n\nPriority storylines by vertical:\n- Clinics: what keeps schedules moving when patients refuse to behave like calendars\n- Retail: how teams stay calm when demand spikes and patience disappears\n- Real estate: what serious follow-up looks like after the first inquiry\n- Education: how admissions feels smoother when reminders and handoffs stop fighting each other\n- Professional services: how intake and approvals stay clear when requests get messy\n- Logistics and fintech: what keeps urgent cases controlled without slowing the business",1778614437036]