[{"data":1,"prerenderedAt":47},["ShallowReactive",2],{"/en/blog/how-to-run-a-pre-mortem-on-your-data-finding-the-assumptions-that-will-fail-firs":3,"/en/blog/how-to-run-a-pre-mortem-on-your-data-finding-the-assumptions-that-will-fail-firs-surround":38},{"id":4,"locale":5,"translationGroupId":6,"availableLocales":7,"alternates":8,"_path":9,"path":9,"title":10,"description":11,"date":12,"modified":12,"meta":13,"seo":23,"topicSlug":28,"tags":29,"body":31,"_raw":36},"f923fb28-868a-4fb1-9aa3-b7092cf430aa","en","1e99dd42-a6f3-4b37-93e6-1c71c080db3f",[5],{"en":9},"/en/blog/how-to-run-a-pre-mortem-on-your-data-finding-the-assumptions-that-will-fail-firs","How to Run a Pre Mortem on Your Data: Finding the Assumptions That Will Fail First","Run a practical data pre mortem before your support metrics review. Surface hidden assumptions (definitions, coverage gaps, automation effects), rank what will fail first, and set decision rules plus monitoring so dashboards do not drive a bad call.","2026-05-26T09:21:15.727Z",{"date":12,"badge":14,"authors":17},{"label":15,"color":16},"New","primary",[18],{"name":19,"description":20,"avatar":21},"Lucía Ferrer","Calypso AI · Clear, expert-led guides for operators and buyers",{"src":22},"https://api.dicebear.com/9.x/personas/svg?seed=calypso_expert_guide_v1&backgroundColor=b6e3f4,c0aede,d1d4f9,ffd5dc,ffdfbf",{"title":24,"description":25,"ogDescription":25,"twitterDescription":25,"canonicalPath":9,"robots":26,"schemaType":27},"How to Run a Pre Mortem on Your Data: Finding the","Run a practical data pre mortem before your support metrics review. Surface hidden assumptions (definitions, coverage gaps, automation effects), rank what will","index,follow","BlogPosting","decision_systems_researcher",[30],"how-to-run-a-pre-mortem-on-your-data-finding-the-assumptions-that-will-fail-firs",{"toc":32,"children":34,"html":35},{"links":33},[],[],"\u003Ch2>Start by naming the decision your metrics are about to “prove” (and how it could go wrong)\u003C/h2>\n\u003Cp>Support dashboards have a talent for sounding more certain than reality. The charts are tidy. The trend line is clean. Someone points at it and says, “So we can reduce weekend coverage,” or “We should expand the bot,” or “We can tighten the SLA.” The numbers prove it.\u003C/p>\n\u003Cp>Then, a few weeks later, the same room is asking why escalations spiked, why a key segment is angry, and why the team is drowning in reopen work that never showed up on the dashboard.\u003C/p>\n\u003Cp>Nobody lied. The data just carried assumptions nobody named.\u003C/p>\n\u003Cp>A \u003Cstrong>data pre mortem\u003C/strong> is how you catch those assumptions before they become expensive. In support operations, it’s a short, structured session where you assume the upcoming decision went badly—and work backward to identify which metric assumption made you feel safe.\u003C/p>\n\u003Cp>Keep it tied to a real decision. You are not “improving data quality.” You are stress‑testing the numbers that are about to steer staffing, hours, automation, or targets.\u003C/p>\n\u003Cp>A scenario that burns teams constantly: headcount is planned off “ticket volume is down 15% and backlog is stable.” The cut happens. Two weeks later the queue explodes.\u003C/p>\n\u003Cp>The root cause is rarely a complicated analysis error. It’s usually a coverage gap.\u003C/p>\n\u003Cp>Chat volume increased after a product change, but chat was captured in a different system and never made it into the reporting set used for staffing. Tickets looked down. Contacts were not down. The dashboard wasn’t wrong—it was confidently incomplete.\u003C/p>\n\u003Cp>That’s why you name the decision first. Metrics don’t exist in a vacuum. They exist to justify action.\u003C/p>\n\u003Cp>What you want out of the process is practical meeting ammo you can attach to the invite:\u003C/p>\n\u003Cp>An assumption inventory in plain language. A ranked list of what’s most likely to fail first. Decision rules that prevent false precision. And a minimal monitoring plan so drift shows up early—before the decision is locked.\u003C/p>\n\u003Ch2>Build your “assumption inventory” in 30 minutes: definitions, coverage, incentives, and automation side effects\u003C/h2>\n\u003Cp>Most support orgs don’t lack metrics. They lack shared meaning.\u003C/p>\n\u003Cp>An assumption inventory turns the silent disagreements into sentences you can test. Keep it tight: start with the \u003Cstrong>three to five metrics\u003C/strong> that will carry the decision. If leadership will see the chart as evidence, it belongs here.\u003C/p>\n\u003Cp>Write assumptions as sentences you could read out loud. If it sounds vague, it will stay vague during the review. If it’s testable, someone can validate it.\u003C/p>\n\u003Cp>Below is a taxonomy that forces the usual suspects onto the table without turning this into an endless audit.\u003C/p>\n\u003Cp>\u003Cstrong>1) Metric definition assumptions\u003C/strong> (what the number actually means)\u003C/p>\n\u003Cp>This is where teams get burned, because definition debates happen \u003Cem>after\u003C/em> the trend is presented. By then the chart has already won the argument.\u003C/p>\n\u003Cp>Examples worth writing down exactly:\u003C/p>\n\u003Cp>“First response time means first human response, not an automated acknowledgement.”\u003C/p>\n\u003Cp>“A reopened ticket counts as new work for volume and workload reporting.”\u003C/p>\n\u003Cp>“Backlog includes tickets in pending customer status.”\u003C/p>\n\u003Cp>Signals definition drift is already happening: step changes right after workflow tweaks (new statuses, routing rules, auto‑responses), or people using different words (“first touch” vs “first response”) as if they’re the same event.\u003C/p>\n\u003Cp>\u003Cstrong>2) Population and coverage assumptions\u003C/strong> (what work is included)\u003C/p>\n\u003Cp>This is your “are we counting the thing we do?” category.\u003C/p>\n\u003Cp>Examples:\u003C/p>\n\u003Cp>“Chat, email, phone, and social contacts are all included in the counts we use for staffing.”\u003C/p>\n\u003Cp>“The dashboard covers every region and language tier we’re responsible for.”\u003C/p>\n\u003Cp>Signals: channel mix shifts materially while headline metrics stay oddly stable; or a fast‑growing segment (new plan tier, new region) is missing from reporting.\u003C/p>\n\u003Cp>Concrete anchor: overall first response time improves and everyone celebrates. Later you learn enterprise customers mostly contact via email, email response time worsened, and chat improved enough to dominate the average. The dashboard looked better; the customers who matter most felt worse.\u003C/p>\n\u003Cp>\u003Cstrong>3) Instrumentation and collection assumptions\u003C/strong> (whether events are captured consistently)\u003C/p>\n\u003Cp>This is less about tools and more about whether key events show up as records and timestamps in the same way over time.\u003C/p>\n\u003Cp>Examples:\u003C/p>\n\u003Cp>“Every inbound contact creates a countable record, even if it’s resolved by automation.”\u003C/p>\n\u003Cp>“Solved time reflects customer resolution, not a status change that parks the ticket.”\u003C/p>\n\u003Cp>Signals: rising missing timestamps or default values, or a new integration lining up with a spike in “near‑zero” response times.\u003C/p>\n\u003Cp>\u003Cstrong>4) Labeling and tagging assumptions\u003C/strong> (where humans meet reporting)\u003C/p>\n\u003Cp>Tags are fragile because they’re applied under pressure—and incentives are real.\u003C/p>\n\u003Cp>Examples:\u003C/p>\n\u003Cp>“‘Other’ is not a dumping ground, and the top issue tags cover most tickets.”\u003C/p>\n\u003Cp>“Agents apply tags consistently enough to compare teams and spot trends.”\u003C/p>\n\u003Cp>Signals: “unknown/other/blank” grows over time (especially during high load), or a tag is heavily used by one queue and ignored by another, making category trends look like product change when it’s actually adoption change.\u003C/p>\n\u003Cp>Common failure: treating tag trends as product truth without checking tag completeness. When backlog rises, tagging becomes optional in practice even if it’s “required” on paper.\u003C/p>\n\u003Cp>\u003Cstrong>5) Incentives and behavior change assumptions\u003C/strong> (metrics reshape behavior)\u003C/p>\n\u003Cp>Support metrics don’t just measure work. They create workarounds.\u003C/p>\n\u003Cp>Examples:\u003C/p>\n\u003Cp>“Agents are not delaying escalations or gaming statuses to protect first response time.”\u003C/p>\n\u003Cp>“CSAT surveys are sent consistently across channels and segments.”\u003C/p>\n\u003Cp>Signals: a KPI improves while reopen rate rises or CSAT falls; or you start hearing “here’s how we keep the dashboard green.” That’s your early‑warning siren.\u003C/p>\n\u003Cp>\u003Cstrong>6) Automation side effects assumptions\u003C/strong> (work moves when bots arrive)\u003C/p>\n\u003Cp>Automation is not just a volume lever. It changes the shape of what’s left—and where it shows up.\u003C/p>\n\u003Cp>Examples:\u003C/p>\n\u003Cp>“Deflected customers don’t reappear as follow‑up tickets later in the week.”\u003C/p>\n\u003Cp>“Bot handoffs are counted as contacts, and misroutes are visible as transfers/reassignments.”\u003C/p>\n\u003Cp>Signals: ticket volume drops after automation, but time to resolution rises because what remains is harder or misrouted; or contact rate per active customer stays flat while tickets drop, suggesting the work moved channels or split into extra touches.\u003C/p>\n\u003Cp>Now add ownership so the inventory doesn’t become a memorial.\u003C/p>\n\u003Cp>For each high‑priority assumption, assign two roles:\u003C/p>\n\u003Cp>A \u003Cstrong>validator\u003C/strong> (who can confirm what’s true today—often the analyst/reporting owner). And a \u003Cstrong>fixer\u003C/strong> (who can change workflow/instrumentation/definitions if it’s false—often support ops, automation, or workflow owners).\u003C/p>\n\u003Cp>This two‑role model prevents the classic stall: everyone agrees something might be wrong, and nobody can move it.\u003C/p>\n\u003Ch2>Run the pre‑mortem workshop: a facilitation script that turns opinions into testable risks\u003C/h2>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Control\u003C/th>\n\u003Cth>Where it lives\u003C/th>\n\u003Cth>What to set\u003C/th>\n\u003Cth>What breaks if it’s wrong\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>Set: Testable Actions\u003C/td>\n\u003Ctd>Action item tracker\u003C/td>\n\u003Ctd>Specific, measurable tests for high-priority assumptions\u003C/td>\n\u003Ctd>Risks remain theoretical, no validation\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Set: Facilitator Role\u003C/td>\n\u003Ctd>Workshop agenda\u003C/td>\n\u003Ctd>Designated facilitator, note-taker, timekeeper\u003C/td>\n\u003Ctd>Chaotic discussion, no clear output\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Set: Timeboxes\u003C/td>\n\u003Ctd>Facilitation script (e.g., 60-90 min total)\u003C/td>\n\u003Ctd>Strict limits for each activity (e.g., 15 min brainstorm)\u003C/td>\n\u003Ctd>Workshop runs over, key steps rushed\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Set: Prompt Sequence\u003C/td>\n\u003Ctd>Facilitation script\u003C/td>\n\u003Ctd>Assume failure > Why did it fail? > What to do?\u003C/td>\n\u003Ctd>Disjointed conversation, blame focus\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Set: Ranked Assumption List\u003C/td>\n\u003Ctd>Shared document/whiteboard\u003C/td>\n\u003Ctd>Top 3-5 assumptions, ranked by impact/likelihood\u003C/td>\n\u003Ctd>No clear priorities, team overwhelmed\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Set: Participant List\u003C/td>\n\u003Ctd>Invite/brief\u003C/td>\n\u003Ctd>Support ops / operator, support lead, analyst / reporting owner, frontline rep, automation / workflow owner\u003C/td>\n\u003Ctd>Missing key perspectives, incomplete risk identification\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Use with Caveats Notes\u003C/td>\n\u003Ctd>Metrics review agenda/report\u003C/td>\n\u003Ctd>Contextual notes for interpreting metrics, based on identified risks\u003C/td>\n\u003Ctd>Metrics misinterpreted, flawed decisions\u003C/td>\n\u003C/tr>\n\u003C/tbody>\u003C/table>\n\u003Cp>These controls look simple because they are. They’re also exactly what keeps a data pre mortem from turning into either (a) an endless audit or (b) a vibes debate.\u003C/p>\n\u003Cp>The “Set: Facilitator Role” line matters more than people think. Without a facilitator, you get seniority gravity: the most confident story becomes the risk list. With a facilitator plus “Set: Timeboxes,” you get outcomes.\u003C/p>\n\u003Cp>Aim for \u003Cstrong>60–90 minutes\u003C/strong>. Short enough that people show up. Long enough to get past the first layer of obvious answers.\u003C/p>\n\u003Cp>Invite a small group with real coverage:\u003C/p>\n\u003Cp>Support ops/operator (how work flows). Support lead (owns the decision). Analyst/reporting owner (owns the view). A frontline rep (edge cases and incentives). Automation/workflow owner (if routing/bots/self‑service are in play).\u003C/p>\n\u003Cp>Set two rules at the start:\u003C/p>\n\u003Cp>This is not a blame session. Broken assumptions are normal in living systems.\u003C/p>\n\u003Cp>Every risk must end as one of three outputs: a test to run, a caveat to state, or a follow‑up to schedule. No scary list that dies in a document.\u003C/p>\n\u003Cp>Use the core prompt: “It’s 60 days later and the decision failed. What happened, and which assumption made us miss it?” Variations of this prompt show up in general pre‑mortem writeups like \u003Ca href=\"#ref-1\" title=\"frameworklist.com — frameworklist.com\">[1]\u003C/a> and \u003Ca href=\"#ref-2\" title=\"qualz.ai — qualz.ai\">[2]\u003C/a>.\u003C/p>\n\u003Cp>Start with \u003Cstrong>five minutes of silent writing\u003C/strong>. Silence prevents the meeting from becoming the first person’s story.\u003C/p>\n\u003Cp>Then go around and capture each failure story as one sentence that includes the decision and the assumption.\u003C/p>\n\u003Cp>Example: “We reduced weekend staffing because backlog looked stable, but weekend chat was undercounted and Monday backlog surged.”\u003C/p>\n\u003Cp>That single sentence hides multiple assumptions:\u003C/p>\n\u003Cp>Backlog represents weekend workload.\u003C/p>\n\u003Cp>Chat is counted the same way as email tickets for staffing.\u003C/p>\n\u003Cp>Next, rank assumptions quickly by two rough scores: \u003Cstrong>decision impact\u003C/strong> and \u003Cstrong>likelihood it’s false\u003C/strong>. You’re not chasing precision; you’re chasing order.\u003C/p>\n\u003Cp>A good anti‑politics trick: do a short silent vote on the top risks before discussion. Titles can join the conversation \u003Cem>after\u003C/em> the list is visible.\u003C/p>\n\u003Cp>Finally, convert your top assumptions into testable actions (“Set: Testable Actions”), and write the “Use with Caveats Notes” you’ll say out loud in the metrics review.\u003C/p>\n\u003Cp>One concrete conversion helps the team feel the difference between “concern” and “control.”\u003C/p>\n\u003Cp>Assumption: “Chat is fully represented in the dataset used for staffing.”\u003C/p>\n\u003Cp>Test: pick a recent week, pull total chat conversation counts from the chat system, compare to what shows in reporting, then split by region and by hour. If the gap clusters outside business hours or in a specific region, you have a real coverage hole—not a philosophical argument.\u003C/p>\n\u003Cp>The warning here is blunt: this is where teams get burned. They identify a high‑risk assumption, schedule the fix “after planning,” and then plan using the broken metric anyway. If the assumption could flip the decision, it deserves same‑week validation or a decision delay.\u003C/p>\n\u003Ch2>Decide what to trust (today): confidence levels, decision rules, and the real tradeoffs\u003C/h2>\n\u003Cp>After the workshop, you’ll want to fix everything. That instinct is good—and dangerous.\u003C/p>\n\u003Cp>Support planning rarely waits for perfect data. If you insist on perfection, you often end up with unspoken uncertainty. That’s worse than an explicit “directional” label.\u003C/p>\n\u003Cp>So the goal is to label confidence clearly and agree on how each metric will be used.\u003C/p>\n\u003Cp>Use a simple ladder:\u003C/p>\n\u003Cp>\u003Cstrong>Decision grade\u003C/strong>: safe enough to base staffing, SLAs, targets, or commitments on. Definitions are stable, coverage is understood, and you have at least one monitoring signal that would tell you if the metric starts lying.\u003C/p>\n\u003Cp>\u003Cstrong>Directional\u003C/strong>: useful for trends and comparisons, not for hard commitments. Known exclusions or recent workflow changes make the absolute level questionable.\u003C/p>\n\u003Cp>\u003Cstrong>Exploratory\u003C/strong>: useful for questions, not conclusions. Treat it like a lead, not a verdict.\u003C/p>\n\u003Cp>A move that changes behavior fast: require every chart in the review to carry a confidence label. Not as bureaucracy—just as visible honesty.\u003C/p>\n\u003Cp>Then set decision rules. Rules feel rigid until you’re in a tense meeting and your brain starts negotiating with itself.\u003C/p>\n\u003Cp>Keep rules tied to common support failure patterns:\u003C/p>\n\u003Cp>If unknown/uncategorized tags rise enough that category rankings become unstable, issue mix becomes directional only. Many teams feel this around ~10% unknown for a couple of weeks, but the threshold should match your baseline.\u003C/p>\n\u003Cp>If a metric excludes a segment that matters to the decision, it cannot be decision grade for that decision. Planning enterprise coverage with a metric that misses enterprise chat/phone is how you “save” headcount and buy churn.\u003C/p>\n\u003Cp>If first response time can be satisfied by automation, don’t use it alone to argue coverage health. Pair it with first human response time, time to resolution, reopen rate, or whichever reflects real work.\u003C/p>\n\u003Cp>After an automation rollout, don’t declare victory on deflection until you can see what moved. Volume down is exciting. It’s also a magician’s favorite distraction.\u003C/p>\n\u003Cp>The automation tradeoff is real. If you wait until automation measurement is perfect, you never ship automation. If you trust deflection blindly, you ship churn.\u003C/p>\n\u003Cp>A grounded compromise: treat deflection as directional unless you can answer three questions with confidence.\u003C/p>\n\u003Cp>Are bot handoffs counted as contacts consistently over time?\u003C/p>\n\u003Cp>Did contact rate per active customer change—or did customers simply show up somewhere else?\u003C/p>\n\u003Cp>Did reopen/transfer/time‑to‑resolution change in a way that suggests misrouting or partial answers?\u003C/p>\n\u003Cp>Concrete anchor: a bot answers refund policy questions, ticket volume drops 12%, and the dashboard looks great. Meanwhile agents say the remaining tickets are harder, reopen rate climbs, and “follow‑up” contacts show up as new tickets with different tags. The numbers say efficiency; the floor feels pain. Both can be true.\u003C/p>\n\u003Cp>One more decision that saves trust: when to freeze definitions.\u003C/p>\n\u003Cp>Freeze definitions when the metric is used for targets, incentives, staffing models, or executive commitments. Changing a definition mid‑quarter without a visible note is how trust dies.\u003C/p>\n\u003Cp>Iterate when the metric is exploratory and you’re still learning what matters. If you change a definition, log it and stop pretending the trend is apples‑to‑apples.\u003C/p>\n\u003Cp>Finally, acknowledge the single‑number versus segmented‑view tradeoff.\u003C/p>\n\u003Cp>Single numbers align teams. They also hide problems beautifully.\u003C/p>\n\u003Cp>If you add only one habit: every headline metric gets one companion segment slice tied to the decision. Enterprise vs everyone else. English vs non‑English. Weekend vs weekday. The slice makes it harder for averages to lie.\u003C/p>\n\u003Ch2>Failure modes: what breaks first in support metrics—and the leading indicators that catch it early\u003C/h2>\n\u003Cp>Support metrics rarely fail with fireworks. They fail like a slow leak: the dashboard stays clean, the story stays plausible, and then you realize you’ve been optimizing the wrong thing for months.\u003C/p>\n\u003Cp>A data pre mortem isn’t complete if it ends with risks. You need leading indicators that tell you when assumptions start cracking—smoke alarms for your reporting.\u003C/p>\n\u003Cp>\u003Cstrong>Failure mode 1: Definition drift in first response time\u003C/strong>\u003C/p>\n\u003Cp>What breaks: an auto acknowledgement starts counting as a response, or a workflow change moves the starting timestamp.\u003C/p>\n\u003Cp>Leading indicators: a sudden spike in near‑zero response times, especially when it lines up with a workflow release.\u003C/p>\n\u003Cp>What to do: split “first touch” and “first human response,” and label the chart so nobody compares across the change without context.\u003C/p>\n\u003Cp>\u003Cstrong>Failure mode 2: Backlog gets “fixed” by status semantics\u003C/strong>\u003C/p>\n\u003Cp>What breaks: new pending statuses make backlog look better even if the customer still experiences delay.\u003C/p>\n\u003Cp>Leading indicators: backlog drops sharply on the day status rules change while time to resolution or reopen rate doesn’t improve.\u003C/p>\n\u003Cp>What to do: publish backlog rules in plain language, and keep a companion view that includes open + pending for coverage planning.\u003C/p>\n\u003Cp>\u003Cstrong>Failure mode 3: Channel mix shift hides a segment problem\u003C/strong>\u003C/p>\n\u003Cp>What breaks: chat grows and is faster, so overall response time improves while email worsens. Enterprise is mostly email, so your most valuable customers get worse service while the dashboard looks better.\u003C/p>\n\u003Cp>Leading indicators: channel share changes materially while headline KPIs “improve.” Watch response time by channel.\u003C/p>\n\u003Cp>What to do: require channel‑segmented KPIs in any review that drives staffing or hours decisions.\u003C/p>\n\u003Cp>\u003Cstrong>Failure mode 4: Tagging sparsity and “unknown” creep\u003C/strong>\u003C/p>\n\u003Cp>What breaks: under load, tagging discipline collapses. Issue trends become a mix of reality and missingness.\u003C/p>\n\u003Cp>Leading indicators: “unknown/other/blank” grows; tag completeness differs wildly by team/shift.\u003C/p>\n\u003Cp>What to do: when unknown crosses your threshold, treat issue mix as directional only. Then ask the operational question: are tags too complex to apply under pressure?\u003C/p>\n\u003Cp>\u003Cstrong>Failure mode 5: Silent automation reroutes\u003C/strong>\u003C/p>\n\u003Cp>What breaks: bots change routing, customers bounce, and contacts appear as follow‑ups or escalations rather than initial tickets. Ticket volume drops and deflection looks great, but work moved.\u003C/p>\n\u003Cp>Leading indicators: contact rate per active customer stays flat while tickets drop; reopen rises; transfers/reassignments rise; “status check” contacts creep up.\u003C/p>\n\u003Cp>What to do: pair deflection with a workload reality check (reopen, transfer, time to resolution). If those worsen post‑rollout, treat deflection as unproven until investigated.\u003C/p>\n\u003Cp>\u003Cstrong>Failure mode 6: Sampling and segment blind spots\u003C/strong>\u003C/p>\n\u003Cp>What breaks: top‑line KPIs improve because a low‑complexity segment grew, while a high‑risk segment worsened. The dashboard isn’t wrong—it’s averaged.\u003C/p>\n\u003Cp>Leading indicators: KPI movement is driven by one segment, or a segment disappears due to low‑volume filters or missing fields.\u003C/p>\n\u003Cp>What to do: define three must‑watch segments. Many support orgs default well with: top revenue tier, fastest growing region, and the channel changing most.\u003C/p>\n\u003Cp>\u003Cstrong>Failure mode 7: Incentives reshape what gets recorded\u003C/strong>\u003C/p>\n\u003Cp>What breaks: teams learn the fastest path to green metrics—premature solves, delayed escalations, or pushing customers into uncounted channels.\u003C/p>\n\u003Cp>Leading indicators: KPI improves while CSAT/reopen worsens; status patterns shift (more solves with shorter handle time but higher follow‑up).\u003C/p>\n\u003Cp>What to do: review incentives when the pattern appears. Don’t blame individuals for responding to the system you built.\u003C/p>\n\u003Cp>Make it operational with a minimal cadence.\u003C/p>\n\u003Cp>Weekly, the reporting owner or support ops operator checks a short set of smoke alarms: unknown tag share, channel mix, reopen, transfer/reassignment, and any step changes that line up with releases.\u003C/p>\n\u003Cp>Monthly, the support lead reviews a one‑paragraph “what changed” note before interpreting trends (staffing shifts, hours changes, routing changes, automation releases). It doesn’t need to be fancy. It needs to exist.\u003C/p>\n\u003Cp>Route alerts to the metric owner and the decision owner. You don’t need to broadcast every wobble to leadership. The point is correction, not public shaming.\u003C/p>\n\u003Cp>If you adopt one lightweight governance habit, make it a metrics change log. When someone later asks “why did the metric jump,” you won’t be stuck relying on memory, which is the least reliable system in your stack.\u003C/p>\n\u003Ch2>Close the loop: a pre‑mortem deliverable pack you can attach to the metrics review invite\u003C/h2>\n\u003Cp>A data pre mortem only changes outcomes if it survives past the workshop. The easiest way to make it survive is to ship a small deliverable pack that travels with the metrics review.\u003C/p>\n\u003Cp>Keep it to one page, or close to it. Leaders read one page. They do not read a novella unless it has dragons.\u003C/p>\n\u003Cp>Attach or paste this pack into the invite or agenda:\u003C/p>\n\u003Cp>The decision statement and the three to five deciding metrics.\u003C/p>\n\u003Cp>The ranked top assumptions in plain sentences (top ten is usually enough).\u003C/p>\n\u003Cp>A confidence label for each deciding metric: decision grade, directional, or exploratory.\u003C/p>\n\u003Cp>Known exclusions that matter (missing channels, regions, tiers, time windows).\u003C/p>\n\u003Cp>Open tests, each with an owner pair: validator and fixer.\u003C/p>\n\u003Cp>“Use with caveats” notes that will be said out loud in the review.\u003C/p>\n\u003Cp>Two leading indicators per fragile metric, plus who gets notified when they move.\u003C/p>\n\u003Cp>If you discover a high‑risk assumption an hour before the meeting, don’t hide it and don’t blow up the agenda. Triage it.\u003C/p>\n\u003Cp>If the assumption could materially change the decision, delay the decision or reroute to a safer metric.\u003C/p>\n\u003Cp>If it changes precision but not direction, proceed with a clear caveat and a committed follow‑up test.\u003C/p>\n\u003Cp>If it’s minor, document it and move on.\u003C/p>\n\u003Cp>Bake this into your planning cadence: run a short pre mortem before quarterly planning and before major automation expansions. Ninety minutes now is cheaper than a quarter of arguing about whose numbers are real.\u003C/p>\n\u003Cp>Primary CTA: Run this pre mortem before your next support metrics review. Start with the assumption inventory and bring the ranked top ten to the meeting.\u003C/p>\n\u003Cp>Secondary CTA: create a lightweight metrics change log and assign an owner per metric so definition drift stops surprising you at the worst possible time.\u003C/p>\n\u003Ch2>Sources\u003C/h2>\n\u003Col>\n\u003Cli>\u003Ca href=\"https://frameworklist.com/academy/how-to-run-a-premortem\">frameworklist.com\u003C/a> — frameworklist.com\u003C/li>\n\u003Cli>\u003Ca href=\"https://qualz.ai/research-guide/techniques/pre-mortem\">qualz.ai\u003C/a> — qualz.ai\u003C/li>\n\u003C/ol>\n",{"body":37},"## Start by naming the decision your metrics are about to “prove” (and how it could go wrong)\n\nSupport dashboards have a talent for sounding more certain than reality. The charts are tidy. The trend line is clean. Someone points at it and says, “So we can reduce weekend coverage,” or “We should expand the bot,” or “We can tighten the SLA.” The numbers prove it.\n\nThen, a few weeks later, the same room is asking why escalations spiked, why a key segment is angry, and why the team is drowning in reopen work that never showed up on the dashboard.\n\nNobody lied. The data just carried assumptions nobody named.\n\nA **data pre mortem** is how you catch those assumptions before they become expensive. In support operations, it’s a short, structured session where you assume the upcoming decision went badly—and work backward to identify which metric assumption made you feel safe.\n\nKeep it tied to a real decision. You are not “improving data quality.” You are stress‑testing the numbers that are about to steer staffing, hours, automation, or targets.\n\nA scenario that burns teams constantly: headcount is planned off “ticket volume is down 15% and backlog is stable.” The cut happens. Two weeks later the queue explodes.\n\nThe root cause is rarely a complicated analysis error. It’s usually a coverage gap.\n\nChat volume increased after a product change, but chat was captured in a different system and never made it into the reporting set used for staffing. Tickets looked down. Contacts were not down. The dashboard wasn’t wrong—it was confidently incomplete.\n\nThat’s why you name the decision first. Metrics don’t exist in a vacuum. They exist to justify action.\n\nWhat you want out of the process is practical meeting ammo you can attach to the invite:\n\nAn assumption inventory in plain language. A ranked list of what’s most likely to fail first. Decision rules that prevent false precision. And a minimal monitoring plan so drift shows up early—before the decision is locked.\n\n## Build your “assumption inventory” in 30 minutes: definitions, coverage, incentives, and automation side effects\n\nMost support orgs don’t lack metrics. They lack shared meaning.\n\nAn assumption inventory turns the silent disagreements into sentences you can test. Keep it tight: start with the **three to five metrics** that will carry the decision. If leadership will see the chart as evidence, it belongs here.\n\nWrite assumptions as sentences you could read out loud. If it sounds vague, it will stay vague during the review. If it’s testable, someone can validate it.\n\nBelow is a taxonomy that forces the usual suspects onto the table without turning this into an endless audit.\n\n**1) Metric definition assumptions** (what the number actually means)\n\nThis is where teams get burned, because definition debates happen *after* the trend is presented. By then the chart has already won the argument.\n\nExamples worth writing down exactly:\n\n“First response time means first human response, not an automated acknowledgement.”\n\n“A reopened ticket counts as new work for volume and workload reporting.”\n\n“Backlog includes tickets in pending customer status.”\n\nSignals definition drift is already happening: step changes right after workflow tweaks (new statuses, routing rules, auto‑responses), or people using different words (“first touch” vs “first response”) as if they’re the same event.\n\n**2) Population and coverage assumptions** (what work is included)\n\nThis is your “are we counting the thing we do?” category.\n\nExamples:\n\n“Chat, email, phone, and social contacts are all included in the counts we use for staffing.”\n\n“The dashboard covers every region and language tier we’re responsible for.”\n\nSignals: channel mix shifts materially while headline metrics stay oddly stable; or a fast‑growing segment (new plan tier, new region) is missing from reporting.\n\nConcrete anchor: overall first response time improves and everyone celebrates. Later you learn enterprise customers mostly contact via email, email response time worsened, and chat improved enough to dominate the average. The dashboard looked better; the customers who matter most felt worse.\n\n**3) Instrumentation and collection assumptions** (whether events are captured consistently)\n\nThis is less about tools and more about whether key events show up as records and timestamps in the same way over time.\n\nExamples:\n\n“Every inbound contact creates a countable record, even if it’s resolved by automation.”\n\n“Solved time reflects customer resolution, not a status change that parks the ticket.”\n\nSignals: rising missing timestamps or default values, or a new integration lining up with a spike in “near‑zero” response times.\n\n**4) Labeling and tagging assumptions** (where humans meet reporting)\n\nTags are fragile because they’re applied under pressure—and incentives are real.\n\nExamples:\n\n“‘Other’ is not a dumping ground, and the top issue tags cover most tickets.”\n\n“Agents apply tags consistently enough to compare teams and spot trends.”\n\nSignals: “unknown/other/blank” grows over time (especially during high load), or a tag is heavily used by one queue and ignored by another, making category trends look like product change when it’s actually adoption change.\n\nCommon failure: treating tag trends as product truth without checking tag completeness. When backlog rises, tagging becomes optional in practice even if it’s “required” on paper.\n\n**5) Incentives and behavior change assumptions** (metrics reshape behavior)\n\nSupport metrics don’t just measure work. They create workarounds.\n\nExamples:\n\n“Agents are not delaying escalations or gaming statuses to protect first response time.”\n\n“CSAT surveys are sent consistently across channels and segments.”\n\nSignals: a KPI improves while reopen rate rises or CSAT falls; or you start hearing “here’s how we keep the dashboard green.” That’s your early‑warning siren.\n\n**6) Automation side effects assumptions** (work moves when bots arrive)\n\nAutomation is not just a volume lever. It changes the shape of what’s left—and where it shows up.\n\nExamples:\n\n“Deflected customers don’t reappear as follow‑up tickets later in the week.”\n\n“Bot handoffs are counted as contacts, and misroutes are visible as transfers/reassignments.”\n\nSignals: ticket volume drops after automation, but time to resolution rises because what remains is harder or misrouted; or contact rate per active customer stays flat while tickets drop, suggesting the work moved channels or split into extra touches.\n\nNow add ownership so the inventory doesn’t become a memorial.\n\nFor each high‑priority assumption, assign two roles:\n\nA **validator** (who can confirm what’s true today—often the analyst/reporting owner). And a **fixer** (who can change workflow/instrumentation/definitions if it’s false—often support ops, automation, or workflow owners).\n\nThis two‑role model prevents the classic stall: everyone agrees something might be wrong, and nobody can move it.\n\n## Run the pre‑mortem workshop: a facilitation script that turns opinions into testable risks\n\n| Control | Where it lives | What to set | What breaks if it’s wrong |\n| --- | --- | --- | --- |\n| Set: Testable Actions | Action item tracker | Specific, measurable tests for high-priority assumptions | Risks remain theoretical, no validation |\n| Set: Facilitator Role | Workshop agenda | Designated facilitator, note-taker, timekeeper | Chaotic discussion, no clear output |\n| Set: Timeboxes | Facilitation script (e.g., 60-90 min total) | Strict limits for each activity (e.g., 15 min brainstorm) | Workshop runs over, key steps rushed |\n| Set: Prompt Sequence | Facilitation script | Assume failure > Why did it fail? > What to do? | Disjointed conversation, blame focus |\n| Set: Ranked Assumption List | Shared document/whiteboard | Top 3-5 assumptions, ranked by impact/likelihood | No clear priorities, team overwhelmed |\n| Set: Participant List | Invite/brief | Support ops / operator, support lead, analyst / reporting owner, frontline rep, automation / workflow owner | Missing key perspectives, incomplete risk identification |\n| Use with Caveats Notes | Metrics review agenda/report | Contextual notes for interpreting metrics, based on identified risks | Metrics misinterpreted, flawed decisions |\n\nThese controls look simple because they are. They’re also exactly what keeps a data pre mortem from turning into either (a) an endless audit or (b) a vibes debate.\n\nThe “Set: Facilitator Role” line matters more than people think. Without a facilitator, you get seniority gravity: the most confident story becomes the risk list. With a facilitator plus “Set: Timeboxes,” you get outcomes.\n\nAim for **60–90 minutes**. Short enough that people show up. Long enough to get past the first layer of obvious answers.\n\nInvite a small group with real coverage:\n\nSupport ops/operator (how work flows). Support lead (owns the decision). Analyst/reporting owner (owns the view). A frontline rep (edge cases and incentives). Automation/workflow owner (if routing/bots/self‑service are in play).\n\nSet two rules at the start:\n\nThis is not a blame session. Broken assumptions are normal in living systems.\n\nEvery risk must end as one of three outputs: a test to run, a caveat to state, or a follow‑up to schedule. No scary list that dies in a document.\n\nUse the core prompt: “It’s 60 days later and the decision failed. What happened, and which assumption made us miss it?” Variations of this prompt show up in general pre‑mortem writeups like [[1]](#ref-1 \"frameworklist.com — frameworklist.com\") and [[2]](#ref-2 \"qualz.ai — qualz.ai\").\n\nStart with **five minutes of silent writing**. Silence prevents the meeting from becoming the first person’s story.\n\nThen go around and capture each failure story as one sentence that includes the decision and the assumption.\n\nExample: “We reduced weekend staffing because backlog looked stable, but weekend chat was undercounted and Monday backlog surged.”\n\nThat single sentence hides multiple assumptions:\n\nBacklog represents weekend workload.\n\nChat is counted the same way as email tickets for staffing.\n\nNext, rank assumptions quickly by two rough scores: **decision impact** and **likelihood it’s false**. You’re not chasing precision; you’re chasing order.\n\nA good anti‑politics trick: do a short silent vote on the top risks before discussion. Titles can join the conversation *after* the list is visible.\n\nFinally, convert your top assumptions into testable actions (“Set: Testable Actions”), and write the “Use with Caveats Notes” you’ll say out loud in the metrics review.\n\nOne concrete conversion helps the team feel the difference between “concern” and “control.”\n\nAssumption: “Chat is fully represented in the dataset used for staffing.”\n\nTest: pick a recent week, pull total chat conversation counts from the chat system, compare to what shows in reporting, then split by region and by hour. If the gap clusters outside business hours or in a specific region, you have a real coverage hole—not a philosophical argument.\n\nThe warning here is blunt: this is where teams get burned. They identify a high‑risk assumption, schedule the fix “after planning,” and then plan using the broken metric anyway. If the assumption could flip the decision, it deserves same‑week validation or a decision delay.\n\n## Decide what to trust (today): confidence levels, decision rules, and the real tradeoffs\n\nAfter the workshop, you’ll want to fix everything. That instinct is good—and dangerous.\n\nSupport planning rarely waits for perfect data. If you insist on perfection, you often end up with unspoken uncertainty. That’s worse than an explicit “directional” label.\n\nSo the goal is to label confidence clearly and agree on how each metric will be used.\n\nUse a simple ladder:\n\n**Decision grade**: safe enough to base staffing, SLAs, targets, or commitments on. Definitions are stable, coverage is understood, and you have at least one monitoring signal that would tell you if the metric starts lying.\n\n**Directional**: useful for trends and comparisons, not for hard commitments. Known exclusions or recent workflow changes make the absolute level questionable.\n\n**Exploratory**: useful for questions, not conclusions. Treat it like a lead, not a verdict.\n\nA move that changes behavior fast: require every chart in the review to carry a confidence label. Not as bureaucracy—just as visible honesty.\n\nThen set decision rules. Rules feel rigid until you’re in a tense meeting and your brain starts negotiating with itself.\n\nKeep rules tied to common support failure patterns:\n\nIf unknown/uncategorized tags rise enough that category rankings become unstable, issue mix becomes directional only. Many teams feel this around ~10% unknown for a couple of weeks, but the threshold should match your baseline.\n\nIf a metric excludes a segment that matters to the decision, it cannot be decision grade for that decision. Planning enterprise coverage with a metric that misses enterprise chat/phone is how you “save” headcount and buy churn.\n\nIf first response time can be satisfied by automation, don’t use it alone to argue coverage health. Pair it with first human response time, time to resolution, reopen rate, or whichever reflects real work.\n\nAfter an automation rollout, don’t declare victory on deflection until you can see what moved. Volume down is exciting. It’s also a magician’s favorite distraction.\n\nThe automation tradeoff is real. If you wait until automation measurement is perfect, you never ship automation. If you trust deflection blindly, you ship churn.\n\nA grounded compromise: treat deflection as directional unless you can answer three questions with confidence.\n\nAre bot handoffs counted as contacts consistently over time?\n\nDid contact rate per active customer change—or did customers simply show up somewhere else?\n\nDid reopen/transfer/time‑to‑resolution change in a way that suggests misrouting or partial answers?\n\nConcrete anchor: a bot answers refund policy questions, ticket volume drops 12%, and the dashboard looks great. Meanwhile agents say the remaining tickets are harder, reopen rate climbs, and “follow‑up” contacts show up as new tickets with different tags. The numbers say efficiency; the floor feels pain. Both can be true.\n\nOne more decision that saves trust: when to freeze definitions.\n\nFreeze definitions when the metric is used for targets, incentives, staffing models, or executive commitments. Changing a definition mid‑quarter without a visible note is how trust dies.\n\nIterate when the metric is exploratory and you’re still learning what matters. If you change a definition, log it and stop pretending the trend is apples‑to‑apples.\n\nFinally, acknowledge the single‑number versus segmented‑view tradeoff.\n\nSingle numbers align teams. They also hide problems beautifully.\n\nIf you add only one habit: every headline metric gets one companion segment slice tied to the decision. Enterprise vs everyone else. English vs non‑English. Weekend vs weekday. The slice makes it harder for averages to lie.\n\n## Failure modes: what breaks first in support metrics—and the leading indicators that catch it early\n\nSupport metrics rarely fail with fireworks. They fail like a slow leak: the dashboard stays clean, the story stays plausible, and then you realize you’ve been optimizing the wrong thing for months.\n\nA data pre mortem isn’t complete if it ends with risks. You need leading indicators that tell you when assumptions start cracking—smoke alarms for your reporting.\n\n**Failure mode 1: Definition drift in first response time**\n\nWhat breaks: an auto acknowledgement starts counting as a response, or a workflow change moves the starting timestamp.\n\nLeading indicators: a sudden spike in near‑zero response times, especially when it lines up with a workflow release.\n\nWhat to do: split “first touch” and “first human response,” and label the chart so nobody compares across the change without context.\n\n**Failure mode 2: Backlog gets “fixed” by status semantics**\n\nWhat breaks: new pending statuses make backlog look better even if the customer still experiences delay.\n\nLeading indicators: backlog drops sharply on the day status rules change while time to resolution or reopen rate doesn’t improve.\n\nWhat to do: publish backlog rules in plain language, and keep a companion view that includes open + pending for coverage planning.\n\n**Failure mode 3: Channel mix shift hides a segment problem**\n\nWhat breaks: chat grows and is faster, so overall response time improves while email worsens. Enterprise is mostly email, so your most valuable customers get worse service while the dashboard looks better.\n\nLeading indicators: channel share changes materially while headline KPIs “improve.” Watch response time by channel.\n\nWhat to do: require channel‑segmented KPIs in any review that drives staffing or hours decisions.\n\n**Failure mode 4: Tagging sparsity and “unknown” creep**\n\nWhat breaks: under load, tagging discipline collapses. Issue trends become a mix of reality and missingness.\n\nLeading indicators: “unknown/other/blank” grows; tag completeness differs wildly by team/shift.\n\nWhat to do: when unknown crosses your threshold, treat issue mix as directional only. Then ask the operational question: are tags too complex to apply under pressure?\n\n**Failure mode 5: Silent automation reroutes**\n\nWhat breaks: bots change routing, customers bounce, and contacts appear as follow‑ups or escalations rather than initial tickets. Ticket volume drops and deflection looks great, but work moved.\n\nLeading indicators: contact rate per active customer stays flat while tickets drop; reopen rises; transfers/reassignments rise; “status check” contacts creep up.\n\nWhat to do: pair deflection with a workload reality check (reopen, transfer, time to resolution). If those worsen post‑rollout, treat deflection as unproven until investigated.\n\n**Failure mode 6: Sampling and segment blind spots**\n\nWhat breaks: top‑line KPIs improve because a low‑complexity segment grew, while a high‑risk segment worsened. The dashboard isn’t wrong—it’s averaged.\n\nLeading indicators: KPI movement is driven by one segment, or a segment disappears due to low‑volume filters or missing fields.\n\nWhat to do: define three must‑watch segments. Many support orgs default well with: top revenue tier, fastest growing region, and the channel changing most.\n\n**Failure mode 7: Incentives reshape what gets recorded**\n\nWhat breaks: teams learn the fastest path to green metrics—premature solves, delayed escalations, or pushing customers into uncounted channels.\n\nLeading indicators: KPI improves while CSAT/reopen worsens; status patterns shift (more solves with shorter handle time but higher follow‑up).\n\nWhat to do: review incentives when the pattern appears. Don’t blame individuals for responding to the system you built.\n\nMake it operational with a minimal cadence.\n\nWeekly, the reporting owner or support ops operator checks a short set of smoke alarms: unknown tag share, channel mix, reopen, transfer/reassignment, and any step changes that line up with releases.\n\nMonthly, the support lead reviews a one‑paragraph “what changed” note before interpreting trends (staffing shifts, hours changes, routing changes, automation releases). It doesn’t need to be fancy. It needs to exist.\n\nRoute alerts to the metric owner and the decision owner. You don’t need to broadcast every wobble to leadership. The point is correction, not public shaming.\n\nIf you adopt one lightweight governance habit, make it a metrics change log. When someone later asks “why did the metric jump,” you won’t be stuck relying on memory, which is the least reliable system in your stack.\n\n## Close the loop: a pre‑mortem deliverable pack you can attach to the metrics review invite\n\nA data pre mortem only changes outcomes if it survives past the workshop. The easiest way to make it survive is to ship a small deliverable pack that travels with the metrics review.\n\nKeep it to one page, or close to it. Leaders read one page. They do not read a novella unless it has dragons.\n\nAttach or paste this pack into the invite or agenda:\n\nThe decision statement and the three to five deciding metrics.\n\nThe ranked top assumptions in plain sentences (top ten is usually enough).\n\nA confidence label for each deciding metric: decision grade, directional, or exploratory.\n\nKnown exclusions that matter (missing channels, regions, tiers, time windows).\n\nOpen tests, each with an owner pair: validator and fixer.\n\n“Use with caveats” notes that will be said out loud in the review.\n\nTwo leading indicators per fragile metric, plus who gets notified when they move.\n\nIf you discover a high‑risk assumption an hour before the meeting, don’t hide it and don’t blow up the agenda. Triage it.\n\nIf the assumption could materially change the decision, delay the decision or reroute to a safer metric.\n\nIf it changes precision but not direction, proceed with a clear caveat and a committed follow‑up test.\n\nIf it’s minor, document it and move on.\n\nBake this into your planning cadence: run a short pre mortem before quarterly planning and before major automation expansions. Ninety minutes now is cheaper than a quarter of arguing about whose numbers are real.\n\nPrimary CTA: Run this pre mortem before your next support metrics review. Start with the assumption inventory and bring the ranked top ten to the meeting.\n\nSecondary CTA: create a lightweight metrics change log and assign an owner per metric so definition drift stops surprising you at the worst possible time.\n\n## Sources\n\n1. [frameworklist.com](https://frameworklist.com/academy/how-to-run-a-premortem) — frameworklist.com\n2. [qualz.ai](https://qualz.ai/research-guide/techniques/pre-mortem) — qualz.ai\n",[39,43],{"_path":40,"path":40,"title":41,"description":42},"/en/blog/decision-meetings-that-dont-lie-a-workflow-for-what-to-measure-what-to-ignore-wh","Decision Meetings That Dont Lie: A Workflow for What to Measure, What to Ignore, What to Do Next","A practical support decision meeting workflow for leaders who need decision grade support metrics, not dashboard theater. Learn readiness gates, bias checks, fair team comparisons, and a decision log,",{"_path":44,"path":44,"title":45,"description":46},"/en/blog/which-numbers-do-you-trust-a-field-test-for-metrics-before-they-mislead-you","Which Numbers Do You Trust: A Field Test for Metrics Before They Mislead You","A practical field test to answer which support metrics can you trust before a leadership review. Learn how to sanity check dashboards, spot definition drift and metric gaming, compare support teams (f",1780761200041]