Ask most companies whether their AI paid for itself and you get an anecdote. Somebody is delighted, somebody else thinks it is overhyped, and the finance team has no line item to point at. That is not a measurement problem, it is a setup problem: almost nobody writes down what they expect before they start, so there is nothing to compare against afterwards.
Take the baseline before you deploy, not after
This is the whole game and it takes about two hours. Before anything is installed, measure the current state of the specific task you are about to change.
- How long does one instance take, measured on a handful of real cases with a stopwatch rather than estimated in a meeting?
- How many instances happen per week, from the system of record rather than from memory?
- How often does it go wrong today, and what did the last three mistakes cost to fix?
- How long does the work wait in a queue before someone starts it?
That last one is the number companies most often skip and most often regret skipping, because queue time is frequently where the real cost lives. A quotation that takes twenty minutes of work but three days of waiting is not a twenty-minute problem.
Four numbers worth tracking, and what each one hides
| Number | What it tells you | What it hides |
|---|---|---|
| Hours returned per week | The headline benefit, in the currency a CFO understands | Whether those hours went into better work or evaporated |
| Acceptance rate | Share of outputs a person used unchanged — the best single health metric | Whether people are accepting because it is right or because checking is tedious |
| Escaped errors | Problems that reached a customer before and after | Slow-surfacing defects; needs a long enough window to mean anything |
| Queue time | How long work waits before it is touched | Seasonality — compare like periods, not last month against this one |
Acceptance rate is the one to watch weekly
Of those four, acceptance rate is the one that moves first and tells you most. It is simply the share of outputs a person used without changing them.
A rate that climbs means the system is being tuned to your work. A rate that is flat and high from day one usually means people have stopped checking, which is a risk rather than a success. A rate that falls after a model update tells you something changed underneath you — which is worth knowing before a customer finds out.
Counting the costs honestly
The licence is rarely the biggest number, which is why licence-only comparisons mislead. A realistic first-year total has five parts.
- 1Licence or usageThe published price, at your real annual volume rather than the headline tier. Model the volume you will have in month twelve, not month one.
- 2IntegrationConnecting to your ERP, PLM, mailbox or document store. On agent-shaped projects this is usually the largest line and the one most often left out of the business case.
- 3Rules and tuningEncoding how your company actually works — your tolerances, your templates, your specifications. This is the work that makes the difference between a generic tool and something useful.
- 4The review time you keepHuman approval is not free and does not go away. Count it. A system that halves the work but requires a senior person to check everything has a different business case than one that does not.
- 5Change managementTraining, documentation, the productivity dip in the first weeks. Small, real, and consistently forgotten.
If the only number in your business case is the licence fee, you are not comparing options — you are comparing price tags.
When to stop
Set the kill criteria before you start, and write them down. Something like: if acceptance rate is below sixty per cent after eight weeks of tuning, or if hours returned are under a threshold you name now, we stop. Deciding this in advance is the only reliable protection against the sunk-cost argument that keeps mediocre deployments alive for years.