We built an estimator that predicts how long a new requirement will take, using the delivery history of a North American education-technology client. Before we ran the test, we wrote down what result would justify shipping it. No method got there. On elapsed days none came near the 15% improvement over the plain median that we had set, and the one effort variant that reached 15.1% failed the second rule we had set in advance. So none passed both, and we didn't ship it. Estimates now appear as the median for the requirement's size band, labelled uncalibrated.
That is a small decision about one tool. It is also the discipline we ask of any AI-led delivery claim: decide what counts as proof before you see the numbers, then report what they say.
Key takeaways
- We tested four estimation methods on 127 finished requirements from one client programme (Anitech backtest, 24 September 2026).
- The rule was fixed in advance: at least 30 cases, at least 15% better than the plain median, and a second pre-set check on the estimates.
- No method passed both tests. The typical miss was about 38 days, on requirements whose median length is about 30 days.
- A prediction that is no better than the median adds false precision, so we show the median and label it uncalibrated.
- The same habit applies to claims about AI: set the bar first, and publish the miss as readily as the hit.
What did we build, and why?
A requirement usually reaches delivery as a title and little else. Someone still has to say roughly how long it will take, because planning, release dates and client conversations all depend on the answer. People guess, and the guesses are rarely checked against what happened.
The client's work-tracking history gave us something rare: 1,011 requirements and 1,244 user stories, with the status changes recorded for each. All the figures in this article come from that one programme. We wanted to know whether the history of past requirements could predict the next one well enough to put a number in front of a product owner.
What was the pass mark?
We wrote the rule down before running anything. A method would ship only if it met every one of these:
- It was tested on at least 30 past requirements.
- Its typical error was at least 15% lower than the error you get by simply quoting the median of past requirements.
- It passed a second check on the estimates themselves, also fixed before the test. We haven't published that rule's definition yet, so we only report whether each method passed it.
The plain median is the right opponent. It costs nothing, needs no model and is what a sensible planner would use anyway. If a method can't beat it by a clear margin, it hasn't earned a place in the workflow. Fifteen per cent is a judgement, not a law. What mattered is that it existed before the results did.
What happened on the backtest?
For each finished requirement, we estimated it using only the work that had finished before it started, then compared the estimate with what happened. That stops the method seeing the future. We tested four ways of estimating: from the requirement's title, from its use-case count, from its story count, and from the story-count size band.
| Method | Elapsed days | Effort |
|---|---|---|
| Title | 0.0% | 1.4% |
| Use-case count | 1.7% | 4.4% |
| Story count | 4.1% | 12.8% |
| Story-count size band | 6.7% | 15.1% |
On elapsed days, nothing came near. The title method matched the median exactly, with a typical error of 37.64 days against 37.64. On effort, one variant reached 15.1%, which looks like a pass until you read the rest of the rule. It failed the second pre-set check. It also depends on knowing how many stories a requirement will split into, and nobody knows that when the requirement is still a title.
So the honest summary is that no method passed both tests. We could have quoted the 15.1% and let the reader assume the rest. We would rather the table was complete.
Why not ship it anyway?
Because a number from a model looks more certain than a number from a median, and here it isn't. The typical error was about 38 days, and the median finished requirement in this programme takes about 30. An estimate that can be wrong by more than the length of the thing being estimated shouldn't be written into a plan.
The history also explains why prediction is hard. In the same data, use-case count has a rank correlation of only 0.10 with delivery days, so how big a requirement looks tells you little about how long it takes. What does move the clock is how many stories it splits into: a large requirement carries about 9.2 times the stories of a small one and takes about 3.2 times as long. That is real signal, but it only appears after the requirement has been elaborated, which is too late for a first estimate.
So we show the median for the requirement's size band instead, labelled uncalibrated, and keep it out of drafted requirements and out of the client's tracker. It tells a planner what usually happens without pretending to know what will happen here.
What does this say about evidence in AI-led delivery?
Most AI products arrive with a number attached, and most of those numbers were chosen after the fact. Three habits stop us doing the same.
- Write the pass mark before the test, with a sample size and the comparison it has to beat.
- Test on work the method could not have seen, and compare against the simplest sensible alternative.
- Publish the miss. A result you only report when it flatters you isn't evidence.
We apply the same standard to our own delivery claims. When we describe the Harness pilot, we say that merge requests were opened, not merged, and that quality outcomes haven't been measured. Our Agentic Harness Orchestration work is governed by rules and human approval gates for the same reason: confidence should follow evidence. ETA, our measurement-first design, is built on the same idea, to instrument work going forward so claims can be checked later. The ETA white paper sets out how.
When does this not apply?
This is one client programme, and only 158 of its 1,011 requirements had a finished, linked delivery history in the 24 September snapshot (157 the day before). The backtest used 127 finished requirements. A method that failed on thin data might pass on richer data. Different work, such as a stable product team on a long-running platform, may behave differently too.
We plan to run the test again after the Requirements Harness has drafted and delivered more requirements, because consistent drafts may make titles more informative. If a method then clears the same bar, we will ship it. If it doesn't, we will say so again.
- Why compare against the median?
- It is the simplest honest estimate: what usually happens for work of this kind. A model that can't beat it by a clear margin adds complexity without information.
- Does this mean AI can't estimate delivery time?
- No. It means these four methods, on this history, didn't clear a pre-set bar. We tested them on one programme, and we intend to retest as better inputs become available.
- Where did the data come from?
- From the status history of one North American education-technology client's work-tracking project, read-only, with no model involved in producing the measurements. The snapshot is dated 24 September 2026.
If you are weighing up an AI delivery claim, ours or anyone else's, start by asking what result would have counted as failure. Our AI consulting team can help you set that bar for your own estate. Get in touch to talk it through.
About the author
How this was written. Drafted with AI assistance from Anitech's internal working papers and edited by Jack Dinneen.
