A fair AI productivity trial counts review, waiting and rework

A practical team experiment that separates perceived ease from observed quality and total effort.

Using AI can feel easier even when a task takes longer. It can also save time on one stage while moving work into review. A team evaluating productivity needs to keep these experiences separate: perceived effort, elapsed time and the quality of the completed work all matter.

METR's early-2025 developer trial found slower completion despite participants' positive impressions. Its 2026 follow-up reports serious selection concerns. Support and consultant research found benefits in particular settings. The evidence supports careful local measurement, rather than declaring that AI always accelerates or always slows work. [METR trial](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/), [update](https://metr.org/blog/2026-02-24-uplift-update/), [support](https://arxiv.org/abs/2304.11771), [HBS](https://aiinstitute.hbs.edu/navigating-the-jagged-technological-frontier/).

In this original exercise, a small documentation team compares two ways of preparing an installation guide. The work includes reading the actual product instructions, drafting, checking every step and revising after a reviewer tries it. The team defines completion as a guide that passes that practical check. A first draft delivered quickly is a stage result, not the finished outcome.

Create several comparable fictional assignments and establish the required content before beginning. Some use a straightforward procedure; others include a version difference or a missing prerequisite. Staff record which tasks use the approved AI tool and which use the existing method. The experiment should avoid giving all easy assignments to AI and all difficult ones to the comparison group.

Record time by stage: understanding the source, drafting, checking, correcting and final review. Waiting time is recorded separately from active attention. If someone does another task while the model runs, the elapsed time and active effort differ. If a colleague reviews the guide, their effort is included. The cost should not disappear because it occurs outside the original author's stopwatch.

A worked example has an AI draft that omits a required restart. The author finishes drafting quickly, but the reviewer later repeats the procedure and identifies the missing step. The revision consumes time and could have confused a customer. Another draft includes the prerequisite correctly but needs awkward language rewritten. These are different failures and should be recorded separately.

Ask participants about effort and confidence before showing the measurements. A task may feel less tiring with AI even if it takes the same time. That can be a worthwhile benefit, provided the team reports it as an experience measure rather than converting it into a percentage speedup. Equally, a more enjoyable process should not excuse a decline in factual accuracy.

The reviewer should not know the production method where a practical blind review is possible. They use the same criteria for both sets of guides: correct steps, prerequisites, accessible wording and successful completion. The team preserves the source version so disagreements can be investigated. It also tests whether an error is reproducible, rather than labelling every reviewer preference a model failure.

Selection requires attention. Staff may volunteer tasks for AI that they already expect it to handle well, or avoid tasks they strongly prefer to do with it. METR's later study highlights why this makes current effects difficult to interpret. The fictional team records which assignments were excluded and why. That does not remove every bias, but it makes the boundary of the result visible.

Repeat a small check after staff have learned the tool and after a material provider change. Do not continually rerun a huge trial without a reason. Maintain several awkward cases as regression checks, and use fresh assignments when testing a new workflow. The team should not report a tuned practice set as proof that the method handles unfamiliar work.

Costs include the authorised licence, secure handling of source material, reviewer attention and maintaining the measurement process. Current provider pricing was not verified for this article. A trial can conclude that AI helps routine guides, adds review cost for version-sensitive ones, or needs a narrower role. None of these conclusions establishes a universal workplace effect.

The most useful report explains where effort moved and whether the completed work improved. It can acknowledge that staff enjoyed the process while showing that a particular task took longer. Honest measurement gives the team a better basis for choosing tools and training than a dashboard that counts generated drafts as finished work.