Tools

How to Measure AI Productivity by Checked Work, Not Tokens

Vadym Chernets · open source · 10 min read

AI agents and work

If you get real work done by handing it to AI agents (Claude Code, Codex, Cursor and the like), no job title, degree or résumé line shows what you can be trusted to deliver. The number many employers reached for in 2026, how many tokens a person burns, shows even less. Delegation Horizon is a free, open-source tool that turns a simple log of the work you hand to AI agents into one dated line:

small-business bookkeeping · Day · ×2 · 2026-Q4

It reads: in small-business bookkeeping, this person regularly hands AI agents pieces of work that would take a specialist about a day by hand, and a check the agents never saw confirms them; every hour of their own attention brings in at least two hours of checked work; scored in the fourth quarter of 2026. You keep the log in any spreadsheet, and the score is worked out on your own computer, by a small Python script or by a web page that runs offline. No account, no key, nothing to pay, and the log never leaves your computer.

Why counting tokens failed

In 2026 several large companies tried to measure how well staff used AI by how much AI they used, a fashion the press named tokenmaxxing. At Meta, an employee-built leaderboard nicknamed Claudeonomics logged 60 trillion tokens in 30 days across more than 85,000 employees (Shopifreaks). At Amazon, "employees were creating AI agents to carry out unnecessary tasks in an attempt to boost their scores", and the tracker was taken offline (CIO). In September Meta said it "will not use AI adoption dashboards or token counts to evaluate impact" (CIO).

A token count measures what went in, not what came out; DORA, Google Cloud's research programme, calls it "purely a measure of input" (DORA). Yet in LeadDev's 2026 survey 58% of respondents said their organisation still measures AI impact by token usage, while only 19% called it effective (LeadDev). Delegation Horizon counts the output instead, in any profession.

Two numbers and one line

Every piece of work you hand to agents becomes one row of the log: its field, how long a qualified specialist would need to do it by hand (its "reference hours"), how many minutes of your own attention it took, which check it went through, and whether it passed.

• The verified delegation horizon is the largest size of work you regularly hand to agents and get confirmed by a check they did not see. It has five rungs: Hour, Day, Week, Month and Quarter (1, 8, 40, 160 and 500 reference hours). Hour and Day need ten pieces of that size in the last 91 days; Week needs six in 182 days; Month six in a year; Quarter three in a year. A bookkeeper and a developer face the same rule.
• The amplification gain is checked reference hours per hour of your attention over the last 91 days, failures included, shown in steps of ×1, ×2, ×5, ×20 and ×60.

Both are kept per field, so the same person can be at Week in financial tools and at Day in web applications. Both carry a date: a rung is earned again every quarter, and from Week upward one of its pieces must come from the last 91 days.

What counts as checked

At least eight in every ten pieces of a rung's size must pass one of four kinds of check: hidden tests, a qualified expert, real use by the people the work is for, or a model from another AI family checking against sources you opened yourself. Your own review and agreement between models of the same family are recorded but never count. Failed and abandoned pieces stay in the log and count against you, and so does a piece someone helped with off the log, a piece whose later audit failed and a result that was later withdrawn. A piece whose mistakes cannot be undone, such as money sent, must pass, and another AI family is never enough on its own to check it. From Week upward, deliberate errors are planted in at least one task before the hand-over; catching them shows you really check, and a missed one blocks Week and above.

The strictness has a reason: people judge their own speed with AI badly. In a 2025 METR study, 16 experienced developers took 19% longer with AI tools, yet afterwards "they still believed AI had sped them up by 20%" (METR).

A worked example

The repository includes the log of an invented bookkeeper, P07: ten pieces of work, each a specialist's day or more, handed to agents between July and September 2026. Scored on 9 October 2026:

P07: small-business bookkeeping . Day (8/10) . x2 . 2026-Q4  (pieces in last 91 days: 10, not verified: 2; gain 4.66; directive share 20%)
  checks: expert 4, hidden_tests 3, self_review 1, real_use 1, other_family_with_sources 1; median accepted size 9.0 h

Eight of ten passed, exactly what Day needs; one piece was checked only by the bookkeeper's own review and one failed an expert's check. The eight checked pieces add up to 80 reference hours against about 17.2 hours of attention on all ten, a gain of 4.66, which falls in the ×2 step. Had the bookkeeper deleted the failed piece, the log would look better and score worse: the rung would rest on nine pieces, miss the count of ten and fall below Hour, while the gain would jump to ×5. Anyone receiving the record sees that at once, because a rescore on the card's date no longer matches. That is why failures belong in the record.

For people who build through agents

• record_card.py reads your spreadsheet and prints the line and a short card: the rung, the count behind it ("8 of 10 pieces of Day size verified in its 91-day window"), the gain, and the kinds of check. --format md makes a card for a profile or an email to an employer; --format record prints just the line for a résumé.
• tools/explain.py (new in version 1.1.0) tells you why your log earns the rung it does: which condition holds or fails and which piece decides it, how many more failures the rung can take, the day it lapses if you log nothing new, and what the next rung still needs. It also shows how little ten pieces prove on their own: eight of ten fit any true pass rate from about 44% to 97%.
• checker.html does the same scoring in a browser, offline; a test compares it with the Python scorer on the worked example and 40 random logs. Both read a spreadsheet saved with semicolons, decimal commas and day.month.year dates, as European spreadsheets save them.
• tools/log_piece.py builds a log line by asking you questions. If you work in Claude Code, a ready hook (a small script Claude Code runs on an event) opens a pending entry every time an agent session ends, so a failed piece is never forgotten. It reads no transcript and sends nothing anywhere; any agent tool that can run a command at the end of a session can use it.
• tools/planted.py seals your list of planted errors with a fingerprint before the hand-over and scores it afterwards, wrong entries and things left out counted apart, and refuses a list changed after sealing.

For the fourth kind of check, another AI family reading your sources, Sidecall puts a piece of work to the other AI tools on your computer.

For managers and HR

kits/ai-usage-in-reviews.md is a ready policy for AI in performance reviews. Instead of tokens or spend it asks for accepted pieces of work with the check each passed; instead of "uses AI daily", the largest piece handed to agents and verified last quarter. Its review conversation has three questions, starting with: what is the largest piece of work you handed to agents this quarter and showed to be right, and how?

A team keeps one log with a made-up label per person (the scorer refuses unknown columns, so real names cannot slip in); examples/team-log.csv shows four people across five fields. Every quarter record_card.py prints a card per person and field. To check one, tools/verify_record.py takes the record and the log and says, field by field, whether the log is the file that was scored, the scorer the same, and the record the same when rescored. audit_sample.py draws a random sample for re-checking from a word the second checker picks after receiving the log, so nobody can guess which pieces will be drawn, and tools/too_clean.py flags logs that look too good: no failures, sizes that barely vary, every piece checked one way.

Reference hours are fixed outside the team: from a task bank, or by two raters who do not know whose work it is. tools/reference_hours.py takes their estimates and gives their geometric mean when they are within 1.5 times of each other, or calls a third rater; kits/reference-hours-raters.md says how to choose and brief them.

To hire by verified work rather than by degree, kits/hiring-test/ holds a paid one-day work test in the candidate's own field, with a hidden check, planted errors and a rater who has not seen the CV. kits/trade-screen.md adds a first-week question of the trade before anyone is admitted, since a profile or job title says less than it seems: of 60 people a survey platform's filters admitted as bookkeepers, 11 got such a question wrong, eight of them by swapping debit and credit.

What the first test with people found

The weakest joint of any measure in "hours of a specialist" is the estimate itself, so it was tested first. In October 2026 a pre-pilot on Prolific, registered on OSF before any data came in, paid 119 working bookkeepers and web developers to estimate how long each of ten tasks would take by hand. Two raters picked at random landed within 1.5 times of each other only 28% of the time in bookkeeping and 25% in web work, against the 70% the plan set. Tasks written to be a week of work got median estimates of 8 to 20 hours, and the same task asked again at the end drew a higher estimate far more often than a lower one.

What follows for anyone using the tool: two strangers' guesses are too noisy to size a piece. Medians of larger panels steady quickly; two separate panels of 15 bookkeepers, or 21 web developers, agreed within 1.5 times in at least 70% of draws. So reference hours come from a task bank with measured times, or from a panel, and the anonymised data, the registered analysis and the exploratory one are all in pilot/ for anyone to rerun.

For researchers

METR measures AI models by a time horizon: the length of task, in expert hours, a model finishes reliably. Delegation Horizon asks the same of a person, and its eight-in-ten share mirrors METR's 80% horizon. A desk pre-pilot on simulated logs tuned the rules (version 1.1), the Prolific pre-pilot above tested the reference hours, and the protocol, preregistration draft and consent form of the full pilot with people are in pilot/. kits/process-vs-experience-study.md and kits/matched-task-study.md are ready protocols for comparing groups on the same tasks, and tools/power.py says how many people such a study needs.

The paper behind the tool also makes dated, checkable bets. In Anthropic's study of about 400,000 Claude Code sessions, sessions that produced code ended in verified success 34% of the time for people in software jobs and 29% for everyone else (Anthropic). Forty-one predictions were registered on OSF on 9 October 2026, each with a probability, among them that METR's 80% horizon for the best public model reaches a working day by the end of 2027 (0.80) and that fewer than two Fortune 500 employers use token counts as an official evaluation criterion during 2027 (0.80). A test shows every registered file is still byte for byte the one filed, and brier.py, predictions/score.py and predictions/skill.py score them every year; a study that is never run is counted as a miss, not quietly dropped.

How to start

You need Python 3.9 or newer and nothing else.

1. Download the repository from the GitHub page (Code, then Download ZIP) or clone it.
2. Open templates/work-log.csv in any spreadsheet (Excel, Numbers, LibreOffice, Google Sheets) and add one row per piece of work you handed to agents: accepted, rejected or abandoned. templates/README.md explains every column in plain words, and examples/c4-log.csv is a filled-in log to copy from.
3. Save it as CSV and run the card, or double-click checker.html if you prefer a browser.

git clone https://github.com/vadimchernets/delegation-horizon
cd delegation-horizon
python3 record_card.py examples/c4-log.csv --as-of 2026-10-09
python3 record_card.py my-log.csv --format md --out card.md
python3 tools/explain.py my-log.csv

Where it comes from

Delegation Horizon is the companion code of the paper After the Programmer Divide: One Measure for People Who Build Through AI Agents by Vadym Chernets (October 2026, SSRN 7589918, DOI 10.2139/ssrn.7589918). The current release, 1.1.0 of 11 October 2026, is tested on every push on Python 3.9 to 3.13. The code is under the MIT licence and archived on Zenodo (DOI 10.5281/zenodo.23267444, which always points to the latest release); everything is at github.com/vadimchernets/delegation-horizon. An earlier paper by the same author, After Chat, asks why so few people have moved from chatting with AI to letting agents work for them; its code, After Chat, puts one question to several AIs and shows where they disagree.

View delegation-horizon on GitHub

Vadym Chernets also builds Poly A1, an app in which AIs from different companies check each other's work.