proof

proof is a tool I wrote to answer one question about an AI assistant of mine: is it actually any good at deciding what to tell me? The code isn't public, and the assistant it came from runs on my personal data, so what I can talk about here is the tool.

Think of a weather forecaster who says it will rain every single day. He catches every rainy day, and he is useless. The only way to see that is to look at the days he did not say rain, and ask whether it rained less on those. Most ways of testing an AI never look at those days.

My assistant chooses what to bring up with me. So I kept a record of everything it considered, including what it decided not to bring up, then checked what really happened and compared the two piles. If the things it flagged go wrong no more often than the things it let pass, it was never telling me anything.

While I was building the assistant, I fooled myself five times in about a week, each time with a clean, believable result about nothing. In one, the part of the assistant that decides what is worth raising (I call it the judge) flagged 19 of 21 items. In another, a "no signal" really meant "we never looked". In a third, an apparent 18 percent improvement was seven things counted over and over. Each became a rule in the tool. The most useful one: when there are too few cases, or it couldn't see what really happened, it prints NOT ENOUGH EVIDENCE instead of a percentage.

The example is 60 pretend support tickets. Some never get closed, the rest get closed a few days to a few weeks after they are opened. The question is whether a ticket will still be open two weeks after the predictor looks. Three predictors all look reasonable. One reads how long a ticket has been sitting, one flips a coin, and one flags everything, so it has the best catch count. A test that only checks whether answers were right would have trouble telling them apart, because all three look defensible from the inside. Comparing the piles, only the first is worth anything.

I also pointed it at real issues from a big public software project: I pulled 400 of them, and 206 were usable, because most of the rest had already been closed within a few days of opening. The elaborate predictor found something real, but no more than a one-line rule about an issue's age, so the tool said it wasn't earning its cost. When I pulled the tool out of my assistant in July, it was telling me NOT ENOUGH EVIDENCE about the assistant's main claim, which was the right answer at that point. It makes cheating on the future hard to do by accident, not impossible, and my examples use hand-written scoring rules, not a language model.

What's next

The real-issues example only uses facts that were fixed when each issue was opened. Rebuilding things that change later, like comment counts, from the issue history is the honest next step.

What I wanted

I was building an assistant that decides for itself what to bring up with me, and I wanted to know if it was any good. The usual way to test an AI is to ask whether its answers were right. That doesn't fit here. A system that chooses what to raise doesn't fail by being wrong, it fails by raising everything, or nothing, and looking fine either way. Only a denominator shows that, meaning the cases it considered and did not raise, and almost nobody keeps those.

In about a week of building that assistant I caught myself believing results that were about nothing, five times. Every one was clean and well formed and looked publishable, and every one became a rule in this library. The assistant works on my personal data, so I'm leaving it out. What's here is the harness I pulled out of it in July 2026, version 0.1.0. It's TypeScript with no runtime dependencies, and the code isn't public.

How it works

Here is the whole path a claim takes, from what the judge is shown to the verdict at the end.

Every candidate the judge considers goes into a ledger, flagged or not, because the unflagged rows are the control group. A claim can't be written without a falsifier, which is the observation that would prove it wrong, and a check date after the as-of date. Otherwise the write throws.

Reality then answers each claim with confirmed (the predicted bad thing happened), refuted or unknown. The thing that answers is an oracle I supply. Unknown, and anything still pending, stays out of every rate.

The report asks four questions, in a fixed order. Is the oracle actually observing things? Do the raw scores separate the outcomes at all? Does the flagged set fail more than the unflagged set? Are there enough distinct subjects to say so? I put them in that order because asking them the other way round is how a broken oracle gets reported as a null result about the judge. That happened to me.

There are two ways in. The live way is ClaimLedger: I write claims as they happen and call resolve() once reality can answer. The other is backtest, which stands the judge at past dates across history I already have and grades it today. Before spending a model call on that, baseRate checks that the outcome actually varies, because if it happens 100% of the time every predictor scores the same. The backtest never grades a claim whose check date hasn't arrived, skips subjects that were already settled, and counts everything it excludes, because a silent filter is how a sample gets rigged.

One more piece is a small check on the claim "no single input caused this". isRealJoin wants at least two strands of at least two different kinds, more than a day apart, each dated and sourced. A candidate that fails is dropped, not downgraded, and if that leaves nothing today then nothing is the honest output. The precedent is a derived-state signal I'd shipped as "unfakeable" that turned out to be produced by a string-truncation bug. I deleted it instead of patching it.

Results

There are two example scripts. The first is a world of 60 made-up support tickets. 14 of them never get closed; the rest are closed 5 to 24 days after they are opened. Every prediction asks the same thing: will this ticket still be open 14 days later? Predictions are made every 21 days along a ticket's life, at most three per ticket, which gives 88 per predictor. Three predictors take part. The alarmist flags everything. The coin gives a random score. The judge reads how long the ticket has been sitting, with score = 1 minus days open divided by 60, and flags anything below 0.35.

I put the 88 predictions per predictor into the slider below. If you have scripts off, the table shows the setting I used, a flag line of 0.35.

The judge holds a lift of between +34 and +57 points at every flag line the harness will certify, from about 0.06 up to 0.88. Push the line further right and the unflagged side drops below 8, and it refuses. The coin's lift is zero or negative at almost every setting. At 0.16 and 0.17 it shows +2 points, which is pure noise, and the separation line still reads backwards. The sufficiency check is a count and not a significance test, so it can't catch that by itself. The separation line is what does. The alarmist, wherever you put the line, has nothing on one side, and the harness says NOT ENOUGH EVIDENCE.

The point of the first example is that the alarmist has the highest raw catch count and is worth nothing. With nothing unflagged it has no control group, so the harness refuses to score it. The coin is backwards, with separation -0.08. The judge shows a +43 point lift, but I built this world so that ticket age predicts stalling, and then gave the judge ticket age. It shows the harness can tell the three apart. It doesn't show that any real judge works.

verdict     NOT ENOUGH EVIDENCE — 88 flagged / 0 unflagged resolved. Any lift here is anecdote.verdict     NO SIGNAL — flagged 58% vs unflagged 68%. The selection carries no information; subtract it.verdict     LIFT +43pp — flagged 93% vs unflagged 50% (flat baseline 65%), over 60 subjects.

The second example uses real data. I pulled 400 issues from the Next.js repository, opened between January and April 2026, and 206 of them were usable: most of the rest had already been closed within three days of opening, before the first prediction date, so there was nothing left to predict. That gave 527 predictions, with the question "will this issue still be open in a fortnight?" The oracle is a lookup, whether closed_at comes before the check date, so a negative verdict is much more likely to be about the judge than about the instrument. The judge is a hand-weighted function. Age dominates, a reproduction in the issue helps, a very short report helps, and an outside reporter makes it linger. It's up against an incumbent, a one-line heuristic that only looks at age, and the comparison flags the same number of issues for both so neither wins by being trigger-happy.

coverage    527/527 observedsignal      separation +0.15, right direction: the estimate carries informationselection   flagged 160/165 · unflagged 320/362 (control) · 206 subjectsverdict     LIFT +9pp — flagged 97% vs unflagged 88% (flat baseline 91%), over 206 subjects.baseline    too close to call (9pp vs 9pp) — a model that only matches a cheap heuristic is not earning its cost

So the judge has a real signal and a real lift, and it still isn't earning its cost, because the heuristic gets the same lift for free. The lift is also small against a high base rate: both groups stay open about nine times in ten. I find that more useful than a result that flatters the judge.

The first run of this example was refused. I'd pulled the newest issues, which spanned twelve days, and the horizon is fourteen, so no check date had passed and the harness wouldn't score an empty set. Reaching for older issues isn't a workaround. You can't grade the future.

What didn't work, and the limits

The leak guard is an API shape, not a guarantee. A determined caller can reach around evidenceAt, and all the library does is make leaking deliberate rather than accidental. The merge-on-write storage can't express a real delete, so it's safe for stores that mark rows closed or superseded and wrong for one that needs to remove them. The sufficiency check is a count, 8 per side and 8 distinct subjects by default, and those two defaults are rules of thumb you can change. It is also not an accuracy benchmark. There's no golden dataset, and ground truth is whatever actually happened. If your system produces four gradeable items a month, it will tell you that, every time you ask.

Both examples score with hand-written functions, so I haven't shown the harness grading a language model. When I pulled it out in July 2026, it reported NOT ENOUGH EVIDENCE about the central claim of the system it came from. That's the intended behaviour, and it's why I trust it.

What is unfinished

The GitHub example only uses fields fixed at creation time. Comment counts and labels change after an issue is opened, so using today's values would be cheating. Rebuilding them as of each date from the issue timeline is the honest way, and it costs an API call per issue. I haven't done it. The tests cover the write-time guard, the unknown handling, the subject count, the storage merge, the leak guard and the join check. It isn't published as a package either.

Stack

TypeScript, no runtime dependencies; tests run with tsx

Modules

ledger, backtest, metrics, join (cross-reference check), jsonl (merge-on-write store)

Status

Prototype, version 0.1.0

Code

Not public