Our model never lost a contest. That was the problem.
For five days, the AI model on my test device had a perfect record. The dashboard's tally read 34 to 0: thirty-four new on-device models had taken over, and the model trained on my desktop had never once won.
None of those wins had been earned. No model had ever been compared with the one it replaced. Each time the device trained a new model, it simply took over and counted as a win.
If you lead a data science team, you've seen this pattern: the champion model that writes its own exam, grades it, and wins every time. This is how I found it in a side project, how we fixed it, and what happened right after the fix, which was the more interesting part.
Kestrel is my nights-and-weekends project, built with Claude as my AI teammate: an open-source Raspberry Pi device for authorized Wi-Fi security research, with a dashboard on your phone and a small learning agent. It's unaffiliated with my employer, and the views here are my own.
What the agent does
Every 30 seconds, Kestrel's learning agent decides how the device should listen for the next window: which channels, how long to stay on each, and in what mode. A scripted sweep is always there as the fallback and the yardstick. A learned model is supposed to earn its place by doing better than that sweep.
New models come from two places. The device fine-tunes one itself every half hour or so of capture, and an optional plugin trains bigger ones on a desktop and sends them back. The question I cared about was simple: which source trains better models?
The perfect record
When Claude and I read the code behind that 34-to-0 tally, it was measuring something else entirely:
- On-device models took over with no test. Each training run wrote straight into the active slot and counted as a "local win".
- Desktop models faced hurdles the local one never did. They had to beat the sweep, beat the local model's lift by 10%, be trained on newer data than the local model, and win outright, since ties went to the local model. Most were never scored at all.
- The exam leaked. Candidates were scored on a random 20% of the logs, but the local model had trained on all of them. On the test device, 94-96% of the "held-out" rows were in its training data. Scored there, its lift over the sweep was +0.57; on the rows no model had seen, +0.17. About a third.
- The desktop never trained the right model. Its launch trigger counted new data since the device's last model. The device retrained every half hour, so that count never got high enough, and the desktop never trained the algorithm the device actually ran.
The tally didn't measure which side trained better. It measured how often the device trained.

One exam, one rule
The fix shipped in Kestrel 1.0.11. It's the champion/challenger setup most ML teams know, with two details that made it fair.
A holdout no model ever trains on. The logs are split by the hour. Each hour's number is run through a hash, and one hour in five is set aside for grading. Because the split comes from a hash and not a random draw, the device and the desktop agree on it without talking to each other, and it never changes as old logs rotate out. Whole hours, not scattered rows, also mean that models with memory are graded on real sequences of events, not shuffled ones.

One rule for every candidate. Every new model is now a candidate, wherever it was trained. On the newest held-out hours (up to 2,000 decision windows; with fewer than 64 the contest waits), the candidate, the model in charge and the scripted sweep are all scored, and the candidate takes over only if it:
- beats the scripted sweep;
- beats the model in charge on the same windows, by a paired difference that clears a significance bar;
- improves on that model's lift over the sweep by at least 10%, so two near-equal models don't swap back and forth.
On-device runs now fine-tune the model in charge instead of starting over, so a strong desktop model becomes the next starting point instead of being overwritten. The desktop trains the algorithm the device actually runs, first. And the tally became what it claimed to be: contests won and lost, by source.
Then it never won
A few days later, the test device's tally was honest, and grim: on-device 0 won, 32 lost. Desktop 0 won, 6 lost. Every contest log said the same thing: not better than the active model, mean difference +0.0000.
Not slightly worse. Exactly equal. So we probed the agent read-only and reproduced one training run on the side:
- The model in charge picked one action on all 192 held-out windows.
- A fresh fine-tune moved its weights by 3.5%, and still picked the same action on 192 of 192. Every per-window difference was exactly zero.
- Of the 32 actions the agent can choose from, the logs held data for 3.
- The reward model, the one that grades the contest, thought the best action on 114 of those windows was one never tried, because it scored untried actions at the average.
That's a loop every data team has met in another form. A greedy policy only takes the action it already prefers, so the data only describes that action, so training can't find a better one, so every new model makes the same choices, so every contest is a tie. A recommender that only learns from what it already showed. A credit model that only ever sees how approved loans turned out. More training can't break that loop, because the missing information was never collected.

Three fixes, in beta now
These are in the 1.0.14 betas (the fair contests above have been in stable releases since 1.0.11):
- Explore a little, on purpose. While the learned model drives, about one window in twenty runs a different allowed action instead, held for two windows (a minute) to get a fair reading. It favours the actions tried least. Those windows are labelled, logged and counted on their own line, so they never flatter or hurt the model's own numbers. The scripted sweep never explores, and the operator's list of allowed actions bounds everything the agent runs.
- Be cautious about the unknown. Until an action has been tried in at least 8 windows, the reward model assumes it does no better than the weakest well-tried action, not the average. A model can't win by betting on actions nobody has tested; exploration is what earns them a fair score.
- Start over now and then. Every 4th on-device run, and any run after 3 contests in a row where the new model chose exactly what the old one chose, trains from scratch with a new random seed and three times the steps. It faces the same contest as every other candidate, and those contests are counted separately.
The contest log now says how often a candidate chose the same action as the model in charge, so a run of identical choices shows up as what it is, instead of a mysterious tie.


First results from the field
The betas are now running on my test device, a Raspberry Pi 5. It's one device and a small sample, so read these as early signs, not a verdict:
- Exploring filled in the gaps. All 32 actions now have data, up from 3. In about 12 hours the device ran 67 exploring windows, all but one in pairs, each logged on its own line.
- Fresh starts are winning. Models trained from scratch have won 4 of 9 contests. In one, the new model's lift over the scripted sweep was +0.93 against the model in charge's +0.61, on 1,247 held-out windows, with a paired difference far past the significance bar.
- Fine-tuning is still stuck. In the latest contests, an ordinary fine-tune chose exactly what the model in charge chose on 100% of windows, and the model in charge stayed. Of the three fresh starts in those 12 hours, two were set off by the "three identical contests in a row" rule, not the every-4th schedule. That's the rule doing its job.
- The desktop hasn't won yet: 0 of 13.
The scoreboard now reads on-device 6 won, 71 lost, and four of those six wins went to models trained from scratch. A lost contest is no longer a mystery: the log says why.
The fleet answers the big question
One device can't settle "desktop or on-device?" alone. Kestrel's anonymous census carries each device's contest wins and losses for the day, by source, with no names or locations, and every device can read the totals: "desktop won X of Y training contests this week". Before 1.0.11, that number would have counted training runs. Now it counts fair contests on data neither side trained on, which makes it worth reading.
For data and engineering leaders
- Ask who grades the champion. If the model in production is scored on data it learned from, every challenger loses. Our incumbent's lift looked three times bigger than it was.
- Make the holdout boring and shared. A deterministic split by time block, that every training job computes the same way, removes a whole class of "the numbers don't match" arguments between teams.
- A model that never loses, or never wins, is a finding. Perfect records and endless ties both mean the contest isn't measuring what you think. Log why each contest ended the way it did.
- Exploration is a budget, not a bug. Cap it, label it, report it separately. Then nobody has to choose between learning and looking good.
Working with an AI teammate is what made this affordable. Claude read the code paths end to end, wrote the read-only probe, and drafted both designs before any code changed. Every decision (the 20% holdout, the 5% budget, waiting for field results before claiming a win) stayed mine.
For investors
"Our model keeps getting better" is one of the most common claims in AI pitches, and one of the least checked. Ask how a new model is compared with the old one: on what data, by what rule, and how often the new one loses. A team that can show you its lost contests has a scoreboard. A team with a perfect record may just have a model that writes its own exam.
If posts like this are useful to you, subscribe and the next one lands in your inbox.
When did a model on your team last lose a fair contest? 🪶
Kestrel is a personal, open-source project built on nights and weekends, unaffiliated with my employer, and for authorized testing only. The user guide and source are public.
Member discussion