8 min read

Our model never lost a contest. That was the problem.

A perfect record usually means nobody is keeping score. How a small learning agent went from 34 unearned wins to fair champion-versus-challenger contests on held-out data, why it then couldn't win at all, and what exploration fixed.
Headline "Our model never lost a contest. That was the problem." beside the Kestrel mark and a contest log reading "keep: not better than the active model", with 34-0, 1 in 5 and 5%.

For five days, the AI model on my test device had a perfect record. The dashboard's tally read 34 to 0: thirty-four new on-device models had taken over, and the model trained on my desktop had never once won.

None of those wins had been earned. No model had ever been compared with the one it replaced. Each time the device trained a new model, it simply took over and counted as a win.

If you lead a data science team, you've seen this pattern: the champion model that writes its own exam, grades it, and wins every time. This is how I found it in a side project, how we fixed it, and what happened right after the fix, which was the more interesting part.

Kestrel is my nights-and-weekends project, built with Claude as my AI teammate: an open-source Raspberry Pi device for authorized Wi-Fi security research, with a dashboard on your phone and a small learning agent. It's unaffiliated with my employer, and the views here are my own.

What the agent does

Every 30 seconds, Kestrel's learning agent decides how the device should listen for the next window: which channels, how long to stay on each, and in what mode. A scripted sweep is always there as the fallback and the yardstick. A learned model is supposed to earn its place by doing better than that sweep.

New models come from two places. The device fine-tunes one itself every half hour or so of capture, and an optional plugin trains bigger ones on a desktop and sends them back. The question I cared about was simple: which source trains better models?

The perfect record

When Claude and I read the code behind that 34-to-0 tally, it was measuring something else entirely:

  • On-device models took over with no test. Each training run wrote straight into the active slot and counted as a "local win".
  • Desktop models faced hurdles the local one never did. They had to beat the sweep, beat the local model's lift by 10%, be trained on newer data than the local model, and win outright, since ties went to the local model. Most were never scored at all.
  • The exam leaked. Candidates were scored on a random 20% of the logs, but the local model had trained on all of them. On the test device, 94-96% of the "held-out" rows were in its training data. Scored there, its lift over the sweep was +0.57; on the rows no model had seen, +0.17. About a third.
  • The desktop never trained the right model. Its launch trigger counted new data since the device's last model. The device retrained every half hour, so that count never got high enough, and the desktop never trained the algorithm the device actually ran.

The tally didn't measure which side trained better. It measured how often the device trained.

Before 1.0.11, an on-device model took over untested while a desktop model faced extra hurdles and was scored on data the local model had trained on. Since 1.0.11, every candidate, from either source, goes through one gate: held-out hours no model trains on, one rule.
Before: two sets of rules. After: one gate for every model.

One exam, one rule

The fix shipped in Kestrel 1.0.11. It's the champion/challenger setup most ML teams know, with two details that made it fair.

A holdout no model ever trains on. The logs are split by the hour. Each hour's number is run through a hash, and one hour in five is set aside for grading. Because the split comes from a hash and not a random draw, the device and the desktop agree on it without talking to each other, and it never changes as old logs rotate out. Whole hours, not scattered rows, also mean that models with memory are graded on real sequences of events, not shuffled ones.

Two days of logs as hour blocks: about one hour in five is set aside by a hash of its number, and no model, on the device or the desktop, ever trains on those hours.
The exam: whole hours, set aside by a hash, never trained on.

One rule for every candidate. Every new model is now a candidate, wherever it was trained. On the newest held-out hours (up to 2,000 decision windows; with fewer than 64 the contest waits), the candidate, the model in charge and the scripted sweep are all scored, and the candidate takes over only if it:

  • beats the scripted sweep;
  • beats the model in charge on the same windows, by a paired difference that clears a significance bar;
  • improves on that model's lift over the sweep by at least 10%, so two near-equal models don't swap back and forth.

On-device runs now fine-tune the model in charge instead of starting over, so a strong desktop model becomes the next starting point instead of being overwritten. The desktop trains the algorithm the device actually runs, first. And the tally became what it claimed to be: contests won and lost, by source.

Then it never won

A few days later, the test device's tally was honest, and grim: on-device 0 won, 32 lost. Desktop 0 won, 6 lost. Every contest log said the same thing: not better than the active model, mean difference +0.0000.

Not slightly worse. Exactly equal. So we probed the agent read-only and reproduced one training run on the side:

  • The model in charge picked one action on all 192 held-out windows.
  • A fresh fine-tune moved its weights by 3.5%, and still picked the same action on 192 of 192. Every per-window difference was exactly zero.
  • Of the 32 actions the agent can choose from, the logs held data for 3.
  • The reward model, the one that grades the contest, thought the best action on 114 of those windows was one never tried, because it scored untried actions at the average.

That's a loop every data team has met in another form. A greedy policy only takes the action it already prefers, so the data only describes that action, so training can't find a better one, so every new model makes the same choices, so every contest is a tie. A recommender that only learns from what it already showed. A credit model that only ever sees how approved loans turned out. More training can't break that loop, because the missing information was never collected.

The loop that kept the agent stuck: a greedy policy runs one action, so the logs only describe that action, so training changes the weights but not the choices, so every contest ties. Three fixes break it: explore a little, be cautious about the untried, and start over now and then.
More training can't break this loop. Different data can.

Three fixes, in beta now

These are in the 1.0.14 betas (the fair contests above have been in stable releases since 1.0.11):

  • Explore a little, on purpose. While the learned model drives, about one window in twenty runs a different allowed action instead, held for two windows (a minute) to get a fair reading. It favours the actions tried least. Those windows are labelled, logged and counted on their own line, so they never flatter or hurt the model's own numbers. The scripted sweep never explores, and the operator's list of allowed actions bounds everything the agent runs.
  • Be cautious about the unknown. Until an action has been tried in at least 8 windows, the reward model assumes it does no better than the weakest well-tried action, not the average. A model can't win by betting on actions nobody has tested; exploration is what earns them a fair score.
  • Start over now and then. Every 4th on-device run, and any run after 3 contests in a row where the new model chose exactly what the old one chose, trains from scratch with a new random seed and three times the steps. It faces the same contest as every other candidate, and those contests are counted separately.

The contest log now says how often a candidate chose the same action as the model in charge, so a run of identical choices shows up as what it is, instead of a mysterious tie.

A strip of 30-second decision windows: most run the learned model's choice, about one in twenty starts a two-window exploration of a less-tried action. Below, three contests in a row with identical choices trigger a training run from scratch.
One window in twenty explores. Three identical contests in a row trigger a fresh start.
The AI Agent page on a phone: Exploring "5% of windows, 21 of 32 actions tried, 6 in the last hour", Model contests "on-device 3 of 10 won, remote 2 of 3 won, from scratch 1 of 3 won", and the learned, baseline and exploring figures on separate lines. Made-up sample data.
Exploring and contests, each on its own line. Made-up sample data.

First results from the field

The betas are now running on my test device, a Raspberry Pi 5. It's one device and a small sample, so read these as early signs, not a verdict:

  • Exploring filled in the gaps. All 32 actions now have data, up from 3. In about 12 hours the device ran 67 exploring windows, all but one in pairs, each logged on its own line.
  • Fresh starts are winning. Models trained from scratch have won 4 of 9 contests. In one, the new model's lift over the scripted sweep was +0.93 against the model in charge's +0.61, on 1,247 held-out windows, with a paired difference far past the significance bar.
  • Fine-tuning is still stuck. In the latest contests, an ordinary fine-tune chose exactly what the model in charge chose on 100% of windows, and the model in charge stayed. Of the three fresh starts in those 12 hours, two were set off by the "three identical contests in a row" rule, not the every-4th schedule. That's the rule doing its job.
  • The desktop hasn't won yet: 0 of 13.

The scoreboard now reads on-device 6 won, 71 lost, and four of those six wins went to models trained from scratch. A lost contest is no longer a mystery: the log says why.

The fleet answers the big question

One device can't settle "desktop or on-device?" alone. Kestrel's anonymous census carries each device's contest wins and losses for the day, by source, with no names or locations, and every device can read the totals: "desktop won X of Y training contests this week". Before 1.0.11, that number would have counted training runs. Now it counts fair contests on data neither side trained on, which makes it worth reading.

For data and engineering leaders

  • Ask who grades the champion. If the model in production is scored on data it learned from, every challenger loses. Our incumbent's lift looked three times bigger than it was.
  • Make the holdout boring and shared. A deterministic split by time block, that every training job computes the same way, removes a whole class of "the numbers don't match" arguments between teams.
  • A model that never loses, or never wins, is a finding. Perfect records and endless ties both mean the contest isn't measuring what you think. Log why each contest ended the way it did.
  • Exploration is a budget, not a bug. Cap it, label it, report it separately. Then nobody has to choose between learning and looking good.

Working with an AI teammate is what made this affordable. Claude read the code paths end to end, wrote the read-only probe, and drafted both designs before any code changed. Every decision (the 20% holdout, the 5% budget, waiting for field results before claiming a win) stayed mine.

For investors

"Our model keeps getting better" is one of the most common claims in AI pitches, and one of the least checked. Ask how a new model is compared with the old one: on what data, by what rule, and how often the new one loses. A team that can show you its lost contests has a scoreboard. A team with a perfect record may just have a model that writes its own exam.

If posts like this are useful to you, subscribe and the next one lands in your inbox.

When did a model on your team last lose a fair contest? 🪶

Kestrel is a personal, open-source project built on nights and weekends, unaffiliated with my employer, and for authorized testing only. The user guide and source are public.