The dashboard said "Unknown." The device was fine. A field bug, found in under an hour

Every engineering team has a drawer of bugs nobody chases. They're weird, they're intermittent, they don't hurt anyone today, and figuring them out would cost a senior engineer most of a week. So they get a ticket, a "can't reproduce", and a quiet spot in the backlog.

This is the story of one of those bugs, and of why I think that drawer is about to get a lot emptier.

Kestrel is my nights-and-weekends project: an open-source Raspberry Pi device for authorized Wi-Fi security research, with a web dashboard on your phone. One evening, right after updating it to the latest release, I watched a card on that dashboard flip from "lots of radios nearby" to "Environment: Unknown. Scanning the area…". Meanwhile, two lines down, it kept reporting new captures. The device was clearly working. The dashboard said it couldn't see anything.

Harmless. Weird. Exactly the kind of thing I'd normally note and move on from. Instead, I asked Claude, the AI teammate I build Kestrel with, to help me find out why. Forty-three minutes after I switched capture off, we had the root cause and a tested fix; two minutes later, a better diagnostic tool than the one we started with.

The hunt

Here's how it actually went, wrong turns included, because the wrong turns are the point.

Step one: build the tool for the question. The existing diagnostic scripts dumped data but didn't answer "is anything wrong with this device right now?", so Claude wrote one that does: a read-only health check that runs from my laptop over SSH and opens with a plain verdict (OK, WARN, FAIL) before any detail. It also follows the project's privacy rules: the device sits in real places, near real people's networks, so every diagnostic reads only counts and ages, never a network name or address, and masks anything identifying before the report leaves the device.

Hypothesis one: it's the update restart. Partly right. The update had restarted everything, which explained why the session counters had reset. It didn't explain the card.

Hypothesis two: the reader can't keep up. The card's number comes from the capture tool's database, and our collector read the whole thing every ten seconds. As the file grew, maybe those reads got slow enough to fail. A plausible story, and it was wrong.

Hypothesis three: permissions. A second tool rebuilt the history minute by minute from the data still on the device. It found the database wouldn't even open: "attempt to write a readonly database." That's a real error, and it sent us down a promising path about file permissions and database modes. Also wrong: the database used a different locking mode from the one we suspected, and the error was a side effect, not the cause.

Hypothesis four: the database fell behind. The history showed something odd. The database's data stopped five minutes into the run, but the file kept being written until capture stopped seventeen minutes in. So Claude read the source code of the capture tool itself (AngryOxide, the excellent open-source project Kestrel builds on) and found that it commits one database row per captured frame. Then came the arithmetic:

• the database held 69,753 rows, written over about 17.5 minutes: roughly 66 rows a second;
• in those first five minutes, the device was seeing about 13,000 frames a minute, about 220 a second, from roughly 600 radios in range.

The capture tool wasn't stuck. It was writing to its database a third as fast as frames arrived, so the database fell further behind every minute: twelve minutes behind by the time capture stopped. Our card read the newest data in that database, so it aged out and said "Unknown", while the capture itself, written to separate files, never missed a beat.

The fix

The fix didn't touch the database at all. The capture tool also writes every frame to a plain capture file, a simple append that keeps up easily. The collector now remembers how far it has read that file and, every ten seconds, parses only what's new, counting each frame's transmitter. No database, no locks, nothing that can slow the capture tool down. The collector's test suite (47 tests) now covers it, including a test that fails if anyone ever points the collector back at the database.

Before shipping it, we ran the new reader against the real capture file from that evening, counts only, on the device. It saw frames right up to the moment capture stopped, at minute seventeen, where the database had stopped at minute five, and about 580 radios in the last five minutes, in line with what the database showed while it was still current. It read the whole 29 MB file in under a second on the Pi; a normal poll reads only what's new.

The health check got upgraded too: it now watches the capture file instead of the database, and it would have named this exact failure on the first try. And the slow database writes themselves are a fix for the capture tool's own code, which we'll make in our copy and offer back to its authors.

What I decided, and what Claude did

The speed is the headline, but the human calls are what made it safe:

• Privacy over convenience. Every diagnostic read counts, never names or addresses, even though dumping the raw data would have been faster.
• No security shortcuts. The permissions theory came with an easy "fix": loosen the folder's permissions. That would have undone a deliberate earlier decision that the privileged capture process never works in a folder an ordinary user can write to. We didn't take it.
• Right-sized scope. Fixing the capture tool itself would force a full reinstall for every user, so the next release gets the safe fix and the deeper one is planned separately.
• Evidence before shipping. The fix doesn't ship until it's been run against the real capture file from that evening.

Claude did the throughput: writing two diagnostic tools on demand (and revising one of them three times as the questions sharpened), reading a third-party codebase, doing the arithmetic, writing the fix and its tests, and changing course each time the data said a theory was wrong.

For engineering leaders: the drawer is about to empty

The interesting part isn't that the bug was found. Good engineers find bugs. It's what finding it cost: well under an hour, mostly spent waiting for a device to answer, instead of the days that made bugs like this not worth chasing.

When investigation gets that cheap, the calculation changes:

• The "weird but harmless" bugs become worth chasing. This one was hiding a real problem in a dependency, one that would have bitten harder in a busier place or on a smaller device.
• Wrong turns stop being expensive. Two of the four theories were wrong. Each cost minutes to test and drop, not a sprint.
• Tooling gets built for the question. Instead of squeezing an answer out of whatever dashboards exist, you build the exact instrument the question needs, then keep it.

The guardrails matter more, not less. Read-only tools, privacy rules written down before the work, and a human who decides what ships are what let you move this fast in a production environment without breaking things.

For investors: reliability just got cheaper

Hardware and field-deployed software die from the long tail: the bug that only happens in a busy café, on one model of SD card, after twenty minutes. Those bugs usually take senior engineers, scarce and slow to hire, days to run down, so small teams can't afford many of them.

What I saw here is a small team (one person, on evenings) running down a three-layer, cross-project bug in under an hour, with a fix, tests and better tooling at the end. That changes the economics:

• Small teams can support real devices in the field. The cost of each field bug drops by an order of magnitude.
• Quality compounds. Every hunt leaves behind a tool, a test and a fixed dependency, so the next one is faster.
• Diligence should look at how teams verify. Ask a team to walk you through their last weird bug: how long it took, what they built to find it, and what's in place so it can't come back. The answer tells you whether their speed will hold.

What's the bug in your drawer you'd chase if it only took an hour? 🪶

Kestrel is a personal, open-source project built on nights and weekends, unaffiliated with my employer, and for authorized testing only. The source is public.