A key everyone knows: how Kestrel's census writes to the BitTorrent DHT, and forgets on its own

Most data has an owner: whoever runs the server, holds the account or keeps the private key.

Kestrel's census has none. It lives in a public data structure that millions of BitTorrent clients already run, using a feature designed for data only its owner can change, and we gave the key to everyone, on purpose. What you get is a whiteboard with no owner: every Kestrel can write on it, anyone can read it, and it wipes itself clean.

Earlier this month I wrote about why the census exists and what it refuses to collect: 20 bytes an hour, opt-in, no server, nothing that points back at a person. This post is the other half: how it actually works, bit by bit, and where it falls short.

Kestrel is my nights-and-weekends project, built with Claude as my AI teammate: an open-source Raspberry Pi device for authorized Wi-Fi security research, with a dashboard on your phone. It's unaffiliated with my employer, and the views here are my own.

A whiteboard in a building everyone shares

The BitTorrent DHT (distributed hash table) is how torrent clients find each other without a central server: one enormous key-value store spread across computers worldwide. Each value is held by the eight or so computers whose IDs sit closest to its key, and anyone can ask them for it.

An extension called BEP 44 lets you store small values there, up to 1,000 bytes. A "mutable" item is signed: its address comes from a public key, and only a valid signature from the matching private key can change it. It was designed for one owner publishing updates, like a feed.

The trick: a key everyone knows

A private key is normally the one thing you never share. The census makes its keys out of public text:

kestrel-census/1|2026-01-01|17

That's the protocol name, the UTC date and a slot number (an invented example: the census didn't exist on that date). Its SHA-256 hash is 32 bytes, which is exactly the size of an Ed25519 private key. From that key comes the public key, and the SHA-1 hash of the public key is the slot's address in the DHT.

Every Kestrel can do that arithmetic, so every Kestrel can write to every slot, and so can anyone else, including a desktop script in the repository that reads the totals. The DHT's computers still check the signature, but it no longer says who wrote. It only proves the writer knew the text, and the text is public.

Putting everything in that one line of text has two nice side effects:

• Every day gets fresh addresses. The date is part of the text, so each UTC day has 64 brand-new slots, each with three spares for when it fills up.
• Versioning for free. A format change older devices would misread gets a new protocol name, kestrel-census/2, and therefore entirely different addresses. Old and new devices can never mix entries in one slot.

20 bytes, bit by bit

A slot's value is a run of 20-byte entries, one per device, sorted. Forty-nine of them fit in the 1,000-byte limit. Every bit is spoken for:

• Token, 8 bytes. One device's entry for one day only: made at the day's first write, kept across reboots, dropped at midnight. From Kestrel 1.0.14 (in beta now) it also carries the board: two bytes spell kb, the third holds a 4-bit family code (Zero 2 W, Pi 3, Pi 4, Pi 5 or "another board"), and five are random. Three bits set aside for memory size are always zero, and only the code leaves the device, never the board's name. Older releases still read it as eight random bytes, so the change needed no new protocol.
• Hour, 1 byte. The UTC hour of the device's last write. This one byte is what makes the census forget, below.
• Release, 2 bytes. Three bits for the major version, four for the minor, six for the patch and three for the beta number. 1.0.14-beta.3 is 0x2073.
• Setup, 1 byte. Two bits for the learning agent's state, three for its algorithm, three for how remote training is doing.
• Contests, 4 bytes. Today's training contests won and lost, on the device and on a desktop, one byte each, stopping at 255.
• Proof-of-work, 4 bytes. It says nothing about the device.

Every field is a small code or a count since midnight. Nothing grows over a device's lifetime, and nothing is free text.

Once an hour: read, merge, put, check

Each device updates its slot itself, once an hour:

1. Read the slot from the computers that store it: the ones it used earlier in the day if at least four still answer, otherwise a fresh lookup. 2. Merge everything it got back: the newest entry per token, only entries from this hour or the one before, and its own fresh entry in place of its old one. 3. Put the result back. The DHT's computers only accept a value with a higher sequence number than the one they hold, so writers need an order without talking to each other. The census uses the clock: minutes since UTC midnight × 100, plus a random 0 to 99, and always above the highest number it has seen. At 14:37 that's somewhere from 87,700 to 87,799. If a computer says something newer already arrived, the writer reads again and retries once. 4. Check, 20 to 60 seconds later. If no copy holds its entry, because another device's write landed on top, it writes once more.

Each writer puts back every entry it saw, not just its own, so the slot heals itself: an entry lost to a collision comes back at its owner's next hourly write.

The random part of that number and the check weren't in the first design. Before writing any of the code, we ran more than a dozen field tests on the real DHT, from a desktop and then from a Pi 5 on a phone's tether. In one, two devices wrote in the same minute, both were told they'd succeeded, and one entry quietly vanished. Now that can't go unnoticed.

Data that forgets on its own

The DHT's computers are supposed to drop items after about two hours, so I expected to lean on that for forgetting. The field tests said otherwise: some still held a test item fourteen hours later, with nobody rewriting it. So forgetting is built into the data instead:

• Two hours of memory. Every writer drops entries from before the previous hour, so a device that goes quiet leaves its slot 60 to 120 minutes after its last write. Readers count "online" from the same hour byte.
• New addresses at midnight. The date is in every slot's text, so at UTC midnight 64 new slots begin, and yesterday's are never written again.
• A new token every day, so one day's entry can't be matched to the next by its token.
• Counters that start over. Contests are counted since midnight, never as lifetime totals.

The trade-off is deliberate. The census can tell you how many Kestrels were active today, and the Census page lists one line per day for the last 30 days. It can never tell you how many different Kestrels were active this month, because tokens don't carry from one day to the next, so the days can't be added up. That's a number we chose not to be able to know.

Privacy in the mechanics, not just the payload

What's in the 20 bytes is half of privacy. The other half is how and when they travel:

• A random minute. Each hourly write happens at a random minute, and the first write after the census starts or comes back (when the phone's tether reconnects, say) waits a random 5 to 30 minutes. Reads are kept out of the write's minute. Timing is information too: a report at the top of every hour, or the moment a phone connects, says something about someone's day. Spreading the writes out has a second benefit: fewer devices land in the same minute, so fewer writes collide.
• One slot per day. Each device picks one random slot for the day and stays in it.
• Quiet on the network. Every query marks itself read-only, asking the DHT's computers not to add the device to their lists of peers. The device answers no queries itself and never announces what software it runs. It also paces itself: at most ten queries a second.
• The tether by default. As the first post explained, the computers storing a slot see the internet address a report comes from. By default the census uses only the phone's Bluetooth tether, so that address is the mobile carrier's, shared with many of its customers, and without the tether the census pauses.
• Small. About 4 MB a day with the defaults, under a daily data budget.

What it can't do

Its limits, plainly:

• The numbers are approximate and unverified. Anyone can read and write the slots, and there's no trusted counter to vouch for a total. The Census page says "approximate and unverified" right above the numbers.
• There's a ceiling. 64 slots, each with three spares, times 49 entries is 12,544 entries a day. Past that, the page says the census is full; a future version can add slots.
• No monthly count of devices, as above. That's the price of daily tokens.
• Late contests fall out. A device's last write of the day lands at a random minute in the last hour, so contests after it aren't counted.
• The release field runs out at 7.15.63. That's a long way off; when it comes, it's a new protocol name.
• IPv4 only.
• It counts only devices that run it. The census is opt-in, ships switched off and takes an explicit confirmation to turn on, and by default a device reports only while its tether is up.

For engineering leaders

• Look for infrastructure that already exists. The DHT has run for years, worldwide, at no cost to anyone who uses it. The cleverest part of the census is that we didn't build the hard part.
• Measure the assumption you'd otherwise build on. "Items expire after two hours" was the plan until a field test showed fourteen. Two of the design's best decisions came from experiments, not from the whiteboard.
• Put forgetting in the data, not in a policy. A retention policy is a promise someone has to keep. A format that can't remember keeps itself.
• Write the format down. The protocol is a public document of under 200 lines, so any reader, not just Kestrel, can count the same way.

An AI teammate made the careful version affordable: the field tests, the protocol document and about 2,500 lines of plugin code with their tests took days, not a quarter.

For investors

No server means no user database to protect and a running cost of zero, however many devices join. The cheapest data to protect is data you never hold, and the cheapest to delete is data that deletes itself. Ask a team what their product remembers when nobody is maintaining it. The best answer is "only what it needs, for as long as it needs it."

If you run Kestrel, the census is one switch on the Plugins page or sudo kestrel plugins enable census, with your phone's tether set up. Like every plugin that goes online, it asks for the authorized-use confirmation first. The board family arrives with 1.0.14.

If posts like this are useful to you, subscribe and the next one lands in your inbox.

What would your product look like if it had to forget on its own? 🪶

Kestrel is a personal, open-source project built on nights and weekends, unaffiliated with my employer, and for authorized testing only. The user guide and source are public.