7 min read

Three weeks, 44 releases: why my AI-built project slowed down on purpose

Eleven releases in five days made a good headline. Three weeks in, the pace has dropped sharply, and that's the plan working. What changed, where the bottleneck moved, and what it means for leaders and investors.
Headline "The pace slowed. On purpose." beside the Kestrel mark, with 14 stable releases, 30 betas, 157 pull requests and 1,567 automated tests.

A couple of weeks ago I wrote that a side project of mine went from version 1.0.0 to 1.0.10 in five days, built with Claude as my AI teammate. People asked the obvious question: does that pace hold, or is it a sprint that ends in a pile of debt?

Here's the answer, sixteen days in. Kestrel has shipped 14 stable releases and 30 betas, from 157 merged pull requests. The automated tests went from 711 to 1,567, and the code roughly doubled.

But the shape matters more than the totals. Eighty-seven of those pull requests landed in the first five days. Nine landed in the most recent seven. The release now in testing is three betas and five days in: about as long as it took to get from 1.0.0 to 1.0.10.

That slowdown is the best thing that has happened to the project, and I chose it. Here's why.

A word on where I'm coming from. By day I'm a VP leading an 110+ person data, analytics and data science organization. Before that I was a CTO who built an engineering and data organization from zero, spent nearly five years at Amazon, and co-founded a venture-backed startup. Kestrel is not that job: it's a personal, open-source project built on nights and weekends, on my own time and hardware, unaffiliated with my employer. The views here are my own.

Releases and merged pull requests per day, October 25 to November 9: a tall burst of stable releases and pull requests in the first week, then fewer, longer betas and a handful of pull requests a day.
Sixteen days of releases and pull requests. The burst is the first week; the long tail is on purpose.

The project, in one paragraph

Kestrel is a build-it-yourself Raspberry Pi device for studying Wi-Fi security. It listens to the networks around it, collects the data researchers use to test password strength, and uses a small machine-learning agent to decide where to listen next. It has a web dashboard for your phone and an optional e-ink "face". It's for authorized testing only, and that's built into the product: capture ships switched off, and nothing that sends data anywhere turns on without an explicit "I'm authorized" confirmation.

Three acts, not one sprint

Looking back, the three weeks fall into three acts, each slower than the last, each for a reason.

Three acts. Act one, build: 1.0.0 to 1.0.10 in five days, 11 stable releases, 19 betas, 87 pull requests. Act two, open up and harden: 1.0.11 to 1.0.13 in three days, 3 stable, 8 betas, 61 pull requests. Act three, prove it: 1.0.14, a week so far, 3 betas, 9 pull requests.
Each act slower than the last, and each for a reason.

Act one: build (five days). The dashboard, the map, the plugins, the e-ink screens, the battery add-ons, updates that install in place. This is the part the first post covered: an AI teammate made writing software cheap, so features arrived as fast as I could decide on them.

Act two: open up and harden (three days). Three releases that were less about features and more about trust:

  • Fair contests for the learning agent. Every new model, whether it was trained on the device or on a desktop, now has to beat the current one on data neither of them trained on before it takes over. What that changed.
  • An anonymous census of how many Kestrels are running, with no server and nothing that points back at a person. Why we built it, and how it works under the hood.
  • Honest status messages: "paused" instead of "error", "last reached 3h ago" instead of "unreachable". Why that matters.
  • A user guide that redraws its own screenshots from every release. How it works.
  • A mirror on GitHub, so people can open issues and send pull requests where they already are.
  • A hardening pass. Before adding more features, I went back over every place Kestrel signs you in, runs something with admin rights or talks to the internet, and made each one stricter.

That release, 1.0.13, also carried the project's first patch from outside: a community member found a bug, worked out the cause and emailed in the fix. It took four betas, each checked on a real device, before it went stable.

Act three: prove it (a week so far). Version 1.0.14 is three betas in. Its biggest change teaches the learning agent to keep exploring: a small share of the time, it tries something other than its favourite choice, so it can find out whether that choice is still the best one. A second patch from a community member arrived as a pull request on GitHub, fixing how some USB Wi-Fi adapters are detected. Neither of these can be judged in an afternoon. The agent's change needs days of running on a real device before its numbers mean anything, and an adapter fix needs the adapter.

Why slower is the right call

The first post argued that the speed was real and the quality could be too. Three weeks in, keeping the quality is what's costing the speed, which is exactly the trade I'd want any team of mine to make.

Hours from the first beta to the stable release, per version: under a day for every release through 1.0.12, 32 hours for 1.0.13, and more than 120 hours and counting for 1.0.14.
From first beta to stable: hours in week one, days now.

Three things changed.

  • The check grew with the product. Since day three, every beta has gone through a written checklist and a read-only check script that runs on the device and reports back. That script has grown from 882 lines at 1.0.10 to almost 2,600 now, and the latest beta's checklist has 167 items, each judged by the script or marked to be checked by hand. I wrote about letting an AI agent do that beta testing; the volume it now handles is the reason the cycle can stay this thorough.
  • The easy-to-verify work is done. A new dashboard card can be built, tested and checked in an evening. Whether a learning agent makes better choices over a week of real use can't. What's left on the roadmap is mostly the second kind.
  • Other people depend on it now. Kestrel has outside contributors, a public mirror and devices that update in place in the field. A bad release no longer costs only my evening; it costs someone else's. So the bar went up, and I'd rather ship one release a week that holds than five that don't.

Where the bottleneck moved

In week one, the bottleneck was execution: turning a decision into tested, documented code. That's the part an AI teammate collapsed. It hasn't come back.

Five steps from idea to stable release: decide, build, test, check on the device, watch it run. In week one the queue sat at build. Now it sits at the two ends: deciding what's worth shipping and safe, and evidence from a real device over real time.
Building stopped being the slow part. Deciding and proving became it.

What's left are the two things it doesn't remove:

  • Evidence from the real world. A device has to run, in a real place, for real hours, and some checks need a person holding a phone or plugging in an adapter. Time on hardware doesn't compress.
  • Decisions. What to build next, what to refuse, whether a change needs another beta, whether a contributor's patch is right as sent. Those were mine in week one and they're mine now. There are just more of them per release, because each release now touches more people.

I think this is the pattern teams will hit as AI takes over more of the building: the work moves to the ends of the pipeline. Deciding well at the front, proving it at the back. The middle is cheap now.

For engineering leaders

  • Measure cycle time, not throughput. Pull requests per day fell by more than 90% here, and the product got better. Time from first beta to a release you trust is the number that tells you whether quality is keeping up.
  • Let the checks grow with the product. The device check nearly tripled in ten days. If your verification is the same size it was at launch, it's checking the product you had, not the one you have.
  • Raise the bar where others depend on you. The moment outside people run your software or send you patches, the cost of a bad release changes. The process should change with it, on purpose, before an incident forces it.
  • Expect the bottleneck to move to judgment and evidence. Plan for senior time at the front of the pipeline and real-world testing at the back, not more hands in the middle.

For investors

An early burst of releases is the easy part of an AI-built product, and it's going to look the same everywhere. The more telling chart is week three.

  • Ask what happened after the sprint. A team whose pace dropped as it added verification, outside contributors and safe updates is building something durable. A team still shipping at sprint speed into real users' devices deserves harder questions.
  • The lasting asset is the process. Code is cheap to write now. Tests, checks that run on real hardware, a written rulebook and a community that sends patches are what make the next hundred releases safe, and they take time to build.
  • Judgment is still the scarce input. Every slowdown here was a decision. That's the skill you're really underwriting when you back a small, senior team with AI leverage.

What's next

Version 1.0.14 goes stable when the device says it's ready, not before. If you run a Kestrel, the betas are open to anyone who turns on beta updates, and the checklists are public on the project's forge.

If posts like this are useful to you, subscribe and the next one lands in your inbox.

When your team's pace drops, how do you tell maturity from trouble? 🪶

Kestrel is a personal, open-source project built on nights and weekends, unaffiliated with my employer, and for authorized testing only. The user guide and source are public.