AI Governance & Making LLMs Predictable in Production
If it can’t fit anywhere else, it goes here.
I shipped an AI test harness the other day that lets Claude run my Unity game from the Editor and execute QA workflows end-to-end. The push for this came from save migration pain. By the third migration in a week I had walked New Game, Save, Quit, Load, Inspect by hand forty times and was visibly losing hair to it.
The same harness drove the save round-trip regression that just signed off 0.2.21.<br> More
Most LLM-powered systems hit a token limit and do the dumbest possible thing: they chop from the front. Take the first N characters, summarize it, discard the rest. Sometimes the logic is slightly smarter: random sample, drop from the middle, oldest-first. But the result is the same. You might silently lose the fact that the player killed the town’s mayor three sessions ago, while keeping five lines of small talk about the weather.<br> More
I’ve been working on AI-powered JIT compilation recently, which in turn led me to test several “autonomous” AI coding agents for research. The pattern across the board is strikingly consistent: impressive demos masking a disappointing reality.
One agent took 9 hours to fix a simple configuration bug, blindly claiming “Fixed!” multiple times without ever actually validating its work. Another wasted 11 commits rewriting documentation when I had explicitly asked for a bug fix.<br> More
Wednesday morning: I had game rules scattered across eight design documents, each making conflicting assumptions about how fights resolve in LAIRD 2. No single source of truth. No clear way to sync them. So I designed a tool I called “claude-interrogate“.
On Wednesday afternoon: I had built a full production MCP server that reads design docs, interviews me Socratically, and generates formatted specs in my voice.<br> More
