Venture 04
Townsfolk
AI companions with bodies in Minecraft and minds on language models.
Partial success▶ Watch the Sky Trials: six models, one riddle (video)
The idea
I wanted a companion in Minecraft who goes about with you like a best friend and thinks for themselves: not a scripted character, but someone who hears what's said, notices what happens, decides what to do, and does it with a body in the world.
The idea from the start was a split. The body knows how to live in the world without being told: walk, open doors, gather, craft, smelt, farm, fight back, eat. The mind is a language model, consulted only when there is something to decide. And what they become is their own: when you sleep, they take stock and rewrite, in their own words, who they are turning out to be and what they know about you.
The solution
Townsfolk is a NeoForge mod for Minecraft 1.21.1 and, since Build 8, a Python server beside it. Two companions, the sisters Robin and Storm, are the heart of it. The project has moved in builds, each one planned, built in parallel by builder agents, reviewed and merged.
The builds so far
- Part 1 · Builds 1 to 6 · before 12 November
A companion and a village
These builds come before the project's git history, which begins with a backup on 12 November, so what they did is known from that snapshot. The mod had a village of independent characters from earlier work, and the companion who became the focus: a real player entity with a body that walks, rides, gathers, crafts, smelts, farms, fights back and never loses what it carries.
Its mind was a model consulted only when there was something to decide, and it grew: notes, a journal, dreams, and taking stock while you sleep. Round 5 added a keeper who watches over them, named places, and wishes.
- Part 2 · Build 7 · 13 to 15 November
Built in packages
The first build in git, and the first built this way: separate packages given to builder agents working side by side, each reviewed, put right and merged. Blueprints and a dry run before building, seeing, named notes, answer blocks and tools, and "show me". Round 7.6 added Blackbird, a director, and gave each voice the right tone.
All of the thinking still lived inside the mod, in Java: 100,203 lines, of which 42,493 were the body, about 18,000 were thinking, 10,000 tangled the two together, and 26,000 were checks.
- Part 3 · Build 8 · 15 November
Thinking moves to a server
The mod was cut down to a body: moving, acting, sensing, running scripts, playing sound. Everything that thinks moved to a new server in Python, behind one contract listing every message between them. The game never waits on the server inside a tick, and a dead server is obvious: the companions stand idle and the game says so.
The thinking came in three levels, named after the psychology of fast and slow thinking: System 1 decides who answers, System 2 is a whole thought, and System 3 does slow work off the answering path. But one System 2 thought still carried everything.
- Part 4 · Builds 9 and 9.5 · 15 and 16 November
The brain remodelled, and the evals →
The one thought was split into a mind and hands, and the telling of the world moved into code. Then, in Build 9.5, it was tested in several ways: live play on a test rig, a scripted half hour, replays, code critics, a rival brain built by Fable 5.2 (Anthropic's frontier model), and six makers' models on the same trial.
The outcome
Partial success.
The companions live in the world, talk, build, keep memories and grow, and since Builds 9 and 9.5 every part of their thinking can be measured, and the model behind it swapped. Six makers' models have each played a companion on the same trial.
It's a partial success because each round of testing has found faults that the last one missed. The latest, knowledge leaking between companions who should be kept apart, is not fixed yet.
The learnings
What went right
- Splitting body from mind, and later the mind into parts, is what made it testable. Once thinking lived on a server behind one contract, I could play the same half hour again, replay real thoughts at another model, or give each companion a different maker's model.
- Building in packages, with parallel builders and a review for each, let a whole build land in a day.
What went wrong
- Speed outran checking. After Build 9's first evening of play, 40% of the things a sister tried to do didn't do what she meant, and two thirds of those failures were told to her as successes. Checks have to look at the world, not at the model's account of it.
- Every new kind of test found a fault the others couldn't: a live play-test rig found a crash in its first ten seconds that six code critics had missed, and the Sky Trials found knowledge leaking between companions.