Venture 04

Townsfolk

AI companions with bodies in Minecraft and minds on language models.

Partial success

▶ Watch the Sky Trials: six models, one riddle (video)

The idea

I wanted a companion in Minecraft who goes about with you like a best friend and thinks for themselves: not a scripted character, but someone who hears what's said, notices what happens, decides what to do, and does it with a body in the world.

The idea from the start was a split. The body knows how to live in the world without being told: walk, open doors, gather, craft, smelt, farm, fight back, eat. The mind is a language model, consulted only when there is something to decide. And what they become is their own: when you sleep, they take stock and rewrite, in their own words, who they are turning out to be and what they know about you.

The solution

Townsfolk is a NeoForge mod for Minecraft 1.21.1 and, since Build 8, a Python server beside it. Two companions, the sisters Robin and Storm, are the heart of it. The project has moved in builds, each one planned, built in parallel by builder agents, reviewed and merged.

The builds so far

  1. Part 1 · Builds 1 to 6 · before 12 November

    A companion and a village

    These builds come before the project's git history, which begins with a backup on 12 November, so what they did is known from that snapshot. The mod had a village of independent characters from earlier work, and the companion who became the focus: a real player entity with a body that walks, rides, gathers, crafts, smelts, farms, fights back and never loses what it carries.

    Its mind was a model consulted only when there was something to decide, and it grew: notes, a journal, dreams, and taking stock while you sleep. Round 5 added a keeper who watches over them, named places, and wishes.

  2. Part 2 · Build 7 · 13 to 15 November

    Built in packages

    The first build in git, and the first built this way: separate packages given to builder agents working side by side, each reviewed, put right and merged. Blueprints and a dry run before building, seeing, named notes, answer blocks and tools, and "show me". Round 7.6 added Blackbird, a director, and gave each voice the right tone.

    All of the thinking still lived inside the mod, in Java: 100,203 lines, of which 42,493 were the body, about 18,000 were thinking, 10,000 tangled the two together, and 26,000 were checks.

  3. Part 3 · Build 8 · 15 November

    Thinking moves to a server

    The mod was cut down to a body: moving, acting, sensing, running scripts, playing sound. Everything that thinks moved to a new server in Python, behind one contract listing every message between them. The game never waits on the server inside a tick, and a dead server is obvious: the companions stand idle and the game says so.

    The thinking came in three levels, named after the psychology of fast and slow thinking: System 1 decides who answers, System 2 is a whole thought, and System 3 does slow work off the answering path. But one System 2 thought still carried everything.

  4. Part 4 · Builds 9 and 9.5 · 15 and 16 November

    The brain remodelled, and the evals →

    The one thought was split into a mind and hands, and the telling of the world moved into code. Then, in Build 9.5, it was tested in several ways: live play on a test rig, a scripted half hour, replays, code critics, a rival brain built by Fable 5.2 (Anthropic's frontier model), and six makers' models on the same trial.

Read onBuilds 9 and 9.5: The brain remodelledOne overloaded thought split into a mind and hands, then tested in several ways, including six makers' models on the same trial.

The outcome

Partial success.

The companions live in the world, talk, build, keep memories and grow, and since Builds 9 and 9.5 every part of their thinking can be measured, and the model behind it swapped. Six makers' models have each played a companion on the same trial.

It's a partial success because each round of testing has found faults that the last one missed. The latest, knowledge leaking between companions who should be kept apart, is not fixed yet.

519commits in five days
4builds in git
236commits on the busiest day
6makers' models tried
010020030012 Nov, Backup: 1 commits112 Nov13 Nov, Build 7: 17 commits1713 Nov14 Nov, Build 7: 54 commits5414 Nov15 Nov, Build 7: 21 commits15 Nov, Build 8: 64 commits15 Nov, Build 9: 126 commits21115 Nov16 Nov, Build 9: 134 commits16 Nov, Build 9.5: 102 commits23616 NovBuild 7Build 8Build 9Build 9.5
010020030012 Nov, Backup: 1 commits112 Nov13 Nov, Build 7: 17 commits1713 Nov14 Nov, Build 7: 54 commits5414 Nov15 Nov, Build 7: 21 commits15 Nov, Build 8: 64 commits15 Nov, Build 9: 126 commits21115 Nov16 Nov, Build 9: 134 commits16 Nov, Build 9.5: 102 commits23616 NovBuild 7Build 8Build 9Build 9.5
519 commits in five days, coloured by the build they belong to. Build 9 alone is 260 of them, about half made overnight while the companions were tested and fixed.Source: git log --all of the Townsfolk repository, each commit placed in its build by the tags build8-start, build8-end and build9-end and the branch build9.5; the round 7 snapshots kept off every branch counted in Build 7; Melbourne dates.

The learnings

What went right

  • Splitting body from mind, and later the mind into parts, is what made it testable. Once thinking lived on a server behind one contract, I could play the same half hour again, replay real thoughts at another model, or give each companion a different maker's model.
  • Building in packages, with parallel builders and a review for each, let a whole build land in a day.

What went wrong

  • Speed outran checking. After Build 9's first evening of play, 40% of the things a sister tried to do didn't do what she meant, and two thirds of those failures were told to her as successes. Checks have to look at the world, not at the model's account of it.
  • Every new kind of test found a fault the others couldn't: a live play-test rig found a crash in its first ten seconds that six code critics had missed, and the Sky Trials found knowledge leaking between companions.