Presenting Seed – learning how language models learn

Lately, I’ve been fascinated by a question: what would it take to create my own language model?

I use AI tools, but I knew very little about what happens inside them. How does a model learn from text? What actually changes during training? And how does a collection of numbers eventually produce something that resembles a conversation?

I wanted to explore those questions by building something I could experiment with on my own computer.

Instead of downloading a pretrained model, I wanted to start with random weights: a model that hadn’t learned any language patterns yet.

In my head, I pictured a baby gradually becoming a toddler. That image gave the project its direction, although the actual process is mathematical: the model learns patterns from text by repeatedly trying to predict what comes next.

My first ambition was modest. If I said, “Hello, how are you?”, could I teach it to respond, “Hello, I’m fine, thanks”?

It sounded simple enough. I was about to discover how much work sits behind even a small answer.

Using AI to build its own classroom

I wouldn’t have known where to begin writing a training engine myself, so I opened Codex CLI and asked it to help build an application.

I wanted somewhere to import training text, adjust a few settings, and press Start training. I also wanted to watch the training progress, experiment with text generation, and eventually open a chat page to talk to whatever I had managed to teach.

The result is called Seed.

Seed is a local language-model training studio with a React frontend and a Node.js backend. Its small Transformer starts from random weights. The training engine is implemented directly in TypeScript, including attention, gradient calculations, and the process that updates the model’s weights.

The default model has roughly 137,000 parameters – the adjustable numbers that training changes. Initially, the model is just small random numbers in each weight.

That makes the first conversation quite different from opening a familiar AI assistant. Seed has to learn its language patterns from the training material I provide.

The default model has roughly 137,000 parameters—the adjustable numbers that training changes. It’s tiny compared with the models we usually call LLMs, but that scale makes it possible for me to experiment locally.

Using AI to help build a tool for training AI felt a little strange at first. It also gave me a practical starting point. I could turn questions into experiments, inspect the implementation, and gradually learn what the pieces were doing.

What does “learning” mean here?

One thing I’ve had to get used to is how much work is hidden inside the phrase “train on some text.”

Seed breaks ordinary text into bytes. During training, it looks at a sequence and tries to predict the next token. Its prediction is compared with the actual next token, and the training process adjusts the weights to make that prediction less wrong.

Then it repeats.

A question-and-answer example such as:

User: Hello, how are you?
Assistant: Hello, I'm fine, thanks.

becomes material for that prediction task. With enough suitable examples and training, the model may learn to produce the expected continuation.

But reproducing an answer it has practised and answering a new question are different achievements.

That distinction gives me something useful to test. Can it handle a slightly different greeting? Does a familiar-looking response still make sense? Has it learned a pattern that carries over, or mainly memorized an example?

Seed lets me watch training and validation loss, which measure prediction error on the training text and a held-out portion. Those numbers help track progress, but I still need to read the output and try unfamiliar prompts.

A lower loss doesn’t tell me whether I’ve built a useful conversational partner.

My artificial toddler needed five and a half days

The first reality check came when I started training on text about basic toddler behaviour.

On my MacBook Pro M4 Max, the estimated training time was 5.5 days.

I had imagined watching a little model learn. Apparently, I also needed to develop some patience.

That estimate led to several rounds of backend optimization. We improved the numerical loops and added native matrix operations through Apple’s Accelerate framework, using BLAS routines for the heavy calculations.

For the same planned training run, the estimate dropped to around four hours.

Both figures were estimates, rather than completed benchmark timings. Even so, the difference changed how practical the experiment felt. A run that could take most of a week became something I could try within a day.

It was still running on the CPU. Making the calculations more efficient had made a substantial difference.

I had started with questions about language and ended up learning why matrix multiplication performance matters.

The first words were… something

With very little training, one early response was:

“I le ay pooo it.”

I can’t claim that as a successful conversation, but it deserves a place in the project’s history. Amusingly, the first response was much the same as one would expect from a toddler.

There’s something entertaining about seeing output like that when you know the model started with random weights. At the same time, it’s a useful reminder of how much we take for granted when using a pretrained assistant.

Recognisable language, sensible answers, and reliable conversation all require learning that my tiny model hasn’t done yet.

I’m still curious about the toddler experiment. Will it pick up the style of the training text? Will it protest when I ask it to wash its hands before eating? Will a convincing answer survive a follow-up question?

For now, those are questions I’m testing, rather than results I can promise.

New Life: teaching through conversation

The project has also grown a second experiment called New Life.

The original training workflow starts with a collection of text. New Life lets me begin with a separate model and teach it through short interactions.

I send a message. Seed learns from that message and generates a reply, which will probably be gibberish early on. Then I can supply the answer I wanted it to give.

For example, I can ask:

What color is the sky?

After its reply, I can teach:

!The sky is blue.

The ! tells the interface that I’m supplying a preferred answer to my latest question. It also recognises corrections such as “No, the sky is blue.”

That question-and-answer pair becomes a training lesson. Seed performs learning updates using the lesson alongside remembered examples, then saves its progress locally. It is continuously learning and saving new weights.

I can ask again, teach again, and see how the output changes.

This is an important difference between Seed’s two chat experiences: ordinary Chat generates responses from the current trained model; New Life updates its model as I interact with it.

It also makes the original baby-to-toddler idea feel more tangible. I’m supplying small examples and watching what repeated teaching produces.

There are limits to the metaphor, of course. A few corrections won’t create a fluent assistant. The model can memorise examples, struggle with variations, and forget things it previously learned. Replaying earlier lessons helps, but it doesn’t guarantee that every lesson will stick.

Those limitations are part of what I want to explore.

What I’m hoping to learn

Seed gives me a place to investigate questions that used to feel abstract.

What changes when I repeat one example many times? What happens when I provide several different ways of saying the same thing? How much does the training material influence the model’s tone? When does more training help, and when does it mainly make the model better at reproducing familiar text?

I also want to become better at judging the results. A funny answer is easy to share. A plausible answer is easy to overinterpret. Testing several unfamiliar questions takes more effort, but tells me more.

I’m still early in this process. Building the application has given me a way to learn through experiments, mistakes, and the occasional deeply questionable sentence.

And yes, I’m still hoping to get a sensible answer to “Hello, how are you?”

The source code is available on GitHub (https://github.com/artur-gajewski/seed). If you know more about language models than I do, I’d welcome feedback on the implementation, the training approach, or useful experiments to try next. Feel free to message me on LinkedIn. I also welcome any contribution to the project.

You’re also welcome to try it out and grow a small model of your own. I’d be interested to hear what its first words turn out to be.