16 歲作者花約一年打造 780B 參數的 Ganymede，分享訓練心得

Hi, I'm Afonso. I'm 16, and for about the past year I've been building Ganymede, my own large language model, mostly on my own under a small lab I started called Merkium.

Most write-ups about training LLMs come from big labs with huge teams. I wanted to share what it actually looked like from the outside, including the parts I got wrong.

1. Sparse beats dense when compute is the limit

Ganymede 1.1 is a mixture-of-experts (MoE) model. It has 780B parameters in total, but only about 20B are active for any given token. A router picks which experts handle each token, so you get the capacity of a very large model while paying roughly the compute cost of a much smaller one.

For someone without big-lab budgets, this was the only way training and serving a model this size was realistic.

2. Data cleaning takes longer than training

Ganymede was pretrained on around 5T tokens of public web text and open-source code. Before any of that went into training, I ran everything through a filter to strip out API keys and personal information.

I expected this to be a quick step. It wasn't. Getting the filter to catch real secrets without throwing away useful code took far more iteration than I planned for.

3. Don't reinvent everything

Ganymede is not a fine-tune of another model. But the architecture follows published open-weight MoE designs, and a few components were adapted from existing open models.

I think being upfront about that matters. Building on good published work saved me months, and there's no reason to pretend otherwise.

4. Safety has to be built in, not added later

In version 1.1 I added an instruction hierarchy in the system prompt, so the model knows which instructions take priority, plus detection for jailbreak attempts and attempts to override its identity.

It's much easier to design for this from the start than to patch it on after people find holes.

5. Compute doesn't have to live in one place

Training ran on 450+ H200 GPUs spread across data centers in France, Portugal and the Netherlands. Coordinating across locations added complexity, but it made the compute I needed actually available.

Where it is now

Ganymede 1.1 is a general reasoning model with a 1M token context window. Reasoning is always on, and it supports streaming and tool calling.

Right now it's free to use, both in chat and through an OpenAI-compatible API, so it works with most existing tools. The rate limit is 30 requests a minute per key.

I'm sure it still has rough edges, and honest feedback is the most useful thing I can get. If you try it, I'd love to hear where it falls short.

Chat: https://chat.merkium.nl
Docs: https://www.merkium.nl/docs

Happy to answer questions about any part of the process in the comments.