Flint 編碼智能體如何在 24 GB RTX 4090 上運行，關鍵在上下文長度

Flint is a new coding-agent model from Recursive, trained from scratch and released under Apache 2.0. Its official compressed file weighs 22.9 GB, and Recursive recommends at least 48 GB of GPU memory for it. On a 24 GB RTX 4090, with an agent loop and four tools, it still fixed a planted bug in five turns. The deciding flag was the context length, meaning how many tokens (word pieces) the model can hold in view at once.

A 39.8B model that computes like a 2.5B one, and still loads whole

Flint has 39.8 billion parameters, the learned numbers that make up the model. About 2.5 billion work on any given token. It is a mixture of experts: a router picks 6 of 256 expert blocks per token, and one shared expert always joins in. Compute stays small, memory does not: idle experts still live on the card.

On the 4090 the loaded file takes 21.3 GiB of the roughly 22.5 GiB the card exposes. Everything else has to squeeze into the gap.

The trap: context length is a memory reservation

"Everything else" is mostly the KV cache. The model stores a key and a value per token in its attention layers, so earlier tokens are not recomputed at every step. That space is reserved at launch, sized to the context you ask for.

Only 14 of Flint's 49 layers attend to the full conversation. The others only look back 513 tokens (a sliding window). Each of the 14 keeps 4 KV heads (sets of keys and values). The memory arithmetic in the Flint deployment notes puts the cache at 28,672 bytes per token: roughly 0.9 GiB at 32K and 1.75 GiB at 64K.

On the 4090, 16K fit with the whole model on the GPU, at 22,316 MiB in use out of 23,028. At 32K the weights loaded, and then the cache reservation was refused:

allocating 87.50 MiB on device 0: cudaMalloc failed: out of memory
failed to allocate buffer for kv cache

That was 87.5 MiB, under a tenth of a GiB. The driver also holds memory outside the process, so the box overstates what you get. With 32 GB, an RTX 5090 ran 64K at about 23.3 GiB.

Recursive's example command passes the full 262,144-token context, the length their 48 GB figure is meant for. On 24 GB you pick a smaller one yourself. 16K covered our one-bug task, and a long session over a big repository will outgrow it.

What the agent loop did

The project: an invoice module that handled a discount percentage as a fraction, two tests that exposed it, and four tools (list files, read file, write file, run tests). Across seven runs on three machines its route never changed. First it listed the files and ran the tests to watch them fail. Then it read the module and its test file in one turn, wrote the fix, reran the suite and got three passing tests. It never touched the tests, and every tool call was valid JSON.

The 4090 wrote about 167 tokens per second, about 20% slower than the 5090's 210. The full loop finished in 5.7 and 6.3 seconds. Below 24 GB we only emulated cards, by capping memory and moving some experts to system RAM. Near 16 GB it answered at 4.6 tokens per second, too slow for interactive use.

Seven runs on one planted bug are a smoke test, not a benchmark. Recursive reports 74.2% on SWE-bench Verified (real bug reports from open source projects), a figure we did not reproduce. We also did not measure what compression costs in quality. Each configuration, down to an emulated 15 GB card, is in the Flint VRAM ladder with peak memory per context.

Two checks before you trust your numbers

Your llama.cpp build. llama.cpp is the engine that runs the file, and Flint's architecture was new to it. Support landed in PR #30928 by alexey-rsi, merged on 21 October, and release b11803 ships it. The chat template also relies on PR #30929, from the same author. A build from before Flint cannot load the model. If your tooling fetches llama.cpp by tag, as our worker (the program on the GPU machine that starts the engine) does, give it a tag such as b11803: a commit hash will not work there.

Your GPU. Our first RTX 5090 generated 27 tokens per second. Same file, same build, a different machine: 210. We could see a 400 W power cap and an older driver; the real cause is still open, and the slow card reported healthy. Run llama-bench (llama.cpp's speed test) on each new machine before you blame the model.

On the client side, this build puts Flint's reasoning inside the answer text, with no reasoning_content field, and those tokens count against max_tokens. Coding tools set up as Recursive suggests look for the separate field. Status: open.

Where to go from here

For short sessions, a 24 GB card at 16K or a 32 GB card at 64K is enough. For long ones, budget more memory.

Recursive built Flint and published its compressed files. alexey-rsi wrote the llama.cpp support. CISC and ggerganov approved PR #30928, and CISC and danbev approved PR #30929. ggerganov also wrote the per-expert scaling (#30861) that the Flint support uses. Our part was deploying it on InferenceKey, measuring it, and writing an SDK example that reruns the bug-fixing loop on your own NVIDIA worker. The example defaults to 64K, so on a 24 GB card change -c 65536 to -c 16384 in main.py.

The serve command, start-up timings and every open issue are in the Flint RTX 4090 deployment report on InferenceKey.