以 6 GB RTX 3050 Laptop 微調 32B 模型，並檢視三項測量誤差

I have one GPU: an RTX 3050 Laptop with 6 GB of VRAM, 34 GB of RAM and an NVMe SSD.
It runs Windows. For the last months I've been building a small engine that trains models
that don't fit, and this post is the honest result for the 32B case — including the
places where I fooled myself and had to publish the correction.

What works

A 32B model (NF4) fine-tunes at 1024 tokens/step on this card.

Why is that interesting? The 32B weights are 19.2 GB. They never fully live in VRAM: they
are mmap'd from disk and streamed block by block, one transformer block resident at a
time, with recompute in the backward. The I/O is paid once per step, not per token —
so longer sequences amortise it. At 8× the context (128 → 1024) throughput goes up 3.1×.

A 14B fits in RAM, so it is much cheaper: 2k/4k steps at ~4–5 s. Dolly-15k SFT for 50
steps drops the eval loss from 1.8755 to 0.9661 (32B) and 2.3747 to 0.9407 (14B).

The full recipe and one-command wrapper:
github.com/byjameson/lcga-local —
tools/finetune.py, LONGCTX_FINDINGS.md, GUIDE_6GB_32B.md.

The three measurements that nearly fooled me

This section is the real point of the post. Every one of these produced a plausible
number that was wrong, and all three were caught only because I went back and checked.

1. I measured a model I never loaded

I published a result saying "the learned block router selects 0.0% content blocks and
is identical to a static window". Then I noticed the evaluation script loaded the
checkpoint before patching the model with the router modules — so nothing was loaded
and I had been benchmarking a randomly initialised router for days.

Fixed, the same model selects 1.7–7.0% content blocks and is 0.005–0.13 PPL worse than
the window. The headline survived in spirit ("learning buys nothing here") but the
reported mechanism was fabricated by a bug. I kept the invalid runs in
results/invalid/ with a note, and rewrote the report.

2. "8.6× faster" was measuring a truncation cap

I reported "32B at 4k context: 382 tok/s, 8k: 616 tok/s". Wrong. The SFT path feeds one
dataset row per step, and --seq there is only a truncation cap. Dolly-15k rows are
43–1964 tokens, median 134. So every "seq 1k–8k" step processed ~134 tokens, and I was
dividing the cap by the step time.

The real numbers, from the LM path where sequences are exactly seq long: 12.0 → 28.5 →
36.7 tok/s at 128 → 512 → 1024, then the memory wall. 3.1×, not 8.6×. The symptom was
visible in hindsight: step time and VRAM were almost flat across seq.

3. The baseline was not budget-matched

For a while my block-sparse attention looked 5 PPL better than "local" attention. It was —
because "local" was allowed 2 blocks and the learned router 4–16. Against a budget-matched
window the learned router adds nothing. This is a widespread way to overstate sparse
attention, and I had walked straight into it.

The other trap: the autotune

The engine grows its resident block window until it hits a VRAM target measured on the
previous step. At seq 2048 it grew to 11 blocks, the next step spilled into Windows
shared memory, and a single step did not finish in one hour. Forcing
--blocks-on-gpu 1..2 --vram-hard-gb 5.3 turns the same point into 3.9 GB and 193 tok/s.
Long context wants fewer resident blocks, not more.

What did not work (and why it's in the repo anyway)

• Long-context training did not improve long-context quality at this budget. With 122,880 tokens, training at seq 512 gave a 2k-context CE of 1.2833; training at seq 2048 gave 1.3033. The lever is data volume, not window length — at least at this scale.
• 2k context does not fit. It is ~0.6 GB away, and the missing piece is the bf16 lm_head (1.56 GB). Its 4-bit variant is broken in the bitsandbytes version here.
• A learned block router trained on plain language modelling degenerates into "sink + the recent window". That is not a bug: on WikiText the dense teacher itself puts almost no mass on distant blocks, so there is nothing to learn.
• On retrieval data it does learn — distilling the dense teacher's content argmax at the answer positions, with no retrieval labels, gives exact-match parity with dense at a 25% block budget (and 6.25% at 8k) where a static window scores zero. It replicates on 0.5B and 3B. But it is a specialist: plain-LM perplexity degrades, and a mixed schedule (1 retrieval step per 7 LM steps) costs +0.6 PPL to keep both.

The takeaway

A 6 GB laptop GPU can fine-tune a 32B model at 1k context, at ~37 tokens/s, in 4.6 GB of
VRAM — if you stream the weights and keep the resident window small. It cannot do 2k yet,
and longer context is not a quality lever by itself.

And the meta-lesson, which took more time than the engineering: check that your
evaluation actually loaded the thing you think it loaded, and that your baseline has the
same budget. Both mistakes look like results.

Everything — code, raw JSON, the invalid runs, the corrections — is in
byjameson/lcga-local.