租用 GPU 微調的成本教訓:填充運算如何推高費用
Your GPU is busy computing padding: cost lessons from fine-tuning on rented cards
作者分享租用雲端 GPU 微調克里米亞韃靼語模型時的成本教訓,指出填充運算及周邊流程故障會增加訓練費用和耗時。短對話填充至 seq_len = 1024 時,有用 token 比例約為 5%;採用 packing 後升至 99%,第二階段成本由舊方法估算的 $331 至 $413 降至 $14 至 $17。文章亦提出按實際處理量估算成本、按工作時長設定檢查點,以及在本機完整演練訓練流程等做法。
In the second post of this series I promised to write up how I rented cloud GPUs and burned money on them without doing the maths. In the third I said the promise still stood. Here it is.
Some context first. I'm building speech and language models for Crimean Tatar, a low-resource language (why, and where the data came from). ASR and TTS train fine on my 24 GB card at home. Full fine-tuning of the language model needs 141 GB or more, so that part runs on rented machines, which works like renting a car: you pay for the hours you have it, whether you're driving or standing on the hard shoulder with the bonnet up.
Over about a month and a half that came to a bit over $300. It wasn't one disaster but a series of iterations with different configurations until the recipe worked. Every mistake below is normal for real work, and I'm writing them down so they get cheaper for the next person, and for me.
I don't name base models, fine-tuning techniques, checkpoints or data sources. GPU classes, vendors and hourly prices are in, because a post about cost without them is useless.
The first run, in numbers
The first rental was an H200 with 141 GB at about $4.60/h, for a two-stage run that the estimate put at 9 to 11 hours and $33 to $40. I stopped it after 11.5 hours and $53, at 15% of stage 2, with about 46 hours still to go. Effective GPU yield over that time was roughly 3 to 4%. There were three independent causes, and any one of them would have blown the budget on its own.
Lesson 1: estimate cost on padded tokens, not on content
The biggest cause was padding. Our SFT conversations are short, median around 20 tokens, and every example was padded to seq_len = 1024. The attention mask and the loss mask make padding harmless for the result, but not for the bill: the matmuls still run over the full sequence, and most of what they compute is thrown away. It's like shipping one apple in a whole crate and paying for the crate. The estimate was based on content, 59M tokens, while the GPU processed 1.17B positions. That's the 20x miss.
What hurts is that the plan literally contained an assumption of about 1.2x padding overhead, and nobody tested it. Testing it takes a minute:
# what share of the compute you pay for is real tokens?
lengths = [len(tok(x).input_ids) for x in sample] # a few thousand rows
useful = sum(lengths)
padded = len(lengths) * seq_len
print(f"useful share: {useful / padded:.1%}") # ours: about 5%
The number that drives cost is steps × micro_batch × seq_len positions processed, divided by real throughput and multiplied by the hourly price, not "corpus size × epochs". If your median example is fifty times shorter than seq_len, no kernel optimisation will save you. The card is multiplying zeros and you're paying by the hour.
The fix was packing. We started packing short conversations into dense blocks, one after another, with a block-diagonal attention mask so packed neighbours can't see each other. There's one thing to watch here: some fast attention paths silently ignore a custom 4D mask, and then your packed examples do leak into each other. We checked that each conversation's logits inside a packed block match its logits computed alone. With that in place the useful share went from 5% to 99%, and stage 2, which the old way would have cost $331 to $413, came out at $14 to $17.
The left bar is an estimate, to be clear. We didn't finish the run the old way to find out.
Lesson 2: checkpoint by wall-clock, and not to a network disk
The second cause was checkpointing. We saved a full 46 GB checkpoint every 400 steps to a network volume, and training blocks while it writes, so the card simply stands and waits for the disk. About 600 GB got written over the run, and the GPU sat at 58 to 68% busy even before you count padding.
You do need checkpoints, because a crash without them sends you back to zero. But the frequency should be set in hours of work, weighing the cost of an hour against the chance of losing one, not in a step count picked by habit. A 46 GB checkpoint makes sense every few thousand steps, not every 400, and it should go to local disk where you can, with the copy to persistent storage happening later.
Lesson 3: feed the card
The third cause was the simplest. Two data loader workers were preparing batches on a 128-core machine, and with short examples the GPU finished a batch faster than the loader could prepare the next one. A fast card with a slow kitchen. Our rule of thumb since then is workers around a quarter of the vCPUs, and a look at GPU utilisation in the first ten minutes. If it isn't near full, something is wrong, and stopping costs less than continuing.
Lesson 4: "it works" and "it pays for itself" are different tests
The frustrating part is that before renting, the whole pipeline had passed a smoke test at home. That test proved the code was correct. It said nothing about throughput, because at home time is free and nobody looks at it. We ran the correctness test and skipped the economics test. There were also about 45 minutes and $3 lost to setup traps before the first training step, which is small, but it all counts.
Not everything was lost. Stage 1 did finish, and its weights were saved.
Lesson 5: rehearse the whole chain, not the pieces
With packing in place I started attempt two, and over September 1 and 2 we went through five machines in a row. Each one died before finishing.
One of them crashed at step 50 of 95, because a fresh change enabled an evaluation that needed about 160 GB on a 141 GB card. In run 1 that evaluation never had time to fire, so nobody had seen the problem. Another was killed by our own watchdog at step 80 of 95, four minutes before the end of the stage: it found a stale "run failed" marker left on the volume by the previous attempt and believed it. The fifth reached 65% of stage 2, ran 4.35 hours for $19.96, and crashed, and I'll come back to that one below.
Looking at those deaths together, I saw the pattern. The training loop itself never crashed. Not once. What crashed was everything around it: launch scripts, checks, watchdogs. Each piece had been tested on its own, but the chain as a whole had never been run end to end.
So now every rental is preceded by a full rehearsal at home, with the same chain, the same watchdogs and the same stage handover, but with a tiny model and for free. One rehearsal took 4.7 hours and caught 6 defects. The nastiest was in gradient accumulation: because of how a model wrapper handled it, the effective learning rate in stage 1 was 16x the configured one, and gradient clipping hid it, so the loss looked fine. In the cloud we would have paid for each of those six.
The stale marker also pointed at a whole class of bugs: stale state on persistent volumes. In a side benchmark, a script saw an old result file on disk, decided it had already downloaded the new one, and destroyed the machine along with the real result. Since then anything that signals run state is tied to a specific run ID.
Lesson 6: don't kill the machine, fix it in place
The most expensive thing in those two days wasn't a bug at all. It was a habit. On every error we did the same thing: destroy the machine, fix the code, start again on a fresh one. A fresh machine reinstalls the environment and re-downloads tens of gigabytes of weights while the meter runs. And when the machine goes, its logs go with it, so the root cause becomes a guess, and sometimes you pay for another machine just to reproduce it.
We wrote the rule down the same day: don't destroy, fix in place and restart the process on the same machine. Think of a flat tyre. You patch it at the roadside; you don't abandon the bike. Unfortunately the rule reached the code an hour too late. When the fifth machine crashed, the rescue script's first action was to lift the pre-shutdown pause, and the copy tool it relied on wasn't installed on the image. The machine was gone within minutes, logs included, and I still don't know why it crashed.
Once the rule was in, it paid off quickly. The next training ran both stages in one go for about $41 for the whole job, compared with $53 for 15% of stage 2 the first time. The run after that was the first real test of the rule: a short on-machine test found 3 defects for $1.50, we fixed them in place, and the whole job took 2.81 hours and $12.91 with zero crashes.
Lesson 7: your watchdog fails in both directions
The watchdog is a separate program that watches training, rescues results and shuts the machine down. Almost every mistake it made cost money, and it erred both ways.
On September 6 it was too quiet. A run on a B300 finished and the watchdog didn't notice, because it had broken silently twice that day. The machine sat with the finished result for 30 to 40 minutes at $7.89/h until I checked by hand. Since then we fire-drill the watchdog before every launch, triggering an event on purpose and confirming that the alert arrives, and I check the machine by hand near the end of the run. The same episode taught us that "run complete" doesn't mean "result safe". The result is safe once it's downloaded and verified.
On September 24 it was too strict. The watchdog sampled GPU utilisation every 5 minutes, and its last readings were 94, 0, 100, 0. The zeros landed on checkpoint saves, when the card is legitimately waiting on the disk. The average came out at 48% against a 50% threshold, so 28 seconds before the end of training it saved a checkpoint and destroyed the machine. The weights survived, all 51.6 GB, but the checkpoint's metadata files were zero bytes and we had to bring up a separate machine to rescue them. The flaw was in the logic itself: GPU utilisation alone can't tell a hang from a save, but the step counter can. The check now looks roughly like this:
# a hang is an idle card AND no progress, not just an idle card
hung = gpu_util_avg < 0.5 and step_now == step_at_last_check
A card is allowed to rest while it saves or finishes. A real hang is when the steps stop.
That left the question of where the watchdog lives. One that runs on the rented machine dies with that machine, which is exactly when you need it. Ours now runs at home and talks to the machine from outside.
Lesson 8: a hard dollar cap can kill a healthy run
Money had the same problem. The first rule was simple: hit the cap, rescue, destroy. On September 2 the watchdog, with a $36 cap, extrapolated from step 10 of 1,875 that the run would cost $35 to $38 and started rescuing a perfectly healthy job. We stopped it in time, but the lesson was clear. At step 10 an extrapolation knows almost nothing about real throughput, and a hard cap that fires later can kill a run that's almost done, taking everything you've already paid for with it. Now, near the limit, the system saves everything first and only then shuts down.
Lesson 9: a price list is not availability
A simpler mistake was trusting the price list. This is what we found once we started checking what was behind the prices:
| vendor | listed | what we found |
|---|---|---|
| DigitalOcean | MI300X 192 GB, $2.59/h | not available in any region; new accounts start with a GPU quota of zero, raised via support; the available alternative was $3.80/h |
| RunPod | MI300X, $2.39/h | 12 create attempts out of 12 refused |
| Hostinger | 192 GB card, $4.50/h | paired with 16 GB RAM and 200 GB disk; we need about 315 GB for checkpoints alone |
| AMD Developer Cloud | MI300X, $1.99/h | available; the later versions trained there |
No complaint at anyone here: a price list shows price, not stock or configuration. The first card didn't exist in any region, and the third came with less disk than our checkpoints alone take. Checking all of this cost under a cent, and since then we check that a card can actually be obtained before trusting a price.
Lesson 10: bill by machine lifetime, then check against the first minutes
The 20x miss wasn't our only accounting error. For the second version our report said $32, because it counted training hours only, while the balance said $41. The machine lived almost 9 hours, not 7, and paid for disks the whole time, so the listed $3.59/h for the card came out at about $4.60/h for the machine. You're billed for the machine's whole life, from creation to deletion, including setup, weight downloads and the final upload, not for the time the loss is going down.
So the estimate now starts from the real step count on the real packed block length, and the ETA gets re-checked against the first minutes of actual steps, because an estimate that has never been compared with real steps is a guess. Cost is counted from machine creation to deletion, disks included. For scale, version 3 on a B300 cost $79.31 for 123M tokens and version 4 on an MI300X about $41 for roughly 200M, and a training cycle now costs $10 to $12. On October 5 four launches failed at startup, at about fifty cents each: caught early, a failure costs pocket change instead of a run.
Lesson 11: measurement bugs are cheaper and more dangerous
The last group cost almost no money, but these bugs are sneakier. A rental mistake at least shows up on the bill; a measurement mistake produces a confident number that is simply wrong. Some appeared earlier in the series (1, 2, 3, 4), and here are the new ones.
Once the recogniser scored a WER near 100% on a book, and it looked like the data was garbage. The cause was the script: the recogniser output Latin and the book text was Cyrillic, so practically every word counted as an error. Before scoring, convert both sides to one script, and do it with the official transliterator, not by eye.
Another time two TTS samples out of two showed a pitch spike, and I started reworking data because of it. Out of 125 takes there were zero spikes. Generation is stochastic, and n = 2 is not a measurement. A related trap is repeating the wrong thing. A proper-name defect didn't show on 60 names in one sentence × 5 takes, although it appeared in about half of real sentences. With 4 names in 8 different sentences it showed at once, p = 0.0013. You have to vary the factor the defect depends on, not the one that's easiest to iterate.
The tooling made one of these mistakes for us. "Keep the last N checkpoints" deleted the best one, and two runs lost their best moment, so the best checkpoint by validation now lives outside the rotation.
The worst one was train/test overlap. 68% of the old dev text also appeared in training, so some version-3 numbers were inflated. That's an exam built from the homework problems: the grade says more about memory than about skill. We built a clean dev set from works that appear nowhere else in the data.
The checklist
Put together, the failure chart tells the story: at first nearly every run broke somewhere, and now failures are rare and cheap.
This is what I'd run through before renting a GPU now:
- [ ] Measure the useful token share on a real sample. If it's far below 90%, pack.
- [ ] Verify packing doesn't leak across examples, on the attention path you'll actually use.
- [ ] Estimate from
steps × batch × seq_len, then compare to the first minutes of real steps. - [ ] Checkpoint frequency in hours, to local disk; keep the best checkpoint outside rotation.
- [ ] Data loader workers sized to the CPU count; GPU utilisation checked in the first ten minutes.
- [ ] Full rehearsal of the whole chain at home with a tiny model.
- [ ] No stale markers or result files on persistent volumes; state tied to a run ID.
- [ ] Watchdog off the rented machine, fire-drilled before launch, hang = no steps, not low utilisation.
- [ ] Near the limit: save first, shut down second.
- [ ] On error: fix in place, keep the machine and its logs.
- [ ] Confirm the card can actually be created before trusting its price.
A cheap cycle isn't a good cycle by default. One version with new data tied the previous one 6 : 5 in a blind comparison (the setup is described in the first post) and didn't ship. But that failure cost $10, not $53, and left full logs behind.
Where this leaves us
We're not stopping cloud training. We're in the data-collection stage now, and training will continue on rented machines, but each run is many times cheaper and doesn't break out of nowhere. The latest speech recogniser came out better than the previous one: more accurate on reference transcripts, and it won more often in the same blind comparison.
The bottleneck has moved. It isn't hardware or money any more; it's live conversational speech transcribed by people. We've learned to count GPUs. What the models are missing is living language.
Fifth post in the series.
Want to help? What the project needs most is native Crimean Tatar speakers. Listen to a short clip and say whether the machine transcribed it right. Read a page and mark which word is Turkish and which is ours. Write a dialogue with the real colour of the language. Record a few minutes of reading. No programming needed, just knowing your language well. Write to ai@ay.mw or on Telegram: t.me/ana_yurt_dev.
If you'd rather support the work financially: ko-fi.com/anayurt. After all of the above, every dollar now goes a lot further.
來源:DEV Community · dev.to


