深入《Kilawatt 報告》：10 次 GPU 故障轉移測試、排除 6 次嘗試並公開所有數字

Disclosure: I'm the founder of Kilawatt Cloud. I ran and timed every drill below on an internal test account, so this is our own test record and not independent reporting.

The question

Chip makers all sell the same promise: more compute, faster. For anyone buying GPUs through software, the next question matters more: what happens when a rented machine disappears halfway through a job?

For an agent that bought the job on its own, a failed job is a lost purchase and a reason not to buy again. I wanted measurements, so I destroyed ten live GPU machines mid-job and timed what the platform did next.

How the drills worked

Eight drills ran through the goose agent app on October 20. Each job was bought over x402 (USDC on Base) through our MCP server, and an operator destroyed the Vast.ai machine by hand after a health check passed. Two earlier drills ran on October 14, with a checkpointing job on a RunPod A100.

I measured two things:

1. Detection: the time from my kill to the platform marking the machine failed.

2. Failover claimed: the time from detection to the platform claiming a replacement on a different supplier.

The results

On October 14, detection took 238 seconds in the first drill (RunPod to Hyperstack) and 121 seconds in the second (RunPod to Vast.ai). In the second, the job resumed automatically from its saved checkpoint 4, 12 seconds after the replacement was ready. It then went on to write checkpoints 5, 6 and 7.

Across all ten drills, the platform placed a replacement on a different supplier every time.

Why the ranges are wide

I recorded the minute I clicked Destroy, not the second, so each detection figure is a window of about a minute. Health checks also run every 2 minutes, which puts detection near 3 to 4 minutes after a kill.

What these drills do not show

1. Replacement-ready time. The database wasn't recording it. Replacements passed later health checks, but I have no timing, so I'm not claiming a recovery time for the October 20 drills.

2. Resume on the October 20 jobs. Jobs bought through x402 don't save checkpoints, so those eight measure detection and replacement only. Resume was shown only on October 14.

3. A median or worst case. Ten drills are too few, so I'm giving exact counts and ranges.

4. A supplier-removal drill. Not run yet.

5. Fully autonomous buying. A person typed each buy prompt and destroyed each machine. This shows the pay-per-job path and failover working together.

The 6 jobs I excluded

I paid for 14 jobs on October 20 and counted 8:

1. Four never booted on one Vast.ai host. They were auto-refunded, and I blocked that host in code.

2. One 600-second window was too short.

3. One was a replacement stopped at the end of its window, which isn't a drill.

What the drills broke

1. The endpoint said "rate limiting requests" for several different refusals. It should name the real limit and when it clears.

2. A job's concurrency slot stayed held until its 15-minute window ended, even after the instance was destroyed.

3. The never-ready refund didn't fire on time for two jobs.

4. Some hosts stall pulling the Docker image. Pre-caching it is on the list.

What's next

Detection is slower than I want at about 2 to 4 minutes, and shortening it is first on the roadmap. After that:

1. Record when each replacement is ready.

2. Run a larger set with median and slowest times.

3. Run a supplier-removal drill and publish it.

The full case study, with the method and the excluded attempts, is here: https://www.kilawattcloud.dev/super-intelligence

About the author: Damian Dixon is the founder of Kilawatt Cloud, based in Los Angeles.