H200 按每 TB/s 小時成本計，在 7 家供應商中有 6 家比 H100 划算

An H200 has exactly as much compute as an SXM H100 and usually rents for more, so on a FLOPS spreadsheet it never makes sense. The H200 vs H100 cost per token question turns on a different column: the H200 moves 4.8 TB/s out of memory against the H100's 3.35, and decode spends its time waiting on memory. Memory is also what the industry is short of: TechCrunch reports that Samsung used its Q2 earnings call to say the shortage worsens in 2027 and stays tight until at least 2028.

So we priced that column, using our GPU table for 2026-10-30.

Decode waits on memory, even at large batches

The FlashAttention paper defines arithmetic intensity as arithmetic operations per byte of memory touched, and says most Transformer operations are held back by memory accesses. Divide a GPU's dense FLOPS by its bandwidth and you get the intensity it can feed: at FP8, about 591 operations per byte for an SXM H100, 412 for an H200, 562 for a B200. Any workload well below that line runs out of bandwidth first.

Researchers at the Barcelona Supercomputing Center and IBM measured where decode sits, with vLLM on an H100. Attention kernels stayed between about 0.5 and 1 operation per byte from batch 1 to the maximum. At batch 512 on OPT-1.3B, memory traffic reached about 1.51 TB/s of a 1.63 TB/s ceiling, while compute sat near 4% of the paper's own compute line. That line, 25.6 trillion floating-point operations a second, is far below NVIDIA's FP8 figure, and decode sits well under both. In their tests on OPT and Llama 2 models, large batches didn't move decode into compute-bound territory.

The table that follows from that

Same provider, same day, SXM H100 against H200 (Vast, Lambda and Baseten list no H200, so they're out):

The break-even is 4.8 / 3.35, so the H200 wins per TB/s whenever it costs under 1.43 times the H100. Together AI, at 1.50 times, is the one provider here where it loses. The B200 wins at all seven, and it's the cheapest row at each.

The cheap-looking row is the trap. RunPod's PCIe H100 lists at $2.89, but NVIDIA's product brief gives that card a peak of 2,000 GB/s, so you pay $1.45 per TB/s-hour: more than RunPod's SXM H100, H200 or B200.

For one model, 70 GB of weights read once per token caps an H100 at 47.9 tokens/s and an H200 at 68.6. At RunPod's prices that ceiling works out to $23.16 per million tokens on the H100 and $21.43 on the H200. Those are batch-1 limits for one stream, so a real stream lands below that speed on both cards. Batching changes the sum: one pass over the weights serves many streams, so batched cost per token can come in far under these prices, and this arithmetic doesn't tell you by how much on either card.

Here's the whole calculation. Put in your own quotes and weights:

"""Rank GPUs for memory-bound decode by dollars per TB/s-hour."""
SPECS = {  # peak memory TB/s and dense FP8 TFLOPS (half NVIDIA's sparse figure)
    "H100 PCIe": (2.0, None),
    "H100 SXM": (3.35, 1979),
    "H200": (4.8, 1979),
    "B200": (8.0, 4500),
}
PRICES = [  # $ per GPU-hour, public on-demand list prices, 2026-10-30
    ("RunPod", "H100 PCIe", 2.89), ("RunPod", "H100 SXM", 3.99),
    ("RunPod", "H200", 5.29), ("RunPod", "B200", 7.99),
    ("CoreWeave", "H100 SXM", 6.16), ("CoreWeave", "H200", 6.31),
    ("CoreWeave", "B200", 8.60),
    ("Nebius", "H100 SXM", 4.50), ("Nebius", "H200", 5.40),
    ("Nebius", "B200", 8.50),
    ("Modal", "H100 SXM", 3.95), ("Modal", "H200", 4.54),
    ("Modal", "B200", 6.25),
    ("Together AI", "H100 SXM", 3.99), ("Together AI", "H200", 5.99),
    ("Together AI", "B200", 8.19),
    # Neither page names the H100's form factor, so SXM is an assumption here.
    ("DeepInfra", "H100 SXM", 2.90), ("DeepInfra", "H200", 3.49),
    ("DeepInfra", "B200", 4.89),
    ("Fireworks", "H100 SXM", 8.00), ("Fireworks", "H200", 8.00),
    ("Fireworks", "B200", 13.00),
]


def per_tb_hour(usd_hr, gpu):
    return usd_hr / SPECS[gpu][0]


def ops_per_byte(gpu):
    return SPECS[gpu][1] / SPECS[gpu][0]  # TFLOPS / (TB/s) = operations per byte


def h200_breakeven():
    """H200 wins per TB/s while it costs less than this many times an SXM H100."""
    return SPECS["H200"][0] / SPECS["H100 SXM"][0]


def max_tokens_per_s(weights_gb, gpu):
    """Ceiling: every weight read once per token, at peak bandwidth, batch 1."""
    return SPECS[gpu][0] * 1000 / weights_gb


def usd_per_m_tokens(usd_hr, weights_gb, gpu):
    return usd_hr / (max_tokens_per_s(weights_gb, gpu) * 3600) * 1e6


if __name__ == "__main__":
    for provider, gpu, usd in sorted(PRICES, key=lambda r: per_tb_hour(r[2], r[1])):
        print(f"{provider:12} {gpu:10} ${usd:5.2f}/h  "
              f"${per_tb_hour(usd, gpu):.2f}/TB/s-h  "
              f"${usd_per_m_tokens(usd, 70, gpu):5.2f}/M tokens at the 70 GB ceiling")

Where this ranking stops being true

Prefill is the other case. Philip Kiely's Inference Engineering says it runs out of FLOPS first, and we didn't measure that. If that's your traffic, an H200 gives you no extra FLOPS over an SXM H100.

Memory size isn't in the table. An H100 holds 80 GB, an H200 141 GB and a B200 180 GB, before any KV cache. If your model doesn't fit, the honest comparison is two cheaper cards against one bigger one, and the per-TB/s number won't settle that.

Peak bandwidth isn't sustained bandwidth. NVIDIA's figures are peaks ("up to 8TB/s" for the B200). That paper's memory ceiling for its card is 1.63 TB/s, under half the SXM figure, though it describes an unusual 64 GB H100, so don't treat that as a discount rate.

More bandwidth doesn't promise proportionally more tokens. The paper found CPU time reaching up to 30% of decode at batch 512 on OPT-1.3B, and a faster memory bus doesn't shrink that.

DeepInfra and Fireworks list their H100 only as 80 GB, so we assumed SXM. If either is a PCIe card the H200 wins by more, so the count holds.

We ran nothing on these GPUs. The tokens-per-second figures are bounds from published specs. Discounts reorder the table too: on Together AI's 91 to 180 day reserved rates ($3.19 and $3.99) the H200 costs 1.25 times the H100 and wins, though its preemptible rates stay at 1.50 times. Some of these rates only come in bulk or as a managed service, such as CoreWeave's 8-GPU instances and DeepInfra's dedicated deployments, so compare within a row, not across providers.

Do this today

Divide your GPU's hourly price by its peak bandwidth, then do the same for the next card up at that provider. If that number is lower and your traffic is mostly decode, benchmark your own model on both before you renew.

We also priced the same seven providers per dense PFLOP-hour, the number that matters for prefill, and the ranking changes; ask in the comments and we'll post the table.

When you pick a GPU for serving, do you compare on bandwidth, memory size or the hourly price?

Prices are InferPrice's public on-demand list prices for 2026-10-30, each from the provider's own pricing page: RunPod, CoreWeave, Nebius, Modal, Together AI, DeepInfra, Fireworks. InferPrice ran the arithmetic and its tests; nothing here was benchmarked on these GPUs.

Sources

• NVIDIA H100 specifications and H100 PCIe product brief
• NVIDIA H200 specifications
• NVIDIA HGX AI Factory components and DGX B200 specifications
• Dao et al., FlashAttention (2022)
• Recasens et al., Mind the Memory Gap (2025)
• TechCrunch on Samsung's Q2 call (2026-07-31)
• Further reading and the prefill rule of thumb: Philip Kiely, Inference Engineering (Baseten Books, 2026), section 2.4

Originally published at inferprice.pages.dev.