追蹤 GPU 與 token 價格的觀察：GPU 租用成本上升，推理價格下跌

Part of my job, and honestly part of my hobby, is watching prices. Not all of them, of course (I mean, what a lunatic must you be), but specific ones: GPU and inference endpoints. A while back I noticed that they differ immensely across providers, so I built a tool. Smart Inference, the router Jenni and I run, reads the price of GPU rentals and LLM tokens from a list of providers around the clock and sends each request to the cheapest healthy option. Since April it has kept every price it ever saw, which by now is roughly 11 million GPU price points and 3.3 million token prices.

I was wondering if my own data would support my theory that there is “overcapacity in the market”. Alas, that is not really what it shows.

I start in June, because before that we had too few providers to say much. From June to October, the median hourly price of the GPUs everybody wants went up:

• H100, on-demand: 3.25 to 3.89 dollars (+20%)

• H200, on-demand: 4.00 to 5.02 dollars (+25%)

• B200, on-demand: 6.11 to 7.42 dollars (+21%)

• B200, spot: 2.14 to 3.78 dollars (+77%)

Meanwhile, even the spot pricing options jumped: H100 in June was offered at a low of 1.04 an hour, and was bumped to 1.81 in October. This fits the overall market trend.

There is one exception, however and that is older hardware: An A100 on spot went from 1.07 to 0.88 an hour back in July and has sat there ever since. So it looks like people are fighting over newer hardware and perhaps if you want to really make a bargain right now, those can be a better choice.

Then I looked at the token side (my inference endpoints) and that’s where it got weird: Across all providers that serve the same model, the median price per million input tokens shifted a lot. DeepSeek V4 was at 1.30 dollars until July and is selling for 0.30 now, about 77% less. GLM-5 went from 1.00 in May to 0.15 in October. Output tokens fell almost as hard (DeepSeek V4: 2.60 down to 1.20, GLM-5: 3.20 to 0.50).

So a steep price drop, even though the models are unchanged and hardware got more expensive. To be fair, not everything got cheaper. Kimi K2.6 and GLM-4.7 stayed roughly the same, and gpt-oss-120b even went up a little at the median. But the big new open models, the ones people actually switch to, lost three quarters of their price or more within a few months of launch.

All in all, hardware got about 20% more expensive and inference got 80% cheaper. Makes no sense, but okay…

I have some theories why that may be: sometimes models are subsidized, other times providers get better at running the models (quantization is a thing) and maybe, just maybe older models are downgraded to lower specs (which could also have an impact on performance).

Theories aside, what it shows, however, is that my own idea that “the market is producing vast overcapacity for AI” is not really the case. At least not according to the data. If that were the case, my own pricing table, which also features “spot pricing”, should get cheaper over time, not more expensive. For current hardware it looks more like the opposite, the GPUs get pricier. It also means that sometimes model providers are eating the costs.

That matters more than you may think: it means that at least for now, it is more expensive to run your own GPU than it is to build on inference infrastructure. If token prices drop faster than GPU prices, renting your own GPU stops making sense for most workloads, and buying tokens from whoever is cheapest this week wins. Six months ago I would have told you that self-hosting on spot GPUs was the smart move for steady traffic. Today, for the big open models, I would check the token price first.

Before I finish this, you should be aware that this observation is only based on our own data: every monthly median comes from two to seven providers, the mix changes from month to month as we onboard additional ones. So even though the data points are complete, we are only comparing a small subset of providers.

Anyway, I love to be proven wrong, so I will keep checking up on the market in the near future. I will keep you posted…