ScaleDown CDM 分類、抽取與摘要模型實測

Last week ScaleDown launched Constrained Decoding Models (CDMs). The launch post got about 1.8M views. The pitch is simple: most production LLM calls don't need prose. They need a label, a field, or a short summary. So instead of a general model writing an answer token by token, each CDM does one job, reads your input once, and returns a typed result. Pricing is $0.05 per million input tokens (OpenRouter lists $0.04 with a 20% promo), and output is free.

The launch post made big claims: "200x faster, 467x cheaper than frontier LLM APIs" and "77.2% on DecisionBench". A thread from Bitroot pushed back the next day. Their point was that the 467x has no stated baseline, DecisionBench has no public source, and nobody should trust a classifier result without a few hundred labelled examples and confidence intervals. That's a fair critique, so I built the test around it.

Here's what I ran, what held up, and what didn't.

The setup

I'm a Lead SWE, so I tested this the way I'd vet any vendor before routing production traffic to it: public labelled data, the same prompts for every engine, and raw results saved to disk.

• Classify: 300 CLINC150 test utterances (10 intents plus 50 out-of-scope) and 110 Bitext customer-support tickets across 11 categories.
• Extract: 50 SROIE receipts (company, date, address, total) and 30 LinkedIn job posts (company, top of pay range, and work type as an enum).
• Summarize: 6 QMSum meeting transcripts (4k to 9k tokens), 2 long ones (31k tokens), and one 369k-token stress doc.
• Baseline: Qwen3 1.7B running locally on Ollama (8-core CPU, no GPU), with the same instructions and JSON-schema-constrained output. I didn't have a frontier API key on this box, so for frontier models I did list-price math from the official pricing pages (checked 22 Oct 2026).

Everything goes through one OpenAI-compatible base URL. Classify also has a Jev-compatible Decisions endpoint, which is the one I used most:

import httpx, os
r = httpx.post(
    "https://api.scaledown.xyz/v1/scaledown",
    headers={"x-api-key": os.environ["SCALEDOWN_API_KEY"]},
    json={
        "state": {"text": "how do I reset my pin"},
        "questions": {"intent": {"type": "choice", "criteria": {
            "pin_change": "wants to change or reset their PIN",
            "balance": "asks for an account balance",
            "oos": "none of the above / out of scope",
        }}},
    },
)
print(r.json()["answers"]["intent"])
# {'type': 'choice', 'choice': 'pin_change', 'confidence': 0.979, ...}

Each response includes a usage.cost field. Mine always worked out to exactly $0.05 per million billed input tokens.

Classify: the strongest result

CLINC150 (n=300, 10 intents + out-of-scope)   accuracy   95% CI          OOS recall
ScaleDown Decisions (classify-1)              97.3%      94.8 - 98.6%    98%
ScaleDown /classify (rubrics)                 96.7%      94.0 - 98.2%    100%
Qwen3 1.7B local                              90.7%      86.8 - 93.5%    86%

Bitext support tickets (n=110, 11 categories)
ScaleDown Decisions                           92.7%      86.3 - 96.3%
ScaleDown /classify                           90.9%      84.1 - 95.0%
Qwen3 1.7B local                              62.7%      53.4 - 71.2%

The confidence intervals don't overlap with the small local model on either set. On CLINC150 the out-of-scope handling impressed me most. It correctly rejected 49 of 50 out-of-scope queries, and only 6 in-scope queries were wrongly pushed to "oos". That's the failure that hurts in a real router.

I didn't need any prompt engineering. I wrote one plain-English line per label, and that was it.

Latency, sequential, measured from a cloud box in India: p50 162 ms, p95 397 ms for Decisions, and p50 134 ms for /classify with reasoning: false. The ticket set had a longer tail (p95 about 1.0 s).

Extract: good spans, and the misses are interesting

Extract takes a standard response_format JSON schema. Each property's description is the search criteria, and every value comes back with a confidence score and character offsets into your input.

schema = {"type": "object", "properties": {
    "company":    {"type": "string", "description": "the hiring company's name"},
    "max_salary": {"type": "string", "description": "the top of the stated salary or pay range"},
    "work_type":  {"type": "string", "enum": ["Full-time", "Contract", "Part-time"],
                   "description": "employment type"},
}}
client.chat.completions.create(
    model="extract",
    messages=[{"role": "user", "content": job_post}],
    response_format={"type": "json_schema", "json_schema": {"name": "job", "schema": schema}},
    extra_body={"top_n": 1, "context_chars": 0},
)

                        field acc   valid JSON   spans verbatim   p50 latency
Receipts (50 SROIE)
  ScaleDown Extract     89.5%       100%         100%             1.43 s
  Qwen3 1.7B local      89.5%       100%         94%              3.55 s
Job posts (30)
  ScaleDown Extract     90.0%       100%         n/a              0.60 s
  Qwen3 1.7B local      81.1%       100%         n/a              2.76 s

Per field on receipts, ScaleDown got 100% of totals right, compared with 78% for the local model. Totals are the field you actually care about. It was weaker on company names (80% vs 88%).

Every span ScaleDown returned appeared verbatim in the source text. The citations aren't decoration. You can highlight exactly where a value came from, which is what Bitroot said to check.

I read every miss, and most were judgment calls rather than hallucinations:

• The SROIE gold "date" often includes the time (21 MAR 2018 17:59). ScaleDown returned just the date.
• It returned KING'S CONFECTIONERY S/B without the registration number the label includes.
• On job posts it sometimes returned the parent company instead of the brand (ManpowerGroup for an Experis post, HCA Healthcare for MountainView Hospital).
• One real error: on one bakery delivery receipt it picked the buyer's name and address instead of the seller's. That template appeared 4 times in my random sample, so it cost 8 field points on its own.

Valid JSON was 100% on both engines, but that's partly because I constrained the local model's output with a schema. The bigger difference is that ScaleDown hands back offsets and a confidence score with no extra work.

Summarize: fast and faithful, but watch the query focus

For summaries I passed the QMSum query as the system instruction and the transcript as the user message.

QMSum meetings, 4k-9k tokens (n=6)   p50       p95       ROUGE-L   summaries w/ invented facts
ScaleDown Summarize                  0.65 s    0.89 s    0.230     0 of 6
Qwen3 1.7B local (CPU)               24.3 s    34.1 s    0.201     2 of 6

I checked every number and name in each summary against the transcript. ScaleDown's held up, including the details: a 2.39 average rating, the drop from 15 to 12.50 euros, and the 50 million euro profit target were all there in the source. The local model said one rating used "1 to 5" when the meeting used a seven-point scale. It also called an over-budget prototype "reasonably priced".

Two weaknesses showed up:

• Query focus. In one meeting it summarised the presentation instead of answering "what did the PM question". On both 31k-token transcripts it summarised an adjacent part of the meeting rather than the exact question. It's a summarizer first and a QA model second.
• The 1M-context claim. I sent a 369k-token document made of 12 concatenated meetings and asked for one or two sentences per meeting. It accepted the input and returned in 14.2 s. The summary, though, covered only one of the two distinct meeting series in the document and skipped the other entirely. So it can ingest that much, but don't rely on coverage at that length.

Is it really 200x faster?

Not against anything I could measure, and not according to ScaleDown's own docs.

Against a 1.7B model on my CPU, it was 1.8x faster for short classification, 2.5 to 4.6x for extraction, and about 37x for 6k-token summaries, where CPU prompt processing dominates. A local CPU model isn't a frontier API, though. ScaleDown's own head-to-head benchmark page reports 2.2x to 13.9x faster than each provider's lineup average (2.4x vs OpenAI, 3.8x vs Anthropic). That's a solid speedup, but it's an order of magnitude below the launch post's 200x. I couldn't find what "200x" was measured against.

Is it really 467x cheaper?

The price is real. Every call billed exactly $0.05 per million input tokens, and output was free. The ratio, though, depends entirely on the baseline, and there's a catch: ScaleDown bills more input tokens than your raw prompt. Label criteria, schemas and internal prompts are counted. My classify calls billed about 481 tokens, while the same task as a plain LLM prompt was 172 tokens. Extract calls billed about 1,380 tokens against 571.

Here's the cost per 1,000 calls at list price, using my measured token counts and leaving out reasoning tokens:

                         classify   extract   summarize (6k tok)
ScaleDown CDM            $0.024     $0.069    $0.317
gpt-5-nano               $0.012     $0.057    $0.333     (0.5x - 1.0x)
gpt-5.4-nano             $0.044     $0.204    $1.300     (1.8x - 4.1x)
gpt-5.4-mini             $0.163     $0.752    $4.860     (6.8x - 15x)
Claude Haiku 4.5         $0.210     $0.931    $6.394     (8.7x - 20x)
Claude Sonnet 4.6        $0.631     $2.793    $19.18     (26x - 61x)
gpt-5.5                  $1.089     $5.015    $32.40     (45x - 102x)
Claude Fable 5           $2.102     $9.311    $63.94     (87x - 202x)

So against the cheapest small models, ScaleDown costs about the same, and on short classification gpt-5-nano is actually half the price. Against flagship models it's 45x to 200x cheaper. You only reach 467x if the frontier model also spends a few hundred reasoning tokens per call. For example, gpt-5.5 classifying with 500 reasoning tokens works out to roughly 670x. That's realistic for a reasoning model doing routing, but it isn't stated anywhere in the launch post. ScaleDown's own docs are more modest: 89x cheaper than the OpenAI average and 161x cheaper than the Anthropic average.

The /classify endpoint accepts up to 50 chunks per request and bills the label set once. If you classify at volume, that removes most of the token overhead I measured.

Limitations: I didn't benchmark Compress or concurrent load this round; my free credits ran out after ~1M input tokens, since the 50M free tokens were a launch-window promo. After that I got 402 Insufficient credits, so budget for a top-up before you run a real eval.

What Bitroot got right

The skeptic thread was right on every point I could check:

• 467x has no stated baseline. I can reproduce it only by assuming an expensive reasoning model.
• I still found no public source for DecisionBench's 77.2%.
• Price is the easy part to verify, and it checks out.

Their main advice was to build a test set of 300+ labelled examples and report intervals, and that advice is what made this test worth anything. With n=300, the gap between 97% and 91% is real. With n=30 it wouldn't be.

My take

I'd route classification traffic to ScaleDown today. It was the most accurate engine I tested, with clean out-of-scope handling, sub-200 ms p50 latency, and no prompt tuning. Extract is a strong default when you need provenance, because every value comes with an offset you can highlight. Summarize is fast and didn't invent facts in my sample, but test its query focus on your own data, and don't trust its coverage of 300k+ token inputs.

The marketing ratios are a different story. If you already run gpt-5-nano, you won't save much. If you run flagship or reasoning models for routing and extraction, the savings are large and real.

Test it on your own data. A few hundred labelled examples will tell you more than any launch post, mine included.