跳到正文
DEV Community· Jacek Garbuliński·· 40 分鐘前AI 評分62

Qwen3.8-27B 處理長文件要多久?實測等待時間、查找表現與成本

How long does Qwen3.8-27B take with a long document?

AI 導讀

作者以單張 H200 實測 Qwen3.8-27B 處理長文件,約 260,000-token 文件的首個 token 等待時間為 52 至 55 seconds,後續問題通常需 1.1 至 1.6 seconds。

正文

A contract bundle or a product manual can go into one request to a language model, so the answer draws on all of it and nobody has to build a search over the documents first. Qwen3.8-27B takes up to 262,144 tokens in one request, about the whole first volume of Bolesław Prus's Lalka (The Doll), a classic Polish novel. On one H200 graphics card it started answering a question about that volume after 52 seconds, and it answered the next question about it after 1.3 seconds. How much of that wait a team pays for depends on how it sends its questions.

This guide shows how long Qwen3.8-27B takes to answer about long documents, how reliably it finds what is in them and what each question costs, from our measurements in Polish and English. The main findings:

  • the first answer took 17 to 18 seconds for a 131,000-token document and 52 to 55 seconds for a 260,000-token one, so twice the text took about three times as long;
  • later questions about the same document came back in 0.6 to 1.6 seconds, with one exception, because the server keeps what it has read;
  • it found every fact we planted in a novel and a manual, and said so when a fact was not there;
  • each question about a 260,000-token document costs about $0.05 of input at $0.20 per million tokens, because every question sends the whole document again;
  • a long document slows the other requests on the same graphics card, and one setting shares the card between them.

These figures come from our own runs on one card: each length was measured once per text and language, on two texts, with questions that look up one fact, and at times with other requests running on the card. They show what working with the model looks like, the size of the waits and what reusing a document already read saves, rather than exact values. Firm numbers would take benchmark work: many runs of each setting, more kinds of documents and questions, and other cards and loads.

It is written for developers deciding whether to send whole documents to a model or to search them first, for teams running Qwen3.8-27B on their own graphics card, and for those calling it through Blixild AI. For documents of up to 32,000 tokens and a smaller model, see our Bielik API guide. The sections cover, in order: what fits in one request, how reliably the model finds a fact, how the wait grows with the length, what later questions cost in time and money, running the model yourself, and where to start.

Qwen3.8-27B's context window holds 262,144 tokens

Qwen3.8-27B is an open model from the Qwen team, published under the Apache 2.0 licence, and Blixild AI serves its FP8 build, in which most weights take one byte each. One request holds up to 262,144 tokens, the model's context window, and the question, the document and the answer all count towards that limit. The first volume of Lalka, 123,000 Polish words, comes to 261,000 tokens, about 2.1 tokens a word; English prose takes fewer, about 1.4 tokens a word in Jane Austen's Pride and Prejudice. Text dense with numbers takes more: our test manual took about 1.7 tokens a word in English and 2.3 in Polish.

Qwen3.8-27B reasons before every answer unless it is told not to, which spends time and tokens on a question that only looks something up. One field turns the reasoning off, and our measurements below were all made that way:

curl https://api.blixild.com/v1/chat/completions \
  -H "Authorization: Bearer $BLIXILD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.8-27b",
       "chat_template_kwargs": {"enable_thinking": false},
       "messages": [{"role": "user", "content":
         "Document:\n\nSection 12. The generator service contract was signed with Voltaris under number VLT-2291.\n\nQuestion: Under what number was the generator service contract signed? Answer briefly, from the document only. If the document does not say, write that it is not in it."}]}'

In practice the whole document stands where the one-line section is, read from a file by the program that sends the request. How well the model uses the whole length is the next question.

Qwen3.8-27B finds a fact at the start, middle and end of a long document

A long prompt is only useful if the model reads all of it, including the middle. We tested that on Lalka, cut at about 4,000, 130,000 and 258,000 tokens, and on a generated company manual full of look-alike inventory and contract numbers, at about the same lengths in English and in Polish. Into each text we planted three invented sentences at 10, 50 and 90 % of its length, such as the inventory number of an antique clock, so the model could not answer from what it had learned about the novel. Each was asked about in its own request, and a fourth question asked about an object the text does not mention.

Qwen3.8-27B found all 27 planted facts and named each one exactly, such as the clock's number INW-4817. To each of the nine questions about the missing object it answered that the document does not say, rather than offering a number from a nearby passage. The test covers look-ups only, one fact per question, once per length, and no summaries or calculations across a document. Finding the fact was not the limit in this test, and the time to get it was.

The time to first token grows faster than the document

Before the first word of an answer, the model reads the whole prompt, so the time to first token, the wait until the answer starts, grows with the document. We measured it on the graphics card itself, without a network in between: 0.2 to 0.3 seconds for 4,000 tokens, 17 to 18 seconds for 131,000 and 52 to 55 seconds for 260,000. Twice the document took about three times as long. Once the answer had started, a short answer of a few words was complete within a tenth of a second; with short prompts the model writes about 164 tokens a second in Polish and 191 in English.

Two bar charts on one scale: the time to the first word of the answer for the first question about a document and for each later question, at 4,000, 130,000 and 258,000 tokens
Measured on one H200 card with Qwen3.8-27B and the first volume of Lalka cut to three lengths, reasoning off. Both charts share one scale; each later question is the mean of three.

Our LLM GPU token speed calculator estimates this wait for other graphics cards and models. For Qwen3.8-27B on an H200 with a 254,000-token prompt it gives 45 to 46 seconds, where we measured 51, so for very long prompts it falls short of the measured wait. These times were taken with the card mostly free; other requests running beside a long prompt lengthen them, as the section on running the model yourself shows. The first wait is paid once per document, provided the next questions can reuse the reading.

Later questions reuse what the server has read

The server keeps a prompt cache, the internal state of a prompt it has already read, and a next request that begins with the same text skips reading that part again. Later questions about the same document came back in 0.6 to 1.0 seconds at about 130,000 tokens and in 1.1 to 1.6 seconds at about 260,000. To get this, put the fixed part first, the instructions and the document, and the question last, and send the document unchanged every time: the server reuses only a beginning that is identical to one it has read. In Blixild AI each customer's cache is kept apart, so only your own earlier requests speed up yours.

For some documents the second question takes as long as the first: one of six long documents in our runs, 17 seconds at 131,000 tokens. The server marks how far it has read with a bookmark, which with this model can only stand at a multiple of 1,600 tokens, up to 1,600 tokens before the end of the request. In the other five, the bookmark fell inside the document, and the next question started from it. When it falls inside the first question instead, the bookmark covers words the second question does not share, so the server reads the whole document again. The questions after it were fast again. A short question after the document makes this rarer.

The cache saves time, not money. Every question is billed for the whole document, so at $0.20 per million input tokens a question about a 260,000-token document costs about $0.05, and ten questions about $0.52. For one-off questions across many documents, a search that sends the model only the relevant passages costs less and answers sooner. How long the server keeps a document between questions depends on how busy the card is, and we have not measured it.

Running Qwen3.8-27B on your own H200

The same measurements serve teams that run the model themselves. We serve it with vLLM, an open-source program that runs models behind an OpenAI-compatible API, on one H200 NVL with 141 GB of memory, with the FP8 build and its cache stored in 8 bits:

  • the model takes about 28 GiB, and 72 GiB set aside for the cache hold about 2 million tokens, seven to eight full-length requests at once;
  • answers to eight requests at once come at about 1,100 tokens a second together, with short prompts;
  • a long prompt is read in chunks, and every chunk holds up the other requests on the card.

vLLM's setting --long-prefill-token-threshold limits how much of a long prompt is read in each step while other requests run. At 1,024 tokens a step, the other requests kept 9 to 14 tokens a second beside a 254,000-token prompt, whose first token came after 63 seconds instead of 51. A lower value would favour the other requests further at the long prompt's cost, and we have not measured it. The figures hold for this card and this build; for another card, the calculator above gives a first estimate. How to put such a server behind keys and prices is the subject of our guide to a self-hosted LLM API.

Where to start with long documents

A team that asks several questions about one set of documents can send the whole set in one request and expect the first answer in about a minute at the full length and the next ones in about a second. A team with one-off questions across many documents is better served by a search that picks the passages first. A team that runs the model itself should set the long-prompt limit before long documents arrive, so that one of them does not stall everyone else. The first steps:

  1. Sign in at Blixild AI, where an API key and $1 of credit to start are ready at once.
  2. Send one of your documents and a question to qwen3.8-27b with reasoning off, as in the example above.
  3. Ask the next questions with the document unchanged and in the same place, and keep the text after it short.
  4. Time the first and later answers on your own documents, and compare the cost per question with a search that sends only passages.

Qwen3.8 27B runs in Blixild AI on our H200 in Poland, at $0.20 per million input tokens and $1.50 per million output tokens. We set up and run a model at a customer's site or in their own cloud account as part of our AI workload hosting.

Sources

  1. Qwen (2026). Qwen3.8-27B-FP8, model card. https://huggingface.co/Qwen/Qwen3.8-27B-FP8
  2. vLLM (2026). Automatic prefix caching. https://docs.vllm.ai/en/stable/features/automatic_prefix_caching.html
  3. vLLM (2026). Engine arguments. https://docs.vllm.ai/en/stable/configuration/engine_args.html
  4. Prus, B. (1890). Lalka. Wolne Lektury. https://wolnelektury.pl/katalog/lektura/lalka/
  5. Austen, J. (1813). Pride and Prejudice. Project Gutenberg eBook #1342. https://www.gutenberg.org/ebooks/1342

來源:DEV Community · dev.to