DeepSeek-V4.1-Flash-1017 評估:60 頁 PDF 的文字擷取遺漏了甚麼
Evaluating DeepSeek-V4.1-Flash-1017 on a 60-Page PDF: What Text Extraction Misses
DeepSeek-V4.1-Flash-1017 在一次 60 頁 PDF 測試中,答對全部 20 題文字及表格題,卻未能提供 5 題只在點陣圖中出現的數值。模型亦正確回應 3 題文件明確未提供的資訊;另一次圖片輸入測試記錄為 5/5 答對,但沒有保留完整對話紀錄。這是一次診斷測試,並非標準化基準測試或受控模型比較。
I recently deployed DeepSeek-V4.1-Flash-1017 across two NVIDIA DGX Spark systems and connected the model to Open WebUI. Once the deployment was working, I wanted to test a common research workflow: attaching a long PDF and asking questions about its tables and figures.
The most informative result was not a headline accuracy score. It was the difference between information present in a PDF file and information available to the model through the document-processing workflow.
I used a 60-page fictional report, ASTER-47, and retained the original prompt and response. This article describes one diagnostic run, what its recorded results support, and what they do not. It is not a standardized benchmark or a controlled comparison of models.
The two-node installation and operating procedures are covered in the companion implementation article. [Link to the implementation article will be added after publication.]
1. Evaluation environment
| Component | Tested configuration |
|---|---|
| Hardware | Two NVIDIA DGX Spark systems (GB10) |
| Model | DeepSeek-V4.1-Flash-1017 |
| Checkpoint | Mia-AiLab EXL3 EAQ v1, 2.9 bpw |
| Inference | TensorFold, two ranks (TP=2) |
| Serving settings | CONTEXT=262144 PARALLEL=1 |
| Web interface | Open WebUI v0.13.3 |
| Document option | Using Entire Document |
| Evaluation material | ASTER-47, a 60-page English PDF |
The serving configuration was based on MiaAI-Lab's two-DGX-Spark repository at the revision used for this deployment.
The 262,144-token context is a configured limit, not a measured result of this PDF evaluation. I did not capture the exact API payload between Open WebUI and TensorFold, nor complete timing and token-usage data for the document test. Consequently, this article makes no claims about throughput, effective context utilization, or where in the internal processing pipeline any raster information was omitted.
2. Evaluation document and actual procedure
ASTER-47 is a custom-generated, completely fictional report about a multi-site virtual monitoring study. Its facility names, event counts, detector statistics, tables, and figures were generated for this test. None represents a real-world measurement or research participant.
The package contains the original PDF (60 pages, 686,543 bytes), a 28-question sheet, a separate answer key, a generation script, a manifest, and five figure pages saved as PNG images.
The questions were organized into three groups:
- Q01–Q20: facts, arithmetic, and reconciliation across selectable text and tables.
- Q21–Q25: specific values and labels printed inside raster figures, intentionally not repeated in the selectable text.
- Q26–Q28: information the fictional report explicitly does not contain, such as measured GPU power or human participants.
For the recorded PDF run, I attached only the report PDF to Open WebUI, selected Using Entire Document, and submitted all 28 questions together in one message. The answer key was not attached. The original model reply was exported as both text and a six-page PDF.
An important methodological detail: the prompt itself said that unsupported values should not be invented. It also explicitly warned that a PDF-to-text-only workflow was not expected to access raster pixels. The synthetic PDF reinforced this point in its own figure descriptions and parser-audit notes. Therefore, a refusal to answer a figure question cannot be interpreted as the model independently discovering that image information was missing.
That distinction matters more than the apparent success of the refusals.
3. Recorded outcomes
| Question group | Observed outcome | Evidence status |
|---|---|---|
| Q01–Q20: text and tables | 20/20 matched the prepared answer key in the recorded run | Full original reply retained |
| Q21–Q25: raster-only values, PDF attachment | Five answers withheld; no requested figure values supplied | Full original reply retained; instructions explicitly prompted caution |
| Q26–Q28: unsupported information | 3/3 responses matched the expected absence of evidence | Full original reply retained; the PDF also explicitly declares these facts absent |
These outcomes describe one bundled 28-question interaction, not 28 independent trials. The key was checked after the run. They should not be combined into a single “PDF accuracy” percentage.
A separate, earlier image-input check was also performed: Figures 6–10 were provided as PNG images, and 5/5 answers were scored correct at the time. However, the complete original transcripts for those five PNG responses were not preserved. I therefore report this as a previously recorded manual result, not as a fully auditable score comparable to the PDF run.
Example A: Reading a table and performing a calculation
Question 9 asked for precision, recall, and F1 to four decimal places using the fictional S2 confusion matrix on page 17:
- TP = 430
- FP = 224
- FN = 110
- TN = 16,156
An excerpt from the original model response reads:
Precision = 430/(430+224) = 430/654 = 0.6575
Recall = 430/540 = 0.7963
F1 = 2(430)/(2·430 + 224 + 110) = 860/1194 = 0.7203
The calculations match the prepared key. This demonstrates successful extraction and arithmetic for this example, not statistical validation of the fictional detector.
Figure 1. Cropped from the actual six-page chat PDF export. The numerical results and wording were not edited.
The response also correctly distinguished the report's 12 illustrative case-log rows from the larger population of 60 positive events in Q20. These examples are more informative than simply listing the overall 20/20 count.
Example B: A value visible in the PDF but unavailable in the text response
Question 22 asked for the value at site FX × Packet in Figure 7 (page 26). The heatmap has the value 0.72 printed inside its pixels; that numeric cell is deliberately omitted from the surrounding selectable text.
Figure 2. The original fictional report's page 26. The cell value is visible in the raster figure.
In the retained PDF-only response, the model marked Q22 “Not accessible” and explained that the cell values were present in the raster image but absent from the text it could use. It withheld the requested numeric answer, as it did for Q21 and Q23–Q25.
Figure 3. Cropped from the actual six-page chat PDF export. The five figure-dependent responses are presented together.
This demonstrates the behavior of this tested end-to-end workflow. It does not prove that the model lacks vision capabilities, that every PDF figure is discarded by Open WebUI, or that a particular parser component is responsible. I did not inspect the final model request payload.
The earlier separate-PNG check was scored 5/5, including 0.72 for this question. That observation is consistent with image input making the missing evidence available, but the absent raw PNG transcripts limit how independently the result can be verified.
Example C: Declining unsupported requests
Questions 26–28 requested real measured GPU power, actual geographical coordinates of a fictional site, and the number of human participants in a supposed field study.
The recorded model reply did not fabricate these facts. However, the fictional report expressly says there were no physical measurements, real facilities, or human-subject experiments. The initial user prompt also prohibited unsupported claims.
Thus, 3/3 describes agreement with an intentionally easy, explicitly signposted abstention test. It is not evidence of general hallucination resistance.
4. What limits this evaluation
Several features constrain the interpretation:
- Purpose-built synthetic document. ASTER-47 was designed to contain explicit, auditable answers and deliberately missing figure-only facts. It does not represent the distribution of real scientific literature.
- One combined prompt and one PDF response. The 28 answers were produced together, so they are not independent samples. There were no repeated runs or confidence intervals.
- Instructional cues. Both the user prompt and the PDF described the raster-only limitation and the absence of real-world measurements. The abstentions are partly a test of following those instructions.
- No inspected internal model payload. The UI was configured to use the entire document, but I cannot attribute the missing raster evidence to a specific component solely from the model's explanation.
- Incomplete PNG provenance. Five direct-image responses were manually scored correct in an earlier check, but their verbatim transcripts are unavailable. A repeated, fully logged image run would be needed for a stronger paired comparison.
- A small document inconsistency. The model noticed that the report's R0 calibration-source wording on page 14 and its validation-selection evidence on page 15 were not fully reconciled. This does not change the recorded Q14 answer-key match, but it is another reason not to treat the synthetic report as a validated external benchmark.
- No performance characterization. The experiment does not establish full-context accuracy, sustained concurrency, memory safety under load, or token-generation speed.
5. Materials and reproducibility
The evaluation materials are intended to be published together: the original PDF, the exact 28-question prompt, the sealed answer key, the report generator and manifest, the figure PNGs, the unaltered original chat TXT, and the six-page chat PDF export.
Evaluation files: [Add the actual GitHub repository or release URL after publication.]
To reproduce the recorded PDF condition, start a fresh conversation, select the same model and document mode, attach the ASTER-47 PDF without the answer key, and submit the exact original combined prompt in one message. Preserve the raw answer before scoring it.
To evaluate image input separately, open a new conversation, attach the relevant Figure 6–10 PNG images, and record each full response alongside its corresponding question. That is a different input condition and should not be merged with the PDF-only results. For a more rigorous comparison, also remove the modality hints, repeat trials, and log the actual API request payload.
The source files and complete chat exports are a better audit trail than a long screenshot embedded in this article. The cropped images above are illustrations only; the unedited exports should remain available alongside the evaluation data.
Conclusion
In this recorded two-DGX-Spark deployment, DeepSeek-V4.1-Flash-1017 answered the synthetic report's text/table questions correctly under the prepared scoring key. For values printed only inside raster figures, the PDF-based interaction yielded appropriately cautious non-answers—but the test prompt and document had already signposted that limitation.
A separately performed direct-PNG check was reported as successful, although its full model outputs were not retained. The strongest supported takeaway is therefore not a general model-accuracy number: it is a practical reminder to verify what a document-processing interface actually makes available to its model before relying on it for scientific figures and diagrams.
AI disclosure: This article documents my own technical setup and results. An LLM was used to help draft, organize, and edit the English text. I reviewed the technical claims against the available original prompt, answer key, document, and chat export.
來源:DEV Community · dev.to


