Youtu-Parsing-Omni 論文提出以單一編碼器解析多種模態
Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities
研究提出 Youtu-Parsing-Omni,以共享權重的單一編碼器解析文件、圖像、音訊及影片,並以統一結構輸出結果。單一 5B 參數 checkpoint 在 OmniDocBench v1.6 得分 96.96 Overall,並在 OmniParsingBench 得分 75.08 Avg.,領先所有開放權重模型。
arxiv:2610.39301
Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities
Published on Oct 29
·Submitted by
沈云航 Yunhang Shenon Oct 30
Authors:
Yunhang Shen,
Zhangquan Chen,
Xiaotian Li,
Zhongpeng Cai,
Qing Zhou,
Zekai Zhang,
Kun Zhang,
Zezhong Wu,
Jian Hu,
Haihan Gao,
Heting Gao,
Xin Li,
Yinsong Liu,
Deqian Jiang,
Di Yin,
Haoyu Cao,
Xing Sun,
Bing Liu,
Yanqiu Qu,
Yunfei Wu,
Kun Yin,
Zhixiang Wei
+8 authors
Abstract
Youtu-Parsing-Omni is a unified multi-modal parser that encodes documents, images, audio, and video into structured spatiotemporal records using shared weights and cross-modal fusion, achieving state-of-the-art results across parsing benchmarks through a self-evolving on-policy distillation procedure.
Generated by thinkingmachines/Inkling-Small
We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.
View arXiv pageView PDFProject pageAdd to collection
Community
![]()
Paper submitter30 minutes ago
Youtu-Parsing-Omni is a compact (5B) omni-modal parsing model. Given a single input — a document page, a natural image, a chart / flowchart, a geometry figure, an audio clip or an audio-visual video — it produces one structured JSON envelope that covers both perception (layout elements, text, tables, formulas, bounding boxes, timestamps, ASR, OCR, acoustic events, camera motion) and cognition (captions, narratives, reports).
| Input | --task |
modality / subtype |
Key contents |
|---|---|---|---|
| Document page | document |
image / document |
layout elements with bbox, text / LaTeX / OTSL tables / Markdown charts / Mermaid flowcharts, reading order |
| Natural image | natural_image |
image / natural_image |
entities and text with bbox, tags, captions, global description |
| Chart | graphics_chart |
image / document |
one chart element: Markdown table, notes, caption |
| Flowchart | graphics_flowchart |
image / document |
one flowchart element: Mermaid, caption |
| Geometry figure | graphics_geometric |
image / document |
one geometric element: points, lines, arcs, shapes, geometric relations and measurements |
| Audio | audio |
audio / – |
vocal / non-vocal segments with timestamps, speakers, ASR, timbre / scene captions, acoustic events |
| Natural video | natural_video |
video / natural_video |
temporal segments with visual elements, actions, interactions, camera motion, audio track |
| Text-rich video | textrich_video |
video / text_rich_video |
segments with OCR + ASR and a Markdown structured_report of the whole video |
Highlights:
- Unified schema – one JSON envelope for seven parsing families, driven by the task prompt.
- Omni encoder – image, audio and interleaved audio-visual video inputs (frames + audio track) in a single model.
- Strong results at a small size – state-of-the-art on OmniDocBench v1.6 (96.96 Overall), best open-weight model on OmniParsingBench (75.08 Avg., second only to Gemini-3-Pro), and competitive with specialized models on chemical-structure (ChemOCR) and music-score (PDMX-Synth) recognition (results).
- Easy to serve – a vLLM plugin, pinned serving settings, task prompts and inference examples are included.
Reply
Edit Preview
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Comment ·Sign up or log in to comment
Models citing this paper 1
#### tencent/Youtu-Parsing-Omni Image-Text-to-Text • 5B•Updated 43 minutes ago• 2.94k• 117
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.39301 in a dataset README.md to link it from this page.
Spaces citing this paper 1
📑 mrfakename/youtu-parsing-omni
Collections including this paper 1
Youtu Collection 14 items•Updated 29 minutes ago• 29
System theme
Company
Website
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co




