跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 24 天前AI 評分60

Youtu-Parsing-Omni 論文提出以單一編碼器解析多種模態

Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

AI 導讀

研究提出 Youtu-Parsing-Omni,以共享權重的單一編碼器解析文件、圖像、音訊及影片,並以統一結構輸出結果。單一 5B 參數 checkpoint 在 OmniDocBench v1.6 得分 96.96 Overall,並在 OmniParsingBench 得分 75.08 Avg.,領先所有開放權重模型。

正文

Image 1: Hugging Face's logoHugging Face


Papers

arxiv:2610.39301

Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

Published on Oct 29

·Submitted byImage 2 沈云航 Yunhang Shenon Oct 30

·Image 3: tencentTencent

- [x] Upvote 2

  • Image 4
  • Image 5

Authors:

Yunhang Shen,

Zhangquan Chen,

Xiaotian Li,

Zhongpeng Cai,

Qing Zhou,

Zekai Zhang,

Kun Zhang,

Zezhong Wu,

Jian Hu,

Haihan Gao,

Heting Gao,

Xin Li,

Yinsong Liu,

Deqian Jiang,

Di Yin,

Haoyu Cao,

Xing Sun,

Bing Liu,

Yanqiu Qu,

Yunfei Wu,

Kun Yin,

Zhixiang Wei

+8 authors

Abstract

Youtu-Parsing-Omni is a unified multi-modal parser that encodes documents, images, audio, and video into structured spatiotemporal records using shared weights and cross-modal fusion, achieving state-of-the-art results across parsing benchmarks through a self-evolving on-policy distillation procedure.

Generated by thinkingmachines/Inkling-Small

We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.

View arXiv pageView PDFProject pageAdd to collection

  • Image 6
  • Image 7

Community

Image 8

shenyunhang

Paper submitter30 minutes ago

Youtu-Parsing-Omni is a compact (5B) omni-modal parsing model. Given a single input — a document page, a natural image, a chart / flowchart, a geometry figure, an audio clip or an audio-visual video — it produces one structured JSON envelope that covers both perception (layout elements, text, tables, formulas, bounding boxes, timestamps, ASR, OCR, acoustic events, camera motion) and cognition (captions, narratives, reports).

Input --task modality / subtype Key contents
Document page document image / document layout elements with bbox, text / LaTeX / OTSL tables / Markdown charts / Mermaid flowcharts, reading order
Natural image natural_image image / natural_image entities and text with bbox, tags, captions, global description
Chart graphics_chart image / document one chart element: Markdown table, notes, caption
Flowchart graphics_flowchart image / document one flowchart element: Mermaid, caption
Geometry figure graphics_geometric image / document one geometric element: points, lines, arcs, shapes, geometric relations and measurements
Audio audio audio / – vocal / non-vocal segments with timestamps, speakers, ASR, timbre / scene captions, acoustic events
Natural video natural_video video / natural_video temporal segments with visual elements, actions, interactions, camera motion, audio track
Text-rich video textrich_video video / text_rich_video segments with OCR + ASR and a Markdown structured_report of the whole video

Highlights:

  • Unified schema – one JSON envelope for seven parsing families, driven by the task prompt.
  • Omni encoder – image, audio and interleaved audio-visual video inputs (frames + audio track) in a single model.
  • Strong results at a small size – state-of-the-art on OmniDocBench v1.6 (96.96 Overall), best open-weight model on OmniParsingBench (75.08 Avg., second only to Gemini-3-Pro), and competitive with specialized models on chemical-structure (ChemOCR) and music-score (PDMX-Synth) recognition (results).
  • Easy to serve – a vLLM plugin, pinned serving settings, task prompts and inference examples are included.

Image 9: image

Image 10: image

Image 11: image

Reply

Edit Preview

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Comment ·Sign up or log in to comment

- [x] Upvote 2

  • Image 12
  • Image 13

Models citing this paper 1

Image 14 #### tencent/Youtu-Parsing-Omni Image-Text-to-Text • 5B•Updated 43 minutes ago• 2.94k• 117

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.39301 in a dataset README.md to link it from this page.

Spaces citing this paper 1

📑 mrfakename/youtu-parsing-omni

Collections including this paper 1

Youtu Collection 14 items•Updated 29 minutes ago• 29

System theme

Company

TOSPrivacyAboutCareers

Website

ModelsDatasetsSpacesPricingDocs

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co