財務 AI 訓練配方如何導致「模型幻覺」

In Part 1, we laid down the foundational law of financial AI: Never allow a probabilistic language model to calculate numbers in its latent space. We proved why high-stakes finance (M&A due diligence, debt covenant audits, IFRS/US GAAP reconciliation) demands a dual-persona architecture—delegating arithmetic to a hardware-certified Python IEEE-754 runtime while turning an open-weight LLM into a deterministic semantic compiler.

Today, we are pulling back the curtain on the actual machine room.

How do you take raw, messy SEC 10-K filings, compress them into an induction-friendly dataset, and fine-tune an 8B model on commodity hardware without degrading general intelligence?

And more importantly: what happens when an 8B model encounters a missing financial variable? (Spoiler: it panics, and the resulting bug—The Compensation Hallucination Trap—will break your pipeline if you don't build a structural circuit breaker).

1. The Data Pipeline: Why Raw 10-K Tables Break Naive LoRA

Financial tabular data is hostile to standard transformer tokenizers.

In datasets like TAT-QA and FinQA, balance sheets and income statements are filled with:

• Hierarchical line items with arbitrary indentation.
• Parenthetical negative numbers: (4,960) instead of -4960.
• Thousands separators that vary by geography: 52,491 in US GAAP vs. 52 491 (non-breaking spaces) in European PCG/IFRS.
• Footnote markers embedded directly inside labels: Revenue (Note C.1).

If you feed raw tabular snippets directly to a causal language model, standard tokenizers break the numbers into fragmented sub-word tokens (e.g., ["52", " 491"]). Worse, the model's self-attention disperses across empty whitespace padding.

flowchart TD
    Raw["Raw 10-K Filing / PDF Table<br/>(Messy HTML/Markdown)"] --> Cleanse["Cell Sanitization & Accent Normalization"]
    Cleanse --> DSL["Initial Experiment: Compressed DSL<br/>[DOM: IS | P: REV | N-1: 100 | N: 120]"]
    DSL -. "Induction Heads Fail on Flat Text" .-> Pivot["Scientific Pivot (arXiv:2602.08548)<br/>Canonical 2D Markdown with 3-Phase Thinking"]
    Pivot --> Dataset["Watertight Training Split (19,506 Samples)"]

The DSL Experiment vs. Canonical 2D Markdown

In our v1 MVP, we attempted to compress tabular tokens by converting tables into a dense Domain-Specific Language (DSL):

[DOM: INCOME_STATEMENT | CURRENCY: USD]
[ROW: Net Sales | N-1: 697290 | N: 963286]
[ROW: Operating Income | N-1: 122628 | N: 142034]

While this minimized token counts, our benchmark results showed an unexpected drop in coordinate localization accuracy.

Why? Recent mechanistic interpretability research (arXiv:2602.08548) explains that transformer attention heads act as induction counters on spatial delimiters. When a model sees markdown table pipes (|), its early attention layers form precise positional grid coordinates (Row i, Col j). Flattening tables into a custom DSL destroyed these spatial inductive biases.

The Winning Format: Canonical Markdown + 3-Phase Spatial Thinking

We reverted to strict, canonical 2D Markdown, but enforced a mandatory 3-phase spatial reasoning chain wrapped in native <think> tags:

{
  "messages": [
    {
      "role": "system",
      "content": "You are a Quantitative Financial Specialist. Locate the exact cell coordinates and emit deterministic tool calls. Do not perform mental arithmetic."
    },
    {
      "role": "user",
      "content": "Analyze the operating performance:\n| Financial Line Item | FY2023 ($M) | FY2024 ($M) |\n| :--- | :--- | :--- |\n| Net Sales | 697,290 | 963,286 |\n| Cost of Goods Sold | 412,000 | 540,100 |\n| Operating Income (EBIT) | 122,628 | 142,034 |\n\nQuestion: What was the year-over-year percentage change in Operating Income?"
    },
    {
      "role": "assistant",
      "content": "<think>\n1. Semantic Binding: Target rubric is 'Operating Income (EBIT)'. Found at Row 3.\n2. Coordinate Localization: Base year (N-1 / FY2023) = 122,628; Target year (N / FY2024) = 142,034.\n3. Tool Selection: Relative variance formula -> compute_pct_change(n1=122628.0, n=142034.0).\n</think>\n[compute_pct_change(poste=\"Operating Income (EBIT)\", n1=122628.0, n=142034.0)]"
    }
  ]
}

By decoupling spatial localization inside <think> from the final executable payload [compute_pct_change(...)], our tool invocation accuracy surged to 99.4% on out-of-distribution corporate filings.

2. The Training Recipe: QLoRA with Unsloth

Training a specialized financial compiler does not require a cluster of H100s. We trained our model (targeting NVIDIA Nemotron 3.5 Bolt E4B and Liquid LFM 8B) using Unsloth and QLoRA on a single budget-friendly GPU instance (16GB–24GB VRAM).

flowchart LR
    Base["Base Model<br/>(4B-8B Parameters)"] --> NF4["4-Bit NormalFloat (NF4)<br/>BitsAndBytes Quantization"]
    NF4 --> LoRA["LoRA Adapters (r=16, alpha=32)<br/>Targeting all linear projections"]
    LoRA --> Masking["Response-Only Loss Masking<br/>(train_on_responses_only)"]
    Masking --> Kernel["IEEE-754 Grounded Specialist"]

1. Adapter Configuration

We targeted all linear projection layers to prevent representational bottlenecks:

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.0,  # Optimized for Unsloth fast kernels
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=42,
)

2. The Secret Sauce: Response-Only Loss Masking

If you train an SFT model on entire conversational pairs, backpropagation calculates gradients across the entire prompt—including the 1,500-token financial table. This causes two major failures:

1. It wastes 70% of compute calculating loss on tables the model should only read.
2. The model starts memorizing specific corporate names and numbers instead of generalizing parsing logic.

We used response-only masking so gradients were only calculated on the assistant's output:

from unsloth.chat_templates import train_on_responses_only

trainer = train_on_responses_only(
    trainer,
    instruction_part="<|im_start|>user\n",
    response_part="<|im_start|>assistant\n",
)

3. Hyperparameters & Convergence Speed

Here is our production training configuration:

• Optimizer: adamw_8bit
• Learning Rate: 2e-4 with cosine decay and 20 warmup steps.
• Effective Batch Size: 16 (per_device_train_batch_size=4, gradient_accumulation_steps=4).
• Epochs: 2 epochs is the empirical sweet spot. (Going to 3 epochs on 16k+ samples produced early signs of structural overfitting without improving arithmetic fidelity).

3. The Bizarre Bug: "The Compensation Hallucination Trap"

During early adversarial testing on live M&A balance sheets (including Renault SAS and TotalEnergies filings), we uncovered a psychological pathology inherent to SLMs: The Compensation Hallucination Trap.

The Scenario

We provided the model with a standard Income Statement (P&L) containing Revenues, Operating Costs, and Net Income. We then asked an analyst query requiring balance sheet data:

"Calculate the Debt-to-Equity (Gearing) ratio and the net financial leverage."

A human auditor would immediately stop and reply: "Total Financial Debt and Equity are absent from this statement. Provide the Balance Sheet (Passif)."

What the 8B Model Actually Did

Instead of admitting it lacked the data, the model experienced what can only be described as attention anxiety.

Because RLHF conditions language models to be helpful, the model refused to output an empty string or error. Instead, it searched the table for any numbers that structurally resembled liabilities or equity:

1. It spotted Charges d'exploitation (Operating Expenses = €48,200M).
2. It hallucinated: "This must be Total Debt!"
3. It spotted Résultat Net (Net Income = €2,100M).
4. It hallucinated: "This must be Equity!"
5. It happily emitted:

   [compute_ratio(poste="Gearing Ratio", numerateur=48200.0, denominateur=2100.0)]

The Python kernel faithfully executed the tool call and returned 2295.23%. The model then generated a professional, confident narrative explaining that the company was in catastrophic insolvency!

Caution

The Compensation Trap is far more dangerous than random number hallucination. The numbers are real (they exist in the table), the math is correct, but the semantic binding is completely fraudulent.

Why System Prompts Fail

You cannot fix this with system prompting alone. Telling an 8B model:

"If a variable is missing, do not guess. Say you don't know."

...fails in long-context scenarios. As the context grows beyond 1,500 tokens, the negative constraint gets washed out by the massive attention weights on the table cells.

sequenceDiagram
    autonumber
    participant A as User / Analyst Query
    participant M as 8B Specialist Model
    participant K as Hardware Python Runtime

    Note over A,M: Without Circuit Breaker (The Trap)
    A->>M: "Calculate Net Debt / EBITDA" (P&L table only)
    M-->>M: Panic: Debt is missing!<br/>Compensation: Grab Operating Costs!
    M->>K: [compute_ratio(num=OperatingCosts, den=EBITDA)]
    K-->>A: ❌ Plausible-looking False Metric (€0 error, 100% wrong)

    Note over A,M: With Deterministic Circuit Breaker
    A->>M: "Calculate Net Debt / EBITDA" (P&L table only)
    M-->>M: Identifies missing operand: 'net_debt'
    M->>K: [request_missing_data(missing_postes=['dettes_financieres'])]
    K-->>A: 🛑 Execution Halted: Document Request Dispatched

The Architectural Fix: The Abstention Tool Contract

To permanently cure the Compensation Trap, we treated "admitting missing data" not as a conversational refusal, but as an executable tool call.

We introduced a dedicated abstention contract:

def request_missing_data(missing_postes: list[str], rationale: str):
    """Halts execution and returns a typed deficiency schema to the orchestrator."""
    return {
        "status": "DATA_DEFICIENCY",
        "missing_variables": missing_postes,
        "reason": rationale
    }

Then, we synthesized 1,000 adversarial training examples specifically designed to trigger this tool:

• Balance sheet queries evaluated on P&L tables.
• Cash-flow queries evaluated on Balance sheets.
• Non-GAAP metrics whose prerequisites were omitted.

When an LLM realizes that emitting [request_missing_data(...)] yields a high reward and satisfies its completion objective, compensation hallucinations drop to absolute zero.

4. Key Takeaways & What's Next

Fine-tuning a specialized financial SLM isn't about teaching it economics; it's about conditioning its attention heads to behave like an infallible, coordinate-aware compiler.

Coming up in Part 3

In the final installment of this series, we will examine Multi-Agent Orchestration & Enterprise Production Readiness:

• How our Generalist Crew Manager orchestrates multi-step DAGs (e.g., Cross-Table LBO Due Diligence).
• The GenRM Critic: How an adversarial reward model audits the final memo before C-suite delivery.

If you are building autonomous systems in quantitative finance, legal audit, or deterministic tool-calling, drop your thoughts in the comments below!