領域持續預訓練前的檢查清單

Your team picked an open weight checkpoint and a corpus. The tempting move is to schedule GPUs and iterate on the mix later. Joel Hron at Thomson Reuters and Fergal Reid at Intercom both describe training as staged pipelines where evaluation sits beside the data, and Hron treats retained capabilities as an explicit objective. I'd treat the run as blocked until you can answer the questions below.

What training stage are you actually running?

Most product teams are not pretraining from scratch. They adapt a base model in layers. Continued pre-training adds more unlabeled domain text on top of an existing checkpoint. Supervised fine-tuning and preference optimization teach labeled behavior. Agentic reinforcement learning teaches tool use inside a harness.

Hron maps Thomson's stack in that order: realign the open source starting point (Imperial College's Snowdon model, from the Qwen 3.5 family), then continuous pre-training on selected Thomson Reuters content (Westlaw, Practical Law, Checkpoint, Reuters News Archive), then fine-tuning and direct preference optimization with human experts, then agentic RL with access to TR applications. Less than 10% of the content corpus had been used for continuous pre-training at the time of the interview, with room to grow.

Reid's path at Intercom started with parameter-efficient experiments such as LoRA and moved toward distributed training on large GPU nodes, combining full supervised fine-tuning with reinforcement learning for harder policies. The stage label matters because each phase needs different eval slices and different failure modes.

Why build the eval set before the run?

Do not treat public benchmarks as a substitute for job-shaped tests. Hron pairs curated training data with "good evals that represent those jobs equally well." Thomson uses CoCounsel Bench (on the order of thousands of legal queries, growing almost daily) plus internal legal benchmarks and public legal benchmarks. For retention, the team tracks roughly ten or eleven general dimensions (legal, tax, journalism, math, coding, instruction following, general reasoning, long context, and others) and describes a technical-paper graphic, due the week after the interview, comparing Qwen, Snowdon, and Thomson on a spider-style plot.

Reid is blunter about offline suites versus live traffic: however carefully a backtest or evaluation is built, what happens in production can deviate from it.

Intercom still invests in backtests (real customer hallucination cases, LLM judges on answer quality) but A/B tests in production against resolution signals before shipping. I'd copy the split: freeze an internal suite that mirrors real tasks before training, plan a production gate for anything that touches users, and accept that backtests alone will not catch every confounder (Reid notes latency can shift resolution rates in ways you would never encode offline).

Sketch for scoping eval buckets before CPT (illustration only, not TR or Intercom code):

# Sketch: define eval lanes before any training job
RETAIN = ["coding_snippets", "math_word_problems", "instruction_following"]
GAIN = ["domain_qa", "tool_use_with_apis", "citation_style"]
HOLDOUT = {"never_in_train": True, "source": "prod_failures_last_90d"}

def gate_release(scores: dict) -> bool:
    for task in RETAIN:
        if scores[task] < scores[f"baseline_{task}"] * 0.98:
            return False  # regressions block ship
    return all(scores[t] >= scores[f"baseline_{t}"] for t in GAIN)

What can regress during continued pre-training?

Catastrophic forgetting stops being an abstract research term once your product depends on skills the base model already has. Hron says Thomson Reuters users still need coding, math, and general reasoning from the open source foundation, and a large share of engineering goes into adding TR data without losing that fluency.

But I think the more important thing is that it does not regress in any of those other areas.

Joel Hron, CTO, Thomson Reuters

He was candid that the result was not zero: he described a very small regression in coding and in math.

The glossary framing matches that practice: measure gains on the new domain and performance on capabilities you need to keep. Mixing general data with domain data can limit damage during continued pre-training, but the mixture ratio is a hypothesis until you eval it.

Before you scale corpus coverage (Thomson started small on CPT fraction), I'd require per-checkpoint reports on both domain lift and retention lanes, not a single average score.

Where can reward hacking show up in post-training?

Once you add RL or multi-objective preference training, the score becomes a target. Reward hacking is when the model satisfies the score through a shortcut that misses the intended task, such as getting answers from an unauthorized source. That matters for agentic RL like Thomson's harness over Westlaw and Practical Law, and for Intercom's offline RL stack built around a resolution model plus additional LLM judges (answer length, bullet-point use, and others).

Reid says designing the objective function is "relatively the smaller part" compared with the infrastructure for running models at scale. I'd read that as a warning: a clever proxy (resolution model, LLM judge, benchmark leaderboard) can drift from user value, and auditing how answers were produced is one way to catch it. Intercom also runs production A/B tests so soft resolutions do not rise while hard resolutions fall, which is one guardrail against turning the product into a deflection machine.

If you augment scarce labels with model-generated examples, treat synthetic data as augmentation, validate against real failures, and do not assume more rows equal better training. Synthetic data inherits its generator's biases and blind spots, and heavy training on model output can amplify errors.

Checklist

• Name the stage (continued pre-training, SFT or DPO, agentic RL) and match eval tasks to that stage before you launch jobs.
• Ship a retention panel (general skills you rely on from the base model) and a domain panel (jobs your product actually sells), both measured at every checkpoint.
• Block promotion when any retention lane drops materially, even if domain accuracy spikes.
• For RL or judge-heavy post-training, log how answers were produced and test for proxy gaming; keep a production or shadow A/B gate when live signal exists.
• If you use synthetic pairs, spot-check against real incidents and existing eval harnesses before they enter the training mix.

More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.