ArchDocs 記錄編碼智能體反覆遇到的已知問題

My coding agents keep running into problems we’ve already found

On November 1, my coding agent ran into an error while writing a design doc. It tried to put inline code inside bold text, and our editor rejected it:

Invalid collection of marks for node text: bold,code

The agent told me the write failed "because I nested code inside bold text", removed the bold and tried again. That worked. A few minutes later, it saved a note to a memory file on my machine saying this was a product bug we should fix.

About an hour later, the same agent tried importing 127 of our own design documents. 102 failed with that same error.

I can't tell from the session whether the agent deliberately put the bug aside. It was still writing the spec, nothing had been built yet, and the design doc's Work Log was empty. The first entry came twenty minutes later.

What I can see is that it treated the error as a problem with what it had written, changed its input and moved on. The bug ended up in a local memory file, with no way to reliably bring it to whoever worked on the editor next.

It had actually happened earlier that morning too. Another session hit the same error while writing a different design doc and worked around it the same way.

When the bulk import failed, the agent finally recorded it in the design doc. Within fifteen minutes, the Work Log had the blocker, the fix, why the fix went outside the plan, and a finding for whoever worked on that code next.

I'm Aylon, and I'm building ArchDocs. Most of the implementation is done by coding agents, and this is one of the problems I'm trying to solve with it. I want to know what we intended to build, what actually happened, how we checked it, and what the next person or agent needs to know.

We use ArchDocs to build ArchDocs. This is one of those changes, including the things that went wrong.

One record per change

A design doc in ArchDocs has a few parts. The spec describes the problem and the decisions we've made. The implementation plan describes how we'll do the work, what we expect to touch and how we'll check it.

While the work happens, the agent records progress, decisions, deviations and findings in the Work Log. At the end, it submits evidence of work covering what was done, how it was verified, what changed from the plan and what wasn't finished.

This is the design doc used throughout the post. The header shows that I decided it shipped, my agent relayed that decision, and no acceptance criteria were configured.

The spec and plan give us something to compare the actual work against. Shipping freezes them, and changing them afterward requires an amendment with a reason. The Work Log and evidence keep the history of what happened during implementation.

We don't create a design doc for every change. Our skills tell agents to propose one when work crosses components, needs someone else's sign-off, will take multiple pull requests or sessions, or introduces a dependency, data store or new personal data. A small change inside one component, a dependency bump or a copy or config edit doesn't need that process.

The change, and the decisions I made

The change was DD-0022, which added importing for existing design docs, RFCs and ADRs.

Before this, documents had to be brought in individually as fresh drafts. We didn't preserve their original author, date or identifier, and importing the same document twice created two copies.

That also meant a team adopting ArchDocs started with an empty history. There was nothing for search or task context to find until someone wrote new documents. In our own repository, we already had 110 specs and 176 implementation plans.

The agent drafted the spec and raised four questions for me to answer.

One was whether imported historical documents could be created directly as Shipped. Normally, our workflow has a person decide when a design doc has shipped. I allowed historical imports through a separately authorized code path.

Another question was about how to handle documents that combine the design and implementation plan.

The agent gave me two options and marked its preference. I wanted a third option. My answer stays attached to the question with my name and a timestamp.

The agent preferred importing the whole document into the spec. That was predictable and would work with AI turned off. The other option was to use AI to split it into a spec and an implementation plan.

I said the person importing should be able to choose.

Version 3 of the spec added that choice: "Keep each document whole" or "Split into spec and plan", with an override for individual documents. Splitting was only available with AI enabled. The spec also recorded that this was decided in Q4 and kept the original options as rejected alternatives.

If someone later asks why the import dialog has that setting, the answer is there.

We didn't require a formal approval step for a change this size. Answering those four questions was the sign-off.

What the agent keeps up, and what stays yours

I wouldn't want to write down every commit, test result and unexpected issue myself while also trying to build the feature. Keeping that record is something I ask the agent to do.

I use Claude Code with the ArchDocs plugin, connected through MCP. The plugin includes skills that tell the agent when to propose a design doc, how to write it, what to record during the work and how to close it out.

The workflow looks like this:

1. The agent drafts the spec and plan, asks questions and retrieves related context. I answer the questions that need a decision from me.

2. During implementation, the agent logs commits, test results, deviations and findings. It leaves a handoff when it stops. With GitHub connected, ArchDocs checks the commit references.

3. The agent writes the evidence of work, including anything it didn't finish. I review the results and decide whether the work shipped.

4. Before related work starts, the agent requests task context. ArchDocs returns relevant records with their sources and status. I can accept or reject constraints proposed by the agent.

For DD-0022, the agent wrote a twelve-task plan covering the API, frontend and API contract, with checks for each task.

The plan also included things we'd learned from previous work. For example, the plugin needed to be released after the API, because its skill would call a tool that didn't exist in production until the API deployed. A new session wouldn't necessarily know that from reading the code.

The plan missed something

Task 6 was where we tried the import against our own documents.

We had 286 files in docs/superpowers/, mapped to 127 documents. The agent imported them into a local organization through the real MCP tool.

25 succeeded. 102 failed.

The agent recorded the failure as a blocker and connected it to the error it had already encountered while writing the spec.

Our editor uses Tiptap. By default, its inline code mark excludes other marks, so markdown like **`createdAt`** failed the schema check and caused the entire document to be rejected.

The feature followed the plan, but it couldn't import four out of every five documents we gave it.

Nine minutes later, the agent logged a fix.

The fix went into a package the plan hadn't included, so the agent recorded it as a deviation. The entry includes what changed and how the import rerun went. I'll get to the “Not found” reference and correction below.

This also affected normal document editing and the agent's own writes. Bold or italic code is valid GitHub-flavoured markdown, and we were rejecting it.

The first fix wasn't complete, though. It stopped rejecting those documents, but the editor ignored the override it relied on. The writes succeeded while silently dropping the bold or italic formatting around the code.

A follow-up commit that night, eb7c428a, extended the code mark itself and added tests checking that those marks survived. The full entry shows the constraint captured from the first fix.

Four minutes after the first fix, the agent captured the finding as a constraint associated with the affected files: keep inline code combinable with bold, italic and links, and keep the round-trip tests for those cases passing.

That doesn't automatically make it an approved rule. A finding captured from the Work Log remains proposed until a person accepts it. ArchDocs labels it "From a Work Log finding, not an approved rule."

The code fix and tests address the bug. The Work Log records what happened and why we went outside the plan. Task context is how that finding can reach the next agent working on those files.

There were two more issues during the import. Two files contained NUL characters that Postgres wouldn't store. After fixing that, the third run imported all 127 documents.

The Notion import also tried attaching images after documents had already been marked shipped and frozen. Image handling had to move inside the import. Both issues went into the evidence as deviations from the plan.

A log the next person can use

By the end of the build, DD-0022 had eighteen Work Log entries.

These covered commits, actual test counts, results from delegated work, review findings and departures from the plan. For this change, Codex handled the implementation from a work order. Claude reviewed the results, ran checks and kept the record.

Some entries captured decisions that would otherwise have been buried in the code.

E-2 had a section called "Calls I made that the plan left open". It listed four small product decisions the agent made during implementation. I could review those decisions directly and change them if needed.

E-13 recorded three existing failures in an unrelated test suite so the next person wouldn't assume they were regressions from the import.

There are limits to this. The log only contains what the agent records, and a finding still has to be captured for it to become a constraint. We learned that the marketing site had tests checking the docs pages and the number of MCP tools. That stayed in a Work Log entry and was never captured as a constraint.

When the agent stopped, it left a handoff.

The handoff links completed work to its supporting entries, lists the decisions waiting on me and explains what to run next.

I used this myself when I came back to write the post and correct the record. My agent called get_journal_handoff first.

It returned E-17, but marked it stale because E-18, which recorded shipping, had been added afterward. It also reported that there were no open questions.

The evidence

DD-0022's evidence of work had four sections: outcome, verification, deviations and not done.

The verification section separates CI results from local runs and labels the import result as reported. The agent wrote all of these results.

For this change, we hadn't configured acceptance criteria. ArchDocs didn't independently compute whether the change passed them. The header says that, and the evidence was a record of the tests and CI results the agent reported.

I checked those results myself using the GitHub CLI.

Run 41310251591, on the final commit, passed the build, unit tests, integration tests, Docker image build and both deploys.

The end-to-end job was skipped. The "5 of 5" Playwright import result in the evidence came from a local run reported by the agent, not that CI run.

The previous run, 41309759807, had failed on two marketing tests that weren't covered by the agent's local checks. The Work Log recorded those failures and the fix. The runs are in our private repository, so I can't link them here.

I made the decision to ship DD-0022. I told the agent:

"Ship it and then tell me how I can try out the import on a few docs."

It attached that statement when marking the doc shipped. The header shows that Claude Code relayed the decision.

Because this doc had no acceptance criteria, the server would have allowed the agent to ship it without that statement. It waited because our skills tell it to wait for a person's decision. Relaying my statement is what put my name on the record.

While checking the record for this post, I found another mistake.

Every commit reference used archdocsio/archdocs. Our repository is archdocsio/monorepo.

ArchDocs had been checking those references against GitHub and displaying "Not found". The same incorrect repository name was also attached to the captured constraint's file paths.

Work Log entries and captured file paths can't be edited. We added Correction entries and a new constraint to replace the original. The old entries remain visible with a "Corrected by" label, and the corrected commit references resolve on GitHub.

The new constraint only replaces the old one once a person accepts it. The correction entry shows the changes.

The next change starts with what we learned

Let's say the next change adds strikethrough support to the editor.

That would touch packages/doc-editor/src/extensions.ts, the same file where we fixed the code mark. An agent could reorganize the extensions while adding strikethrough and accidentally remove the behavior that lets code combine with bold, italic and links.

We'd be back to rejecting imports and agent-written documents.

The next agent needs to know a few specific things: preserve that behavior, keep the round-trip tests passing, and test markdown like **`createdAt`**.

When an agent follows our skills, it requests task context before planning, using a description of the work and the files it expects to touch. ArchDocs retrieves the decisions, constraints, open questions and design docs associated with those files, including their sources and status.

Nobody has to tell it to go read DD-0022.

We tested that query after correcting the repository name, using the real repository and the two files the strikethrough change would touch. We haven't implemented that change. This was a demonstration of what the next agent would retrieve.

The constraint appears first, matched by file path, with the original Work Log entry and its current status.

Before the correction, the query found the constraint through text search, in third place. It couldn't match the file paths because the repository name was wrong.

After the correction, it was the first result, matched by path.

The query used no catalog objects. The file paths on the constraint were enough. The full result also includes the original capture, labelled to show that a newer item proposes to replace it.

A human reviewing the next change can retrieve the same context, check the constraint against the implementation and decide whether to accept it.