Decision Ledger:無需 LLM 偵測 Jira 和 Confluence 中互相矛盾的決策
Detecting contradictory decisions in Jira and Confluence without an LLM
Decision Ledger 是面向 Confluence 和 Jira 的 Forge app,使用不呼叫 LLM 的 TF-IDF 和規則,偵測 Jira 工單或留言是否與有效決策矛盾。它會整理頁面和留言中的決策、在工單顯示相關決策,並以約 325 行程式碼實作自動檢查。
AI disclosure: This article was written with AI assistance (Claude), and the app it describes was built with AI assistance too. The code is copied from the public repository, and every example output below was produced by running that code.
Teams make a decision in a Confluence page ("Use PostgreSQL for the billing service"), and three months later someone opens a Jira issue that quietly undoes it ("Switch to MongoDB for the billing service"). Nobody is doing anything wrong. The person filing the issue just never saw the page.
I built Decision Ledger, a Forge app for Confluence and Jira, to catch that. It collects decisions from pages and comments into one ledger, shows related decisions on every Jira issue, and warns when an issue or comment looks like it contradicts an active decision.
- Repo: https://github.com/agebreakco-prog/decision-ledger
- Demo video (3 min): https://youtu.be/w9mOL597N-s
The interesting constraint: the matching makes no LLM calls. It's TF-IDF and a handful of rules, about 325 lines in src/lib/match.js. This post explains why, and how it works.
Why not just ask an LLM?
I considered it. Three things pushed me to a deterministic approach:
- It runs on every issue and every comment. That's a lot of calls for a team of 50, and each one would cost money or quota. TF-IDF over a few hundred decisions costs microseconds and fits inside Forge's free allowance.
- People need to know why. A warning that says "this might conflict (0.73)" gets ignored. A warning that says Decision chose "postgresql" for billing service; this proposes "mongodb" instead can be checked in two seconds.
- False positives are expensive. An app that posts wrong warnings on Jira issues gets uninstalled. Rules are predictable: when they're wrong, you can see which rule fired and fix it.
The app does have a Rovo agent for open-ended questions ("what did we decide about refunds?"), and that uses Atlassian's own AI. But the automatic checks are plain code.
Step 1: topic tokens, not all tokens
The key idea is to split each text into two kinds of words:
-
Topic words say what it's about:
postgresql,billing,service,refunds. -
Stance words say what we're doing about it:
use,adopt,stop,drop,ban,not.
If you put stance words into the similarity vectors, "Use Postgres" and "Stop using Postgres" look less similar than they should, and two unrelated "use X" decisions look more similar. So stance words are removed from the topic vector and used separately for polarity.
// Words that express a stance (do / don't) rather than a topic. They are
// excluded from topic vectors but drive polarity detection.
export const POSITIVE_STANCE = new Set([
'use', 'adopt', 'keep', 'enable', 'allow', 'require', 'choose', 'chose', 'pick', 'select', 'switch', 'migrate',
'approve', 'introduce', 'add', 'standardize', 'standardise', 'mandate', 'continue', 'support', 'build', /* ... */
]);
export const NEGATIVE_STANCE = new Set([
'avoid', 'stop', 'drop', 'deprecate', 'remove', 'ban', 'forbid', 'disallow', 'reject', 'abandon', 'sunset', /* ... */
]);
export const NEGATIONS = new Set(['not', 'no', 'never', 'without', 'nor', 'neither', 'none', 'cannot']);
/** Topic tokens: stopwords, stance words and negations removed, then stemmed. */
export function topicTokens(text) {
return rawTokens(text)
.filter((t) => t.length > 1 && !STOPWORDS.has(t) && !NEGATIONS.has(t))
.filter((t) => !POSITIVE_STANCE.has(t) && !NEGATIVE_STANCE.has(t))
.filter((t) => !/^\d+$/.test(t))
.map(stem);
}
Running it:
topicTokens('Switch to MongoDB for the billing service') -> [ 'mongodb', 'bill', 'servic' ]
topicTokens('Use PostgreSQL for the billing service') -> [ 'postgresql', 'bill', 'servic' ]
The stopword list also contains words like decided, agreed, team and quarter. They appear in almost every decision and carry no topic.
The stemmer is a 14-rule suffix stripper ("Porter-lite"). It's not linguistically great, but it's stable and predictable, which matters more here. To keep the UI readable, the app maps stems back to the words the user actually typed, so it shows "billing", not "bill".
Step 2: TF-IDF and cosine similarity
Each decision becomes a TF-IDF vector over its topic tokens. Document frequency is computed across the ledger, so a word that appears in every decision (say, your product name) gets a low weight automatically:
export function buildIndex(decisions) {
const df = new Map();
const docs = decisions.map((d) => {
const toks = topicTokens(decisionDocText(d));
for (const t of new Set(toks)) df.set(t, (df.get(t) || 0) + 1);
return { d, toks };
});
const N = docs.length;
const idf = (t) => Math.log((N + 1) / ((df.get(t) || 0) + 1)) + 1;
for (const doc of docs) doc.vec = vectorize(doc.toks, idf);
return { docs, idf, N };
}
export function vectorize(tokens, idf) {
const tf = new Map();
for (const t of tokens) tf.set(t, (tf.get(t) || 0) + 1);
const vec = new Map();
let norm = 0;
for (const [t, c] of tf) {
const w = (1 + Math.log(c)) * idf(t);
vec.set(t, w);
norm += w * w;
}
norm = Math.sqrt(norm) || 1;
for (const [t, w] of vec) vec.set(t, w / norm);
return vec;
}
Sublinear TF (1 + log(c)) stops a long page that says "billing" twelve times from dominating. The IDF is smoothed (+1 on both sides) so a ledger with three decisions doesn't produce infinite or zero weights. Vectors are sparse Maps and L2-normalised, so cosine similarity is just a dot product over the shared keys.
This alone powers the Related decisions panel on Jira issues. Shared Jira labels give a small boost (+0.1 per shared label), and each result lists the matched terms so the user can see why it's there.
Similarity alone can't detect a contradiction, though. "Use PostgreSQL for billing" and "Switch to MongoDB for billing" are very similar (0.59 in my test ledger). That's exactly why they conflict: they're about the same thing and they disagree. The rules handle the "disagree" part.
Step 3: five explainable contradiction rules
detectConflict(text, decision) computes the similarity, then runs five checks. Each one that fires adds a plain-English reason and raises the confidence. The result is flagged only when confidence reaches 0.45.
Rule 1: different choice, same scope
parseChoice() pulls "use X for Y", "migrate Y to X" and "X over Z" patterns out of each sentence with a few regexes:
parseChoice('Use PostgreSQL for the billing service')
-> {"choice":"postgresql","scope":"billing service","rejected":[]}
parseChoice('Expose the public API as GraphQL over REST')
-> {"choice":null,"scope":null,"rejected":["rest"]}
If both texts have a choice and a scope, and the scopes overlap by at least half while the choices share nothing, that's a conflict:
if (ca.choice && cb.choice && ca.scope && cb.scope) {
const scopeOverlap = overlap(tokenSet(ca.scope), tokenSet(cb.scope));
if (scopeOverlap >= 0.5 && overlap(choiceA, choiceB) === 0) {
reasons.push(`Decision chose "${ca.choice}" for ${ca.scope}; this proposes "${cb.choice}" instead.`);
confidence = Math.max(confidence, 0.6 + 0.3 * scopeOverlap);
}
}
Note that negated phrases ("stop using X", "do not use X") are removed before choice parsing. Otherwise "do not use MongoDB for billing" would be read as choosing MongoDB.
Rule 2: a rejected alternative comes back
"GraphQL over REST" records REST as rejected. If a new issue picks REST, or asks for it with positive stance and enough topical overlap, it's flagged with Decision explicitly ruled out "rest". There's a guard so that text which also rejects REST doesn't trigger it.
The reverse also counts (rule 2b): if the new text rejects what the decision chose ("stop using PostgreSQL"), that's a conflict too.
Rule 3: opposite stance on the same topic
polarity() scores a text +1 (for), −1 (against) or 0 (unknown) by counting stance words and flipping any that have a negation in the three words before them ("do not use"). A bare negation such as "no production deploys" counts as −1.
if (pa && pb && pa !== pb && similarity >= 0.3) {
reasons.push(
pa > 0
? `Decision is in favour ("${truncate(d.text, 80)}"); this text argues against it.`
: `Decision rules this out ("${truncate(d.text, 80)}"); this text argues for it.`
);
confidence = Math.max(confidence, Math.min(0.95, 0.35 + similarity));
}
There's a variant (3b) for bans. "No production deploys on Fridays" has polarity −1, but "Deploy the hotfix to production on Friday evening" has polarity 0, because "deploy" isn't a stance word. If the decision is a ban, the new text has no negation of its own, and similarity is at least 0.35, it's treated as proposing the banned thing.
Rule 4: antonyms on the same topic
A short list of mutually exclusive pairs: sync/async, public/private, weekly/monthly, mandatory/optional, build/buy, opt-in/opt-out and so on. Some are prefixes (automat, centraliz) so they match several word forms.
This rule gates on the larger of the IDF-weighted similarity and a plain unweighted one. The comment in the code explains why:
// Gate on the larger of the IDF-weighted and the plain term overlap: IDF shifts as the ledger grows,
// and a few shared words ("order confirmation email") are enough context for an antonym clash.
The final gate
// Gate: everything except an exact choice/scope clash needs some topical overlap.
if (similarity < 0.12 && confidence < 0.6) confidence = 0;
return { conflict: confidence >= 0.45, confidence: round(confidence), similarity: round(similarity), reasons };
Superseded decisions are excluded from the pool entirely, so once a team records "we moved billing to MongoDB" and supersedes the old decision, the warnings stop.
What it outputs
Here is a five-decision ledger and some new issue titles run through findConflicts(). These are real outputs from the repo code:
| New text | Flag | Confidence | Reason |
|---|---|---|---|
| Switch to MongoDB for the billing service | yes | 0.90 | Decision chose "postgresql" for billing service; this proposes "mongodb" instead. |
| Add a REST endpoint for partners | yes | 0.75 | Decision explicitly ruled out "rest". |
| Deploy the hotfix to production on Friday evening | yes | 0.90 | Decision rules this out ("No production deploys on Fridays"); this text proposes doing it. |
| Move support on-call to a monthly rotation | yes | 0.90 | Opposing terms on the same topic: "weekly…" vs "monthly…" |
| Auto-approve refunds up to $1000 without finance review | yes | 0.84 | Decision is in favour ("Refunds over $500 require finance approval"); this text argues against it. |
| Add dark mode to the settings page | no | – | – |
Where it falls short
Being explainable also makes the failures easy to see. Some real ones:
Wording sensitivity. "Auto-approve refunds up to $1000 without finance review" is flagged. "Auto-approve refunds up to $1000 without review" is not. Without the word "finance", the IDF-weighted similarity to "Refunds over $500 require finance approval" drops to 0.27, just under rule 3's 0.3 threshold. Tested against that decision alone, with flat weights, it scores 0.33 and is flagged. So the same sentence can pass or fail depending on what else is in the ledger. That's the cost of corpus-dependent weighting.
No synonyms. "Which database does billing use?" finds the PostgreSQL decision only through "billing". The word "database" never appears in the decision, and there are no embeddings to connect "database" with "PostgreSQL".
Odd reasons from regex parsing. For "Keep the support team on a weekly on-call rotation", the choice parser reads "support team" as the choice and "weekly on-call rotation" as the scope. The flag is still correct, because the antonym rule fires too, but the first reason it prints reads strangely.
English only. The lexicons, stemmer and regexes all assume English.
I decided these were acceptable for a warning system. It only needs to be right often enough to be worth a glance, and it must rarely be loudly wrong. The thresholds were tuned against the demo scenario and the extraction tests (50 tests in total, run with node:test). An embedding model would fix the synonym problem. If I add one, I'd use it only to find candidates and keep the rules for the final decision and the reason.
Getting decisions into the ledger
Matching is only useful if the ledger fills itself. Extraction (src/lib/extract.js) is rules too:
- Native Confluence decision items (
/decisionin the editor), only in the DECIDED state. - Everything under a heading called Decisions, Decision log, Outcomes or What we decided.
- Decision-log tables with a Decision column.
- Lines like
Decision: …,Agreed: …, and sentences like "We decided to …".
Rationale is split off at because / since / due to / so that, so "Use PostgreSQL for the billing service because we need ACID transactions" stores the why separately. The rationale is included in the TF-IDF document for relatedness, but contradiction checks use only the decision text, so a rationale that mentions an alternative can't trigger a false flag.
Takeaways
- Separate topic from stance before you compute similarity. It helps more than any tuning.
- TF-IDF tells you what two texts share. You need something else to tell whether they agree.
- A small set of rules, each with its own reason string, is easy to debug, cheap to run and easy to trust.
- Write down the misses. They show you where a heavier model would actually help.
The full source, including the extraction rules, the Forge manifest and the tests, is at https://github.com/agebreakco-prog/decision-ledger.
來源:DEV Community · dev.to