Monomem 與 agentmemory 的 LongMemEval-S 評測差異未達統計顯著
I re-ran a rival's memory benchmark. It tied mine. Here is the interesting part.
Monomem 與 agentmemory 在 LongMemEval-S 評測中的分數差異未達統計顯著,作者將結果視為平手。作者以同一數據檔重跑 agentmemory 自行發佈的評測,重現 84.7%;Monomem 得分 86.0%(470 題中答對 404 題),exact McNemar test 的 p = 0.29。
I build Monomem, shared memory for AI coding agents. It records which app wrote each note and holds suspicious writes until a person looks.
On LongMemEval-S, a standard long-term-memory test, a well-known memory project (agentmemory) publishes 84.7%. I re-ran its own evaluation on the same data file and got 84.7% back, exactly.
- Monomem scored 86.0% (404 of 470 questions).
- Question by question, 14 were answered only by Monomem and 8 only by agentmemory.
- That gap is not significant (exact McNemar test, p = 0.29). It is a tie.
Two more things I will not hide:
- Monomem's ranking was tuned on half of this question set, so the fairer figure is the other half: R@5 97.5, Rall@5 86.2. The repo says so.
- Another system, Vestige, reports a higher score than mine (its own number, which I have not re-run). It is named in the docs.
Why post a tie? Because memory search is turning into a commodity. What I think still matters is trust: who wrote a note, whether an outside web page could have poisoned it, and whether you can undo it. Monomem's own attack check (monomem check --attacks) replays 26 known poisoning attacks on a copy of your store. Two are known misses (a reworded override, and self-poisoning), and the README lists them next to the headline number. It is pattern matching, so a reworded attack it does not know can get through.
Everything runs on your computer in one file: no cloud, no account. MIT licence. Beta, not independently security-reviewed.
Reproduce the numbers: monomem bench --longmemeval ... --per-question out.ndjson (instructions in docs/BENCHMARK.md). If you can beat or break it, tell me.
Repo: https://github.com/calasdan-sketch/monomem
Using it at work? Team setup and support: https://calasautomations.com/devtools/monomem/#teams
來源:DEV Community · dev.to