VEKTOR 2.0.2：LongMemEval 記憶基準測試結果

Desk Trip Planner

VEKTOR 2.0.2 How we got there, via our custom Minimax matrix and a lot of testing

We measured where answers were being lost, changed one thing at a time, and kept a no-change control beside every test.

With the same open models throughout, the shipped build scored 89.2% on a paired check of 120 random LongMemEval questions, against 84.2% for the build before it. On the full 500 questions it scored 84.0%, the full run with the 30 preference questions taken from the shipped build (27 of 30).

Minmax: Each row is a candidate setting and each column is a kind of question:

Temporal, counting, updated facts, single-session lookups, preferences.

We run a candidate on a small set of questions from every column and score it by its worst column, since a setting that lifts one category by breaking another is not an improvement.

The winner is the row whose weakest column is highest. Each candidate runs next to a no-change control, because the same run twice moves a few answers by chance.

Each also runs against a guard sample of questions that were already right. Extraction is cached per question, so a test that only changes recall or the answering step takes minutes, not hours.

We tune on one set of questions and confirm on another. That is how the recall sizes (12, 80 and 120), the saving instruction, and the answering effort were chosen, and how the reranker, the paper-style expansion and duplicate removal were ruled out.

The newly built switch is in the preferences panel. The rest of the release upgrades are listed at the bottom.

Benchmark results: https://vektormemory.com/research

LongMemEval is a public benchmark of long chat histories with questions about them. It has six categories and 500 questions. We ran every question through the real memory code, with a fresh database per question, and graded the answers with the benchmark’s own rules.

Every test had a control that changed nothing, because the same run twice moves questions by chance: on 90 missed questions, 8 turned right on a plain repeat. And every change was checked against a sample of questions that were already right, so a gain in one place could not hide a loss in another.

Dates written by code

The first deduction came from reading the misses. The model was asked to turn “last Friday” into a date, and it guessed. So code now does it, from the conversation’s own date: “last Friday” becomes 2023–03–10, “four weeks ago” becomes a target day, and the events near it are marked in a sorted timeline. Temporal questions went from 63.8% to 80.3% in the first lab run (graded by Llama 3.3 70B).

In Preferences the switch is called “Remember when things happened”. Every saved fact also starts with the day it was said, at no extra cost.

The right amount of memory for the question

The second deduction came from counting the misses by cause. Of 61 multi-session misses, 52 were “how many” questions that had been shown only 12 memories. The answer was in the history, and only part of it was returned.

Recall size now depends on the question. Plain questions return 12 memories (it was 5). Counting and totalling questions about your own history return up to 80. Order, duration and relative-day questions return up to 120. A short tested check recognises each kind, so a plain lookup such as “how much does the plan cost” keeps the small size.

The results, on the same saved memory and the same judges: multi-session questions went from 54.1% correct at 12 memories to 75.9% at 80. Knowledge updates, where a fact changed over time, went from 74.4% at 5 memories to 85.9% at 12 (that curve graded by Llama 3.3 70B). Recall took the same time at 5 and at 12.

Saving the specifics

The third deduction: about 34 of the 90 questions still wrong were never saved at all. The saving step wrote summaries and dropped the named items. One added instruction now keeps every named item from both sides of a conversation, with list items and their position, and never keeps secrets or health records.

On the 52 previously missed questions it fixed 16, against 5 for a repeat with no change, and 39 of 40 right answers stayed right. It stores about 1.7 times as many memories and makes no extra model calls.

We measured the path users actually run

Our test harness had been calling the raw memory core, not the recall the app uses, which fuses keyword, meaning, graph and sentence search.

Measured through the app’s own recall, 29 of the 90 misses moved to right, against 8 for the control. Together with the saving change, 47 of the 90 were fixed against 8, and 30 of 32 guard questions stayed right.

On a paired check of 120 random questions, the shipped build scored 89.2% against 84.2% for the earlier build with the gpt-oss judge, and 85.8% against 82.5% on the majority of three judges.

Finding near-duplicates

The remaining misses are mostly lost counts. We traced them to the exact conversation turns that hold the evidence. Across 40 multi-session questions and 101 evidence turns, the saving step kept 94.1%, the app’s recall returned 77.2% at its normal size and 90.1% at double the size.

Then we tested the ideas that should have closed that gap. A cross-encoder reranker kept the same score at 24 memories and lost at 12. Expansion borrowed from the SmartSearch paper found no new evidence on top of our graph and sentence search.

Removing near-duplicates lifted the evidence returned from 77.2% to 84.2%, but the answers did not improve. A period computed in code for “in the past two weeks” made no difference. A preference-focused saving instruction scored no better than the control.

Each result narrowed the search. With nearly all the evidence in front of it, the answering model still miscounts, so counting is limited by the answerer, not by retrieval.

Calibrating the judges

Preference questions are graded against a rubric, and two judges disagreed. We read all 30 answers ourselves. The gpt-oss judge agreed with a careful reading on 25 of 30, and the Qwen judge on about two in three. We graded preference with the judge that matched the audit and published the audit file. Under that judge, preference is 83 to 90%, close to the 90% other memory systems report.

We also stopped re-running everything. Weighting a small set of missed questions and a sample of right ones to their real sizes reproduces a full run (81.9% estimated against 82.0% measured), so a change can be judged in minutes.

Update to 2.0.2, open Preferences and turn on “Remember when things happened”. Then ask your memory a question about time, or how many of something you did.

What the numbers say from the oracle

We used the oracle variant of LongMemEval, which hands the system only the conversations that hold the evidence. The full variant buries them among about 50 sessions. Scores on the two are not comparable, and published results are usually on the full one.

Each row is one run, and cloud models are not fully deterministic. Every claim above rests on a control or a paired run.

Settings were tuned on the same questions we report. The random paired check reduces that overlap but does not remove it.

Everything here uses open-weight models available. Saving and answering: gpt-oss:120b. Grading: Qwen3.8–27B, gpt-oss:120b and nemotron-3-ultra. The earlier GPT-4o-mini grader has been retired and hence not used on this run.

Judges are models, and the preference audit is one reader’s. The file is public so you can check it.

We tried a larger answering model (beam, 501 billion parameters) on the questions still wrong. It fixed more of them, and the difference from gpt-oss was within the noise of the test, so it is not the shipped setting.

We would put forth a perspective that these tests were reproduced on smaller models; better recall/retrieval results will be achieved with larger frontier models in the field.

Everything else upgraded in 2.0.2

Cards and Desk

Answers as cards: plans, comparisons, checklists, timelines, charts, slide decks, photo strips and tiles, journey maps and where-to-eat sections. A Look button cycles four looks, and a fifth uses your Studio brand.

A plan is a living document. Press a button on a plan card and the reply updates that same card as it arrives.

Add and suggest inside the card. Checklists, trip days, steps, timelines, card sets, boards and food lists take new entries by hand, or show four ideas that fit the plan. Tables, comparisons, numbers and budget sliders take entries but no ideas, since ideas there would be invented figures.

Start-screen cards: a Today row for what is overdue or due soon, forms for goals, tasks, reminders, projects, trips, reviews, recaps and budgets, and a box for your own notes or numbers.

Save (as a note in Jot) and Email on every card answer. Long answers become a report card with Expand, Share and a side reader.

Copy table and Copy as picture on every table, so a table keeps its shape in Docs, Word, Notion, Sheets and Medium.

Panels and tools

Find anything (Ctrl+K): panels, memories, notes, tasks and books in one pop-up, with tabs, recent picks and a New menu.

Customise the left rail: hide panels, reorder them, saved in your browser.

Connectors as a tile grid. A Connect an app button on the MCP panel with the one command and a Copy button.

Agent activity feed in Transparency. Routines gallery in Workflows with eight ready-made routines. Persona cards in the Agent Profile.

Four more Convert tools: TypeScript to JavaScript, JSON5 to JSON, JSON with comments to JSON, and Markdown with code blocks to one working page.

Memory and answers

Better answers when a fact has changed: ordinary recall returns 12 memories instead of 5.

Better answers to counting and totalling questions: up to 80 memories, so “how many” questions see the whole history.

Order questions such as “which happened first” are now recognised, so they get the timeline and the wider recall.

Off by default: keep the original wording of past chats with VEKTOR_RAW_CONVERSATION=on. Keys, tokens, card numbers and passwords are blanked, lines that read like health records are never kept, and pieces are deleted after 180 days. On the 56 assistant questions it scored 53 against 51 without it, so it is a choice, not a default.

Models and speed

The local model reads the start of a turn up to about 30% sooner. Set VEKTOR_OLLAMA_NUM_BATCH=default to go back.

OpenRouter: more than a dozen newer models, plus Kimi K3.1 Pro, Meta Muse Wonder and LiquidAI LFM2.5 24B. Ollama Cloud: Beam.

Saving a conversation can use Ollama Cloud.

Polish

One icon style across the app. A lighter, greyer teal theme. The notes side panel stays closed until you open it.

Card buttons act at once and no longer leave their sentence in the message box.

Version 2.0.2 is live in the downloads page: www.vekmem.com

Run your own race, and keep your data sovereign

Vektor Memory

AI

Generative Ai Tools

Longmemeval

Ai Memory