作者為 cognee 建立 PubMed 連接器，剖析 AI 助理記憶中的「遺忘」難題

I wanted to see whether an AI assistant could keep up with a fast-moving corner of cancer research instead of answering from whatever it saw during training. So I picked one narrow question: how do KRAS G12C lung and colorectal tumours become resistant to sotorasib and adagrasib? Then I built a PubMed connector for cognee, the open-source memory layer for AI agents, as part of Mergetober.

This post covers how to use it, what actually happened when I pointed it at real PubMed data, and three things about NCBI's API that surprised me.

The problem with "just ask the model"

Resistance to KRAS G12C inhibitors is an active research area, with new case reports and preprints every month. A model's training data is frozen. Pasting a few abstracts into a chat window doesn't scale and is gone the next session.

What I wanted was memory: a store that pulls in the relevant literature, keeps it up to date on its own, and turns it into something an agent can reason over (entities and relationships, not just text chunks). That's what cognee does with documents. It was missing a way to get PubMed into it.

Ten lines to a literature memory

The connector is a dlt source you hand to cognee.remember. You choose what to ingest with an ordinary PubMed query, exactly what you'd type into the PubMed search box.

import asyncio

import cognee
from cognee_community_connector_pubmed import pubmed_source

TERM = '"KRAS G12C" AND (adagrasib OR sotorasib) AND resistance'


async def main():
    await cognee.remember(
        pubmed_source(term=TERM, mindate="2024/01/01"),
        dataset_name="kras_g12c_resistance",
        primary_key="id",
        write_disposition="merge",
        max_rows_per_table=0,
    )
    answer = await cognee.search(
        query_text="Which secondary KRAS mutations cause resistance to G12C inhibitors?",
        query_type=cognee.SearchType.GRAPH_COMPLETION,
        datasets=["kras_g12c_resistance"],
    )
    print(answer)


asyncio.run(main())

No API key is required, because PubMed is public. If you create a free NCBI key, the rate limit goes from 3 to 10 requests per second:

export NCBI_API_KEY="your_key"      # account.ncbi.nlm.nih.gov → Account settings → API Key Management
export NCBI_EMAIL="you@example.org" # NCBI asks tools to send a contact address

What happened on real data

Before involving an LLM at all, I wanted to see the ingest side behave, so I ran the sync twice against live NCBI and counted the requests.

• The query matches 253 PubMed records in total. Limiting it to records added since 2024 left 155.
• The first sync stored all 155 in about 7 seconds, using just two EFetch batches.
• The second sync made zero EFetch calls and finished in about 2.4 seconds. Nothing had changed, so nothing was downloaded or sent to the LLM again.
• 36 of the 155 had structured abstracts. The connector keeps those sections (Background, Methods, Results, Conclusions) as headings, which gives cognify much cleaner text to work with.

Three of the papers mention Y96, a mutation in the switch-II pocket where these drugs bind. One of them, a 2026 case report in The Oncologist (PMID 42474245), describes a patient whose tumour developed both Y96S and Y96C after about nine months on a G12C inhibitor. Adding cetuximab and trametinib to sotorasib didn't hold it back; progression-free survival was 1.9 months.

This is the kind of finding I want an assistant to remember and connect: drug → target → resistance mutation → which combinations didn't work. A frozen model can't know about a paper from this summer, but the memory layer now does.

Under the hood

E-utilities is a two-step API. esearch.fcgi turns a query into PMIDs, and efetch.fcgi returns the full XML for up to 200 of them at a time. The connector parses that XML into Markdown (title, authors, journal, DOI and PMCID, labelled abstract sections, MeSH terms) and writes one row per article. cognee then ingests each row as a document and runs entity extraction on it.

Getting that basic loop working took an afternoon. Making it correct took longer, because of these three surprises.

1. ESearch stops at 10,000 results

ESearch never returns more than 10,000 PMIDs for a query. My first test, plain CRISPR gene therapy, matches 10,930 records. A naive loop would silently drop the last 930.

The fix: if a date window has more than 10,000 hits, split it in half and search each half, repeating until every slice fits. On the CRISPR query that took 13 requests and listed all 10,930 records with no duplicates.

2. Entrez dates are in US Eastern time

Incremental sync uses the Entrez date (edat), the day a record entered PubMed. After each run the connector saves the last day it covered and searches from there next time.

NCBI assigns those dates in US Eastern time, though, so a run just after midnight UTC can miss records added late on the previous Eastern day. The connector re-reads one day of overlap and skips PMIDs it already has, so the overlap costs one extra search rather than re-ingesting anything.

3. A retraction doesn't change the date

A memory layer has to be able to forget, especially in medicine. If a paper is retracted, you don't want your assistant citing it next month.

But retracting a paper doesn't change its Entrez date, so a date-based cursor would never notice. Each run also searches (your query) AND retracted publication[pt] and removes any article you already ingested that has since been retracted.

Deleted records work the same way. Each run re-lists the PMIDs your query currently matches, and anything that has disappeared gets a tombstone row ({"id": "pubmed:<PMID>", "_deleted": true}). dlt hard-deletes those rows, and cognee's orphan cleanup removes them from the graph and vector store.

There's one safety rule. If PubMed suddenly returns nothing at all while articles are known, the connector assumes an outage rather than "everything was deleted", and skips the cleanup. It also only saves its progress after a run fully succeeds, so a crash halfway through just means the next run repeats the work.

What it doesn't do (yet)

• Abstracts only. Full text lives in PubMed Central, which is a separate API.
• Corrections to an existing record aren't picked up, because a revision doesn't change the Entrez date. NCBI tracks revisions with a different date field (mdat), which would be a natural next step.
• Very broad queries are expensive. Something like cancer matches millions of records. Narrow the query or set mindate before your first run.

What I learned

The interesting part of a connector isn't fetching data, it's deciding what changed and what should be forgotten. Each of the three issues above came from asking "what happens on the second run?" instead of only testing the first.

Try it

The connector is in review as topoteretes/cognee-community#275, for issue #4807. Until it's merged, you can install it from my branch:

pip install "cognee-community-connector-pubmed @ git+https://github.com/priyansh-narang2308/cognee-community@feature/pubmed-connector#subdirectory=packages/connector/pubmed"

If you're working on research assistants or literature-review tools, I'd love to hear which queries you'd point it at.