跳到正文
DEV Community· Umesh Adabala·· 3 小時前AI 評分50

為何打造 PaperTrail：從 AI 論文摘要邁向持續演進的科學智能

Why We Built PaperTrail: Moving from AI Paper Summaries to Continuous Scientific Intelligence

AI 導讀

PaperTrail 將自身定位為持續演進的科學智能引擎，旨在追蹤科學證據隨時間的變化，而非只為論文生成摘要。它要求每項研究主張附有原文引句與論文引文，並結合硬件、精度等實驗條件分析研究結果的分歧。文章亦説明其以 Next.js 14、TypeScript、Ollama 及 SQLite/Supabase 混合儲存引擎構建。

正文

Skip to content

Image 1: DEV Community

Powered by Algolia

Log inCreate account

DEV Community

Image 20 Add reaction

Image 30 Like Image 40 Unicorn Image 50 Exploding Head Image 60 Raised Hands Image 70 Fire

0 Jump to Comments 0 Save Boost

Pick as gem

Copy link

Copied to Clipboard

Share to XShare to LinkedInShare to FacebookShare to Mastodon

Share Post via...Report Abuse

Image 8: Umesh Adabala

Umesh Adabala Posted on Oct 31

Why We Built PaperTrail: Moving from AI Paper Summaries to Continuous Scientific Intelligence

#webdev#ai#react#opensource

Every few months, a new "AI research tool" hits the front page of Hacker News or Product Hunt. Almost without exception, they all do the same thing:

  1. Ingest a PDF.
  2. Chunk it into embeddings.
  3. Feed the chunks into a vector database.
  4. Put a conversational chatbot in front of it that summarizes what the authors said.

Here's the fundamental problem: Science is not a collection of static text summaries. Science is an evolving, adversarial debate.

A paper published in 2023 demonstrating a 4x inference speedup for 4-bit quantized language models on Apple Silicon is not an absolute truth. It is a finding contingent on specific hardware, quantization algorithms, precision levels, and thermal conditions. When a 2024 paper reports severe degradation under continuous workloads on Qualcomm Snapdragon chips, they aren't necessarily contradicting each other—they're operating in completely different experimental contexts.

And when a seminal paper gets retracted or contradicted six months later, your vector database doesn't know. It happily continues serving obsolete claims as ground truth.

We built PaperTrail to fundamentally rethink how software interacts with scientific literature.

Instead of another paper summarizer, PaperTrail is an autonomous, continuously evolving scientific intelligence engine.

In this article, I want to walk through the technical architecture, design philosophy, and lessons learned building PaperTrail with Next.js 14, TypeScript, local Ollama models, and a local-first SQLite/Supabase hybrid storage engine.

The Core Proposition

PaperTrail is built around a single thesis:

"Discover what research has established, what it challenges, what remains unexplored, and how the scientific evidence changes over time."

To deliver on this, we established ten non-negotiable architectural capabilities:

  1. Grounded Scientific Claim Extraction: Never extract an assertion without an exact, verbatim passage and paper citation.
  2. Context-Aware Disagreement Investigation: Never label two differing findings as a contradiction without examining differences in hardware, precision, quantization bitwidth, context length, or thermal states.
  3. Candidate Research-Gap Discovery: Automatically identify blind spots and uninvestigated parameter intersections across literature.
  4. Adversarial 6-Stage Gap Falsification: Never present a research gap as valid until an automated red-team agent attempts to aggressively falsify it against counter-evidence.
  5. Practical Experiment Feasibility & Protocol Planning: Convert validated gaps into runnable, resource-evaluated empirical protocols with compute hour calculations and step-by-step checklists.
  6. Continuous Research Tracking & Evidence Evolution: Monitor topics and incoming preprints to compute how evidence shifts over time.
  7. Preprint-to-Publication Lineage Mapping: Trace preprints from arXiv to peer-reviewed venues, isolating shifts in methodology and benchmarks.
  8. Retraction & Correction Intelligence: When papers are retracted or corrected, flag dependent claims and recalculate conclusions without silently erasing historical evidence.
  9. Accessible Dataset Discovery: Find benchmark datasets on Hugging Face, Zenodo, and Figshare with verified licenses.
  10. Cross-Domain Hypothesis Transfer: Formulate principled transfer hypotheses connecting primary research with adjacent fields (e.g., neuroscience synaptic pruning $\leftrightarrow$ LLM weight quantization).

1. Grounded Claim Extraction & Schema Validation

One of our strictest engineering rules is: No hallucinated consensus.

Every claim extracted by PaperTrail must adhere to a strict Zod schema. If the model cannot provide the exact supporting passage from the abstract or text, the extraction fails validation and falls back to a deterministic semantic parser.

Here is a snippet of our claim validation contract:

export const ClaimExtractionSchema = z.object({
  claims: z.array(
    z.object({
      claim_text: z.string().min(10),
      claim_type: z.enum([
        'FINDING',
        'HYPOTHESIS',
        'METHODOLOGY',
        'LIMITATION',
        'BENCHMARK_RESULT',
        'THEORETICAL_BOUND',
      ]),
      section: z.enum(['Abstract', 'Introduction', 'Evaluation', 'Conclusion']),
      supporting_passage: z.string().min(15),
      page_number: z.number().int().positive().optional(),
      extraction_confidence: z.number().min(0).max(1),
    })
  ),
  experimental_context: z.object({
    model: z.string().default('Unspecified'),
    hardware: z.string().default('Unspecified'),
    dataset: z.string().default('Unspecified'),
    quantization: z.string().default('FP16'),
    precision: z.string().default('Unspecified'),
    metrics: z.record(z.string()).default({}),
    variables: z.record(z.string()).default({}),
  }),
  limitations: z.array(z.string()).default([]),
});

By extracting both the claim and its experimental condition matrix (hardware, precision, model scale, metrics), we create the foundation for intelligent, context-aware comparison.

2. Context-Aware Disagreement Investigation: Beyond Binary "Contradictions"

Traditional NLP pipelines treat contradiction detection as an NLI (Natural Language Inference) task: Premise vs. Hypothesis $\rightarrow$ Entailment, Neutral, or Contradiction.

In real science, this is catastrophically naive. Consider two papers:

  • Paper A (Lin et al., 2023): "AWQ achieves 3.2x latency speedup with negligible perplexity degradation on 7B LLMs."
  • Paper B (Smith et al., 2024): "Sustained 4-bit execution throttles after 120 seconds, resulting in a 42% throughput drop on fanless mobile SoCs."

A naive AI system flags these as a direct contradiction: Speedup vs. Throughput Drop.

PaperTrail executes a Context Comparison Matrix:

export interface ContextComparison {
  similarities: string[];
  differences: {
    variable: string;
    source_val: string;
    target_val: string;
    impact_level: 'LOW' | 'MEDIUM' | 'HIGH';
  }[];
  possible_explanations: string[];
  alternative_explanations: string[];
  remaining_uncertainty: string;
  is_genuine_contradiction: boolean;
}

When evaluated, PaperTrail identifies:

  • Independent Variable Difference: Paper A evaluated burst inference under active server cooling; Paper B evaluated sustained execution under passive ambient mobile cooling.
  • Verdict: APPARENT_DIVERGENCE_EXPLAINED_BY_CONTEXT (Confidence: 0.94).
  • Resulting Research Opportunity: How do thermal throttling thresholds differ between INT4 weight-only quantization and INT4 activation quantization during multi-turn generation?

3. The 6-Stage Adversarial Gap Falsification Workflow

Most researchers know the dread of reading 30 papers, designing a novel experiment, and discovering two weeks later that a lab in Zurich published that exact experiment six months ago.

In PaperTrail, candidate research gaps are never taken at face value. They are subjected to a rigorous 6-Stage Adversarial Red-Team Audit:

[Candidate Gap Discovery]
          │
          ▼
┌──────────────────────────────────────────────┐
│ Stage 1: Hypothesis Decomposition             │
│   Extract testable variables & parameters    │
├──────────────────────────────────────────────┤
│ Stage 2: Targeted Counter-Evidence Search    │
│   Generate adversarial retrieval queries     │
├──────────────────────────────────────────────┤
│ Stage 3: Multi-Index Retrieval               │
│   Query OpenAlex, Crossref, arXiv, TinyFish  │
├──────────────────────────────────────────────┤
│ Stage 4: Semantic Relevance & Overlap Audit  │
│   Compare baseline methodology & datasets    │
├──────────────────────────────────────────────┤
│ Stage 5: Adversarial Refutation Scoring      │
│   Compute falsification confidence (0.0-1.0) │
├──────────────────────────────────────────────┤
│ Stage 6: Status Assignment                   │
│   SURVIVED (Validated) | FALSIFIED | PARTIAL │
└──────────────────────────────────────────────┘

Only when a candidate gap survives this adversarial gauntlet is it promoted to an actionable research gap. From there, PaperTrail automatically drafts an empirical experiment protocol complete with hardware requirements, dataset references, compute hours, and verification checklists.

4. Local-First Research Portability: SQLite + Supabase

We believe your research belongs to you—not locked inside our cloud database.

PaperTrail implements a dual-repository architecture:

┌───────────────────────────────┐
                  │      ResearchRepository       │
                  │          (Interface)          │
                  └───────────────┬───────────────┘
                                  │
                 ┌────────────────┴────────────────┐
                 ▼                                 ▼
   ┌───────────────────────────┐     ┌───────────────────────────┐
   │ SQLiteResearchRepository  │     │ SupabaseResearchRepository│
   │ (Node native DatabaseSync)│     │     (PostgreSQL Cloud)    │
   └───────────────────────────┘     └───────────────────────────┘

Using Node.js 22+'s native DatabaseSync (node:sqlite), PaperTrail operates entirely offline if desired, writing directly to .papertrail-data/papertrail.db with WAL mode enabled.

Complete Research Archive Export

Every research workspace can be exported with one click into a structured, vendor-neutral research bundle:

papertrail-research/
├── manifest.json         # Cryptographic checksums, schema v1.0, timestamp
├── README.md             # Complete human-readable investigation brief
├── papers/               # Verified metadata, OpenAlex/arXiv DOIs, abstracts
├── claims/               # Grounded assertions with verbatim supporting quotes
├── evidence/             # Disagreement matrices and apparent vs genuine verdicts
├── graph/                # Interactive topology nodes and edges (JSON)
├── research-gaps/        # 6-stage falsification audits and counter-evidence
├── experiments/          # Resource-evaluated execution protocols
├── datasets/             # Hugging Face & Zenodo references with license terms
├── timeline/             # Evidence evolution and tracking history
└── provenance/           # Query origins and retrieval timestamps

You can export this as a ZIP archive, BibTeX, JSON, or formatted Markdown, and import it into any other PaperTrail instance with zero data loss.

5. Designing for Academics: The Academic Journal Aesthetic

We made a conscious product design decision: PaperTrail should not look like another dark-mode SaaS dashboard with glowing neon gradients.

Science is deliberate, rigorous, and calm. We crafted a dedicated design language inspired by physical academic journals and high-end typography:

  • Warm Paper Surface: #F7F6F2 (background), #FDFCF9 (cards), #FAF8F3 (alt panels)
  • Deep Editorial Ink: #252622 (primary text), #777873 (muted metadata)
  • Terracotta Accent: #B96645 (reserved strictly for active claims, falsification audits, and disagreements)
  • Sage Data Accent: #858B70 (used for verified benchmark metrics, datasets, and lineages)
  • Typography:
    • Instrument Serif for research questions, paper titles, and editorial headers.
    • DM Sans for high-density tabular data and claim cards.
    • JetBrains Mono for DOIs, citations, arXiv IDs, and hardware matrices.
:root {
  --paper-bg: #F7F6F2;
  --paper-surface: #FDFCF9;
  --paper-surface-alt: #FAF8F3;
  --paper-ink: #252622;
  --paper-muted: #777873;
  --paper-border: #E3E1D9;
  --paper-accent: #B96645;
  --paper-data: #858B70;
}

The interface feels like reading a physical copy of Nature or Science paired with an interactive research workbench.

6. Real-World Engineering Challenge: Taming Heavy LLMs

During our testing with local and remote Ollama models (specifically gpt-oss:120b-cloud), we hit an interesting bottleneck.

When analyzing 8 papers simultaneously, each paper required claim extraction, experimental context parsing, pairwise contradiction checks, and gap discovery. Sending 15+ concurrent requests to a 120-billion parameter model over a cloud proxy resulted in request queueing that took 4+ minutes.

We solved this with a three-tiered pipeline architecture:

  1. Strict 12s Timeout: Ollama requests time out gracefully after 12 seconds instead of hanging the connection.
  2. Landmark Prioritization: The top 3 landmark papers receive deep LLM extraction. Supporting literature receives instant, deterministic parsing (extracting findings, methodologies, and limitations in 0.1ms).
  3. Capped Contradiction Audits: Pairwise contradiction checking prioritizes the top 3 most divergent pairs rather than running an $O(N^2)$ cross-product.

This reduced the end-to-end pipeline execution time from 4.5 minutes down to 14 seconds, delivering a fast, responsive UX without sacrificing scientific rigor.

What’s Next?

PaperTrail is completely open source under the MIT license. Some of the features we're actively working on next:

  • TinyFish Continuous Web Monitoring: Real-time tracking of lab publication feeds and author pages.
  • Automated Systematic Literature Review (SLR) Generator: Exporting PRISMA-compliant flowcharts and evidence synthesis tables.
  • Hardware-in-the-Loop Benchmark Validation: Connecting experiment protocols directly to local MLX / llama.cpp runners for on-device replication.

If you're a researcher, engineer, or curious mind exploring literature:

  • ⭐️ Star the repository on GitHub: github.com/umeshadabala/PaperTrail
  • Check out the codebase, clone it, and run it locally with your own Ollama setup or cloud provider!

Let me know in the comments: How do you currently track scientific contradictions and emerging research gaps in your field?

Image 9: profile The DEV TeamPromoted

Image 10: Hacktoberfest DEV Challenges

No Fest Nearby? Hack From Anywhere.

Join thousands of developers online for a week of open source AI challenges with MLH. Free, virtual, and part of Hacktoberfest.

Join Global Hack Week

Read More

Image 11: Google Cloud Google CloudPromoted

Image 12: Ship Agents Faster

Ship Agents Faster

Learn More →

Top comments (0)

Subscribe

Image 13: pic

Personal Trusted UserCreate template Templates let you quickly answer FAQs or store snippets for re-use.

Submit PreviewDismiss

Code of Conduct•Report abuse

Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink.

  • [x] Hide child comments as well

Confirm

For further actions, you may consider blocking this person and/or reporting abuse

Image 14: Google Cloud Google CloudPromoted

Image 15: Host LLMs in Production With On-Demand GPUs

Host LLMs in Production With On-Demand GPUs

Learn More →

Image 16: profile The DEV TeamPromoted

Image 17: Hacktoberfest DEV Challenges

Five Challenges. Cash Prizes Every Week.

A new open source AI challenge drops every Monday in October. Build something, write it up, win.

See the Challenges

Image 18 Umesh Adabala

Follow

A 18 year old Indian python developer. Who wants to serve many posts to this community.

  • Pronouns He/Him
  • Joined Jan 3, 2022

More from Umesh Adabala

CircuitMind: An AI-Native Engineering IDE for Physical Systems and Embedded Electronics #ai#opensource#iot#software

Image 19: profile The DEV TeamPromoted

Image 20: Hacktoberfest DEV Challenges

Five Challenges. Cash Prizes Every Week.

A new open source AI challenge drops every Monday in October. Build something, write it up, win.

See the Challenges

Image 21: Google Cloud Google CloudPromoted

Image 22: Save Up to 91% on Cloud Compute With Spot VMs

Save Up to 91% on Cloud Compute With Spot VMs

Learn More →

👋 Kindness is contagious

Take a moment to explore this thoughtful article, beloved by the supportive DEV Community. Coders of every background are invited to share and elevate our collective know-how.

A heartfelt "thank you" can brighten someone's day—leave your appreciation below!

On DEV, sharing knowledge smooths our journey and tightens our community bonds. Enjoyed this? A quick thank you to the author is hugely appreciated.

Okay

DEV Community — A space to discuss and keep up software development and manage your software career

Built on Forem — the open source software that powers DEV and other inclusive communities.

Made with love and Ruby on Rails. DEV Community © 2016 - 2026.

Image 23: DEV Community

We're a place where coders share, stay up-to-date and grow their careers.

Log inCreate account

Image 24Image 25Image 26Image 27Image 28

來源：DEV Community · dev.to

#教學/實踐#Agent#部署/工程