Miedocuentista：完全本機運行的 AI，將照片轉化為鬼故事、電影劇本和小説

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

Miedocuentista ("fear-teller") is a spooky storytelling machine that runs entirely on my laptop. You feed it images one at a time and it writes the story frame by frame: a short film script, a film story, or a novel with one chapter per image.

How it works in one breath: a small vision model looks at each image and lists what it sees (objects, setting, mood), I can correct that list and add my own twists, and a small language model writes the next scene while a rolling "story memory" keeps the plot, characters and open threads alive between scenes. After the final image, I run a Final AI Story Editor that reads the whole story, looks for plot holes and timeline problems, and rewrites every scene so they connect, trying to keep the storyline intact.

How I interpret the theme. The ghost in this machine isn't hiding inside the model. It is what the machine sees in your photographs (every frame is treated as a haunted scene) and what it remembers between scenes: the story memory literally haunts each new scene with everything that came before. It has a Halloween interface, a horror default, and a Día de los Muertos tone for warmer, remembrance-style stories.

What inspired it. It started from one of the challenge's own ideas, a ghost story generator. My first version was simple: one photo in, one ghost story out, using a small vision model and a small language model on my laptop. But one photo only gives you one scene, and stories, films and novels are made of many frames. So I extended it into a frame-by-frame studio where every new image continues the story. I also wanted to build the front end of a future AI video workflow: storyboard images go in, a structured script comes out (visuals, sound, dialogue, transitions) that other open-source tools could later turn into video. And I wanted all of it to run on my own machine, so my photos and stories never leave it.

Why a pipeline, not just a toy. The script format separates VISUAL, SOUND, dialogue and TRANSITION on purpose, and every scene is stored as structured JSON. My longer-term goal is to feed this into open-source image/video and text-to-speech workflows (storyboard in, script out, shots next). That hand-off is not built yet; today it produces the script, story or novel and exports .txt/.md.

Demo

Screenshots, in the order of the workflow:

Code

👻 Miedocuentista

A fully local, open-source AI story studio that turns a sequence of images into a short story, a film script or a novel, one frame at a time.

Miedo (fear) + cuentista (storyteller): the fear-teller. Built for the Hacktoberfest Open-Source AI Challenge, Week 4: Ghost in the Machine.

Everything runs on your own computer through Ollama. No cloud, no API key, no cost, and your images never leave your machine.

👻 Miedocuentista

The outputs from my test runs (problems included) are in the tested_outputs folder of the repo.

Run it yourself (needs Ollama and Node.js 18+):

ollama pull moondream
ollama pull qwen2.5:3b
git clone https://github.com/shristi005/Miedocuentista.git
cd Miedocuentista/backend
npm start

Then open http://localhost:3002. There is no npm install step; the backend has zero dependencies.

How I Built It

Open models (via Ollama, all local):

moondream (about 1.7 GB, open weights): the "eyes". For each frame it answers four short questions (caption, objects, setting, mood). Small vision models follow short questions far better than one long prompt.
qwen2.5:3b (about 1.9 GB, open weights): the "writer", the "memory keeper" and the "editor".

App: Node.js backend with zero npm dependencies, plain HTML/CSS/JS frontend, and one JSON file per project. Frontend and backend are separate folders.

The parts I'm proudest of:

Rolling story memory. A 3B model on a small GPU can't re-read a novel for every scene. After each scene, the model merges the old memory with the new scene into a note of about 140 words (characters, setting, events, open threads). The note stays the same size however long the story gets, and only the previous scene's ending is passed word for word.
Regenerate / undo that restores the story memory. Each scene saves the memory from before it, so regenerating or undoing the last scene brings back the exact earlier state.
Final AI Story Editor. Pass 1 reads the complete story and writes an edit report (canon, plot holes, timeline errors, repetition, weak transitions, per-scene fixes). Pass 2 rewrites each scene using that report plus the neighbouring scenes. The original is always kept, with an Edited/Original toggle.
Human in the loop. The vision model makes mistakes, so every frame's caption and element chips are editable before they shape the story.

Challenges I Hit (and Why I Think They Happen)

I tested with a three-frame lighthouse story: a boat and pier, a stone archway ringed with red roses and an urn, and a keeper's room with a journal. The pipeline worked end to end (image to elements to scene to memory to next scene to editor), but a 3B model on a 4 GB GPU has real limits. I'd rather show them than hide them. The causes below are my best diagnosis from reading the outputs and my own prompts, not proven facts.

Problem I saw	What it looked like	Likely cause	Planned fix
Copy-pasting earlier text	The same paragraphs ("The room spun, making her dizzy for a moment…", "The handprint on the urn matched the others…") show up in all three chapters	My prompt passes the last ~1,200 characters of the previous scene word for word. Small models tend to copy what is sitting in their context, and I set no repetition penalty	Pass a one-line summary of the last beat instead of raw text; set a repeat penalty; detect duplicated sentences and regenerate
The story stalls and loops	Chapters keep re-describing the lantern, the midnight, the room and the handprint instead of moving on	I tell the writer to use every visible element, neighbouring frames overlap, and the model has no plan for what NEW thing happens	A "beat planning" step: decide one new event and one new piece of information, then write from that plan
Memory feeds on itself	Repetition gets worse in later chapters, and the timeline resets (she arrives at the dock again)	The story memory is summarised by the same small model from text that is already repetitive, so the repetition is baked into the memory. There is no location/time tracking	Structured memory (character sheet, event list, current place and time), facts the user can pin, and de-duplication before summarising
Headings and numbering ignored	Two blocks labelled "Chapter 2" and a third with no heading; chapter titles reuse the book's title	I ask the model to write the heading itself, and 3B models follow formatting loosely	The app inserts the chapter number; a separate tiny call writes a new chapter title
Prompt wording leaks into the prose	Phrases like "the journal from the previous scene" appear inside the story	Labels in my prompt (PREVIOUS SCENE, reader-added elements) get echoed	Rename labels, clean leaked phrases after generation, add a worked example
Script mode leaks its template	`NAME:` printed literally, `CUT TO / FADE OUT / SMASH CUT` copied as-is, the prompt's input block echoed	I used placeholders in the format instructions and the model copied them	Replace placeholders with one concrete example scene; strip stray labels
The Final Editor can't fix it	The edited version still has repeated passages	The editor is the same small model that wrote the problems, so it cannot reliably spot them	A deterministic de-duplication pass first; a larger model for the editor only
Vision mistakes	The 1.7 GB vision model sometimes misses or invents objects	Tiny model	Already mitigated: every caption and chip is editable before it shapes the story. Next: a stronger vision model

What this taught me. Small local models are good at structure and weak at long, coherent prose. Most of what went wrong is engineering around the model (what I put in its context, what I let it decide, what I check afterwards), not a flaw in the idea. The rolling memory and two-pass editor are my first answer to that; the table above is the second round.

Future Improvements

Quality guardrails: a repetition score for every scene, automatic regeneration when a scene copies earlier text, and a "must change something" rule for each scene.
Planning before writing: generate a short beat (new event, new information, emotional turn) from the story memory and the frame, then write the scene from that beat.
Better memory: structured JSON memory with a character sheet, timeline and open threads, plus facts the author can pin so names and relationships never drift.
Stricter formats: the app owns chapter numbers and scene headings; script output is cleaned and validated.
Bigger or better models: because the models are environment variables, a 7B or 8B model (on a machine with more VRAM) can be dropped in. I want to benchmark a few open models side by side on the same frames.
In-browser editing of any scene, and a side-by-side diff between original and edited versions.
Video-workflow hand-off: export each scene as a shot list (VISUAL, SOUND, dialogue, duration) so open-source image/video and text-to-speech tools can pick it up. Not built yet.
A Día de los Muertos "ofrenda" mode: remembrance stories from old family photos, kept fully on the owner's machine. Today the Día de los Muertos option is only a tone setting.

How It Can Be Used

Today, Miedocuentista is best treated as a first-draft and idea engine, not a finished-story generator. You get structure and momentum quickly, then you edit.

Short films and YouTube videos: drop in storyboard frames or concept art, get a scene-by-scene script with visuals, sound and dialogue to rewrite. Later, the same structure is meant to feed AI video workflows.
Novel and short-story outlines: turn a mood board or a folder of reference images into chapters to react to.
Game and world-building: turn level screenshots or location art into lore and quest text.
Classrooms and writing groups: the same photos, different frames, different stories; compare and discuss.
Private projects: because everything is local, it is suitable for personal photos and unreleased art.

How: (1) pick short film, feature film or novel; (2) choose script or story; (3) add your frames one by one and correct the elements; (4) mark the last frame; (5) run the Final Editor; (6) export .txt or .md and finish by hand.

Why Does Open Innovation Matter?

Your photos stay yours. Stories get built from personal images: old family photos, sketches, unreleased concept art. With local open-weight models, none of it is uploaded anywhere.
It works with the lights (and the internet) off. No API key, no rate limit, no per-token bill. The UI even avoids external fonts and CDNs.
It's not locked inside a service that might disappear. Projects are plain JSON files I own. If a hosted API changes its pricing or shuts down, my stories and my workflow survive.
Swappable models. The vision and story models are environment variables, so another open model can be tried without touching the code, and a stronger machine should give better stories. (So far I have only tested moondream and qwen2.5:3b.)
It made a tiny-model design possible. The rolling memory and two-pass editor exist because I had to make a 3B model on a 4 GB GPU work. A closed API with a huge context window would have hidden that problem instead of letting me solve it.
Free to run, which matters for students and indie creators.

The Spirit of It

I gave it three images: a boat with a lantern at a lighthouse pier, a stone archway ringed with red roses, and a keeper's room with a journal. Out of those three photos, with no plot from me, it wrote a novel in my first test run whose final chapter ends on a twist I did not plan: the light the heroine had been chasing for fifteen years was her missing brother all along.

The outputs from my test runs (problems included) are in the tested_outputs folder:

Novel output (3 chapters)
Short film script output (3 scenes; the weakest result, see the Challenges section)
The three input frame images