GooDesign 停止讓 LLM 改寫提示詞，剖析意圖流失問題與修正方案

The failure report that changed how we build goodesign is the kind you can reconstruct from almost any prompt-rewriting pipeline, because they all break the same way. Picture a completely ordinary session: a user uploads a photo of their living room, types a five-word prompt — "make it cozy, keep the rug" — and gets back a render where the rug was gone, the sofa had visibly grown, and the whole scene had slid from afternoon daylight into warm evening lamplight.

Nothing about that output was broken in the model sense. The image was sharp, the shadows were plausible, every chair leg touched the floor. It just wasn't what the person asked for. The rug — the one thing they had explicitly protected — was the first thing to go.

We traced it, and the culprit wasn't the image model. It was us. An LLM sitting in the middle of our pipeline, added months earlier to "help" users who type vague requests, had rewritten those five words into a 140-word template before the render ever started. This post is about what that rewrite broke, why prompt rewriting loses intent in the first place, the two-layer architecture we replaced it with, and how we test for intent loss now. If you build anything where a user types words and a generative model draws pictures — AI room design, virtual staging, photo editing, anything — this is the failure mode I'd put at the top of your list.

The prompt that came back wrong

Here is roughly what our expansion layer produced from "make it cozy, keep the rug":

"A cozy living room interior with warm ambient lighting, plush textiles, soft throw blankets, layered cushions in warm earth tones, an inviting oversized sofa, a soft evening glow from table lamps, photorealistic, professional interior photography."

Read it next to the original and the damage is easy to miss, because nothing in it is wrong. That's what makes prompt rewriting dangerous. Line by line:

• The user's only hard constraint — keep the rug — is gone entirely. It isn't paraphrased or weakened. It's just not there.
• "Cozy" got decoded as evening lighting. That is one interpretation of cozy. It was not this user's interpretation, and it happened to require changing the one thing in the photo the user liked.
• The template stacked enough soft-furniture adjectives that the renderer, averaging all those simultaneous demands, quietly reallocated "softness" — the rug became throw blankets and cushions.

The individual edits all looked defensible. Warm light is cozy. Big sofas are cozy. Nothing in the output contradicted the word "cozy." It contradicted the person. And because the rewrite happened silently between the text box and the renderer, the user had no way to know their words were never the words the model saw.

Why rewriting loses intent: words are not specifications

The first thing to understand is what a short prompt actually contains. In a room-design prompt, users mix at least three kinds of content:

1. Intent — what they want changed ("make it cozy")
2. Constraints — what must not change ("keep the rug")
3. Rendering parameters — the physical facts a renderer needs: light direction, camera height, material names, counts

Users are good at the first two and usually silent on the third, because the third is our job, not theirs. The expansion layer was written as if all three were missing. So it inflated rendering parameters (lighting adjectives, furniture descriptions), inflated intent into a full aesthetic manifesto, and dropped the constraint — because constraints are the one thing an expansion model has no reason to carry.

That last point deserves emphasis. "Keep the rug" is negative space in a sentence. It describes what should not happen. A language model trained to add helpful detail has essentially zero gradient toward preserving an instruction to not add detail. In our own logs, once we went looking, the keep-clause was reliably the first casualty of rewriting. Intent and parameters survived rewrites in various mangled forms; constraints tended to simply vanish.

The second problem is the word-to-image gap. "Cozy" does not map to any geometry. When a diffusion model reads "cozy," it resolves the adjective through its training priors — the statistical average of every cozy living room in its captioned training data. That average skews warm, textile-heavy, and evening-lit. When our LLM rewrote the prompt, it wasn't translating the user's intent into renderer language, because there is no such translation. It was replacing the user's private meaning of "cozy" with the training distribution's public meaning. The rewrite didn't add information. It substituted a different person's taste and called it help.

Third: rewriting has a direction bias. Language models are rewarded for fluent, detail-rich, confident text, so a rewriter normalizes unusual input toward the middle of the distribution. The more specific and unusual the user's request, the more the rewrite sands it down. Vague prompts get inflated; specific prompts get flattened. Both directions move the output away from the user and toward the average.

And there's a quiet authority transfer nobody signs up for. Once your pipeline decodes "cozy" into a fixed list of signifiers — lamps, throw blankets, earth tones — you have decided, for every user, forever, what cozy means. Some of your users mean candles and rain on the window. Some mean a deep sofa and a bright, crisp winter-lit room. A rewrite can't know. It can only pick one and disguise the choice as neutrality.

The fix: a verbatim layer and a parameter layer

We did not remove the LLM from the pipeline. Truly vague input ("fix this room") genuinely benefits from defaults. What we changed is where the model is allowed to act, and in what shape.

Layer 1: the user's text is immutable. Whatever the user typed goes to the renderer byte-for-byte, first in the composed prompt. No synonym upgrades. No reordering. No cleanup of typos, no expansion of "cozy" into "warm, intimate, inviting" — that triple-synonym move is precisely how you get averaged mush, because each synonym is another vote for the mean. If the user's sentence is short, the render starts from a short sentence, and the fill comes from Layer 2 where we can see it.

Layer 2: structured supplement, not prose. A small, cheap model call classifies what the user actually specified, then emits a strict JSON object of rendering parameters — light source and direction, time of day, camera height, up to about four material anchors. It may only fill fields the user left empty. The JSON is serialized into a short bracketed hint that gets appended after the user's text, never interleaved with it, never substituted for it. Our materials guide makes the same point from the user side: one named material per surface beats paragraphs of mood adjectives, because past roughly 30 words of stacked demands, the model starts averaging them. The supplement layer exists to add the technical floor a photo-real render needs — nothing more.

Constraints leave prose entirely. "Keep the rug" no longer lives in the prompt text, where it competes with forty adjectives and loses. It gets extracted into a machine-readable keep-list that drives the edit-in-place controls directly: locked regions, preserved wall color, preserved window geometry. A constraint expressed as code is a constraint the pipeline cannot paraphrase away.

The supplement layer runs under four rules:

1. Never contradict. If the user said "bright daylight," the layer is forbidden from appending a warm-evening hint. User wins, always.
2. Never touch the user's substring. The composed prompt is user_text + hint, and the boundary is tested.
3. Idempotent. Running the supplement twice changes nothing.
4. Valid JSON or nothing. If the classifier is unsure, it emits an empty object and the render proceeds on the user's words alone. Uncertainty degrades to silence, not to creativity.

If you've read our public prompting guide, this architecture will look familiar, because the user-facing advice describes the same shape: lead with an action verb, name one style direction, list 2–3 anchor pieces with a finish, give placement cues, and finish with an explicit keep list. The pipeline now respects that structure instead of flattening it.

One honest trade-off: with rewriting gone, a genuinely lazy prompt produces a more literal — sometimes plainer — render than it used to. We think that's correct. The old way felt magical partly because it was making decisions the user never delegated.

How we test prompt integrity now

You can't fix intent loss you can't see, so the first thing we built was visibility: the user's raw text is logged separately from the model-facing composed prompt, permanently. Every support conversation about "that's not what I asked for" now starts with a diff.

Then we built a regression suite around the concept of annotated intent. Every prompt in the golden set carries tags:

{
  "prompt": "make it cozy, keep the rug",
  "must_keep":    ["rug", "wall color", "window geometry"],
  "must_change":  ["overall mood"],
  "must_not_introduce": ["evening lighting", "rug removal"],
}

On every change to the model, the hint layer, or the composer, the golden set re-renders and gets scored: a VLM judge checks each tagged constraint, and a rotating human pass spot-checks the judge, because VLM judges are themselves models with their own lazy priors — they will happily grade "the rug is still there" as true if the rug's color changed. The judge scores; humans arbitrate.

The boundary cases in the set matter more than the happy paths:

• Constraints-only prompts. "Don't change anything, just brighten it." The pipeline must treat this as a lighting edit, not a redesign invitation.
• Contradictory prompts. "Cozy but bright daylight." The user wins; the hint layer stays silent rather than picking a side.
• Non-English prompts. The product runs in 8 languages, and user words pass through untouched in all of them. The supplement layer may localize its hints, but it may not translate the user's sentence into English first — that's a rewrite with extra steps.
• Counting language. "Two nightstands, one large piece of art." Exact counts survive; "some" and "a few" from the hint layer never override a stated number.
• Junk. Empty strings, emoji-only input, 400-word rambles. Ramblers get length-capped from our side — the user's first N words survive, our words get cut.

Finally, the keep-list is treated as an API contract. If a render violates a tagged keep-constraint, that's a bug with a reproducing case, not a vibe to shrug at.

What changed after launch — shown with images, not dashboards

I'm suspicious of posts that claim "+38% user satisfaction" without a methodology, so I won't invent one. What I can show is what the pipeline produces now. Our public gallery is 35 before/after pairs — 7 rooms across 5 transformation types, every pair generated from a single input photo. These two pairs are straight from it; notice in each that the fixed features of the room survive the transformation.

What actually changed, described honestly and qualitatively:

• User adjectives now show up as themselves. When someone's prompt survives to the renderer intact, the render reflects their version of "cozy," not the template's.
• Keep-constraints mostly survive. Not 100% — on cluttered scenes the model still sometimes fights a locked region — but violations are reproducible from the golden set now, which is the difference between a bug and a ghost.
• Composed prompts got much shorter, because a large fraction of their former length was ours, not the user's. Shorter prompts also mean cheaper renders and, usefully, easier diffing when something goes wrong.

What didn't change, because no amount of prompt hygiene fixes it:

• Renders still can't measure. If the actual wall is 3.4 meters and the sofa in the render reads as 2.6, nobody in the pipeline knows. Scale is estimated from pixels; a tape measure is still the final word before anyone buys furniture.
• The model still drifts on mirrors, fine text, and complex ceiling lines. Those are model limitations, not prompt-layer ones.
• Vague input still produces default-flavored output. We made the defaults honest and inspectable. We didn't abolish them, and users who type three words and expect a miracle will still get a competent render of somebody's taste — just with their own words, verbatim, at the front of it.

Three rules for anyone shipping a similar pipeline

1. Log the user's raw text separately from the model-facing prompt — forever. The single most useful debugging artifact we have is the pair (what the user said, what the model saw). If you only log the second, every intent-loss bug is invisible by construction.

2. If you must expand user input, expand into structured parameters, never prose. JSON fields can be unit-tested, diffed, suppressed, and audited. Free-text expansion can only be regretted. The entire class of "the LLM overrode the user" bugs shrinks to "the classifier misfiled a field," which is a much smaller animal.

3. Move constraints out of language and into product surfaces. A "keep this wall" checkbox beats a keep-clause any rewriter can eat. Every constraint you can express as code instead of tokens is a constraint the pipeline cannot paraphrase away. The prompt is the worst place to store the user's non-negotiables — it's the one part of the stack whose entire job is to change.

None of this is exotic. The hard part wasn't engineering; it was accepting that "helpful" and "faithful" are different objectives, and that we had optimized for the one users never asked for.

Try it on your own room

If you want to see how the pipeline behaves on your photos: goodesign takes one room photo and renders a redesign in about a minute. Your first 5 credits are free, no credit card required, and 1 credit is 1 render. And if you'd rather write prompts by hand, the five-part pattern — action, style, anchors, placement, keep list — is laid out in the prompting guide linked above.

This article was drafted with AI assistance and edited/fact-checked by the Goodesign engineering team.