MarketMirror 作者用三個 AI 模型模擬 2,999 名買家，發現回答模型比撰寫人物設定的模型更影響結果

I'm building MarketMirror, an open-source tool that tests ads, prices, messages and sales pitches on simulated buyers before any money goes into a real campaign. It ends with a plain-language report that says go, iterate or no-go.

The obvious objection to any "synthetic customers" tool is: aren't you just measuring the language model's mood? I wanted to see that effect directly, so I ran one market with three model providers at once. Every number below is simulated. Fitloop, the product, is an invented fitness app.

The setup

The advertised offer was a 7-day free trial with no card, then USD 4.99 a month. The brief and the landing page still asked for a card at sign-up, on purpose: the test is meant to catch that.

I requested 3,000 people in six audiences across India, the Philippines and Kenya. They belonged to five communities: two WhatsApp groups, two Facebook groups and r/budgetfitness. A third each were written and interviewed by Claude Sonnet, Codex (gpt-6-luna) and Gemini 3.8 Flash. In the backend's settings:

LLM_POOL=claude_code:sonnet,codex:gpt-6-luna,antigravity:gemini-3.8-flash-low
LLM_POOL_STRICT=true          # keeps rounds and tests on each person's own provider (until a provider runs out; see below)

Each provider writes an equal share of every audience and interviews its own people. It then answers for them in the feeds, social rounds, survey and pitch, so one model speaks for a person from start to finish, unless a provider runs out mid-run (which happened here; see below). In total, 2,999 people were written and 2,987 were interviewed. The one person whose calls failed is counted, not filled in.

How one simulated person is built

Each audience is spread over slots (age, gender, city, income, life stage), so you don't get 500 copies of the same 24-year-old. Then, for each person:

1. a 150-250 word persona;
2. a stereotype guard, in code with no model call, so that no two people in an audience share both a job and a city (in this run it changed 20 people);
3. a 12-question life interview: how money is this month, what phone and data plan they have, a time they got scammed, who they ask before spending, what annoys them in ads;
4. follow-ups and a decision card: triggers, stoppers, dealbreakers, trust cues, and where what they say differs from what they do.

Here's one person as the model wrote him. Yash is 42 and lives in Pune. He drives an app cab and runs a mobile-accessories stall at weekends. He once lost ₹6,000 to a crypto course, so now he reads reviews and asks his wife and mother before buying. His says-vs-does line: he says he'd do 15 minutes of exercise while waiting for rides, but his shifts run long, so he probably won't.

What they went through

./mm cli feedtest --simulation $SIM --creatives @creatives.json   # add --seeds 3 for a range; this run used one seed
./mm cli survey --simulation $SIM --offer "7-day trial, no card" \
  --messages @messages.txt --split --pricing USD
./mm cli pitch --simulation $SIM --script @pitch.md --rebuttals @rebuttals.md

In practice, a single market-test config runs all of this.

• Feed test: five ads placed among organic posts in Facebook, Instagram, TikTok and Reddit feeds. Each person sees one ad, assigned at random. In total there were 5,140 person-platform sessions.
• Social rounds: four Reddit-like rounds on OASIS produced 3,565 actions, and 1,259 of those were "do nothing".
• Survey: answers come back as free text. Semantic similarity rating (SSR, arXiv 2510.08338) then compares each answer against five anchor statements using local embeddings, which gives a 1-5 distribution without another model call. When you ask a model for a number directly, you get narrow, unrealistic rating distributions. The SSR paper reports that rating the model's own words comes much closer to real survey panels.
• Pitch: every person hears the pitch, raises an objection, gets my rebuttal, and decides again.
• Analyst panel: five simulated analysts read the run independently.

The headline (simulated)

The signals disagree, and the report says so. The survey alone points to no-go, while the feed test and the analysts point to iterate.

The models disagreed

The stand-in row wasn't planned. Gemini's login ran out of quota for about four days mid-run, so Claude played the Gemini-written people (MM_PROVIDER_STANDIN=antigravity=claude). The results label those people "antigravity (played by claude)". That accident became the most useful comparison in the run: people written by Gemini but played by Claude answered like Claude's people, not like Codex's.

So in this run, the model doing the answering shaped the survey far more than the model that wrote the person. All three groups still named trust as the top objection. I read every rate above as a range between two temperaments, not as one market's answer. The dashboard's "Do the models agree?" section is built for this. It splits results by provider with 95% ranges and checks whether the would-convert ranges overlap and whether the top objection and best message match. Agreement is a robustness check on the simulation, not evidence about real buyers. This run can't tell me whether Codex's people are more realistic or just more negative. Only real data can settle that.

The pitch made it worse

Before the rebuttals: 1.1% yes, 64.0% maybe, 34.8% no. After them: 1.1% yes, 44.6% maybe, 54.2% no. The pitch scored 23 out of 100. The trust rebuttal moved 44.9% of the people who raised trust, and the price rebuttal moved 8.9%. The report's fix was to stop arguing and instead show the exact charge after day 7, the cancel steps and the local price. That matched the analysts' themes: no local payment option (UPI), a USD price competing with free YouTube, and a card wall that kills the trial.

What it took

Writing 2,999 full-depth people took 2 h 40 min through three login gateways. That includes a restart midway to apply a fix; a cache meant finished people weren't rewritten. The social rounds then kept hitting usage limits, so I moved the rest of the run to an Anthropic API key. From there it took 40 minutes for four rounds, 23 for the survey of 2,999 plus its retest, and 37 for the pitch. MarketMirror didn't meter the tokens used inside the rounds, survey and pitch, so I can't give you a cost for this run.

Limits you should know

• The people are synthetic. Detailed biographies don't make them representative.
• Attention, fatigue, social proof and landing friction include hand-set priors.
• The spread across seeds measures simulation variation, not predictive confidence. Test-retest (76.7% gave the same verdict here) measures consistency, not agreement with humans.
• Fitloop is invented, so none of this has been checked against real outcomes. backtest and calibrate exist for that, but they need your paired real data.

Try it

git clone https://github.com/DarkWorldCoder/market-mirror.git
cd market-mirror
bash install.sh
./mm configure
./mm provider check
./mm start
./mm cli quickstart --out scenarios/acme

Quickstart asks about your product, buyers and channels. It prepares the people, then stops before the remaining tests so you can review them. Preparation itself uses model calls, so start with a small cohort. From Claude Code, Codex, Cursor or Gemini, you can install the plugin or MCP server and ask in plain words.

The repo is at https://github.com/DarkWorldCoder/market-mirror (AGPL-3.0). If you've run real surveys or ad tests and have opinions on how to validate something like this, I'd like to hear them.

I wrote this with help from an AI assistant and checked every number against the run's saved results.