SmolVLA 野餐機械人微調後，未見指令措辭成功率達 78%

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

In Week 1, an open robot model (SmolVLA) packed my picnic only when I used its exact training sentence. This week I fine-tuned it on a few demos, each labeled with the ways people actually ask, and priced the fix.

For the price of a coffee, the robot learned how I talk.

	Before	After 1 demo per item
Phrasings I taught	31%	80%
Phrasings it never saw	29%	78%
Demos needed		4 in total
GPU cost		about $1.50

Who would pay: anyone whose robot meets people who did not write its training data.

Customer	What they say	What they get
Café with a kitchen arm	"grab me the oat milk"	A model that understands the staff
Care home	"could you get my cup?"	A robot that does not need exact commands
Robot vendor abroad	Their customers' dialect	A fine-tune per market, priced per job

Demo

Upper clip of each pair: the published model. Lower clip: after one demo per item, same scene.

Rows	Request
Top two	"pack the {item} for our picnic" (taught)
Bottom two	"the basket still needs the {item}" (never seen in training)

The results

Six fine-tuned models (1 to 40 demos per item) plus the published one, all tested on the same 420 episodes.

Demos per item	Taught	Never seen	Original sentence	6 untrained items
0 (published)	31%	29%	100%	87%
1	80%	78%	95%	60%
3	86%	74%	90%	70%
5	84%	74%	88%	70%
10	86%	77%	92%	72%
20	92%	79%	98%	68%
40	92%	78%	98%	68%

1. One demo per item does most of the work

	Phrasing	0 demos	1 demo
A ●	grab the {item} and put it in the basket	23/40	35/40
B ●	{item} in the basket	3/40	29/40
C ●	pack the {item} for our picnic	1/40	31/40
D ●	could you put the {item} in the picnic basket?	23/40	33/40
E ○	toss the {item} into the basket	8/40	33/40
F ○	the basket still needs the {item}	7/40	25/40
G ○	can you get the {item} into the basket for me?	18/40	32/40
H ○	move the {item} over to the basket	13/40	34/40

E to H were frozen before training, under rules that ban the verbs and sentence shapes of A to D.

2. The twist: more demos did not help with new words

	1 demo	40 demos
Taught phrasings	80%	92%
Never-seen phrasings	78%	78%
Orange juice, never-seen	36/40	23/40

Same scene, same request. Likely reason: every demo carried the same five sentences, so more demos taught more ways to move, not more ways to be asked. One training run per N and cream cheese went the other way, so this is a strong hint, not a proof (see also VLA or IL? by @fn8211).

Lesson for the side hustle: collect more phrasings, not more demos.

3. The fine print: it forgets what it was not taught

Original sentence	0 demos	1 to 40 demos
4 trained items	100%	88-98%
6 untrained items	87%	60-72%

The fine-tune only saw four items, so the model drifted toward them. Ketchup, the first item my robot ever packed in Week 1, still works in 6 to 8 of 10 scenes, but loses this one. Next fix to try: mix in a few demos of every item.

What It Costs

	Time	Money
One fine-tune (shared GPU, 3000 steps)	48 min	$1.50
One fine-tune (GPU alone, 1000 steps, estimated)	10 min	$0.30
Whole experiment (6 models, about 4,400 evaluation episodes)	6 h	$11
Buying the GPU instead		$10,000+

Validation stopped improving after step 1000 for 3+ demos per item (training chart below), so a real job can stop early.

How I Built It

Part	Choice
Model	`lerobot/smolvla_libero`, 450M parameters
Data	Public `lerobot/libero` demos, first N per item, 4 items
The trick	A wrapper around `lerobot-train` relabels each sample with one of 5 sentences at random; no new recordings
Training	Whole model, 3000 steps, batch 32, lr 2.5e-5
Checkpoint choice	Best of steps 1000/2000/3000 on held-out scenes, taught phrasings only
Test	420 episodes per model, LIBERO-Object in MuJoCo, same scenes and seeds for all
Hardware	1x A100 80 GB on RunPod, training and evaluation side by side in tmux

Gotcha: the pod showed 256 CPUs but had 16; evaluation threads starved training until I set OMP_NUM_THREADS=1 (setup guide).

Limits: one training run per N; simulation only; phrasing G turned out closer to D than my rules intended.

Code

Hacktoberfest 2026

A month-long exploration of open-weight AI, VLA, and Physical AI.

Goals

Explore open-weight AI models
Build VLA/Physical AI projects
Participate in DEV Challenges
Document experiments and learnings

Projects

hacktoberfest-2026/
├── dev-challenge/          # Hacktoberfest DEV Challenge entries
│   ├── weekend-challenge/  # Shared Workbench Reset Assistant (local open-weight VLM)
│   ├── week-1/             # Picnic Packing with SmolVLA (paraphrased instructions in LIBERO)
│   └── week-2/             # Teaching the Picnic Robot My Words (SmolVLA fine-tune on a rented GPU)
├── global-hack-week/       # Global Hack Week: Hacktoberfest challenge write-ups
│   ├── 01-registration.md  # Registration challenges
│   └── github/             # GitHub challenges (Codespaces, repo management, Copilot)
├── devrelay/               # DevRelay usage notes and reviewed agent session transcripts
└── models/                 # Shared local model weights; excluded from Git

Development Environment

Component	Configuration
Host	Windows
CPU	Intel Core Ultra X7 358H; 16 logical CPUs visible in WSL
GPU	Intel Arc B390 GPU; Windows driver 32.0.101.8356
RAM

…

Where	What
`dev-challenge/week-2`	Relabeling wrapper, training and evaluation scripts, all tables, chart and video scripts
Hugging Face Hub	All six fine-tuned models with a model card

Why Does Open Innovation Matter?

Open piece	Why the business needs it
Open weights	You cannot fine-tune what you cannot download
Open data	I relabeled public demos instead of recording any
Open benchmark	I could measure what one demo buys and what it breaks
Hourly GPUs + small model	The customer pays for minutes, not a GPU or a per-call API
A model file	The customer runs it offline on their own robot, free of anyone's pricing

No paying user yet. The first person I would show it to: anyone who has watched a robot arm ignore a perfectly reasonable request.

My Agent Session

I built this with Claude Code over a week. The session below is a curated slice: the experiment design, the GPU run and its one bug, the results, and the charts and videos.

Note: I worked with the agent in Korean. This is a condensed English translation of the key moments, not a verbatim transcript.

Disclosure: I designed the experiment, ran the training and every test episode, and reviewed every result. An AI coding agent helped write the scripts, drafted phrasings E to H under rules I set (I reviewed and froze them before training), analyzed the results, made the charts and videos, and drafted this post.