AeroLoop：會保留失敗策略的空氣質素智能體

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

AeroLoop — The Air-Quality Agent That Keeps Its Own Ghosts 👻

What if an AI agent remembered every mistake it ever made?

Not just its successful predictions. Not just its latest model weights. Every failed experiment, every rejected idea, every strategy that looked promising but couldn't survive a proper test.

That's AeroLoop.

AeroLoop is a self-improving AI agent that predicts PM2.5 air pollution across London. It saves every prediction before ground truth exists. When the actual reading arrives, the agent scores itself, examines its prediction errors, and attempts to improve its estimation strategy.

But there's a twist: it doesn't just learn from its mistakes. It preserves them.

Every strategy is versioned. Successful experiments become the new incumbent; unsuccessful ones are saved as vN-rejected and displayed on the dashboard as literal ghosts 👻. Nothing is silently overwritten.

The inspiration came from the idea that an intelligent machine shouldn't just remember what worked. It should remember what failed, why it failed, and what it learned from the experience.

Air pollution is already a ghost story: PM2.5 is the invisible residue of traffic, heating, and weather that lingers in a city long after its source is gone. AeroLoop adds another ghost to the story: the history of an AI learning from its own mistakes.

How the loop works

1. Predict: Use only features available at prediction time.
2. Remember: Save the prediction before ground truth exists.
3. Evaluate: Match the prediction with the actual reading when it arrives.
4. Reflect: Let an open-weight LLM examine the agent's historical errors and propose strategy improvements.
5. Challenge: Evaluate the candidate on a chronological holdout set that was never used for tuning.
6. Decide: Keep the candidate if it improves holdout MAE. Otherwise, preserve it as a rejected ghost.

The strategy is an intentionally small, interpretable linear correction over the CAMS model output:

pred = (
    pm25_model
    + humidity_coef * (humidity - 50)
    + wind_coef * (wind - 5)
    + precip_coef * precipitation_mm
    + bias
)

The LLM provides the imagination; the holdout provides the judgment.

Any proposed parameters are clamped to sensible bounds. If the model is unavailable or returns invalid output, a deterministic heuristic can propose a candidate instead. The report records exactly which model and parameter source were used. No silent fallbacks.

What actually happened

I ran a simulated London day with 4 stations × 30 hourly ticks, producing 120 matched prediction/reading pairs.

Here are the results:

The final result:

Holdout MAE: 3.94 → 2.99 — approximately 24.1% improvement.

One particularly interesting subplot: TabPFN initially beat the hand-tuned formula outright. Gemma's incremental tuning subsequently edged ahead. A foundation model and a four-parameter formula competed on the same unseen data, with exactly the same evaluation gate.

And when Gemma proposed increasing the bias to 4.5, the holdout said no.

That strategy became a ghost instead of silently replacing the previous version.

Important: These results come from a deterministic synthetic demo dataset, not real-world measured pollution readings. The holdout is chronological and was kept separate from strategy tuning.

Demo

To reproduce the demo:

1. Start the backend.
2. Open the dashboard.
3. Click Simulate 24h.
4. Click Self-Improve.
5. Watch the agent evaluate, improve, and preserve its rejected strategies.

The demo can run entirely locally with Ollama.

Code

GitHub repository: https://github.com/udirobert/aeroloop

Clone and run:

git clone https://github.com/udirobert/aeroloop.git
cd aeroloop/backend

python -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt
cp ../.env.example .env

uvicorn main:app --port 8000

Then open the dashboard and run the simulation.

Two important endpoints:

• POST /demo-day — generates a complete day of prediction and ground-truth memories using deterministic synthetic data.
• POST /self-improve — runs the complete analyse → propose → holdout-gate → keep/reject loop.

How I Built It

AeroLoop combines open data, open-weight AI models, local inference, and a persistent agent memory.

The reasoning engine:

• Gemma 3 27B IT: The primary reasoning model, used to inspect historical prediction errors and propose updated strategy parameters.
• NVIDIA Nemotron Mini: Available through Ollama for fully local inference.
• Gemma via OpenRouter: A documented alternative endpoint.
• TabPFN (Prior Labs): An independent tabular foundation-model challenger, trained on the tuning split and evaluated on the same untouched holdout.

The model configuration is switchable, allowing different reasoning engines to compete without changing the core agent architecture.

The data layer:

• Open-Meteo provides CAMS PM2.5 forecasts and weather features.
• SQLite stores predictions, ground-truth readings, strategy versions, and experiment results.
• A chronological evaluation pipeline prevents future information from leaking into training or tuning.

The self-improvement mechanism:

The LLM doesn't directly rewrite the application or modify arbitrary code. It proposes parameters for a small, interpretable estimation formula.

Candidates are validated, bounded, evaluated against the incumbent, and either promoted or rejected.

This makes the system easier to inspect, reproduce, and trust.

Why Does Open Innovation Matter?

AeroLoop would be a very different project if its intelligence, memory, and evaluation process were locked inside a proprietary API.

Open innovation made several things possible:

1. Running with the lights (and internet) off.

The reasoning engine can run locally through Ollama using Nemotron Mini. No hosted inference service is required for the local demo.

2. Freedom to change the brain.

The architecture supports different OpenAI-compatible endpoints. Gemma, Nemotron, and other compatible models can be tested without rebuilding the agent.

3. Memory that belongs to the project.

Predictions, readings, strategy versions, and rejected experiments live in a local SQLite database. The agent's history isn't trapped in a vendor's infrastructure.

4. Independent challengers.

TabPFN can compete against the LLM-guided formula under exactly the same evaluation rules. The system doesn't care which model wins; it cares whether the candidate performs better on unseen data.

5. Reproducibility and transparency.

The data sources, strategy parameters, model choices, fallbacks, and evaluation results are inspectable. If a model fails, the system records what happened rather than pretending everything worked.

Open innovation made it possible to build an agent whose intelligence can be exchanged, whose memory can be inspected, and whose improvements can be independently challenged.

My Agent Session

The agent's experiment history is preserved in its SQLite database, including the proposals, parameter changes, holdout scores, and rejected versions.

I haven't included a DevRelay session recording for this submission.

The version history itself demonstrates the agent's decision-making process: every improvement has evidence, and every rejected experiment remains part of the record.

The Spirit of It

The most interesting moment wasn't when the agent improved.

It was when the agent failed.

Gemma proposed increasing the bias from 4.2 to 4.5. The candidate looked like another reasonable incremental adjustment. But the holdout evaluation showed that it performed worse.

So AeroLoop rejected it.

The strategy was saved as v7-rejected and appeared on the dashboard as a ghost.

It wasn't deleted. It wasn't quietly forgotten. It remained visible as part of the machine's history.

That is what I wanted the Ghost in the Machine theme to mean.

We often build AI systems that celebrate their latest answer and hide everything that came before it. AeroLoop takes the opposite approach.

Its ghosts are reminders that intelligence isn't just about getting things right. It's also about remembering when you got things wrong.

The LLM is the imagination. The holdout is the judge. The graveyard is the audit trail. 👻

Prize Categories

[Add the applicable partner prize categories here once confirmed against the challenge rules.]

Built for Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

Repository: https://github.com/udirobert/aeroloop