在 8GB Celeron 筆電上打造自己的本地 AI 編碼助手

I've started working on a project I've wanted to experiment with for a while: building my own local AI assistant, with the eventual goal of having it help me with development work.

There is one slight complication.

My development laptop isn't exactly an AI workstation.

We're talking:

• Intel Celeron N4000
• 2 cores / 2 threads
• 8 GB RAM
• Intel UHD Graphics 600
• Windows 11

No dedicated GPU. No 32 GB or 64 GB of RAM. No monster Ryzen or Core i9 sitting under the desk.

Just a little Celeron doing its best.

And somehow... it's working.

First, What Do I Mean by "My Own LLM"?

I'm not training a multi-billion-parameter foundation model from scratch.

That would require a completely different level of hardware, data and infrastructure.

Instead, I'm building my own LLM-powered assistant around an existing open-weight model.

The model handles inference, while I'm building the software around it: the interface, conversation handling, developer tools, file access, safety boundaries, prompts, context management and eventually things such as retrieval and tool use.

That distinction is important because the model itself is only one part of an AI application.

My goal isn't:

«Build ChatGPT from scratch.»

It's much more practical:

«Build a local AI assistant that is actually useful to me as a developer.»

And importantly, build it in such a way that the underlying model can eventually be replaced without rebuilding the entire application.

The Hardware Constraint Is Part of the Project

Most local AI discussions quickly reach hardware specifications that make my laptop quietly hide in the corner.

Large models simply aren't realistic on this machine.

With only 8 GB of system RAM and a 2-core/2-thread Celeron N4000, every decision has to take the available resources into account.

That means using:

• small models;
• quantised GGUF models;
• conservative context sizes;
• sensible thread counts;
• lightweight infrastructure;
• and plenty of benchmarking.

I'm currently targeting the sort of 1B–3B model range that has a realistic chance of running on the hardware.

I don't expect enormous token speeds.

That's not really the point.

If I can ask my own locally running assistant to explain some JavaScript, diagnose a small bug, generate a regex or inspect a controlled set of files, and it gives me a useful answer, then it has done its job.

In fact, I think getting useful local AI inference running on this hardware is an interesting engineering problem in itself.

Instead of throwing more compute at the problem, I have to ask:

How much useful AI can I squeeze out of the computer I already have?

Getting the First Local Response

The project uses "llama.cpp" as the local inference runtime.

One of the first major milestones was simply getting from:

Prompt

to:

My laptop
↓
llama.cpp
↓
quantised GGUF model
↓
response

without sending that prompt to a cloud AI service.

It sounds like a tiny milestone compared with everything I eventually want to build.

It didn't feel tiny.

There is something rather satisfying about looking at a Celeron N4000 generating tokens locally and thinking:

«You're actually running an LLM.»

The machine definitely isn't going to break any inference speed records, but it doesn't need to.

It works.

That gave me the foundation for everything else.

Turning the Model Into an Application

Running a model from a terminal is interesting, but I don't just want a terminal demo.

I want an application.

So the architecture is gradually becoming something like:

Frontend
↓
My backend
↓
AI provider layer
↓
llama.cpp
↓
Local model

That separation is deliberate.

I don't want the whole application tightly coupled to one particular model or runtime.

The frontend shouldn't care whether the response eventually comes from the tiny local model I'm using today, a better local model on future hardware, or even an optional cloud provider.

The surrounding application is the part I'm really building.

That includes things such as:

• conversation context;
• local conversation storage;
• system instructions;
• developer workflows;
• controlled file access;
• model configuration;
• tool handling;
• error handling;
• and eventually retrieval.

If I replace the model later, those parts shouldn't suddenly become useless.

It Already Has a UI

I've also moved beyond interacting with the model exclusively through the command line.

The project now has the beginnings of its own chat interface, including user and assistant messages, loading and error states, Enter-to-send behaviour and protection against accidental duplicate submissions.

I've also started working on conversation context, so follow-up messages can include previous messages instead of treating every prompt as an entirely new conversation.

That starts turning the project from:

«"Look! A model generated text!"»

into:

«"Okay... this is beginning to behave like an actual application."»

And that transition is probably one of my favourite parts of building it.

The Bit I Really Want: Developer Mode

The long-term reason I'm particularly interested in this project is development.

I already spend a lot of time working with JavaScript, PHP, Python, PowerShell, HTML, CSS, JSON, regex and various other bits of web development.

So I don't need my local assistant to know absolutely everything.

I'd rather have it become useful at the relatively small jobs I repeatedly encounter.

Things like:

Explain this function.

Why is this JavaScript behaving like this?

Generate a regex for this.

Summarise this log.

Convert this data to JSON.

Find the likely cause of this error.

Inspect these files and propose a small change.

That last one is where things become especially interesting.

I don't want an AI model to have unrestricted access to my filesystem and simply change whatever it wants.

The model's output should be treated as a proposal, not authority.

So I'm experimenting with controlled file access and human approval.

The basic idea is:

Developer request
↓
AI inspects allowed context
↓
AI proposes a change
↓
Human reviews it
↙ ↘
Reject Approve
↓ ↓
Nothing Apply exact
changes proposed edit

I've already reached the point where a coding test can make it through the workflow to the human approval gate.

That's a really important boundary for me.

The goal isn't an AI that gets free rein over my projects.

It's an AI that can say:

«"I think this is the change you need. Shall I apply it?"»

That is much closer to the developer tool I actually want.

Small Models Also Force Better Engineering

One thing I'm beginning to appreciate is that weak hardware exposes inefficiencies very quickly.

On a powerful system, an unnecessary extra inference pass might just be mildly annoying.

On a Celeron?

You notice. 😅

If a tool error causes the model to reread a file, regenerate context, retry something unnecessarily and go through several inference passes, that becomes expensive very quickly.

So optimisation isn't just:

«Make the model generate more tokens per second.»

It's also:

«Make the application ask the model to do less unnecessary work.»

That's a much more interesting problem.

I'm looking at things such as reducing repeated file reads, keeping tool errors concise, avoiding unnecessary context, controlling context size and measuring how many inference passes a coding task actually requires.

Sometimes the best optimisation is simply not asking the LLM to think about something twice.

Benchmarking Instead of Guessing

I'm also putting together a fixed benchmark for the assistant.

The plan is to test the same set of tasks against different configurations, including things such as:

• grammar correction;
• summarisation;
• JSON transformation;
• regex generation;
• JavaScript explanation;
• Python explanation;
• PHP explanation;
• deliberate coding bugs;
• multi-step instructions;
• and hallucination resistance.

Each test can then record things such as response quality, runtime, tokens per second, tool calls, retries and errors.

That gives me an actual baseline.

Then I can change one variable at a time:

Baseline
↓
Change one thing
↓
Run the same benchmark
↓
Compare quality + performance
↓
Keep it or revert it

That feels much more useful than endlessly changing model parameters until something feels faster.

And on hardware this constrained, I don't necessarily want the fastest configuration anyway.

If configuration A gives me 2 tokens per second but consistently useful coding answers, while configuration B gives me 3 tokens per second and writes nonsense, configuration A wins.

Local AI Doesn't Have to Mean Huge AI

I think that's probably the biggest thing this project is teaching me so far.

When people talk about running LLMs locally, it's easy to focus entirely on parameter counts, GPUs and increasingly enormous models.

But a useful AI tool doesn't necessarily need to be enormous.

It needs to be good enough at the task you're giving it.

My Celeron isn't going to run the same models as an RTX-equipped workstation.

That's fine.

I'm not trying to make it.

I'm trying to find the point where a small local model plus carefully designed software becomes useful.

And honestly, seeing an 8 GB, 2-core Celeron laptop go from:

"There's no way I'm running useful AI on this."

to:

"Wait... it just inspected that and reached my approval screen."

has been incredibly satisfying.

Where I'm Going Next

There is still plenty to build.

The immediate work is less glamorous than shouting "AI!" at everything.

I need to improve error recovery, create a proper benchmark baseline, finish the provider abstraction, improve conversation handling and continue tightening the controlled developer workflow.

Later, I can experiment with retrieval, better models, more developer tools and perhaps dedicated coding models when the hardware allows it.

But I'm deliberately not making those requirements for version one.

V1 doesn't need autonomous agents, image generation, voice, enormous context windows or a giant model.

It needs to do a few things reliably.

If this little machine can eventually help me debug code, understand files, generate regex, explain errors and safely propose edits to my projects, I'll consider that a success.

And if I eventually move the application onto significantly better hardware?

I can swap out the little model for something considerably more capable while keeping the architecture I've already built.

For now, though, I'm rather enjoying finding out just how far I can push an 8 GB Celeron N4000.

The little CPU that probably never expected "run an AI coding assistant" to appear in its job description. 😅