我為 Android 開發了一款無需雲端的 AI 助手 Snor

Running language models locally on Android, keeping conversations on-device, and exploring what happens when AI doesn't depend on a server.

Most AI applications have something in common.

You type a message, send it somewhere, wait for a server to process it, and receive a response.

It's convenient. It's also an architecture we've become so accustomed to that we rarely question it.

What if the model didn't need to run on someone else's infrastructure?

What if your phone could handle the inference itself?

That question is what led me to build Snor, an Android application designed around local AI inference.

Explore Snor

The idea: AI without the usual cloud dependency

I wanted to experiment with something different from the typical AI chatbot architecture.

Instead of making a remote API the center of the application, I wanted the phone itself to be capable of running language models.

That introduces an entirely different set of engineering problems.

You're no longer just building a chat interface around an API.

You're dealing with:

• Model loading and memory consumption.
• CPU and GPU inference.
• Context window management.
• Native Android integration.
• Generation latency.
• Device-specific performance limitations.

The interesting part is that all these constraints become part of the application design.

Running LLMs directly on Android

Snor is built around local model inference.

One of the technologies involved is llama.cpp, which makes it possible to run compatible quantized language models using GGUF files.

The application also explores LiteRT-LM for on-device inference.

This means model execution can happen directly on the device rather than requiring every prompt to be sent to an external AI provider.

For smaller devices, this distinction matters.

A model that works comfortably on a desktop might be completely impractical on a phone with limited available memory.

That's why model selection, quantization, and resource management are important parts of the project.

Why smaller models?

One of the more interesting directions I've been exploring is lightweight models.

Instead of assuming that bigger models are always better, I've been focusing on the SmolLM2 family, particularly:

• SmolLM2 135M
• SmolLM2 360M

These models aren't intended to compete with large frontier models.

Their value is in exploring what can be accomplished with relatively small models under mobile hardware constraints.

There are trade-offs, obviously.

Smaller models have limited reasoning capabilities, and they aren't suitable for every task. They can also produce incorrect or incomplete answers.

But that limitation is part of the engineering problem.

A useful local AI application shouldn't pretend that a tiny model can do everything.

It should manage expectations, handle output carefully, and acknowledge when it cannot provide a reliable answer.

Privacy is an architectural decision

One of the main reasons for building Snor around local inference is data control.

When inference happens locally, conversations don't need to be transmitted to a remote model provider.

Snor is designed around:

• Local-first AI inference.
• No account requirement.
• No default cloud dependency.
• On-device conversation storage.
• Optional cloud AI integrations when users explicitly choose them.

This distinction is important.

Privacy shouldn't just be a marketing statement added to an application description. It should influence how the software is architected.

Local inference provides a different relationship between the user, the application, and the model.

The engineering challenges are more interesting than the chat interface

The UI is only one part of the project.

The difficult work happens underneath it.

1. Native model integration

Running native inference engines inside a Flutter application requires communication between Dart and native code.

Snor uses native integration to connect its application layer with the inference engines.

This creates additional responsibilities around model lifecycle management, memory handling, and safe execution.

2. Model lifecycle and concurrency

Native inference isn't something you want to manage casually.

Loading a model, generating a response, clearing context, and disposing of native resources need predictable execution.

Concurrent operations can cause serious problems if the underlying native resources aren't protected.

This is why synchronization and controlled model lifecycle management are important parts of the implementation.

3. Working within mobile hardware limitations

Android devices don't have unlimited memory or desktop-class cooling systems.

Model size, available RAM, context length, and inference settings all affect the experience.

A mobile AI application needs to account for those constraints rather than assuming every device behaves the same way.

4. Making smaller models useful

A smaller model shouldn't be given an unlimited conversation history and expected to perform like a much larger model.

Snor explores bounded conversation history, model-specific context limits, and output constraints to keep interactions manageable.

The goal is to build a more predictable experience around the capabilities of the model actually running.

Local-first doesn't have to mean cloud-never

I don't think local AI and cloud AI need to be competing philosophies.

They solve different problems.

Local models offer more control over data and can work without an internet connection.

Cloud models offer access to significantly larger models and more demanding capabilities.

Snor explores a local-first approach while allowing optional cloud integrations.

The important distinction is that cloud usage should be a deliberate choice rather than an invisible requirement.

What I'm exploring next

Snor is an ongoing experiment in making local AI practical on Android.

Some areas I'm interested in improving include:

• Better inference performance across different Android devices.
• More reliable model downloads and loading.
• Improved handling of partial generation and interrupted inference.
• Smarter context management.
• Better lightweight model selection.
• A more polished mobile AI experience.

I find this space particularly interesting because mobile hardware keeps evolving, while efficient inference techniques and smaller models continue to make local AI more practical.

There's still a lot to figure out.

Try Snor

If you're interested in running language models locally on Android, you can explore the project here:

Visit Snor

I'm particularly interested in hearing from developers experimenting with llama.cpp, GGUF models, Flutter, and on-device inference.

The bigger question I'm exploring is simple:

How much useful AI can we actually fit inside a phone, without depending on someone else's server?