LocalLens：在瀏覽器幕後運作的隱形 AI 自動化智能體

Hacktoberfest Open-Source AI Challenge: Ghost in the Machine Submission 👻🌼

*This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine*

What I Built

I built LocalLens, a privacy-first, on-device AI browser extension (MV3) that acts like an invisible digital spirit inside your browser—turning natural language commands into autonomous DOM interactions without transmitting a single byte of screen state to remote servers.

Theme Interpretation & Inspiration

The "Ghost in the Machine" theme represents an invisible, autonomous entity operating silently behind the physical interface. Most AI agents act like cloud phantoms—remote entities hovering over third-party servers, demanding full access to your DOM trees, credentials, and raw screenshots.

LocalLens flips this paradigm. It brings the "ghost" entirely into your own machine. Operating local perception via client-side WebGPU and quantized open-weights, LocalLens silently watches the page, redacts sensitive PII on render like an invisible filter, and executes complex clicks, inputs, and navigations behind the scenes.

I built this for my developer friend Ankit, who needed an intelligent agent to automate repetitive administrative portals and documentation forms—without giving third-party cloud APIs visibility into private keys or student records.

Demo

Code

LocalLens

LocalLens is a privacy-first, locally-deployable browser automation agent. It bridges a Chrome MV3 extension with a Vision-Language Model (VLM) backend to autonomously execute natural language tasks on any webpage (e.g., "Submit the login form").

🏗️ Architecture

The system is split into a robust frontend extension and an intelligent Python backend:

1. Browser Extension (Frontend)

• Content Script (content.ts): Runs on the webpage. Walks the DOM, identifies interactive elements, stamps them with stable data-agent-id tags, calculates bounding boxes, and redacts PII (emails, cards, SSNs) client-side before it ever leaves the browser. It also acts as the actuator, executing clicks and typing based on backend commands.
• Background Worker (background.js): The secure relay. It captures visible tab screenshots and relays messages between the popup and the content script.
• Popup (agentLoop.ts): The orchestration brain. It fetches the UI Graph context from the content script, packages…

How I Built It

LocalLens is built around a fully local open-source AI architecture that operates entirely within your machine:

• In-Browser Perception (WebGPU + ONNX Runtime Web): Lightweight vision and token classification models run directly inside the active browser tab context. The extension maps interactive DOM elements and instantly redacts sensitive PII (emails, authentication headers, API keys) before any action planning takes place.
• Local Planning Engine (FastAPI Backend): A lightweight local backend interfaces with open-weight LLMs (such as Qwen 2.5 or Gemma) hosted locally via Ollama or vLLM. It parses natural language intent into structured JSON execution routines.
• Execution Engine: Dispatches sanitized JSON action sequences back into the active browser tab context to execute clicks, scrolls, and text inputs automatically.

Why Does Open Innovation Matter?

Open-source AI and client-side runtimes are what allow this "ghost" to stay strictly inside your local machine:

1. Guaranteed Data Sovereignty: Closed cloud APIs force users to stream private page states over the network. Open-weight models and ONNX WebGPU runtimes keep the entire perception pipeline on-device, guaranteeing zero data exposure.
2. Predictable $0 Operating Cost: Running open-weight models locally removes API token billing and rate-limiting, allowing continuous, background automation completely free of charge.
3. Model Flexibility: Users can swap open-weight models based on their hardware specs (e.g., fast 3B parameter models for laptops vs. larger models for workstations).
4. Offline Capability: Once weights are downloaded, LocalLens can perform structured DOM navigation and form handling completely offline.

My Agent Session

The Spirit of It

The moment LocalLens truly felt like a "ghost in the machine" was during initial testing with Ankit. He typed a prompt into the extension interface, hands completely off the keyboard, and watched as the extension parsed a dense, multi-input administrative form, masked sensitive credentials in real-time, and auto-populated fields step-by-step.

Seeing complex DOM operations unfold locally—with zero network calls to cloud LLMs—felt like watching an invisible assistant control the browser from within the hardware itself.

Prize Categories

• Best Use of Gemma (If using Gemma as your local model)
• Best Use of GitHub Copilot