AppTutorial 為 Windows 提供本機 AI 螢幕操作教學

Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle Submission 💸

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

This app provides a "tutorial" for users using a complicated app or software to guide them through a set of instructions for their desired or picked from one of the preset instructions. Many people, including myself, have had trouble dissecting what an app does and figuring out how to navigate it. Although searching up your problem on the web could be a solution, it usually is inconvenient and lacks interactivity, unlike this app.

1. Prerequisite:

install Ollama from ollama.com, then:
ollama pull qwen2.5vl

2. Get the code

git clone https://github.com/codename-ryan/appTutorial.git
cd appTutorial

3. Install + run

python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -e .
python -m apptutorial.app

Demo

Code

appTutorial

An AI-powered, on-screen software tutor for Windows, running on a local open-weight vision model (no cloud AI, no API key). Press a hotkey on a confusing app (Excel, an IDE, a web app) and appTutorial captures the active window, identifies the app and your intent, and walks you through tasks step by step — verifying each step against your live screen and highlighting where to focus.

Status: v1 built (Excel is the deeply-tested app). See SPEC.md for the full spec.

How it works

Hotkey (default Ctrl+Shift+T) over the app you want help with.
An open-weight vision model (default: Qwen2.5-VL via local Ollama) identifies the app and suggests tutorials; pick one or type your own goal.
The model plans the steps; you get one step at a time with a see-through highlight on exactly where to act.
Each step is verified automatically: deterministic local checks first (Excel…

How I Built It

For this project, I used the Open-weight model Qwen2.5-VL to identify the app, plan the tutorial, and for the user's action identification. Ollama hosts the model and exposes an endpoint and then it applies JSON-schema-constrained decoding to get the structured output.

My Agent Session

I consulted many of my friends and close acquaintances to try out and test the app. They provided overall positive reviews and a vow to use this app to facilitate their work.