使用 C# 和 ILGPU 構建微型 GPT:由 OpenAI Codex 驅動
Building a Tiny GPT with C# and ILGPU — Powered by OpenAI Codex
作者以 C# 和 ILGPU 建立開源教育項目 my-gpt,實作字元級 GPT,並由 OpenAI Codex 負責大部分開發工作。模型採用 2 層 Decoder-only Transformer,嵌入維度為 64,上下文長度為 64 個字元。項目使用 GPU 執行模型的前向及反向計算,並以 CPU 參考計算和數值梯度檢查驗證實作。
GitHub: https://github.com/tingwei628/my-gpt
Introduction
I've been exploring how GPT models work internally, from basic mathematical operations to Transformer architecture and GPU acceleration.
Inspired by Andrej Karpathy's minGPT, I started my-gpt, an open-source educational project that implements a small, character-level GPT using C# and ILGPU.
One of the most interesting aspects of this project is how it was developed:
Most of the implementation was carried out by OpenAI Codex.
Rather than writing every component manually, I provided the requirements, hardware constraints, and feedback, while GPT through Codex handled much of the coding, educational documentation, training operations, and verification.
The goal was not to build a production-ready LLM, but to explore two questions:
- Can we implement and train a small GPT using C# and GPU acceleration?
- How far can an AI coding agent like OpenAI Codex take a technical project involving neural networks and GPU programming?
Why C# and ILGPU?
Most educational Transformer implementations use Python and PyTorch.
These frameworks are excellent for machine learning, but I wanted to explore a different approach.
As a C# developer, I was interested in understanding how neural networks could be implemented using the .NET ecosystem.
That's where ILGPU comes in.
ILGPU is a high-performance JIT compiler for GPU programming in .NET. It allows developers to write GPU kernels using C# and execute them on supported GPU backends, including CUDA.
In this project, C# handles the application logic, model architecture, and training workflow, while ILGPU executes the computationally intensive mathematical operations on the GPU.
This provides an opportunity to explore GPU programming without switching the entire project to Python or C++.
OpenAI Codex as the Main Development Partner
A major motivation behind this experiment was to explore AI-assisted software development beyond ordinary application code.
OpenAI Codex was the primary implementation tool throughout the project.
My role focused on defining the project goals, providing hardware constraints, evaluating results, and giving feedback.
Codex handled much of the technical implementation, including:
- Transformer architecture and GPU kernels
- Forward and backward propagation
- Model training and optimization
- Checkpoint saving and loading
- Text generation and inference
- Documentation and educational examples
- Mathematical correctness checks
Additional AI agents were also used to review educational clarity, GPU mathematics, and reproducibility.
This made the project an interesting experiment in AI-assisted engineering.
Writing code is only part of building a machine learning system. The implementation also needs to produce mathematically correct results, train consistently, and behave as expected during inference.
For that reason, the project includes independent CPU reference calculations, numerical gradient checks, and other validation tests.
These checks provide evidence that the implementation behaves as intended, although they do not replace broader independent testing.
How the Tiny GPT Works
The model follows a decoder-only Transformer architecture, similar in principle to GPT models, but with a much smaller configuration.
Instead of processing complex text tokens, it uses a character-level tokenizer.
Each character becomes a token, and the model learns to predict the next character based on the previous characters.
For example:
Input: the fox
Target: next character
The model generates text by repeatedly predicting a character and appending it to the existing sequence.
Model Architecture
The main processing pipeline looks like this:
Input Text
|
v
Character Tokenization
|
v
Token Embedding + Position Embedding
|
v
Transformer Block 1
|-- Layer Normalization
|-- Multi-Head Causal Self-Attention
|-- Residual Connection
|-- Feed Forward Network
|-- Residual Connection
|
v
Transformer Block 2
|
v
Final Layer Normalization
|
v
Linear Projection
|
v
Next-Character Logits
|
v
Text Generation
Model Configuration
The project uses a deliberately small configuration to keep training and experimentation manageable.
| Parameter | Value |
|---|---|
| Architecture | Decoder-only Transformer |
| Transformer layers | 2 |
| Embedding dimension | 64 |
| Attention heads | 4 |
| Feed-forward dimension | 256 |
| Context length | 64 characters |
| Precision | FP32 |
| Optimizer | AdamW |
| Default learning rate | 0.003 |
| Tokenization | Character-level |
Although this model is tiny compared with modern language models, it includes many of the same fundamental building blocks.
These include embeddings, causal self-attention, layer normalization, residual connections, and feed-forward networks.
GPU Acceleration with ILGPU
One of the core objectives was to implement the mathematical operations required for GPT training using GPU kernels.
The project uses ILGPU for operations such as:
- Matrix multiplication
- Softmax
- Layer normalization
- Attention calculations
- Gradient computations
- AdamW parameter updates
The CPU handles tasks such as reading training data, managing commands, and saving checkpoints.
The GPU performs the model's forward and backward computations.
The implementation separates the model structure from the mathematical kernels.
For example:
Model.cs organizes the Transformer computations, model parameters, forward propagation, and training workflow.
Kernels.cs contains the GPU operations used to execute the mathematics.
Reference.cs provides independent CPU calculations for comparing results against GPU computations.
This separation makes it easier to explore how high-level neural network operations map to lower-level GPU execution.
Training and Backpropagation
The training process follows the standard next-token prediction approach.
For each training sequence, the model predicts the next character and compares the prediction with the actual target.
The training loop follows these steps:
Training Text
|
v
Forward Propagation
|
v
Calculate Cross-Entropy Loss
|
v
Backward Propagation
|
v
Compute Gradients
|
v
AdamW Optimizer
|
v
Update Model Weights
|
v
Repeat
Unlike frameworks that automatically differentiate the entire model, this implementation explicitly handles the backward computations required by its GPU operations.
The project also supports saving checkpoints containing model weights, optimizer state, vocabulary information, and training progress.
This allows training to be resumed later.
Verifying the Mathematics
GPU code can execute successfully while still producing incorrect numerical results.
To reduce this risk, the project includes several checks:
- Comparing GPU outputs against independent CPU calculations
- Numerical gradient checking
- Testing causal attention masking
- Verifying batch isolation
- Testing checkpoint save-and-load behavior
These checks are particularly important when implementing neural network mathematics without relying on a mature automatic differentiation framework.
Learning GPT Step by Step
In addition to the working model, my-gpt includes seven lessons designed to explain the implementation gradually.
| Lesson | Topic |
|---|---|
| 0 | GPU Programming with C# and ILGPU |
| 1 | Neural Network Fundamentals |
| 2 | Matrix Operations |
| 3 | Transformer and Attention |
| 4 | Mathematical Verification |
| 5 | GPT Training and Backpropagation |
| 6 | Text Generation and Inference |
The lessons begin with simple GPU vector addition and gradually introduce weights, gradients, matrix multiplication, attention, and training.
By the final lesson, these individual components are combined into a working character-level text generation system.
The idea is to make it possible to follow the mathematics and execution flow, rather than treating the model as a black box.
Running the Project
The current project targets Windows with a compatible NVIDIA CUDA GPU and the .NET 10 SDK.
First, clone the repository:
git clone https://github.com/tingwei628/my-gpt.git
cd my-gpt
Run the mathematical demonstrations and validation checks:
.\my-gpt.cmd math
.\my-gpt.cmd attention
.\my-gpt.cmd check
To train a new model:
.\my-gpt.cmd train lessons/my-gpt/quickstart.ckpt 100
Once a compatible checkpoint has been created, launch the interactive interface:
.\my-gpt.cmd
Then enter a prompt such as:
the fox
The model will generate additional characters using its learned weights.
The repository does not include pretrained checkpoint files, so a fresh clone requires training a model or supplying a compatible checkpoint before interactive generation.
A short 100-step run is useful for exploring the workflow, but it does not guarantee good text generation quality.
Current Limitations
This project is designed for education and experimentation, not production use.
The current model has several limitations:
- A relatively small training corpus
- Character-level rather than subword tokenization
- A context window of only 64 characters
- Limited language generation capabilities
- A small Transformer architecture
The model is trained on simple English patterns and can learn to reproduce similar structures.
It is not intended to compete with modern large language models or function as a general-purpose chatbot.
The more important result is having an implementation where the computations can be inspected, tested, and modified.
What I Learned
The most interesting part of this project was not simply getting a small GPT to generate text.
It was exploring how an AI coding agent can help implement a system involving GPU programming, neural network mathematics, model training, and verification.
OpenAI Codex played the central role in turning the project requirements into a working implementation.
At the same time, this experiment reinforced the importance of validation.
AI-generated code still needs mathematical checks, reproducible tests, and careful evaluation, especially when working with low-level numerical computations.
Another takeaway is that C# and ILGPU can be useful tools for exploring neural network internals, even though Python remains the dominant ecosystem for machine learning.
What's Next?
There are several directions worth exploring:
- Improving GPU kernel performance
- Experimenting with FP16 or mixed precision
- Implementing KV caching for inference
- Exploring BPE tokenization
- Supporting larger model configurations
- Improving evaluation on unseen text
- Comparing performance against other implementations
These are potential future improvements rather than features already completed.
Final Thoughts
If you're interested in C#, GPU programming, Transformers, or AI-assisted development, feel free to explore the source code.
來源:DEV Community · dev.to