Attention Mechanism
Q·KT → softmax → weighted V |
Autoregressive GPT
Token-by-token generation |
LoRA Fine-tuning
Low-rank weight injection |
Word Embeddings
Contrastive learning → semantic clusters |
DPO Alignment
Preferred vs. rejected → policy update |
RAG Pipeline
Retrieve → augment → generate |
Flash Attention
Tiled O(N) memory computation |
Quantization
Float32 → Int8 = 4x compression |
Mixture of Experts
Sparse routing to specialist MLPs |
BPE Tokenizer
Iterative pair merging → vocabulary |
BERT
Bidirectional attention + [MASK] prediction |
KV-Cache
Memoize keys/values — stop recomputing |
Beam Search
Tree search with top-k pruning |
RoPE
Position via rotation matrices |
PPO (RLHF)
Clipped policy gradient for alignment |
State Space Models
Linear-time selective state transitions |
Convolutional Net
Sliding kernels → feature maps |
Diffusion
Noise → data via iterative denoising |
GAN
Generator vs discriminator minimax |
VAE
Encode → sample z → decode |
RNN vs GRU
Vanishing gradients and gating |
Optimizers
SGD vs Momentum vs Adam convergence |
Batch Normalization
Normalize activations → stable training |
Dropout
Kill neurons → prevent overfitting |
QLoRA
4-bit base + full-precision adapters |
GRPO
Group-relative rewards, no critic |
REINFORCE
Log P(a) × reward = gradient |
Checkpointing
O(n) → O(√n) memory via recompute |
PagedAttention
OS-style paged KV-cache memory |
Model Parallelism
Tensor + pipeline across devices |
> All 30 algorithms have animated visualizations. Full 1080p60 videos in [Releases](https://github.com/Mathews-Tom/no-magic/releases).
> Video source scenes in [`videos/scenes/`](videos/scenes/) — built with [Manim](https://www.manim.community/).
### Rendering Videos Locally
All visualizations can be rendered from source. System dependencies: `cairo`, `pango`, `ffmpeg`, and optionally `gifsicle` for GIF optimization.
```bash
# Install Manim (one-time)
pip install -r videos/requirements.txt
# macOS system deps (one-time)
brew install cairo pango ffmpeg gifsicle
# Ubuntu/Debian system deps (one-time)
sudo apt-get install -y libcairo2-dev libpango1.0-dev ffmpeg gifsicle
```
**Using the Python renderer** (`render_all.py`):
```bash
# Render all 30 scenes — full 1080p60 MP4 + 480p GIF previews
python videos/render_all.py
# Render specific scenes only
python videos/render_all.py microattention microgpt microlora
# Full MP4s only (no GIFs)
python videos/render_all.py --full-only
# GIF previews only (faster)
python videos/render_all.py --preview-only
# Custom quality (low/medium/high/4k)
python videos/render_all.py --quality medium
# Skip GIF optimization step
python videos/render_all.py --preview-only --skip-optimize
```
**Using the shell renderer** (`render.sh`):
```bash
bash videos/render.sh # all scenes (MP4 + GIF)
bash videos/render.sh microattention # single scene
bash videos/render.sh --preview-only # GIF previews only
bash videos/render.sh --full-only # MP4s only
```
Output lands in `videos/renders/` (MP4) and `videos/previews/` (GIF). Full rendering details in [`videos/README.md`](videos/README.md).
## Philosophy
Modern ML education has a gap. There are thousands of tutorials that teach you to call library functions, and there are academic papers full of notation. What's missing is the middle layer: **the algorithm itself, expressed as readable code**.
This project follows a strict set of constraints:
- **One file, one algorithm.** Every script is completely self-contained. No imports from local modules, no `utils.py`, no shared libraries.
- **Zero external dependencies.** Only Python's standard library. If it needs `pip install`, it doesn't belong here.
- **Train and infer.** Every script includes both the learning loop and generation/prediction. You see the full lifecycle.
- **Runs in minutes on a CPU.** No GPU required. No cloud credits. Every script completes on a laptop in reasonable time.
- **Comments are mandatory, not decorative.** Every script must be readable as a guided walkthrough of the algorithm. We are not optimizing for line count — we are optimizing for understanding. See `CONTRIBUTING.md` for the full commenting standard.
## Who This Is For
- **ML engineers** who use frameworks daily but want to understand the internals they rely on.
- **Students** transitioning from theory to practice who want to see algorithms as working code, not just equations.
- **Career switchers** entering ML who need intuition for what's actually happening when they call high-level APIs.
- **Researchers** who want minimal reference implementations to prototype ideas without framework overhead.
- **Anyone** who has ever stared at a library call and thought: _"but what is it actually doing?"_
This is not a beginner's introduction to programming. You should be comfortable reading Python and have at least a surface-level familiarity with ML concepts. The scripts will give you the depth.
## What You'll Find Here
The repository is organized into three tiers based on conceptual dependency:
### 01 — Foundations (11 scripts)
Core algorithms that form the building blocks of modern AI systems. GPT, RNN, BERT, CNN, GAN, VAE, diffusion, embeddings, tokenization, RAG, and optimizer comparison.
See [`01-foundations/README.md`](01-foundations/README.md) for the full algorithm list, timing data, and roadmap.
### 02 — Alignment & Training Techniques (9 scripts)
Methods for steering, fine-tuning, and aligning models after pretraining. LoRA, QLoRA, DPO, PPO, GRPO, REINFORCE, MoE, batch normalization, and dropout/regularization.
See [`02-alignment/README.md`](02-alignment/README.md) for the full algorithm list, timing data, and roadmap.
### 03 — Systems & Inference (10 scripts)
The engineering that makes models fast, small, and deployable. Attention variants, Flash Attention, KV-cache, PagedAttention, RoPE, quantization, beam search, checkpointing, parallelism, and SSMs.
See [`03-systems/README.md`](03-systems/README.md) for the full algorithm list, timing data, and roadmap.
## How to Use This Repo
```bash
# Clone the repository
git clone https://github.com/Mathews-Tom/no-magic.git
cd no-magic
# Pick any script and run it
python 01-foundations/microgpt.py
```
That's it. No virtual environments, no dependency installation, no configuration. Each script will download any small datasets it needs on first run.
### Minimum Requirements
- Python 3.10+
- 8 GB RAM
- Any modern CPU (2019-era or newer)
### Quick Start Path
If you're working through the scripts systematically, this subset builds core concepts incrementally:
```text
microtokenizer.py → How text becomes numbers
microembedding.py → How meaning becomes geometry
microgpt.py → How sequences become predictions
microbert.py → How bidirectional context differs from autoregressive
microbatchnorm.py → How normalizing activations stabilizes training
microlora.py → How fine-tuning works efficiently
microdpo.py → How preference alignment works
microattention.py → How attention actually works (all variants)
microrope.py → How position gets encoded through rotation
microquant.py → How models get compressed
microflash.py → How attention gets fast
microssm.py → How Mamba models bypass attention entirely
```
Each tier's README has the full algorithm list with measured run times for that category.
## Inspiration & Attribution
This project is directly inspired by [Andrej Karpathy's](https://github.com/karpathy) extraordinary work on minimal implementations — particularly [micrograd](https://github.com/karpathy/micrograd), [makemore](https://github.com/karpathy/makemore), and the `microgpt.py` script that demonstrated the entire GPT algorithm in a single dependency-free Python file.
Karpathy proved that there's enormous demand for "the algorithm, naked." `no-magic` extends that philosophy across the full landscape of modern AI/ML.
## How This Was Built
In the spirit of transparency: the code in this repository was co-authored with Claude (Anthropic). I designed the project — which algorithms to include, the three-tier structure, the constraint system, the learning path, and how each script should be organized — then directed the implementations and verified that every script trains and infers correctly end-to-end on CPU.
I'm not claiming to have hand-typed every algorithm from scratch. The value of this project is in the curation, the architectural decisions, and the fact that every script works as a self-contained, runnable learning resource. The line-by-line code generation was collaborative.
This is how I build in 2026. I'd rather be upfront about it.
## Contributing
Contributions are welcome, but the constraints are non-negotiable. See `CONTRIBUTING.md` for the full guidelines. The short version:
- One file. Zero dependencies. Trains and infers.
- If your PR adds a `requirements.txt`, it will be closed.
- Quality over quantity. Each script should be the **best possible** minimal implementation of its algorithm.
## License
MIT — use these however you want. Learn from them, teach with them, build on them.
---
_The constraint is the product. Everything else is just efficiency._