strands-microgpt: A GPT From Scratch as a Strands Model Provider 🧠

Published: March 26, 2026

What if I told you the entire GPT algorithm — autograd, attention, Adam optimizer, training loop, and inference — fits in 815 lines of pure Python with zero dependencies? No PyTorch. No NumPy. No CUDA. Just math, random, and the raw algorithm.

That's strands-microgpt: Karpathy's atomic GPT, enhanced with winning techniques from OpenAI's Parameter Golf competition, packaged as a proper Strands Agents model provider.

The Idea

"Everything else is just efficiency." — Andrej Karpathy

Karpathy's original gist showed that a GPT is just math. strands-microgpt takes that idea and makes it useful by wrapping it in the Strands Model interface — so you can plug a from-scratch transformer into any Strands agent.

graph LR
    A[Text Dataset] -->|Tokenize| B[MicroGPT Engine]
    B -->|Autograd + Adam| C[Trained Model]
    C -->|Strands Model Interface| D[Agent]
    D -->|Tool Calls| E[Train / Generate]

    style B fill:#f96,stroke:#333,stroke-width:3px
    style D fill:#9cf,stroke:#333,stroke-width:3px

Three Ways to Use It

1. Standalone Engine — Pure Python, Zero Deps

from strands_microgpt import MicroGPT

model, tokenizer, docs = MicroGPT.from_dataset()
model.train_on_docs(docs, tokenizer, num_steps=1000)

for name in model.generate(tokenizer, num_samples=10):
    print(name)

That's it. No pip install torch. No CUDA drivers. Just Python.

2. As a Strands Model Provider

from strands import Agent
from strands_microgpt import MicroGPTModel

model = MicroGPTModel(num_steps=1000, temperature=0.5)
agent = Agent(model=model)
agent("Generate some names")

The model trains on first use (lazy init), then generates character by character through the streaming interface. It implements the full Model protocol — stream(), format_request(), format_chunk() — proving that the Strands Model interface is truly universal.

3. As Agent Tools

from strands import Agent
from strands_microgpt import microgpt_train, microgpt_generate

agent = Agent(tools=[microgpt_train, microgpt_generate])
agent("Train a GPT on names for 500 steps, then generate 10 names")

Your Bedrock/OpenAI/Ollama agent can now train a GPT from scratch via tool calls. Meta? Absolutely.

The Architecture: 815 Lines of Algorithm

strands_microgpt/
├── engine.py           # Value (autograd), Tokenizer, MicroGPT
├── microgpt_model.py   # Strands Model interface  
└── tools/
    ├── microgpt_train.py     # @tool for training
    └── microgpt_generate.py  # @tool for inference

The Autograd Engine

Every computation builds a graph. Every backward() call walks it:

from strands_microgpt import Value

a = Value(2.0)
b = Value(3.0)
c = a * b + a    # builds computation graph
c.backward()     # backpropagates gradients
print(a.grad)    # 4.0 (dc/da = b + 1)

The Value class supports +, *, -, /, **, relu(), exp(), log(), tanh(), and backward() — everything you need for a transformer, implemented in ~80 lines.

The Transformer

GPT-2 architecture with modern enhancements:

Component	Implementation
Normalization	RMSNorm (no bias)
Activation	ReLU² (smoother gradients)
Attention	Multi-head causal with GQA
Position	RoPE (rotary embeddings)
Optimizer	Adam with cosine warmdown

v2: Parameter Golf Enhancements

The engine incorporates winning techniques from OpenAI's Parameter Golf competition — a contest to build the smallest working language model:

Winning Techniques Baked In

• ReLU² activation — smoother gradients, implicit soft gating
• GQA (Grouped-Query Attention) — fewer KV heads, saves parameters
• Tied embeddings — input/output share weight matrix
• RoPE — rotary position embeddings, no learned position matrix
• U-Net skip connections — encoder/decoder with learned skip weights
• Logit soft-capping — prevents logit explosion (tanh cap=30.0)
• Cosine warmdown LR — smooth final convergence
• Gradient clipping (max_norm=0.3) — training stability
• Learnable residual scales — per-layer attention/MLP modulation
• Q/K RMSNorm + learnable Q gain — better attention dynamics
• Zero-init output projections — muP convention

These aren't just theoretical — they're the techniques that won the competition by achieving the lowest perplexity per parameter.

Why This Matters

1. The Strands Model Interface is Universal

If something can generate tokens, it can be a Strands model. strands-microgpt proves this by implementing the full Model protocol with a pure-Python transformer. The same interface that talks to Claude Opus talks to 815 lines of math.exp().

flowchart TD
    Interface[Strands Model Interface]
    Interface --> Bedrock[Amazon Bedrock]
    Interface --> Anthropic[Anthropic API]
    Interface --> OpenAI[OpenAI]
    Interface --> Ollama[Ollama]
    Interface --> MicroGPT[🧠 MicroGPT<br/>815 lines of Python]

    style MicroGPT fill:#ff9,stroke:#333,stroke-width:3px

2. Understanding > Using

You can use GPT-4 without understanding it. But training a transformer from scratch — watching the loss drop, seeing coherent text emerge from random weights — that's when the magic stops being magic and starts being math.

3. Tools Teaching Tools

An agent powered by Claude can use microgpt_train to train a GPT from scratch. A large model orchestrating the training of a tiny model, all through the same tool interface. It's agents all the way down.

Quick Demo

pip install strands-microgpt

from strands_microgpt import MicroGPT

# Train on names dataset (default)
model, tok, docs = MicroGPT.from_dataset()
losses = model.train_on_docs(docs, tok, num_steps=1000)

print(f"Parameters: {model.num_params}")
print(f"Final loss: {losses[-1]:.4f}")
print(f"Techniques: {', '.join(model.techniques)}")

# Generate
for name in model.generate(tok, num_samples=5):
    print(f"  → {name}")

Output:

Parameters: 4,832
Final loss: 2.1847
Techniques: ReLU², GQA, tied_embeddings, RoPE, U-Net_skip, logit_cap, cosine_warmdown

  → Aaliyah
  → Kaelyn  
  → Brayden
  → Norah
  → Ezekiel

4,832 parameters. Pure Python. Generating plausible names from scratch.

Checkpoints

Save and load models for later inference:

# Save after training
model.save_checkpoint("my_model.json", tokenizer)

# Load later — no retraining needed
model, tokenizer, meta = MicroGPT.load_checkpoint("my_model.json")
samples = model.generate(tokenizer, num_samples=10)

The checkpoint format is plain JSON — human-readable, versionable, portable.

Custom Datasets

Train on anything — poems, molecules, DNA sequences, code:

docs = open("shakespeare.txt").read().split("\n\n")
tokenizer = Tokenizer.from_docs(docs)
model = MicroGPT(vocab_size=tokenizer.vocab_size, n_embd=32, block_size=64)
model.train_on_docs(docs, tokenizer, num_steps=5000)

Links

• PyPI: pip install strands-microgpt
• GitHub: cagataycali/strands-microgpt
• Karpathy's Original: The Gist
• Parameter Golf: openai/parameter-golf
• Strands Agents: strandsagents.com

---

A GPT is just math. strands-microgpt is the proof.