Skip to content

strands-microgpt

Karpathy's pure-Python microGPT as a Strands Agents model provider.

PyPI GitHub

Overview

A complete GPT implementation — autograd engine, transformer, tokenizer, Adam optimizer, training, and inference — in 815 lines of pure Python with zero dependencies. Packaged as a Strands model provider.

Three Ways to Use

  1. Standalone engine — MicroGPT.from_dataset() → train → generate
  2. Strands Model — drop-in Model interface for any agent
  3. Agent Tools — microgpt_train / microgpt_generate tool calls

Install

pip install strands-microgpt

Quick Start

from strands_microgpt import MicroGPT

model, tok, docs = MicroGPT.from_dataset()
model.train_on_docs(docs, tok, num_steps=1000)
for name in model.generate(tok, num_samples=10):
    print(name)

from strands import Agent
from strands_microgpt import MicroGPTModel

model = MicroGPTModel(num_steps=1000)
agent = Agent(model=model)
agent("Generate names")

from strands import Agent
from strands_microgpt import microgpt_train, microgpt_generate

agent = Agent(tools=[microgpt_train, microgpt_generate])
agent("Train on names for 500 steps, generate 10")

v2 Parameter Golf Enhancements

Incorporates winning techniques from OpenAI's Parameter Golf competition:

Technique Benefit
ReLU² Smoother gradients
GQA Fewer KV heads, saves params
Tied embeddings Input/output share weights
RoPE Rotary position embeddings
U-Net skip connections Encoder/decoder with learned skips
Logit soft-capping Prevents logit explosion
Cosine warmdown LR Smooth convergence
Gradient clipping Training stability