autoresearch (Karpathy)

What it is

A minimal framework for autonomous AI-driven ML research. Give a Claude/Codex agent a single-file LLM training setup and a fixed 5-minute time budget per experiment; let it modify =train.py, run the experiment, check the metric, keep or discard, and repeat overnight. You wake up to a log of ~100 experiments and (hopefully) a better model.

The human programs program.md (a lightweight skill/instructions file) rather than touching any Python. The agent handles everything else.

Key Design Decisions

Single file to modify

  • The agent only edits train.py — the GPT model, optimizer (Muon + AdamW), and training loop.
  • prepare.py is fixed (data prep, tokenizer, eval utilities). The agent never touches it.
  • This keeps scope manageable and diffs reviewable.

Fixed 5-minute time budget

  • Every experiment runs for exactly 5 minutes (wall clock, excluding startup/compilation).
  • ~12 experiments/hour → ~100 experiments overnight.
  • Makes experiments directly comparable regardless of what the agent changed (model size, batch size, architecture, optimizer). Fair comparison across architectural changes.
  • Finds the model most optimal for your specific platform and time budget.
  • Downside: results are not comparable across different hardware.

Single metric: valbpb

  • Validation bits per byte — lower is better.
  • Vocab-size-independent, so architectural changes are fairly compared.

program.md as the human interface

  • The human's job is to write and iterate on program.md — essentially a "research org" specification.
  • The agent reads program.md at startup and acts on it.
  • The interesting meta-question: what research org code achieves the fastest progress?

How to Run

uv sync                  # install deps
uv run prepare.py        # one-time data + tokenizer setup (~2 min)
uv run train.py          # single manual experiment (~5 min)
# Then spin up Claude Code in the repo and prompt it to start an experiment

Requirements: single NVIDIA GPU, Python 3.10+, uv.

Why This Matters

  • Closes the loop between LLM capability and ML research: the agent writes code, trains, evaluates, and iterates — all without human intervention per cycle.
  • The human moves up one level: programming the research process (program.md) rather than individual experiments (train.py).
  • Karpathy's framing: "You are programming the program.md Markdown files that provide context to the AI agents and set up your autonomous research org."
  • Natural extension: multiple agents, richer program.md, longer horizons.

Relation to Claude Code

  • Designed to work with Claude Code (or Codex) with permissions to edit files in the repo.
  • The program.md file is "essentially a super lightweight skill" — analogous to SKILL.md files in ~/ClaudeBrain/skills/.

Mental Models

Backlinks