autoresearch: An AI That Runs LLM Experiments While You Sleep

🔧 autoresearch


What it is	An AI agent that edits model code, trains it, and keeps what improves the loss
Stack	miolini/autoresearch-macos fork, llama.cpp (local GGUF), OpenCode
Status	Working, with caveats on an small Mac
Code	github.com/karpathy/autoresearch

Karpathy’s autoresearch automates the loop every ML researcher runs by hand: change the model, train it, check if it got better, keep or revert. Give an agent this repo overnight and you wake up to a log of experiments and a slightly better model. The main repo wants an H100, but a community fork runs on Apple Silicon, and the agent itself can be a tiny local model instead of a paid frontier one. That’s the setup I wanted: a fully offline overnight research lab on a laptop.

What I learned

The repo is three files on purpose. train.py is the only thing the agent edits, prepare.py does the fixed data work, and program.md is the human’s file: your instructions to the agent. The whole “research org” is a folder.
Every experiment runs exactly five minutes. Wall clock, no matter what the agent changes. That fixed budget means runs are comparable on your machine, and autoresearch finds the model that works best for your specific hardware, not someone else’s.
One metric, lower is better. Everything is judged on val_bpb, validation bits per byte. It’s independent of vocabulary size, so the agent can fairly compare a smaller model against a bigger one, or a new attention pattern against an old one.
About 100 experiments a night. Five-minute runs, and the agent keeps or discards each one automatically. You don’t touch anything until morning, when you read the log. It’s a different way to do research: you write a Markdown skill, and the agent does the bench work.
On a Mac you need the fork. The main repo hardcodes FlashAttention-3 and expects CUDA. miolini/autoresearch-macos swaps in PyTorch’s native attention, disables torch.compile paths Metal doesn’t support, and tunes batch sizes for MPS. Same files, same loop, runs on Apple Silicon.
The agent brain can be a 700MB local model. Point a coding agent at a llama.cpp server and it works like Claude or Codex, just weaker and free. LiquidAI/LFM2.5-1.2B-Thinking-GGUF at Q4_K_M is about 730MB of RAM, DevQuasar/WeiboAI.VibeThinker-1.5B-GGUF at Q4_K_M about 1.1GB. Both are RL-trained “thinking” models, which helps for this kind of step-by-step editing loop.
The catch: a 1.5B model is a mediocre research assistant. It will follow program.md loosely and break experiments more than Codex/Claude/GLM/Kimi would. And when training and the model share an small Mac, they compete for memory. Expect fewer useful experiments than Karpathy’s demo, but the loop still works, and it’s fully private and free.

How it works

Install llama.cpp and serve a small model on port 8080:

brew install llama.cpp
llama-server -hf LiquidAI/LFM2.5-1.2B-Thinking-GGUF:Q4_K_M

That downloads the GGUF once and starts an OpenAI-compatible server. Check it with:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"LiquidAI/LFM2.5-1.2B-Thinking-GGUF:Q4_K_M","messages":[{"role":"user","content":"hi"}]}'

Then set up the research repo and verify the training loop once, by hand:

git clone https://github.com/miolini/autoresearch-macos
cd autoresearch-macos
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync
uv run prepare.py   # downloads data, trains tokenizer, ~2 min
uv run train.py     # one 5-min experiment; confirm val_bpb prints

Finally, point your coding agent at the local server. In OpenCode, add the provider to opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llama-local": {
      "npm": "@ai-sdk/openai-compatible",
      "options": { "baseURL": "http://localhost:8080/v1", "apiKey": "local" },
      "models": {
        "LiquidAI/LFM2.5-1.2B-Thinking-GGUF:Q4_K_M": {
          "name": "LFM2.5 1.2B Thinking"
        }
      }
    }
  }
}

Open the agent in the repo, disable permissions, and prompt: “Have a look at program.md and let’s kick off a new experiment.”

Steal this if: you want to see what autonomous research looks like without renting a GPU, and you’re fine with a laptop that stays warm overnight. Skip it if you expect frontier-agent quality from a 1.5B model.

autoresearch: An AI That Runs LLM Experiments While You Sleep

🔧 autoresearch

What I learned

How it works

Related TMFNK Content