Contrastive Language Models: Fast Decisions Without Generating a Word
A Stanford and NVIDIA team led by Jacky Kwok just posted Contrastive Language Models (Sep 23, 2026), with code on GitHub and weights on Hugging Face. The pitch: most of what an agent does is pick, not write. So train a model to pick.
CLM is a System One model. It never generates text. Given a state and candidate actions, it scores each candidate by embedding alignment and picks the winner. Training uses a bidirectional InfoNCE loss that pulls each state toward its true action and away from the rest of the batch.
The trainable footprint is tiny. Each side is a frozen Qwen3-8B encoder plus a projection head of around 20M parameters, one for states and one for actions. Embeddings are precomputed once, so a full pre-training run takes an hour on a single RTX 4090. Cheap enough to rerun scaling sweeps most labs could afford.
The data recipe runs in three stages. Pre-training aligns broad semantics on about 60M Nemotron Q&A pairs. Mid-training sharpens judgment on about 30M synthetic hard negatives from Gemini 2.5 Flash-Lite. Post-training grounds decisions in about 1M agentic trajectories, with 40 percent Q&A replay so general judgment survives.
Hard negatives work best second, not first. Pre-training alone already scores 52.1 percent top-1 on held-out questions it never saw hard negatives for, and short mid-training lifts that to 69.2 percent. Training with hard negatives from the start peaks at 62.4 percent and overfits.
Zero-shot, CLM-8B matches Jev on computer-use, gaming, and tool-calling tasks at up to 9x lower latency. Gains are largest with many candidates (WikiRacing) or reusable actions (their T-Rex and Mario demos, with Mario running about 4x faster). Latency barely moves from 1 to 1,024 candidates, 36ms to 44ms, where constrained decoding hits 4.3 seconds.
Fine-tuned as a verifier, it sets records on two agentic coding benchmarks. Sampling solutions with Opus 5 and Fable 5, then letting CLM pick, reaches 81.6 percent on 38 held-out DeepSWE tasks and 87.6 percent on 30 held-out Terminal-Bench 2.1 tasks, at 4 to 6x faster inference. Jev as verifier scores below the pass@1 baseline on these long-horizon tasks.
You might think a scorer this cheap replaces the generator. It does not, and the team says so plainly. The record numbers come from fine-tuned heads, not the released checkpoint zero-shot. CLM only ranks candidates you hand it, its probabilities are relative to that set, and the heads lock to Qwen3-8B embeddings. Bad candidate sets still produce confident bad picks.
Verification scales predictably, which is the part I keep coming back to. Test loss falls as a power law in compute, data, head size, and encoder size, with encoder size helping most. A multimodal CLM-35B is announced for early October.
The takeaway: If you run agents, the cheapest upgrade available is a dedicated picking layer. Serve the reference head behind their TypeSafe-compatible API and let CLM rerank best-of-N outputs or route tool calls before you pay for a bigger generator.
Related TMFNK Content
- Harness Engineering for Self-Improvement: Lilian Weng’s Survey Weng names weak evaluators as the bottleneck in agent self-improvement, and CLM is a concrete answer to that complaint.
- Is Grep All You Need? How Agent Harnesses Reshape Agentic Search The earlier piece on file-system harnesses pairs with this one: the harness proposes candidates, a verifier like CLM picks among them.
- Papers with Code Is Back: Where ML Research Meets Leaderboards and Repos Where to watch whether the DeepSWE and Terminal-Bench numbers survive independent reproduction.
Crepi il lupo! 🐺