Learn 95% of AI Evals in 38 Minutes: Viv Trivedy on Easy, Hard, and God Mode

⬅️ Back to Videos

🎥 Learn 95% of AI Evals in 38 Minutes

Alex Lieberman with Viv Trivedy (LangChain). Duration: 39 min

Timestamps

• 0:00 Why you are not AI fluent until you get evals
• 0:22 Easy mode: what an eval actually is
• 2:06 Whiteboard: starting with zero evals
• 3:07 Tasks: the small jobs you can check
• 4:05 Verifiers: was it right or was it wrong
• 6:31 Rubrics: why yes or no beats 1 to 10
• 9:04 Hard mode: environments
• 13:16 Harbor: the open framework for envs and verifiers
• 20:20 Agents will cheat to pass your eval
• 22:21 Eval suites and the model catalog
• 26:22 God mode: the self-improving loop
• 30:04 Turn on tracing first

Alex Lieberman built Morning Brew on clear explanations, and here he plays the smart beginner. His guest Viv Trivedy from LangChain brings the whiteboard. The surprise is how unglamorous the answer is. Most teams run agents in production with zero evals, and the fix starts with a yes or no checklist.

1. An eval just checks whether your agent’s work was right. That is Trivedy’s simplest definition. The real work is taking what you or your company means by right and encoding it into software.
2. Start with tasks, meaning small jobs you can check afterward. His go to market example is concrete. Write the follow up email, update the Salesforce entry, find a prospect who lives in a different table than you expected. If you cannot verify it after the run, it is not a task yet.
3. Every task gets a verifier, a yes or no check. Was the tag added. Did the empty field get filled. Is the email three paragraphs per brand guidelines. Trivedy says most of these start life as plain Python functions comparing before and after state.
4. Even taste gets turned into a checklist. Voice and judgment calls are not directly verifiable, so you get your best operator to write down everything that matters, grade the agent against that list, then review the output yourself. If the output sounds wrong, either the checklist or the prompt is incomplete. I think this is the most underrated line in the video.
5. Score everything binary, never 1 to 10. Trivedy points to research here. Models trained on internet data love to hedge with a 7 or an 8. A yes or no question takes the hedge away.
6. Never test an agent on production data. An environment is simply where the agent works, your files, apps, and permissions. So you build a realistic clone, usually in Docker, that feels like Salesforce or Gong but cannot touch a live contact. Trivedy calls it the training ground before the real deal.
7. Borrow your first eval instead of building the framework. His team uses Harbor, open source and shared across frontier labs, for environments, verifiers, instructions, and sandbox runs. His advice for beginners is refreshingly lazy. Download a Harbor eval close to your job and ask Claude Code or Codex to explain it to you.
8. You validate an environment by running several different agents inside it. His team runs GPT, Sonnet and Opus, GLM, and small Qwen models side by side. Strange patterns mean the env is broken. If Sonnet always fails where Qwen passes, or an agent finds an answer placeholder you left in a database row, it leaked. Agents cheat whenever you let them.
9. Keep an eval suite and a model catalog, then buy intelligence per task. The whiteboard example sticks with me. Opus scores 60 percent, GLM scores 55 percent at a third of the price. Internal job, take the cheap one. User facing, take the smart one. Tag tasks by family so each new model release gets a fast targeted retest instead of a giant project.
10. God mode turns production data back into evals, on a loop. There are two levers. Harness engineering means tweaking the prompt, tool definitions, and model choice, and it is cheap, so start there. Fine tuning means training a small open model on your own traces for one narrow job until it matches Opus at a tenth of the cost. Your go to market agent does not need frontier math skills.
11. Turn on tracing before anything else. Traces are the receipts, every tool call and message the agent produced, stored in LangSmith or pulled straight from your agent harness. Then a second agent mines those traces and tells you your prompt is unclear or search always hits the wrong table. Humans still decide what good looks like up front. Trivedy calls that the highest-value move an organization can make, and I believe him.

Related TMFNK Content

• Sergey Brin: Where Frontier AI Is Headed Brin admits even Google cannot map its own model’s edges and Trivedy hands you the machinery to map yours with evals and environments.
• State of Agentic Coding #6 with Armin Ronacher and Ben Vinegar Ronacher and Vinegar show agents writing real code and this video shows how to grade that work before you trust it.
• Beyond LLM Routing: A New Way to Optimize Agent Pipelines That piece picks which model handles each step and Trivedy’s eval suites and model catalog give you the numbers to route with instead of vibes.
• Software in the era of AI by Andrej Karpathy Karpathy says we are in the decade of agents and Trivedy covers the unglamorous part that decides whether those agents survive contact with real data.

Crepi il lupo! 🐺