Creator perspective · MoneyThe full loop to build is: instrument, trace, eval, and iterate. Define what you want, build it, measure how well it works, ship it, monitor it in production, and iterate based on what you see.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss argues failures should seem fair: when a task fails, it should be clear what the agent got wrong and why (a principle from Anthropic's eval teams). The problem with one-off fixes is you don't know whether the fix actually worked or whether it broke something that was working before — if you just run the agent again on a couple of examples and eyeball it, you're back to vibes. You need a structured way to compare before and after when making fixes, which is what experiments are for.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyWithout evals, switching models means weeks of manual testing to determine if the new model is better or worse. With evals, you run the suite, compare the scores, and know within hours. The field moves fast with new models dropping every few months, which makes evals matter more and more.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss explains that the power of experiments is controlled comparison: take the same inputs and the same evaluators so the only thing that changed is the agent's prompts. With controlled comparison, any difference in scores is attributable to your change. You're still wondering a little whether a score change came from your prompt change, because the agent is non-deterministic and could have done 10 web searches instead of five — but controlled comparison eliminates a major source of variation, which is the test cases.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyThe speaker argued that if you automate evals before understanding your failures, you create an eval that measures what is easy to measure instead of what actually matters.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss argues that you need requirements first — before you can categorize failures, you need to know what success looks like. You can't say an agent doesn't work if you haven't defined what 'works' means. Success criteria seem obvious when written down, but lots of teams never write them and instead ship an agent and react to complaints. Writing requirements down turns vague disappointment into specific testable criteria, and those criteria turn into your evals.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyIf you don't derive eval criteria from observed errors in traces, the right criteria won't occur to you — you can't come up with them from whole cloth.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyYou can't switch models without evals — the field moves fast with new models dropping every few months. Without evals, switching means weeks of manual testing with no certainty; with evals, you run the suite, compare scores, and know within hours whether a new model is better.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Creator perspective · MoneyThe speaker observed that most agent outputs appear fine at a glance, but failures are only visible when reading the traces — in his sample, three of thirteen traces were doing something weird.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyEvals turn a vague definition of good into a measurable bar, so 'did it get better' can be answered with proof rather than vibes.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Report · MoneyThe source argues that debugging voice agents requires seeing the conversation beyond just the transcript: audio, transcript, and trace all in one place.
AI Engineer · The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize
Report · MoneyBecause evals come from LLMs, they provide not just a score but an explanation; feeding those explanations back into a coding agent lets it make the software better automatically. Once you have cleverly defined what good looks like, you can close the software development loop without doing anything — the agent climbs the hill by itself.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI
Report · MoneyThe traditional solution to not being able to read every trace is evals: either code or, better, another LLM that looks at an entire agent trace and scores whether it was good or bad by a definition you define.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI
Creator perspective · MoneyFrye argues the scheduler process is semantically single-threaded and becomes the gatekeeper to the GPU, so it can be a host-side bottleneck. The guiding principle: at every moment some kernel should be running on the GPU, and CPU-side decisions should never block GPU progress. He notes it is very easy to accidentally write a host-device sync that blocks GPU progress, even though some host-device information passing is necessary. The majority of performance engineering effort should go into scheduler design and operation, not just GPU work; operating multiple schedulers in parallel is not worth the lock/coordination complexity on GPU resources.
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Report · MoneyBoth speakers run a terminal all day with eight or more agent sessions, each focused on a different project with its own context and tools. They drive by each session, directing the next one while another works in the background, and running multiple sessions compounds quickly.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyAll of this is converging towards the same primitives that you need to be very familiar with. Combine hooks, goals, loops, and schedules as building blocks to create fully autonomous systems and automate tedium out of your workflow — e.g., schedule a goal every week and it will run that goal. If a task recurs and you have access to Claude and scheduled tasks, you shouldn't keep doing it manually. When you're constantly correcting Claude's output, reach for goals and loops. When the AI is failing, fix the system, not the code itself. Never tell the AI to 'try harder' — it doesn't know what you mean by that. You can rage at the AI — it's good for the soul and cathartic; the speakers have a whole channel called Bot Hate.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Report · MoneyVoice coding is one of the highest-leverage moves both speakers have made in recent years. It eliminates typing friction and moves at the speed of thought (Zach: ~90 WPM typed, ~190 WPM with tools like WhisperFlow), changes how you interact with Claude and your computer, and its speed advantage compounds across multiple tabs and background agents. It's also useful beyond code — for responding to colleagues and customers — though quality and formatting still matter, and architectural ideation by voice can be the largest lift.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Report · MoneyUsing an LLM as a judge is probably the most popular technique with evals and is kind of the gold standard, either pairwise or pointwise. With an LLM judge, you take whatever the output is and send it to a judge LLM that asks: what do you think? Is this close enough? But LLM judge systems are themselves non-deterministic, so the question becomes how do you trust it and how do you build a judge. LLM-as-a-judge architectures sometimes pass and sometimes fail on the same input, making it hard to ensure the right answer. It's the speaker's go-to because it's cheap and easy.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyTool call results should be treated as non-instructional data — just data, not instructions to follow. This came up in the context of the Claude Code prompt-injection catch: the HTML contained commented text telling the model to ignore all previous instructions, and the harness caught it.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyTejas frames the core distinction: evals give ahead-of-time reliability while harnesses give just-in-time reliability, run by the user who prompts them. He maps this to compiled languages: JavaScript is unsafe because it's just-in-time (string '2' + integer 1 yields '21'), whereas TypeScript catches the error at build time. Locking in on both harnesses and evals lets you go far with AI, often at low cost using less capable models with a great harness and eval set.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM