Creator perspective · MoneyNick Nisi explains that once you build verification loops that are durable, reliable, repeatable, and uniform, the main benefit is that a failure starts to drive future work on its own — without such loops, a failed agent run cannot drive future work and requires human intervention.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyNick Nisi recommends maintaining checks for everything that came before — unit tests, evals, smoke tests, and whatever the application needs — and giving the model the ability to interpret and run all previous checks so that when new features are added, all prior checks run and catch regressions. He also suggests deploying to a preview environment first and verifying the change there before promoting to production, then immediately monitoring after promotion.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyIf you have verifiers in place, your source code doesn't matter as much because you can rewrite it and prove the new version works.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyZack Proser argues that constraints can equal freedom: if agents have the right constraints, they will come up with creative solutions and actually solve what you said. That's why you give them the outcome — he points to Nick's project experimenting with outcome-driven development. He also notes that teams might score differently or have different criteria, so the approach is super flexible.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Report · MoneyLaurie Voss notes that once registered, every call your agent makes is traced automatically — you don't have to write a log line or trace for every single thing the agent does. The Claude Agent SDK already instruments tools and web searches for you, and you don't have to add any logging to your agent for tracing to work — all the useful information shows up without doing anything.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Report · MoneyLaurie Voss explains that without traces you would only get a mysterious output saying something was wrong; traces are where you can see the tool call failing and the agent trying to do something it isn't given permission to do.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Report · MoneyLaurie Voss argues that defining good is not solely an engineering job: if you are defining good for your product, you have to talk to the non-technical stakeholders in your organization because they are the domain experts who know what good looks like. Engineers have specific guesses and biases about what makes a good agent and good output, and won't always agree with the domain experts. Customer-facing people need to be in the room when you define what your agent looks like.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Report · MoneyLaurie Voss warns that if you automate before you understand how your agent fails, you'll end up measuring what's easy to measure instead of what actually matters.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Creator perspective · MoneyLLM judges have all the flaws of LLMs: unpredictable, never the same twice, can make mistakes, cost money, and can be confidently wrong. Because they can be confidently wrong, you have to check them.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Report · MoneyLaurie Voss explains that changing a system prompt can silently break other behaviors, creating a whack-a-mole cycle of regressions, and that evals fix this by giving a trackable number.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Report · MoneyLaurie Voss states that synthetic data is a great way to get started testing an agent, and once you have real users with real traffic, real traffic is the gold standard for testing an agent — get it incorporated into your dataset as fast as you possibly can.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Creator perspective · MoneyEvals exist to solve the problem of knowing whether your agent is good and whether a change made things better or worse. They let you stop regressions when making changes without reading every output yourself, and fix the whack-a-mole problem by giving you a number you can track, compare, and act on.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyThe last principle is verification: you want a second pair of eyes, and this builds on the principle of accountability because you are accountable for that review as well. The principle of verification is building more feedback loops so you don't have blind spots, but without relegating any of the accountability. Verification is just extra information from a different source to avoid biases and avoid getting stuck on your own thinking and processes, but it doesn't remove any of the responsibility or accountability.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyMonzo's approach to AI harnesses is explicitly designed for financial use cases where trust matters, and the recipe may not transfer to less trust-critical contexts.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyAsking a stochastic machine to produce a consistent output means you are likely not going to get it. If the agent generates new code for every session, you get non-determinism: same question, different answer.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyThe trust users have in your product, agent, and its output is what drives usage, adoption, and improvement through the loops of tracing and evolution — so design harnesses so customers can trust the results.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Report · MoneyWhen asked where a number came from, agents typically answer with vague references like 'we talked about this' rather than an actual verifiable paper trail — the speakers describe this as the default experience in every agent conversation.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyIf you wait for agents to get better instead of building deterministic guardrails, the agent will create stuff on the go — you may get the right code at one point in time, but by run 100 it's inventing code that didn't need to be reinvented.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyThe one takeaway the speakers want everyone aligned on: we are ultimately the responsible, accountable people for whatever AI agents do. 'Agents face no real world consequences, you do' — a quote the speaker loved but can never attribute. When someone says 'that's what Claude gave me,' it may feel fine, but we face the consequences of what agents do. The speaker gets hired and paid partially because he is accountable for the results he produces for the company.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyAI will likely not be perfectly accurate, so design mechanisms so that whatever process runs, you can trust the results to a degree.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo