Explore

Mapping eval findings to specific prompt changes

Laurie walks through concrete examples of mapping eval findings to prompt changes: because the judge said 'makes no recommendation', the improved prompt asks for an explicit recommendation; because the judge said 'mentions risks with no evidence', the prompt asks for risks with supporting data; because faithfulness found invented numbers, the prompt says to use only numbers that appear in your research; and because reading traces found the agent trying to save files all the time, the prompt tells it to reply with the report instead. The improved prompts live in prompts.ts, with an improved version of each prompt or the option to write your own.

Method, not a verified protocol.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Step 7: use trusted evals to improve the agent and prove the fix

Once you have evals you trust, use them to actually improve the agent and prove the fix worked. In Phoenix you can filter evals (e.g., 'not actionable'), select them, and add them to a dataset, then run experiments against only the failing evals instead of the entire set — which saves thousands of tests per run on a decent-sized test set. Every prompt change should map to a finding from the evals, and you should not guess: come up with specific things for the agent to do based on previous eval results.

Method, not a verified protocol.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Use faithfulness instead of correctness when the judge lacks fresh data

Laurie Voss recommends using faithfulness instead of correctness when the judge lacks fresh data: faithfulness asks whether the report makes sense given the data the agent was given, and can work with exactly the same data the agent was given.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Run agent-driven error analysis over traces

Laurie Voss walks through the error-analysis workflow: open the coding agent, tell it where Phoenix is, have it read traces, come up with failure categories, count them, and write each category back onto traces as annotations; approve the plugin's tool permissions and let it write notes to disk. His run took about nine minutes. An annotation is a human-readable string attached to a trace.

Method, not a verified protocol.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Don't write vague criteria — write down exactly what makes a report actionable

Laurie Voss discourages writing 'a good response is helpful and accurate' because a judge doesn't know what helpful and accurate mean in this context, and it can't make a yes or no decision from that. Instead, write down exactly what makes a report actionable: it makes a specific recommendation, it names concrete risks, and it looks forward, not just back. He notes this is where a lot of people don't put in enough effort — the explicit criteria section of the rubric.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Annotate traces in Phoenix to build human ground truth

To annotate in Phoenix, attendees open up a trace, click the little pencil button, and add annotations. Laurie Voss created an annotation in advance called 'human actionable' that can be set to actionable or not actionable, then quickly clicked through all traces and set it on each one. This is where a domain expert would get involved: throw them at Phoenix with 10, 20, or 50 traces and have them read the outputs and give you a golden dataset. The speaker sat down and labeled all 16 of his reports properly against the same criteria he gave the judge, and those are sitting in the seed project so attendees don't have to do it.

Method, not a verified protocol.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Build several small evals, not one giant super eval

Laurie Voss advises building several small evals today rather than one giant super eval; cramming every possible failure into one big LLM-as-a-judge eval is unreliable and complicated without solving the problem.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Build a shared domain (plan mode) with the agent first

Francesco recommends first building a shared domain with the agent — calling it plan mode — before anything else, then setting up consistency.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Design a branching harness where decisions branch out and join back

Design a harness with a branching structure where decisions branch out and join back in, so the model can verify assumptions with you or figure out how things fit into the overall system.

endorsed advice

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Define the outcome and success metric first, then track whether AI supports it

When evaluating AI adoption, first define what outcome you're looking for and how you measure success; e.g. if the goal is less back-and-forth with a BI team, validating a new dashboard, or fewer internal ask tickets because a number looks wrong, track that metric and see if AI can support generating that outcome.

endorsed advice

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Move from skill-based prompts to a separate tool when the use case becomes critical

Move from a skill-based prompt approach to a separate tool when the use case becomes critical enough that the trade-offs of the non-deterministic approach are not acceptable.

endorsed advice

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Never have an athlete do an exercise that causes pain

Grubbs's rule: never have any athlete do any exercise that causes pain; when selecting exercises, ask why you are doing it and whether it is creating pain — if so, you probably shouldn't be doing it.

Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs

Produce independently checkable certificates rather than claimed independence

Clark recommends building systems that produce independently checkable certificates so others can verify the evaluation themselves, arguing it would be a real shame if all AI safety were left in the hands of the labs or friends of the labs who merely claim they are independent.

Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club

Full approach flow: opener, 'how's your day going?', playful misinterpretation, laughter, observation/assumption, personal input

Coach Kyle demonstrates a complete approach sequence: open with a basic compliment ('I saw you walking by, I thought you looked very cute and fashionable'), ask 'how's your day going?', playfully misinterpret her answer (e.g., 'You dog walker?'), inject laughter, then make an observation/assumption about her (e.g., 'you probably do something artsy') and share personal input about his own experience.

Method, not a verified protocol.

Coach Kyle · Don't know what to talk about with women? Watch This

Six guideline questions to kickstart tokenomics refinement

To kickstart your process internally within your organization, you can either talk to the speaker or his team, or you have six questions to start asking. These are like guideline questions for you to start asking to refine your tokenomics. The questions: (1) Who, team, spend what? (2) What must you test to use a cheaper model? (3) What's your base load? (4) What's the default model for each team? (5) Which workload has a person waiting that you need something faster? (6) How much of what we generate is useful?

Method, not a verified protocol.

AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Distill a strong model down to a small model: start expensive, generate data, fine-tune across sizes

Bootstrap a fine-tuned model by starting with an expensive model that can already do the task: deploy it at small scale to generate your training dataset, then distill its outputs down to a 7B-scale model and fine-tune the collected data at different model sizes to see what fits best. Fine-tune models to specialize use cases once you reach scale.

AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Vectorize tools and skills; retrieve relevant ones via vector similarity search

Nacho Martinez proposes creating vector embeddings out of the tools and skills; when the agent loop needs to retrieve useful skills or tools, do a vector similarity search against the vector store, pull the most relevant skill or tool, solve the problem, then offload it again — one of the ways to keep the context window relevant.

AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle

Use context summarization: summarize the conversation and restart from the summary

Nacho Martinez describes context summarization as tackling context bloat: you ask the model to generate a short summary of everything said in the context window, then start the next conversation from that summary.

Contested by Jake on Figma

AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle

Practice skill distillation: turn past conversations and runs into reusable skills

Nacho Martinez recommends skill distillation — taking previous conversations and runs with an AI agent and creating something meaningful out of them, such as a skill MD file placed into the skill box — describing it as a technique to improve a frozen model's learning over time by controlling the data layer rather than the weights.

AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle

Vectorize tools and skills; retrieve them via similarity search and offload after use

Nacho Martinez proposes creating vector embeddings of tools and skills, using vector similarity search against the vector store to pull the most relevant skill or tool when the agent loop needs one, and offloading it again after solving to keep the context window relevant.

AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle