Explore

Decompose tasks cleanly with per-task agents and done-verification prompts; avoid giant multi-day PRs

The tired way — one giant massive PR from a single agent session running three or four days being told 'fix this, fix this, fix this, make it better' — is being replaced. What works essentially perfectly: decomposing tasks really cleanly, giving each task an agent, and in the prompt giving exactly what it needs to do to know it is actually done and to verify the state of done — those will just rip.

AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS

Prompt-based verification is fragile; prefer a deterministic script that proves the work and closes the loop

Putting a lot of verification instructions in a prompt is what a lot of skills and saved prompts do, but if you forget anything or things change, it leaves a lot up to interpretation. Instead, have a script that can prove the work out in different ways so you can run it over and over as a deterministic check. The script feeds its output back into the agent so it knows exactly what was wrong and how to fix it quickly, closing the loop. The agent can spin the script, run it again, check, verify, and repeat over and over until there are no more errors. LLMs are not deterministic — giving 30 agents the exact same prompt has potential for variance in outcomes — so pinning verification down to a single command backed by actual bash commands and bash calls is significantly more deterministic.

AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS

A separate supervising agent should enforce policy on worker agents

There should be a separate supervising agent that decides what the worker agent gets to do, creating a separation of concerns where the supervisor can't do any of the work itself. If a user tells an agent to go read the CEO's email, that is outside the scope of that agent and should be rejected as intent. A request to read somebody else's email would probably get caught in deterministic policies right away because it's not a vague thing and good controls exist for it. Telling an agent to forward the CEO's email to the outside world is vaguer and is the kind of thing that ideally comes back to a supervising agent, which would say that's probably a bad idea and no, you can't do that.

AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth

Without intent, we can't get to agent autonomy; give the agent the plan as context with a mission identifier for audit

The creator describes that without intent — some capacity to relay the user's or business's intent — we can't get to agent autonomy. The creator recommends giving the agent the plan — what you're trying to do — as context, so that a mission identifier tracks through everything and authorization decisions are made in the context of that mission. A mission identifier also lets you audit and see all the things that happened and the decisions made in the context of one mission. Today there isn't a good concept of agent identity or the mission — things make decisions without knowing which agent instance is doing it or what mission it's involved in.

AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth

Gate CI builds on acceptance criteria: guardrails must pass, targets must stay above a threshold

In the demo, Voss gates the build on acceptance criteria: mentions-ticker is a guardrail (the agent must talk about the right company) and actionability is a target (the average must stay 0.8 or better). He gates on the average score rather than a single report because an LLM judge can be a bit noisy on a single report.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

LLM judge biases: position, length, confidence, and self-preference

Laurie Voss describes several biases in LLM judges. Position bias: given two options, depending on the model, they tend to prefer whichever comes first or last — so swap the order of options presented to a judge so it's not always looking at them in the same order. Length bias: judges prefer long answers. Confidence bias: they prefer answers that sound fluent and confident. Self-preference bias: a model judging its own output tends to rate it higher — so judge haiku with sonnet, and ideally judge haiku with something from OpenAI, to avoid self-preference bias.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Capability evals that pass 100% become regression evals that gate releases in CI

When a capability eval passes reliably (100% of the time), Voss says you don't throw it away — you turn it into a regression eval. A regression eval is the kind you put into CI: it used to pass 100% of the time, and if it suddenly stops, you call that a failure and stop the release.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Check for regressions: did anything that used to pass break?

The important thing when improving an agent is whether anything that used to pass now breaks. In Phoenix you can filter down to the passes split to check whether previously passing reports still pass. In Laurie's case, all five reports that passed previously still passed after he improved his prompt. He also notes that when a test passes 100% of the time you don't throw it away — you turn it into a regression eval.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Order of operations for improving an agent: data, prompts, model, hyperparameters

When improving an agent there is an order of operations based on how useful a change will be. First is data quality: if the underlying data is bad, hard to read, or difficult to interpret for the agent or the judge, nothing else you tweak will be useful. Second is prompts: once data is good, move to the prompt level by changing your rubric or agent. Third is model choice: it has a lot of leverage but changes many things at once, and the usual change (a bigger model) makes things more expensive — a trade-off you don't always want. Fourth is hyperparameters: they are so useless that modern models have stopped allowing you to tune them — you can't pass temperature to modern Claude models, and if you're tempted, it's the least useful thing you can pass.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Precision vs. recall for judges: prioritize recall in most cases

Precision asks: when the judge says 'not actionable', how often is it correct? Recall asks: of all the reports that aren't actionable, how many did the judge catch? The two sound very similar but measure different things. In most cases you should prioritize recall, because high recall catches everything but will give you false alarms. A false alarm costs you a minute or two when you look at a report that turns out to be fine, but a missed failure gives a bad report that reaches a customer. If recall goes up, precision will go down, and vice versa — you have to trade one for the other. For your application, you have to decide whether discarding some actually-correct things is better or worse than getting false alarms where you flag something important that turns out not to be.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Treat eval prompts as code: they are extremely sensitive to wording

Laurie Voss advises treating eval prompts as code, because eval prompts going into a custom eval are extremely sensitive to small changes in wording. As an illustration, he forgot to put XML tags around his examples; with examples as plain text, the judge returned a string of not-actionable results because it was confused about what was data and what was examples. When he wrapped the examples in XML tags, everything suddenly became actionable — with no other change to data, instructions, input, or output. The Coca-Cola report was the example that flipped when the prompt changed, showing the judge got better with the examples.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Split evaluators by dimension; avoid the 'God evaluator'

Laurie Voss warns that a trap with evaluators is trying to write one evaluator that does everything — the 'God evaluator'. The best practice is to split evaluators into as many dimensions as you are testing — one for faithfulness, one for correctness, one for actionability. He also advises knowing which one of your evals is a guardrail and which one is a North Star.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Once live, keep evaluating: run evals on a sample of real traffic and alert on trends, not single failures

After launch, Voss says you should keep evaluating: run your evals on a sample of real traffic and alert when scores drop and stay down. He warns against alerting on every single failure — one failure would be noise — and says only a trend in failures is worth alerting on.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

How many eval examples you need: 10-20 in dev, hundreds for production

Laurie Voss advises that 10 to 20 examples give a rough direction of where things are going and are very useful in development, but for production you want hundreds of examples — it takes a long time but is very worth it for accuracy. To halve the error bars on your accuracy you need to quadruple the number of examples. Going from 25 to 100 examples is really helpful and not that much effort, but going to 400 requires a big jump in effort and gets into territory you're never going to bother with. The bigger the better, and the more production failures you throw into the dataset, the better.

AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Write the calculation once and run it every time; keep arithmetic out of the model's head

Write the calculation once and then run it every time you need it. The model can pick tools, but your harness must make it very strict that it will not do any arithmetic or try to solve these problems by parsing tokens. If calculations aren't in code, the arithmetic is in the model's head and gone once you start a new session, and new code gets generated for every new question or session until it becomes a mess. Asking the same question and getting different answers is non-determinism — you're asking a stochastic machine to produce a consistent output, so you're likely not going to get it.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Establish a shared domain with the model so you talk in the same terms

Francesco Galletta recommends sharing a domain with the model and agreeing on a plan so you and the model talk in the same terms, noting that your definitions likely differ from what the model knows because its training differs from your context, and that going back and forth with the model really makes a difference in getting a shared understanding.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

A planning harness should refuse vague input and force every needed data point

Francesco Galletta explains the harness should not go into yes-man mode and implement an unclear idea; instead it is guardrailed to force the user through every single data point needed to come up with a plan, helping them drill down into what actually needs to be done and challenging assumptions they may not have known they had.

AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Pitching as a whip: arm care through mobility and assessment

For a baseball pitcher, the arm is a whip — many big leaguers are super skinny, with force coming through the hand and shoulder. From an arm care point of view, a key priority is good mobility: assess T-spine mobility, the ability to separate (dissociate) shoulders and hips, and shoulder external/internal rotation control. The elbow will be taxed in some way regardless.

her coaching framework

Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs

Hitter development: strength foundation, fast contraction, then transfer

Grubbs outlines a pyramid for hitters: first assess a good foundation of strength through the lower body, posterior shoulder, or lat to stabilize, plus the ability to contract and relax. Develop rate of force development through plyos and jumps on top of that foundation. Once strength and fast contraction are established, the hitter can contract fast to do their job, and you can get creative with transfer of training — the 'sexy stuff' like rotational med ball throws.

her coaching model

Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs

Split API keys by department and use case to enable measurement

Eugene Cheah advises starting to split up API keys by departments and by major use cases so you know where all your token spend is, noting that splitting keys this way informs your workload decisions from then on. He stresses that if you are not measuring your token spend, you should ignore everything else in the talk and start with measuring first, because the biggest ROI is just from measuring, and that token usage should be measured through separate token keys. For those not yet tracking token spend and token-view split, he points to Featherless's open-sourced tokenomics tracking tool, which he describes as a fancy way of splitting up API keys and tracking which API key spends how much. Attendees can scan a QR code for updates on the tool and fill a form to be notified about the talk's.

AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI