Method · MoneyUse faithfulness evals when the judge lacks fresh data: faithfulness asks whether the report makes sense given the data the agent was given. Run it with three inputs — the input (what the user asked), the output (the report), and the context (the search results and notes). Pull both the raw search results and the research notes in your getAgentRuns helper; a first version that only gave research notes caused the judge to flag real numbers as made up. Expect faithfulness to run slower than correctness because the judge reads every search result.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyQuery Phoenix for spans with parent ID null to get only top-level spans (the agent runs); the repo's getAgentRuns helper hands back each run's span ID, input, output, plus the research turns output. Use createEvaluator from the Phoenix evals package with an ordinary function, marking it as a code eval so no LLM is involved. Code eval functions receive input and output, check each requested ticker appears in the report, and return a label plus a score with the missing ticker named so the failure explains itself. Run them via npm run eval code — they execute almost instantly because code evals don't wait for an LLM.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyInstall the Phoenix plugin/skill for your coding agent, point it at your traces to look for weird things, have it read traces, come up with categories of failures, count them, and write each category back onto its trace in Phoenix as an annotation; let it write notes to disk if it asks.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Recommendation · MoneyUse all three kinds of evals at the same time — they are complementary, not competing approaches. Use code evals when the answer is deterministic: formats, lengths, required fields, banned phrases. Use an LLM judge when you need understanding: is it accurate, is it grounded, is it useful. Use humans for failures you haven't seen before and to check that your LLM judges are judging correctly.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyEvery eval follows the same pattern: get the runs, evaluate them, then log the results back to Phoenix via log span annotations, which attaches each result to its span. The correctness evaluator is one import and one line of setup; call evaluate to get back a label, score, and explanation, and log results the same way as the ticker check. Phoenix evals in TypeScript use the Vercel AI SDK to talk to models, so any model the AI SDK supports can be a judge.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Recommendation · MoneyWhen designing a harness, include on purpose those gates so you can still be accountable for what is being produced. Make sure any important judgment call or disambiguation — not the trivial ones — you can still get hold of. Without accountability gates, the only answer to 'what happens?' is 'the stochastic machine said it was fine.'
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Recommendation · BodyBecause professional baseball players compete nearly every day across a 162-game season, lifting must happen on game days; if you skip game-day lifts, you won't lift at all during the season.
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
Recommendation · MoneyBack to the first step: measure. Know the token spend per department and per team so you can see which team is performing better, how much tokens they use, and what works.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Recommendation · MoneyA common approach: deploy an expensive model at small scale, generate your dataset from it, then reduce dramatically to a smaller model (e.g., 7B scale).
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Recommendation · MoneySplit API keys by department so you can see where token spend is going, and track it down to the team and use-case level rather than just organization level. A single shared API key across the whole organization makes token spend unmeasurable. Featherless AI is open-sourcing a tokenomics tracking tool that splits API keys and tracks per-key spend.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
RecommendationSet a budget on agent flows based on whatever your budget might be to prevent runaway sessions; relying on Claude alone to detect misbehaving sub-agents without a budget risks runaway sessions. When enabling multi-agent sessions via the API, set up a budget, and know you can extend it if you end early without spending it.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic
RecommendationSet a very specific goal at the beginning of a flow and copy it consistently into each sub-agent's context across all phases to prevent goal drift.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic
RecommendationPrompt Claude to give you reports or update a specific report as the run progresses (typically markdown files); have Claude write to its outputs directory so you get durable downloadable files via the v1 files API; stream the durable event stream in real time or list events via a static API endpoint to monitor what agents are doing and decide whether to take action; use thread message sends and receives to inspect inter-agent communication and verify middle-management agents aren't doing something unwanted with their underlings.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic
RecommendationKick off long-running multi-flow projects and walk away, checking back later rather than waiting synchronously; when a lead agent's flow finishes, start another flow to chain large projects across multiple flows, and a lead agent with full state of where the flow is in its process can inspect where it left off and kick off a new flow incorporating both new information and past history.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic
RecommendationUse the suite of abstractions that ensure credentials never have to touch the sandbox — an incredibly important decoupling. Claude takes on the persona of whatever credentials you give it, so credential handling determines your agent's effective identity and permissions.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic
RecommendationYou can interrupt a running flow session to correct its direction mid-run with a direct stop command, and when new external information changes the trajectory of a run, interrupt the session and provide that information as a direct instruction referencing the new material, rather than burning two days of tokens and starting from scratch. Note: you can't inject a single step halfway through and rerun later steps.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic
RecommendationFan out across all services and dedicate an individual agent context to each service so each agent is only responsible for its small scope, yielding a much better answer and guaranteeing every service was checked.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic
RecommendationWhen a teammate's exploration contains something you like, select both their layer and yours, then ask the agent to combine them into a new code layer, naming the exact feature to take from each (e.g., 'create a new code layer, combining the grid clustering from the grid layer and the delivery feature from the delivery code layer'). The result is a single code layer with the best of both explorations.
Figma · Explore ideas in code | Figma
RecommendationExtracted layers are fully editable — move components, adjust typography, or make other design adjustments — and changes can be pushed back to the code layer so they make their way into the code.
Figma · Explore ideas in code | Figma
RecommendationWhen exploring different design directions, ask the agent for multiple explorations in a single prompt; specify one variation explicitly and let the agent surprise you with the others. Example prompt: 'Make three variations of this and add a new delivery button to the map page. When clicked, the user can choose their address. The first one should be an Omni search address field, but surprise me with the other two.'
Also discussed by Kova on AI Engineer, Magic Patterns (guest) on Y Combinator (unclassified source — not counted until classified). Not counted as independent sources.
Figma · Explore ideas in code | Figma