Recommendation · MoneyLaurie Voss advises running npm run batch to execute 15 queries in a row, since one test is not enough data to draw conclusions. He also recommends including edge-case queries in the test set: comparing two companies, non-public companies, different industries, and fake tickers. Synthetic LLM-generated queries are only a seeding method, so a human should review the list.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Recommendation · MoneyWrite down exactly what makes a report actionable: it makes a specific recommendation, it names concrete risks, and it looks forward, not just back — things a judge can actually check. Your criteria come from your error analysis: 'only summarizes data with no interpretation' is a problem seen when reading traces, and it goes in the rubric because it's a problem you're trying to avoid.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyTracing takes about five lines: call register from the Phoenix OTEL package to set up tracing and point it at Phoenix. In TypeScript, use manually instrument, which hands you back a copy of the instrumenter — use that copy because the Claude Agent SDK is an ES module with read-only exports. Import the Claude Agent SDK from the instrumentation file, never straight from the package.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Recommendation · MoneyLaurie Voss advises workshop attendees to ask themselves what would make them act on one of these reports and what they saw in their own traces, write that down, and let it become their rubric. He also notes that the criteria and examples in the rubric file can be changed.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Method · MoneyPass the session ID back in as resume so the next turn remembers everything the last turn saw. Put the two turns together in one trace, wrapped in trace agent to get one parent span encompassing all the data. Without wrapping you would get two sibling spans, which is fine but slightly harder to read.
Method, not a verified protocol.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Recommendation · MoneyFrancesco recommends being hand-wavy with reviews prior to functional tests passing, but never pushing anything into production that hasn't passed all your tests.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Recommendation · MoneyFrancesco recommends that when creating a harness, you build gates not only for validating things but also for correcting the course and making sure you can do so safely.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Recommendation · MoneyApproach AI adoption from a highly impact-based point of view rather than 'we should use AI for this because AI can something, something, something'.
endorsed advice
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Method · BodyA structured template for in-season lifting: lower body day one pairs a knee-dominant squat pattern with a jump, plus a Nordic or Razor Curl supersetted with a 3D lateral lunge for the groin, and mobility/core work. Upper body day one pairs a horizontal push with a dynamic horizontal push, an upper back movement with a vertical pull, and optional grip work. Lower body day two uses a split squat pattern or trap bar deadlift with a split squat jump, hip-dominant posterior chain work (hip thrust or RDL) with groin work, foot and calf work, plus core and mobility. Upper body day two uses a vertical push (landmine press) supersetted with a rotational mid-back row and a horizontal row for upper back/scap work, plus biceps, triceps, and arm care in coordination with the training staff. Exercise
Consequential creator advice requires independent verification before Today can promote it.
Method, not a verified protocol.
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
Recommendation · BodyGrubbs passes along advice to young women entering men's pro sports: 'make it about the role, not about the gender, like just dominate the position that you're in.' When asked how she'd get respect from male athletes, she answers 'yeah' and just goes and does it. She also coaches nervous interviewees: 'these are your talking points, go in, dominate.'
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
MethodTo find the most likely failure in language models, use perplexity: search for scenarios with low log perplexity and high toxicity (the top left corner of the search space). If the most likely failure (toxicity above some threshold) is really likely, study it and put in place mitigations.
Method, not a verified protocol.
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
RecommendationAI systems should be built with zero-trust architectures so that infiltrating one system does not allow infiltrating everything else.
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
RecommendationYou should continue the validation effort after deployment, not just before; this is sometimes called runtime monitoring or runtime assurance. You want to validate what's going into your decision system, i.e., the sensor inputs; work with Airbus was done on integrity monitoring of vision-based systems. You also want to monitor what is happening in the internals of the system, which has led to work on mechanistic analysis for an aircraft system. And you also want to model and monitor the outputs of the system.
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
RecommendationContinuously monitor AI benchmarks for distribution shift, just as you would monitor AI systems, and use quantitative methods to identify shift in language — because definitions of harms drift fast as new generations introduce new slang while benchmarks move slowly.
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
Recommendation · MoneySplit the dataset because people are lazy to filter — get at least a golden trace dataset, which may be much smaller, and fine-tune that with a much heavier weightage.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Recommendation · MoneyEfficiency in the memory layer comes from the number of entities that you create. Keep the memory layer simple: don't create too many abstractions or too many components. Experiment first with only short-term memory and long-term memory — just two structures.
creator's recommendation
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Recommendation · MoneyNacho Martinez advises deciding your own memory layer architecture based on the use case rather than adopting a fixed design.
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Recommendation · MoneyNacho Martinez recommends keeping the context window as small as possible, since the smaller it is, the better the model solves the problem.
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
RecommendationImplement human-in-the-loop feedback in a flow by giving Claude a custom tool called 'request human feedback' that Claude calls whenever it needs input.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic
RecommendationThe speakers recommend using Claude Code (UltraCode) when sitting at your laptop typing into the terminal interactively, and Cloud Managed Agents (Flows) when you have a repeatable thing that should happen on a regular basis and want to integrate against an API — treating UltraCode as the single one-off instance and Flows as the layer to reuse and run agents on a trigger schedule.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic