Report · MoneyLaurie Voss explains that adding Phoenix tracing to an agent project is minimal: call register from the Phoenix OTEL package, which sets up tracing and points it at Phoenix. Frameworks already have instrumentation built in, so the register command just turns it on and points it at your Phoenix instance (e.g., your laptop).
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyAsking a machine to follow all instructions and make no mistakes will probably work 99% of the time, but the 1% edge cases are the real risk. Determinism via back-end code covers these out of the box.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyWhen a calculation needs to be repeated reliably, convert it from agent-driven to a script. Letting the agent decide categorization, aggregates, refunds, and expenses yields slightly different results every time and wastes money and time.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyWhen reviewing agent work, review the set of tasks the agent plans rather than approving every individual change; let the agent do its own thing and review the results at the end. Harness design should be intentional about this.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyIf you don't have a way to validate, you end up checking everything, checking nothing, or having no sense of what about-right looks like — and if you don't have that point of reference, your harness won't have it either. You need to build external validation points for your harness. For verification, add notes in the decision — an external point of reference added to some of the stuff the model thought.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneySubmitted documents like bank statements are facts; agent reasoning or information pulled from memory must be treated as assumptions; unstructured thoughts in chat must be labeled as interpretations.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyThe boundary between a skill and a separate tool is when the use case becomes critical enough that the trade-offs of the non-deterministic approach are not acceptable. Felippe describes the prompt-based (skill) approach as: 'Hey, please follow all of this and make no mistakes. Please, please, please.' Prompt-based approaches work because agents are good enough nowadays and will probably perform your instructions 99% of the time — but the question is what about that 1%.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyBecause you have code that is tested, you have a trail you can follow backwards for evidence. Tests run in your environment with no agent thinking involved, and skipping them sabotages the analysis.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyBuild a harness when you feel the use case is worth enough — when you want reproducibility, or when the results you are getting using pre-built harnesses are very frequently getting it wrong. There's no clear answer to how generic or specific a harness should be or how often to build one — it comes down to use case value, number of users, and problem criticality. The decision on whether to build a harness comes down to how many users you believe will go through that use case and how critical the problem is that you're seeing.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyEvidence goes into the code itself: track the provenance and the method behind every number, not just its source. Gather as much evidence as you can and assign the right level of importance to each piece, the same way you would curate a dataset for training.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyTreating all information the same without distinguishing facts from estimates leads to fatal errors in financial analysis, because a single wrong assumption can invalidate the whole model. Enforce the hard-truths-vs-estimations distinction through harness design.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · BodyGrubbs argues that in professional baseball, if players don't lift on game day they effectively never train — skipping game-day lifting across a 162-game season means losing size and building no strength foundation. She frames game-day lifting as something done 'literally based on necessity.'
Garage Strength · How The Best Pro Baseball Players Actually Train w/ MLB Strength Coach Stephanie Grubbs
PrincipleMichael explains the Swiss cheese model from aviation safety: multiple layers of defense each have vulnerabilities (holes), and safety requires ensuring the holes don't line up so a critical scenario cannot pass through all layers. He details the aviation layers (strategic separation, air traffic control, collision avoidance system) and applies the model to LLMs (training data control, runtime monitoring, chain-of-thought monitors).
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
PrincipleTurning everything over to computers with no humans in the loop causes systems to go off the rails when something weird happens; keep humans in the loop for AI systems, including the AI admin idea.
Y Combinator · AI Safety Fact vs. Fiction | YC Paper Club
Report · MoneyBefore deciding on GPU count or deployment, ask what model quality and what tokens-per-second speed you actually want. Customers typically target 20-30 tokens per second (ChatGPT speed) up to 60, depending on how long the chain of tasks is.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Principle · MoneySplit API keys by department and by major use case so you know where all your token spend is. Know the token spend per department and per team so you can start seeing which team is performing better and what works. Check which teams spend the most tokens and whether they are productive. Use per-team token usage data to advise on dedicated capacity purchases, whether for open or closed models.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Creator perspective · MoneyThe speaker is not expecting the audience to start building their own custom harnesses from scratch because great solutions already exist in the market (e.g., Cloud Code, Codex). But you can build custom harnesses for specific problems without them being as sophisticated as creating a new Cloud Code. Simplify the problems you are trying to solve and then build custom harnesses that attach onto that, rather than running all your work in one sophisticated custom harness. Consultants are a perfect use case: they can create one that works for a financial agent and then adapt it for different customers, data types, or constraints.
creator's guidance; consultant case is hypothetical
AI Engineer · Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Report · MoneyOliver Kharraz describes how ZocDoc still operates its destination-website model for finding and booking doctors, but has evolved into an infrastructure layer that payers, AI platforms, and consumer companies can plug into for seamless appointment scheduling.
The a16z Show · Why AI Could Finally Fix Healthcare Access | Zocdoc CEO Oliver Kharraz
Creator perspectiveCJ Avilla explains the orchestration model: a lead agent creates a plan, automatically spins sub-agents up and down based on its own capability, and accomplishes discrete verifiable units of work across phases without the user managing interconnections.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic
Creator perspectiveStoring API keys as plain text on disk (even hidden files or secret directories) risks the credential leaking, whether via a user prompting Claude or your product to extract it, or through prompt injection or user manipulation. Putting an API key into a markdown file given to Claude is definitely bad — do not do that.
AI Engineer · Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic