Report · MoneyLaurie Voss explains that the span kind tells you what type of step it is — an agent tool, an agent, a tool call, an LLM, and so on. In tool calls and web searches, the span input is JSON, and that structured data makes the first eval easy to write. Spans also include timing and token counts, and often your agent will record what it costs — these are the fields to pull out and grade when writing evals.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyThe principles build on top of each other and form a logically sensible order: build a shared domain (plan mode) first, then set up consistency, then evidence, then accountability, then finally verification.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyTreating every input as a fact when some are estimates can invalidate a whole financial model. The model cannot tell them apart on its own, so you must enforce the distinction through harness design.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyInclude review gates on purpose when designing a harness so you can remain accountable for what is produced; without them, your only answer is 'the stochastic machine said it was fine.' Capture important judgment calls and disambiguations — not trivial ones — at those gates.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyThe main takeaway for designing accountability mechanisms is to use few gates with very high signal. Breaking tasks down in a way that you can reason about the issues and how they connect is a good example of a high-signal gate. You can be a little hand-wavy with reviews before functional tests pass, but you wouldn't push anything into production that hasn't passed all your tests.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneyIf you don't make sure calculations run in code, the arithmetic is in the model's head and is gone once you start a new session. Make the harness strict so the agent doesn't do arithmetic by parsing tokens.
AI Engineer · Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Principle · MoneySet up your test cases, evals, and examples, then do the search process from there. Find the cheapest model that passes most of your evals, then see how you iterate and fine-tune from there. It's also not that hard to just download 10 models from Hugging Face and try them as an alternative to using a provider.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Principle · MoneyYou generally only want to pay for tokens-per-second speed when the model is in front of someone (developers or customers). Background agents running at midnight don't deserve premium speed because no one is waiting for the output.
AI Engineer · The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Principle · MoneyThe speaker argues that removing one low performer (an 80) raises the team's average more than adding another high performer (a 140), so your bar will rise much faster by eliminating the duds than by adding new hires. He frames getting to a super high-performing team as coming from not letting people stay, and building a better team through elimination — get out everybody who sucks, and what you're left with is awesome. To increase the IQ per square foot, remove everything that is not high IQ. He notes this lesson requires boldness to act on.
Alex Hormozi · My honest advice on how to raise your team’s bar
Principle · MoneyWhen making hard people decisions, the speaker advises asking whether you want local empathy (for the individual) or global empathy (for the business and team). Shifting from local to global means you deal with the local pain while everyone else gets the global benefit; the reverse is that you get the local benefit and everyone else suffers the global pain. He encourages being willing to take the emotional pain of doing what's required to protect the high-performance culture you want to build.
Alex Hormozi · My honest advice on how to raise your team’s bar
Principle · MoneyThe speaker argues for a slightly easier way in, because judging people from one or two conversations and a document is limited, whereas judging their output is much better. Assuming someone has gone through training, you can just look at their output to evaluate them. He advises shying away from allowing someone to explain away a lack of output, noting that mediocre people are often good at selling their excuses.
Alex Hormozi · My honest advice on how to raise your team’s bar
Principle · MoneyBecause culture is built from the bottom up, the worst person on the team becomes the bar you have signed off on. One of the easiest ways to improve a team's culture is therefore to remove the people who perform poorly — the speaker frames this as the core mechanism of raising the bar.
Alex Hormozi · My honest advice on how to raise your team’s bar
Principle · MoneyThe speaker defines the actual action of raising the bar as lowering your tolerance for people doing mediocre work. Practically, he advises not giving nine chances — give a few chances, and after that, that's it.
Alex Hormozi · My honest advice on how to raise your team’s bar
Principle · MoneyHarish Gaggar explains that sensitive data (documents, API keys) can be accidentally typed or uploaded into AI systems, so it should be blocked and redacted the moment it is ingested — above roughly 90 percent severity — so secrets never reach LLM models, the memory DB, tracking/eval tables, or tracing systems.
AI Engineer · Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma
Principle · MoneyHarish Gaggar recommends a centralized MCP gateway where every tool call request — from a local client or an agent — is authenticated before reaching MCP servers, which then control which tool executes the query.
AI Engineer · Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma
Creator perspective · MoneyExtracting plain text from PDF tables formats them badly and can make the agent fetch wrong values (e.g., fees); transforming tables into CSV with spanned fields duplicated made a big difference in answer correctness.
AI Engineer · Your Fridge Is a Valid Payment Device — Eleonora Drykina & Nina Žižakić, In The Pocket
Creator perspective · MoneyThe obvious retrieval move is dense embeddings — turning the question and every chunk into vectors and comparing distance — which is great for meaning. But rulebook codes, message names, and exact identifiers look like noise to a dense embedding, while to the member asking they are often the essence of the question. With BM25 it is the mirror image: exact tokens work but paraphrases would not. The team combined both in one query call and fused them with reciprocal rank fusion, whose nice property is that no score calibration between the two legs is needed. The pipeline takes dense top 50 and BM25 top 50, fuses to top 30, re-ranks for section coherence, and feeds the top 10 to the agent.
AI Engineer · Your Fridge Is a Valid Payment Device — Eleonora Drykina & Nina Žižakić, In The Pocket
Creator perspective · MoneyEval questions come from real sources — rounds of user testing with actual members plus behavioral guidelines for what AI systems must and must not do (no advice, stay neutral about competitors, say clearly when something isn't present in the rule books). The team runs about 86 end-to-end tests with promptful: a full agent, an LLM judge against a rubric, a check that the right source document was cited, plus the original retrieval-only evaluation. LLM judge evals have a noise floor — you need to rerun before you trust the delta. Sometimes the rubric itself is the bug, not the model: a hallucinated date the judge kept punishing turned out to be real text sitting in the rulebook.
AI Engineer · Your Fridge Is a Valid Payment Device — Eleonora Drykina & Nina Žižakić, In The Pocket
Principle · MoneyThe section path stored on each chunk is used both to render citations and to fetch an entire section in order (sorted by chunk index) when the agent needs more context.
AI Engineer · Your Fridge Is a Valid Payment Device — Eleonora Drykina & Nina Žižakić, In The Pocket
Creator perspective · MoneyDense embeddings capture meaning but treat rulebook codes and exact identifiers as noise, while BM25 handles exact tokens but not paraphrases; the team combines both in one query call, fuses with reciprocal rank fusion (dense top 50, BM25 top 50, fused to top 30, re-ranked for section coherence, top 10 to the agent), and notes RRF needs no score calibration between legs.
AI Engineer · Your Fridge Is a Valid Payment Device — Eleonora Drykina & Nina Žižakić, In The Pocket
