Principle · MoneyKent C. Dodds endorses 'Tell me about the last time this happened' as a better validation question, alongside Mom Test-style questions like 'When was the last time you were stuck?' and 'How's your experience been while waiting?' — questions about actual experience rather than evaluation of an idea.
AI Engineer · Build the Right Thing: Product Engineering (Part 1) — Kent C. Dodds, EpicProduct.engineer
Principle · MoneyWhen validating an idea, don't ask users future or hypothetical questions like 'Would you pay?' or 'How much would you pay?' — nobody can tell the future, and such questions produce unreliable answers. They also turn into 'complimenting': the person just says yes to everything to get out of the conversation. Instead, don't talk about the solution at this stage; start with what the user did before — what is the past. A better validation question is 'What did you do instead?' and 'What did it cost you to do that?'
AI Engineer · Build the Right Thing: Product Engineering (Part 1) — Kent C. Dodds, EpicProduct.engineer
Creator perspective · MoneyJim Clark describes Docker's sandbox approach: a gateway endpoint is placed into each sandbox and funnels all MCP traffic. Every agent harness should have a single internal MCP gateway URL (e.g., mcpgateway.docker.internal) through which all resources and tool calls move. The gateway manages which tools, resources, and prompts are given to each sandbox, and the sandbox configuration controls what the agent can do inside that sandbox with things like MCP.
AI Engineer · How Many Credentials Should Your AI Agent Have? Zero. — Jim Clark, Docker
Principle · MoneyKent C. Dodds describes the Kano model as a mental model for prioritizing software changes, with five categories of which two should never appear in your code base or product at all. Basic needs: nobody will be satisfied until they're fully implemented, no matter how much you've implemented. Performance needs: once you hit a basic level people are okay, but continuing to deliver on them makes people happier and happier — though there's probably a limit. Delighters: people aren't even expecting these, so even barely implemented they generate a lot of satisfaction, and you can continue to add delighters until they eventually cross over into performance needs.
partial (the limit on performance needs is hedged with 'probably')
AI Engineer · Build the Right Thing: Product Engineering (Part 2) — Kent C. Dodds, EpicProduct.engineer
Principle · MoneyKent C. Dodds explains that not all feature requests create value in the same way, introducing the Kano model — a mental model from a 1984 paper about attractive quality and must-be quality. There are four quadrants on two axes (implemented and satisfied) and five categories, two of which should never appear in your codebase or product at all. The three categories that appear on the graph are delighters, performance needs, and basic needs. The graph has no units, so the specifics of where things fall are a bit hand-wavy.
AI Engineer · Build the Right Thing: Product Engineering (Part 2) — Kent C. Dodds, EpicProduct.engineer
ReportGemma 4 models are Apache 2 licensed, meaning users can download them, use them in their company, fine-tune them, and expand on them however is most useful.
AI Engineer · Research to Reality with Google DeepMind — Paige Bailey, Google DeepMind
Creator perspective · MoneyWhen coming up with output choices, keep them simple — binary is best, two clear labels. With a numeric scale you would have to write a whole bunch of rules for the LLM to know the difference between a three and a four; it doesn't know that automatically. Give the judge pass-fail and it will be pretty good at deciding what is a pass and what is a fail. If you really need more categories, you can use a partial result, e.g., a 0.5 score for a half-assed job, but by default go for zero or one.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyInclude edge cases in your test data: non-existent tickers, multi-part questions, and jailbreak attempts. Edge cases might be 1% of your traffic, but they are the 1% that ends up being a PR disaster in the press.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyAfter categorizing failures, sum them up into a table of root cause frequency — e.g., looks good, possible hallucination, reasoning gap, unverifiable data, missing recommendation. It's tempting to fix the most frequent failure first, but you should balance between frequency and severity instead. If one time in 100 the agent gives the user instructions on how to make a bomb instead of a financial report, you fix that one first because it's the most severe possible failure — multiply severity by frequency to decide what to fix first.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyLLM judges exhibit position bias (favoring whichever option comes first, though some models prefer the last), length bias (scoring longer responses higher even when extra length is filler), confidence bias (believing wrong responses stated confidently), and self-preference bias (models tend to like their own output when generating and judging with the same model).
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyThe speaker asked Alex to build an eval that ensures he is never giving instructions about how to build a bomb, and noted that such a basic eval is something that should have been in place from the beginning.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI
Principle · MoneyQuality issues can exist even when there are no error spans: spans can return 200 and say things are fine while the app is misbehaving, so relying only on error spans to detect agent problems misses these quality issues.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI
Report · MoneyExisting verifiers use smaller, dumber models like O4Mini or GPT-4o as the LLM judge, don't use rubrics (the most critical thing for assigning credit), either skip relevant screenshots or overflow the context window by looking at all of them, and sometimes don't even look at the final answer or action history.
AI Engineer · Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase
Report · MoneyThe application layer provides a latency budget (SLO) for how quickly tokens should be returned, which is the primary constraint under which inference deployment operates; it covers time-to-first-token (finishing pre-fill) and time-per-output-token / inter-token latency (during decodes).
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Principle · MoneyBlaine says a principle is a jig or a working tool used to make a product — supposed to make it easier to make the product, especially with other people. A design principle is only useful if its opposite isn't obvious, and a useful principle is specific enough to be an opinion you could actually disagree with — that's what helps teams. He illustrates with a hypothetical automated email: he thinks it should be playful and fun while his product manager thinks it should be authoritative, and every time they meet about it they'll be wrong unless they agree on the principle for how they talk to people.
Commercial context: The creator describes this as their own product.
Design Better · Blaine Billingsley: OpenAI designer on the collaboration problem AI hasn't solved yet
Creator perspective · MoneyKarthik identifies five misconceptions: more memory is not learning (just more memory and more expensive); shared state is not shared knowledge (sharing an MD file doesn't mean shared knowledge); RAG-ing everything is not learning or quality (it's garbage that LLMs burn more tokens on for worse results); fine-tuning is not scalable (you cannot save and persist lessons agents have learned); and a bigger context window is the surest way to the quickest, most expensive failure because context rots. Every one of these is an architecture gap.
AI Engineer · Agent Memory Is Solved. Agent Learning Isn't. — Karthik Ranganathan, Yugabyte
Creator perspective · MoneyWhen Agent A hands off work to Agent B by sharing only the output (an MD file, PDF, or document), the reasoning behind the output is lost. This loses the dead ends and the number of failures before success, so the receiving agent re-derives everything from scratch, burning tokens and money.
AI Engineer · Agent Memory Is Solved. Agent Learning Isn't. — Karthik Ranganathan, Yugabyte
Creator perspective · MoneyVector search returns chunks of code that often miss imports, tool calls, or the calling code above them, so the chunks don't give enough context for the agent to solve the bug. In one eval row, the vector agent made 26 searches and still couldn't piece together how three functions interacted across the file. Vector search was more expensive because it made more searches — returning a chunk, searching again, returning a chunk, searching again — and all those LLM calls piled up. Vector search gives proximity to the correct code, but agentic search has the connective tissue to connect the actual logic between chunks of code.
AI Engineer · Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust
Principle · MoneyEvals comprise four main things: a dataset, a task, a scoring system, and experiments. A dataset is the data you pass into your AI system — usually golden standard cases, edge cases, and failure modes. A task defines how the AI system should behave on input/output, usually including a system prompt and the chosen model. A scoring system tells you whether the output is good or bad, with the PM or engineer writing the criteria. Every configuration of a specific dataset, task, and scoring system equals one experiment.
AI Engineer · Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust
Creator perspective · MoneyLaura frames the initial org chart as a deliberate best guess: the point is to get it wrong so the champion corrects it live, which also surfaces who actually matters and who has influence.
30 Minutes to President's Club | No-Nonsense Sales · #608 - How Mapping the "Shadow" Org Chart Got Me to 305% of Quota