Principle · MoneyAl Chen's principle: it's not enough to say you bumped the package version to a specific number — you have to provide evidence that the exploit is no longer actually in the application code. Similarly, with backup/restore, you can't just bump the version and assume the exploit is gone; you have to do the extra checks.
endorsed
AI Engineer · Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco
Principle · MoneyAl Chen explains that one way to build trust with security compliance teams is to not just say 'trust us, here's the answer' — you have to show the evidence and the work on how you arrived at the answer. When AI vendors give an answer, they have to show the evidence and the work on how they arrived at it; showing your work is the real lever for compliance velocity when working with a bank.
endorsed
AI Engineer · Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco
Creator perspective · MoneyAl Chen explains that most banks think they are not leaking sensitive information to internal employees, users, or outside their VPC, but there may be side doors that are open. Leaks come in the form of PII and MNPI that need to be redacted, and there are many strategies: at the SDK level, at the gateway level, and using various middleware. In practice, the UI might show customer information with sensitive fields redacted.
described
AI Engineer · Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco
Principle · MoneyWhen building customer-facing agentic solutions in regulated industries, reduce unpredictability and reproduce outcomes as much as possible, then figure out failure modes and unwanted results so you can improve upon them. Banks face financial liability, legal liability, and a bunch of risk when putting out LLM-powered customer-facing solutions, so accuracy of information is critical for the institution. Don't put out inaccurate or hallucinated information in front of your customers.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Principle · MoneyFully automating agent improvement with no human time investment produces slop, low quality, and unsatisfied customers; asking a coding agent to 'fix my agent, make no mistakes' is an anti-pattern. Bringing human taste and domain expertise in while automating what can be automated lands at the ideal balance.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Principle · MoneyThe foundation of agent improvement is a loop: trace all agent steps into an inspectable log, monitor production traces to find what goes wrong and right, build datasets representing production behavior, run experiments by varying one element at a time, and continuously close the loop by feeding production learnings back into datasets.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
PrincipleWhen evaluating models for RAG, force the model to associate each piece of information in its answer with a source — this can force it to realize something wasn't in its sources. Check whether the model admits when information is not source-provided; a model that knew information wasn't source-provided but still gave it (e.g., 'no source provided beyond interactions with carpenter bees') is a signal about hallucination tendency, and an answer without a source may indicate hallucination.
AI Engineer · The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft
Creator perspectivePamela Fox explains that RAG is when you ask an LLM to answer a question based off some context, and the model must stay grounded in the provided context and not answer from its weights. Smaller/less powerful models are much more likely to answer from their weights and pretend it was part of the context in RAG.
AI Engineer · The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft
Creator perspectivePamela Fox argues that benchmarks alone don't tell you how good a model is for your particular scenario; you have to use it in your scenario. She recommends testing models against your own scenarios rather than relying only on quality and safety benchmarks, and figuring out what task you want to do and whether the model will do a good job at it.
AI Engineer · The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft
PrincipleWhen executives raise concerns, Larry David refused to compromise and threatened to quit rather than dilute his vision: 'this is what the show is. This is my vision. If you don't like it, I quit.' But the narrators caution against blindly copying this: the lesson isn't to be a blowhard and take no notes ever — for most people that approach is failure. Only let a creator 'be a genius' if you've verified they're actually a genius; most people with absurd ideas are wrong. Most founders with contrarian ideas are wrong, but the few who are right are really right, so evaluate each on merit.
Acquired · Seinfeld
Creator perspectiveAmeisen describes how interpretability serves alignment: use interpretation to check whether character training worked — look inside to see if the model is behaving well only because it's being watched. A model that only behaves well when observed will behave poorly when it doesn't have evidence of being monitored. Use interpretation to get at unverbalized parts of the model that only exist in intermediate results, identify cases when models represent things they don't say, and prevent them. When a model does something really weird and you're perplexed, investigate: look inside the model's intermediate results to understand weird behavior and use what you find to improve alignment of this model and future models.
Lenny's Podcast · Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)
Report · MoneyThe speakers argue that tools are what allow an agent to physically interact with the computer and verify its own work — checking whether a join actually worked, whether storage updated, or whether a command ran start to finish. Without these tools, the agent cannot see the results of its actions and must rely on human interpretation.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyNick Nisi frames the purpose of verification tooling as reaching a trusted 'done' state faster and with fewer interaction turns with the agent. The moment an agent realizes it is not done and refuses to drop the work on the floor is when the entire verification system pays off. He also emphasizes giving agents tools to prove to themselves they're done and to show evidence the human can trust, so returning to the work doesn't require re-prompting and restarting the loop.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Principle · MoneyThe speakers endorse that verification tools create reproducible evidence that acts as a regression check — next time the page changes, the agent can run it again and verify nothing broke, keeping the team moving forward.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyThe speakers endorse that once you build durable, reliable, repeatable, uniform loops, failure starts to drive future work — the agent determines on its own that something is wrong and takes another stab. If an agent fails in a loop while you're busy, it can self-correct: run the command, verify again, and confirm it's good without you coming back to the PR at end of day. They call agents self-healing on single lanes of work foundational for speed and velocity.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyNick Nisi explains that relying on prompts alone without verification tools yields the confabulations LLMs are designed to give you, because they process text and try to make you happy. For every detail you don't fill in about the desired outcome, the agent will infer or make something up, and its assumption may not match yours. An agent without clear goals will spend tokens and might reach the expected end, but there's no guarantee of it.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyNick Nisi argues that every failed verification run is valuable: each miss adds a new line to the failure map, reveals why the agent didn't reach the goal, and gets the team one step closer to a fix because the failure can be captured and repaired. He also notes that once a bug like the wrong-email bug is caught and proven, it won't recur.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Principle · MoneyThe speakers endorse that the goal is not to wrap determinism around LLMs — agents are intelligent and capable and you want them to be creative. With the right constraints, agents come up with creative solutions that actually solve the stated problem (constraints can equal freedom). They recommend giving agents the outcome rather than prescribing how they achieve it (outcome-driven development), noting being overly prescriptive can lead to lower performance in some cases.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyThe speakers frame verification not as an optional add-on but as a foundational concept that benefits the whole team. When agents can prove their own correctness, the entire team moves faster and ships safer, more reliable software. Without it, teams get stuck in a loop of the agent claiming 'done', the human re-checking, and re-prompting while juggling other agents.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyNick Nisi explains that once you build verification loops that are durable, reliable, repeatable, and uniform, the main benefit is that a failure starts to drive future work on its own — without such loops, a failed agent run cannot drive future work and requires human intervention.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS