Principle · MoneyRoute AI system usage based on reliability and materiality, ensuring high-risk rated systems are only used in regions where you are confident of performance. Only make the deployment decision once you are confident about the region; otherwise it may be too expensive due to downstream failure costs.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Principle · MoneyEvaluate the full AI system based on risk, not just one component. A narrow evaluation can miss failures that matter in the real world. When extending evaluation to the whole deployment system, test whether retrieval brings back the right information, what happens when a tool fails in the agentic system, whether human review catches mistakes, and whether the system can finally recover from failures.
Also discussed by Y Combinator (unclassified source — not counted until classified). Not counted as independent sources.
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Creator perspective · MoneyTanmay Sah argues that AI failures have structure and that some sub-dimensions are more critical than others: human mediation (human-in-the-loop effectiveness) and harm detection (monitoring) are strongly associated with harm, while scale and system design appear frequently but are not strongly associated with harm. He recommends prioritizing sub-dimensions by harm potential and allocating investigation resources accordingly rather than treating all equally. Deployment controls rank most critical, followed by business impact, then technical failure; securing deployment controls for high-risk systems gives assurance failures can be handled in advance, and deployment controls are far more important than other sub-dimensions.
associations as reported by the guest from their incident analysis
AI Engineer · 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Creator perspectiveFirst step in pairing: trace each face back to its calligraphic root (broad nib vs pointed nib / expansion) and pair faces sharing the same structural tradition. High-contrast modern serifs (Bodoni style) pair well with grotesques because modern serifs structurally evolved into grotesques.
Design Better · Tom Foley: How to pair typefaces and why the best brand fonts go unnoticed
Creator perspectiveAfter structural tradition, compare proportions — x-height is a major factor, along with ascenders and cap height — then match stem weights and overall weight range/color/texture. This three-step filter (calligraphic root, vertical metrics, weight) narrows pairings quickly, especially when pairing functional with expressive typefaces.
Design Better · Tom Foley: How to pair typefaces and why the best brand fonts go unnoticed
Creator perspective · MoneyThe team tried to reapply the same mechanism learned on RAG to the contract project: get the answer first, then have the agent find it back (invert the problem). When you have a well-known technique for one sub-task (like document retrieval from RAG), reapply that same mechanism to a new domain-specific project. When no public benchmark exists for a domain-specific agent task, build a custom test set by inverting the problem: create the answer first, then test whether the agent can find it back. When building an agent for a task with no benchmark, identify the sub-steps of the agent and determine which ones already have known techniques versus which ones need new evaluation approaches.
AI Engineer · How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Principle · MoneyAdit Abraham's top advice for evaluating document-processing tools: the number one thing to care about is the distribution of data — test on data representative of your own documents rather than trusting generic benchmarks. He also advises validating that your eval dataset actually holds before running any model-vs-model comparison.
The a16z Show · The Missing Infrastructure for AI Agents: Documents | Reducto’s Adit Abraham
Principle · MoneyAdit Abraham emphasizes that Reducto cares about not just accuracy but determinism and traceability — customers need to know where an answer came from. Traditional computer vision was strong at this, and word-level citations let agents reconcile conflicting documents and land on the truth.
The a16z Show · The Missing Infrastructure for AI Agents: Documents | Reducto’s Adit Abraham
Principle · MoneyThe Cache Program teams use a 'team brain' — a common knowledge storage where engineers' product feature knowledge becomes accessible to all team members, so a newly joined engineer can tap into the same prompt context as a five-year veteran.
AI Engineer · AI Cut Our Research Time From 7 Days to 15 Minutes — Gareth Yoder, Vanguard
Principle · MoneyChris Dreyer went from legal in general to personal injury because 70–80% of his revenue was already coming from that space — a natural push. He argues people in the agency space struggle because they go too broad — it's market fit, not product-market fit.
Ahrefs · How I Make $32M A Year Dominating This SEO Niche
Principle · MoneyYou cannot automate a process that does not exist. Before handing work to an agent, build a standardized process around the org function, then define the steps to hand that work off to an agent running in the background. Without standardization you also cannot build evals or know whether the agent is doing the work right.
Latent.Space · AI for the Fortune 2000: How Tenex Builds an AI-Native SDLC — Alex Lieberman & CJ Hess
Principle · MoneyRob Walling describes the most common ways founders stumble onto an unsolved problem: scratching your own itch, hitting a problem at your day job, solving a problem of a spouse/relative/colleague, being on the receiving end of a poor customer experience, and finding a problem online in communities like Facebook groups, Quora, Slack groups, and support forums.
Startups For the Rest of Us · Episode 854 | How to Find Startup Ideas (Sneak Peek Chapter from My New Book)
Principle · MoneyGaryVee argues that skills are easy to train but embedding drive and hunger is the hardest part, and that if employees were as ambitious as their boss they wouldn't work for them. He says you can't force a penguin to be a giraffe — you can only maximize the giraffe's penguin-ness — and that over-infatuation with turning a person into something they're not wastes time. No matter how much he trains someone, they're not going to the NBA. Instead, reverse engineer people to be the best version of themselves, push them as far as you can, and when they hit their cap, move on.
GaryVee · The Biggest Opportunity Most People Are Ignoring
Principle · MoneyGaryVee argues quantity is a secret, under-admitted ingredient in breaking through — it's at-bats at making the piece of content that changes your life — but quality must also be part of the equation, since people who make 10,000 low-quality pieces never make it. Quality is subjective while quantity is not.
GaryVee · What to Post When You Run a Small Business
Principle · MoneyGaryVee asserts that accountability is the greatest trait: the second you go into accountability for what's wrong in your relationship, work, or life — saying 'it's on me' rather than blaming the other person (the 'fun part') — things become unbelievable. He notes that blaming the other person means missing the moment of seeing what's actually wrong.
GaryVee · The Leadership Mistake I Made for 20 Years
Principle · MoneyAnything that can charge money or spend money has to be confirmed — treat that as a best practice for agent workflows.
AI Engineer · Stop Telling Agents How — Dan Barrett, Raymond Weitekamp & Jose Montes de Oca, OpenProse
Principle · MoneyRather than dictating every step, declare the how by composing intermediary contracts along the way. Think of these as checkpoints in a DAG: each step fulfills one contract, which triggers the next until the final contract is met. Agents stay flexible on how to achieve each intermediary.
AI Engineer · Stop Telling Agents How — Dan Barrett, Raymond Weitekamp & Jose Montes de Oca, OpenProse
Report · MoneyWhat's different from other chat- or agent-based flows is that all runs are compared against a contract — think of it as a fresh session every time getting benchmarked against the contract you've written declaratively.
AI Engineer · Stop Telling Agents How — Dan Barrett, Raymond Weitekamp & Jose Montes de Oca, OpenProse
Principle · MoneyBenchmarking fresh sessions against a declarative contract gets models out of the self-confidence loop. You don't want the same model checking the thing that was done; spawning new sessions and bouncing them off a contract is a better pattern.
AI Engineer · Stop Telling Agents How — Dan Barrett, Raymond Weitekamp & Jose Montes de Oca, OpenProse
Principle · MoneyDan Barrett endorses spawning new sessions and bouncing them off a contract as a better pattern than having the same model check the thing that was done, because it gets the models out of the self-confidence loop.
AI Engineer · Stop Telling Agents How — Dan Barrett, Raymond Weitekamp & Jose Montes de Oca, OpenProse