Creator perspective · MoneyTechnique 3: take an example of a contract that has been signed, remove a clause, and the agent is supposed to say 'Hey, there's something missing here.' The missing-clause technique is very hard because different suppliers have different templates and content, so knowing what should be in which contract requires thinking; these were mixed with existing contracts that should not show any flags. The missing-clause approach is the most powerful approach they had: it generates a lot of contracts, and it revealed the agents were missing most of the clauses that had been removed — a very strong benchmark. When an agent is missing most of the clauses that have been removed, that finding itself is a very strong benchmark result.
AI Engineer · How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Creator perspective · MoneyTechnique 5: turn an example into many using the concept of templates — it's the most powerful technique but very hard; Air Canada's experts have different kinds of templates with blanks that should be filled by the expert. Use expert templates with blanks as the basis for generating synthetic test contracts. Generate an array of fill values for each template blank instead of filling blanks manually, enabling generation of many contracts. Deliberately generate some tuples that break rules and some that do not, so you know exactly what should be flagged as a breach. Check template-generated contract evaluations with code, without involving any LLM. Reuse templates that experts already use in their normal workflow to generate test contracts, reducing friction. Expect roughly a four-week
AI Engineer · How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Creator perspective · MoneyTechnique 2: start from rules instead of creating an agent that provides flags — take a known rule (e.g., a cap above $1 million that every Air Canada contract should follow), create a clause that breaks the rule, generate the contract around it, and check the agent finds the flag back. The rule-based technique's problem is that you need to know the rules; coverage of everything cannot be guaranteed, so the team also came up with their own rules. Using rules from experts and documentation plus their own rules, the team generated 275 tests where they know the test can run and know the answer. When you know a specific rule, create a clause that breaks the rule, generate the contract around it, and test whether the agent finds the flag. When you have limited access to domain experts, use
AI Engineer · How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Report · MoneyThe biggest impact of AI on the speaker's content is especially with captions — their captions now are so different from before because the AI can tell them which hook types aren't working for interviews and suggest trying different styles, then report which performed well and which didn't. To make A/B testing more apples-to-apples: take a longer video, create several cut-downs, test different caption experiments across them, and use the API to compare engagement, since the base content is more similar.
Personal experience plus a recommended method.
AI Engineer · The AI Content System Behind 200M Views — Sarah Chieng, Cerebras & Cova Lee, OpenAI
Principle · MoneyAdit Abraham argues that if you have a harness where the agent is not only extracting information but also correcting itself and reviewing its outputs, you can achieve higher accuracy than human annotation — he calls this the first time agents can be better than humans at structured extraction (via Reducto's Deep Extract). Human-annotated ground truth often contains errors (missing content, dropped rows, lazy partial fills) that go undetected because there's no comparison point.
The a16z Show · The Missing Infrastructure for AI Agents: Documents | Reducto’s Adit Abraham
Principle · MoneyThe speaker recommends comparing a creative's organic views to your account's norm: if a post performs well relative to your norm, that is the creative to put ad money behind, because content cannot get views unless the creative is relevant. He claims this should save an extraordinary amount of money, noting many people waste money on social ads whose creative will not convert.
GaryVee · Everything You Need to Know About Social Media in October 2026
Principle · MoneyGaryVee advises leaders to empower people underneath them: instead of suppressing a direct report with bigger natural strengths in parts of the job, become an accelerator of them. He frames suppression as an anti-pattern and suggests auditing whether you are a suppressor or an accelerator.
GaryVee · The Leadership Mistake I Made for 20 Years
Principle · MoneyGaryVee describes his practice of 'kind candor' — putting the word kind in front of candor — which he says completely reactivated his organization. He observed his executives using radical candor as a weapon to suppress people or be jerks with their words, and advises not using candor as an excuse to be a jerk. He notes most companies need kind candor for the kind part, while his organization weirdly needed it for the candor part.
GaryVee · The Leadership Mistake I Made for 20 Years
Principle · MoneyGaryVee describes his operating model: he works for the 2,800 people his company employs — they don't work for him — and he asks how he can bring them value, believing that serving their value first gets him everything he wants.
GaryVee · The Leadership Mistake I Made for 20 Years
Principle · MoneyFor an intelligence producer, rent is one part of the cost. The operator still has to turn compute cost into work the customer can actually use. To compute full cost per useful output, count everything spent to deliver the service and divide by the output the customer can actually use. Operating capability affects both sides of that equation: what you spend and how much useful work you can deliver.
AI Engineer · Same GPU, 50% More Output: Why the Operator Matters — Oliver Kennon, SemiAnalysis
Principle · MoneyThe host explains that once you save a search, every new person fitting the criteria automatically appears in it, so you never have to recreate the search — you just respond to new prospects.
30 Minutes to President's Club | No-Nonsense Sales · #616: How I Use LinkedIn Sales Navigator to Find Warm Leads (Crash Course)
Principle · MoneyWhen evaluating an acquisition-prediction system against the live web, searching 'now' leaks future knowledge (the acquisition is already visible), producing a false 100% true positive rate. The fix: use Snapshot to set the snapshot date to just before the outcome was announced, so the only information informing the system doesn't leak the exact confirmation of that outcome. For example, when designing a system to understand company acquisitions, you might search one day, one week, or one month before the acquisition to check whether the search would have found the relevant signals.
AI Engineer · Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa
Principle · Money'Status comes from records, not model pros.' Show me the receipts — the actual tangible proof that it did what was expected. Don't just accept 'yeah, it passes, it looks good' from an agent — there's nothing to back that up; demand actual screenshots, videos, test results, and smoke tests.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Principle · MoneyIf your web app does not already have a bespoke CLI that allows any agent to fully flex the functionality you care about verifying, you'll have a much rougher time and an unpaved path using prompts alone. Give each agent the ability to say: yes, I actually built that feature, it is actually performant, it didn't cause any errors, and the application looks correct.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Principle · MoneyThe tired way — one giant massive PR from a single agent session running three or four days being told 'fix this, fix this, fix this, make it better' — is being replaced. What works essentially perfectly: decomposing tasks really cleanly, giving each task an agent, and in the prompt giving exactly what it needs to do to know it is actually done and to verify the state of done — those will just rip.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Principle · MoneyPutting a lot of verification instructions in a prompt is what a lot of skills and saved prompts do, but if you forget anything or things change, it leaves a lot up to interpretation. Instead, have a script that can prove the work out in different ways so you can run it over and over as a deterministic check. The script feeds its output back into the agent so it knows exactly what was wrong and how to fix it quickly, closing the loop. The agent can spin the script, run it again, check, verify, and repeat over and over until there are no more errors. LLMs are not deterministic — giving 30 agents the exact same prompt has potential for variance in outcomes — so pinning verification down to a single command backed by actual bash commands and bash calls is significantly more deterministic.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyThere should be a separate supervising agent that decides what the worker agent gets to do, creating a separation of concerns where the supervisor can't do any of the work itself. If a user tells an agent to go read the CEO's email, that is outside the scope of that agent and should be rejected as intent. A request to read somebody else's email would probably get caught in deterministic policies right away because it's not a vague thing and good controls exist for it. Telling an agent to forward the CEO's email to the outside world is vaguer and is the kind of thing that ideally comes back to a supervising agent, which would say that's probably a bad idea and no, you can't do that.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Creator perspective · MoneyThe creator describes that without intent — some capacity to relay the user's or business's intent — we can't get to agent autonomy. The creator recommends giving the agent the plan — what you're trying to do — as context, so that a mission identifier tracks through everything and authorization decisions are made in the context of that mission. A mission identifier also lets you audit and see all the things that happened and the decisions made in the context of one mission. Today there isn't a good concept of agent identity or the mission — things make decisions without knowing which agent instance is doing it or what mission it's involved in.
AI Engineer · An AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
Principle · MoneyIn the demo, Voss gates the build on acceptance criteria: mentions-ticker is a guardrail (the agent must talk about the right company) and actionability is a target (the average must stay 0.8 or better). He gates on the average score rather than a single report because an LLM judge can be a bit noisy on a single report.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Principle · MoneyLaurie Voss describes several biases in LLM judges. Position bias: given two options, depending on the model, they tend to prefer whichever comes first or last — so swap the order of options presented to a judge so it's not always looking at them in the same order. Length bias: judges prefer long answers. Confidence bias: they prefer answers that sound fluent and confident. Self-preference bias: a model judging its own output tends to rate it higher — so judge haiku with sonnet, and ideally judge haiku with something from OpenAI, to avoid self-preference bias.
AI Engineer · How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI