Recommendation · MoneyData set items should contain the input plus the expected output, which can be a sample reply, expected behavior, or keywords to be covered. Include out-of-scope requests in your data set that you want the agent to correctly defer, such as refusing to give investment advice or answer weather questions. Use data sets as a stable benchmark, either for offline iteration or as part of your CI/CD.
endorsed
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyYou typically end up with different datasets to measure specific things or specific areas of quality in your agent fleet. Maintain a red-teaming dataset that looks for prompt injection attacks and any attack that would allow someone to circumvent guardrails and obtain information they shouldn't get. Maintain a dataset specific to a use case, e.g., knowing which cases have been disputed and whether the dispute has not been solved. Curate datasets that are specific not only to the use case but also to the technical requirements you have for that agent.
endorsed
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyBuild evaluation criteria anchored on datasets: a set of questions and expected outputs that allow you to measure the quality of outputs of your LLM or AI solution. Think of datasets as your test cases if you come from a software development background. Do not ship software into production without passing it through QA — merging straight into main or prod is not a best practice. Datasets give you the evidence that you passed the agent through a series of steps or requirements that measure all the things you want to capture. The expected output in a dataset does not necessarily need to be the answer the agent will give; you can measure the intent of the agent, what the agent is supposed to do, or behavior.
endorsed
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyDoneyli recommends not asking the SME to invent categories — instead ask for unfiltered raw commentary on the traces. Then use an LLM to find all failure modes or categories that are patterns across the expert's comments, because LLMs are great at pattern matching and extracting information from text. This keeps it simple for the domain expert and uses LLMs for what they are great at.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyDoneyli recommends measuring agent outputs across both LLM-as-a-judge scores/evaluators and deterministic evaluators, and building a library of evaluators so incoming traces automatically get their scores calculated — or alternatively having scores calculated directly from the client application in real time and surfaced back to the system. The more balanced the mix, depending on the categories you identify, the better: if you rely 100% on LLM-as-a-judge, there's a non-zero chance some outputs will be inaccurate because that's the nature of LLMs. Deterministic checks always produce exactly the same output for the same inputs — a code evaluator gives the same output 100% of the time for the same input, not 95% or 80% — enabling a blended assessment of agent quality.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyIn regulated industries, make sure you have a ground truth data set that is validated by your customers. Run human-in-the-loop systems when shipping agents in regulated industries. Run a fully human-in-the-loop system first, then over time slice the data into parts you are confident to automate versus parts where you always want a human in the loop.
anecdotal
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyFor each node in a DAG, ask how valuable it would be to have an agent drive the node versus a human drive it. The final decision node in a lending flow should probably be human driven, since it is essentially the check against the policy. A node may be better suited for automation if you can check it against policy easily, run it in a bounded loop, trace it, and fall back to a human.
AI Engineer · How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase
Recommendation · MoneyKormos recommends building representative data sets and evaluating them, defining what good looks like. Run evaluations in repeated series as a statistical analysis, looking at outputs in volume rather than as direct comparisons. Combine evals with traces — the runtime logs of what happens — as a confidence mechanism complementing traditional software testing.
AI Engineer · How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase
Method · MoneyUnder the 30 connection rule, if the target is posting on LinkedIn you warm them up by engaging and commenting before pitching; if they're not posting, you hit them right with the pitch. For prospects with no LinkedIn activity or signals, Kade goes generic and sends a more standard message but still gives ideas and signals — though most of those people don't respond.
Method, not a verified protocol.
30 Minutes to President's Club | No-Nonsense Sales · #615: How to Master Social Selling on LinkedIn to Book Meetings | Kade Hinkle | 30MPC Hall of Fame
Method · MoneyPrathik Naidu lays out a sequence for AI deployment in an acquired business: build your tech and capture the workflows, commit a lot of team time into making it work, partner very closely with the company's team and iterate to validate your product, potentially train new models, examine all the unique workflows and data within the business, and then put the unlocked capacity to work to generate real business impact.
Method, not a verified protocol.
AI Engineer · Why Buying Companies Beats Selling Them AI — Prathik Naidu, Long Lake
Recommendation · MoneyCentralize verification authority in a web service when running multiple agents. Have agents report completed work through an MCP connection and ask a verifier 'Am I done? Please verify it'. Verify a PR was created correctly by going to the GitHub API and checking that the PR exists, is healthy, and all merge checks pass. When verification fails, redirect the agent with a list of the specific things that are still failing.
endorsed
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyKeep receipts from every run — this value cannot be overemphasized. For an e-commerce app, save receipts of the entire user journey: login, page hits, errors seen, catalog load time, cart actions, and cart abandonment. Every run receipt can be fed to LLMs, or you can set up a system running all night long that aggregates the latest receipts. An overnight aggregation system can identify patterns like eight failures in the caching module and issue a PR that night to fix it properly.
endorsed
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyIf you have a web app and don't already have a custom verifier CLI, building one is one of the highest leverage things you can start with for your codebase. Also build a foundational eval that shows what good looks like in your application.
endorsed
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Anecdote · MoneyThe speakers describe that a fixed app can still lie: after reloading, the page could say 'waitlisted' while the screenshot says 'you're in, your seat is confirmed' — a mismatch that is a failure. Confirmation text that breaks the contract (not exactly one-to-one with what the model saw) causes the judge to fail even when the feature worked. They recommend avoiding tightly coupling verification to the exact text on the screen and proving things worked without that tight coupling, noting the answer is system-one models.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyIf your web app lacks a bespoke CLI that lets any agent fully flex the functionality you care about verifying, you'll have a much rougher time using prompts alone. Build a bespoke CLI (like their 'Snack' tool, run with pnpm snack) that lets any agent verify the functionality you care about.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyBuild a script that proves the work deterministically across a number of checks and feeds output back to the agent. Pin verification to a single command backed by actual bash commands because LLMs are non-deterministic and 30 agents with the same prompt produce variance. The agent can run the script repeatedly — run, check, verify, run again — until there are no more errors.
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyRequire actual screenshots, videos, test results, and smoke tests as proof that an agent did the work. Accepting an agent's verbal claim that 'it passes, it looks good' without backing evidence leaves you with nothing to verify the work.
endorsed
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyGive your agents the ability to see all aspects of performance — networking, traffic, rendering, etc. If you're shipping a web app, give your agent Chrome DevTools. If you're shipping native clients (iOS or OSX apps), attach alternative observability projects, with more coming out every day. The common foundation across all platforms is that you need to give your agent the ability to see absolutely everything you would normally see as a developer. The more eyes you give your agent, the more it will be able to take over from you.
endorsed
AI Engineer · Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Recommendation · MoneyDenys recommends having technical implementations and detection systems to safeguard your own data cleanliness. Unless you have good systems to detect duplicate records, you just keep getting the same bad quality data. Once you can detect duplications, you no longer incentivize things like sending claims multiple times with different dates or the same claim. As an organization, you have to have lines of defense in order to make sure that missing records get flagged.
AI Engineer · Insurance Claims Are AI's Messiest Data Problem — Denys Linkov, Wisedocs
Recommendation · MoneyWith claims data extracted, insurers can analyze what affects loss ratio, claim types, geographies, injury types, and doctors involved; map which lawyers, doctors, chiropractors, and physiotherapists work together; analyze which medical conditions co-occur (mapped to ICD codes) and compare write-ups from different doctors; analyze patient travel within provider networks; estimate treatment cost changes over time (e.g., a 10% year-over-year increase for a treatment type for a given doctor); and do counterfactual analysis on network changes, billing changes, and standard-of-care procedures.
Consequential creator advice requires independent verification before Today can promote it.
AI Engineer · Insurance Claims Are AI's Messiest Data Problem — Denys Linkov, Wisedocs