37,617 results · Most relevant
Use LLM-as-a-judge, i.e., another LLM checking what the LLM did, as a scalable evaluation approach. Use code-based evals for classification tasks where you can check for an exact match. Use deterministic checks for behaviors like detecting all-caps messages.
endorsed
Define scores for your use case that are automatically calculated whenever a new interaction or trace is captured by the system. Score whether the information given to the customer is compliant with banking information — e.g., if the customer asks about the weather, that is not compliant. Score for failure modes and for guardrails blocking answers given back to the customer. Use scores to filter traces for specific intents, e.g., filtering for traces where customer intent was investment advice.
endorsed
Doneyli describes two main ways to create golden datasets: have a team maintain a spreadsheet of things to test on the agent and upload it as a CSV, or add items that are already part of your production traces. He frames the latter as 'dogfooding' the dataset — building it from production traces as long as you've determined they are good examples of desired future outputs.
Doneyli recommends evaluating agents against specific evaluators such as source recall, language match, correctness, and non-unsolicited upsell, and defining minimum score thresholds for each evaluator: if a score doesn't clear the bar, it's not considered a pass to go into production.
Enable PII identification so any data deemed PII is automatically masked before it gets persisted into Langfuse. Financial institutions do not want to persist credit card information or social security numbers in an observability system; they want that data kept in systems of record they own. You cannot control what your users input into the system, so you must catch sensitive data (like credit card numbers in clear text) when instrumenting and building your evals. Use a masking function that immediately masks any defined PII so it is not persisted in plain text in your system.
endorsed
Data set items should contain the input plus the expected output, which can be a sample reply, expected behavior, or keywords to be covered. Include out-of-scope requests in your data set that you want the agent to correctly defer, such as refusing to give investment advice or answer weather questions. Use data sets as a stable benchmark, either for offline iteration or as part of your CI/CD.
endorsed
You typically end up with different datasets to measure specific things or specific areas of quality in your agent fleet. Maintain a red-teaming dataset that looks for prompt injection attacks and any attack that would allow someone to circumvent guardrails and obtain information they shouldn't get. Maintain a dataset specific to a use case, e.g., knowing which cases have been disputed and whether the dispute has not been solved. Curate datasets that are specific not only to the use case but also to the technical requirements you have for that agent.
endorsed
Build evaluation criteria anchored on datasets: a set of questions and expected outputs that allow you to measure the quality of outputs of your LLM or AI solution. Think of datasets as your test cases if you come from a software development background. Do not ship software into production without passing it through QA — merging straight into main or prod is not a best practice. Datasets give you the evidence that you passed the agent through a series of steps or requirements that measure all the things you want to capture. The expected output in a dataset does not necessarily need to be the answer the agent will give; you can measure the intent of the agent, what the agent is supposed to do, or behavior.
endorsed
Doneyli recommends not asking the SME to invent categories — instead ask for unfiltered raw commentary on the traces. Then use an LLM to find all failure modes or categories that are patterns across the expert's comments, because LLMs are great at pattern matching and extracting information from text. This keeps it simple for the domain expert and uses LLMs for what they are great at.
Doneyli recommends measuring agent outputs across both LLM-as-a-judge scores/evaluators and deterministic evaluators, and building a library of evaluators so incoming traces automatically get their scores calculated — or alternatively having scores calculated directly from the client application in real time and surfaced back to the system. The more balanced the mix, depending on the categories you identify, the better: if you rely 100% on LLM-as-a-judge, there's a non-zero chance some outputs will be inaccurate because that's the nature of LLMs. Deterministic checks always produce exactly the same output for the same inputs — a code evaluator gives the same output 100% of the time for the same input, not 95% or 80% — enabling a blended assessment of agent quality.
In regulated industries, make sure you have a ground truth data set that is validated by your customers. Run human-in-the-loop systems when shipping agents in regulated industries. Run a fully human-in-the-loop system first, then over time slice the data into parts you are confident to automate versus parts where you always want a human in the loop.
anecdotal
For each node in a DAG, ask how valuable it would be to have an agent drive the node versus a human drive it. The final decision node in a lending flow should probably be human driven, since it is essentially the check against the policy. A node may be better suited for automation if you can check it against policy easily, run it in a bounded loop, trace it, and fall back to a human.
Kormos recommends building representative data sets and evaluating them, defining what good looks like. Run evaluations in repeated series as a statistical analysis, looking at outputs in volume rather than as direct comparisons. Combine evals with traces — the runtime logs of what happens — as a confidence mechanism complementing traditional software testing.
Under the 30 connection rule, if the target is posting on LinkedIn you warm them up by engaging and commenting before pitching; if they're not posting, you hit them right with the pitch. For prospects with no LinkedIn activity or signals, Kade goes generic and sends a more standard message but still gives ideas and signals — though most of those people don't respond.
Prathik Naidu lays out a sequence for AI deployment in an acquired business: build your tech and capture the workflows, commit a lot of team time into making it work, partner very closely with the company's team and iterate to validate your product, potentially train new models, examine all the unique workflows and data within the business, and then put the unlocked capacity to work to generate real business impact.
Centralize verification authority in a web service when running multiple agents. Have agents report completed work through an MCP connection and ask a verifier 'Am I done? Please verify it'. Verify a PR was created correctly by going to the GitHub API and checking that the PR exists, is healthy, and all merge checks pass. When verification fails, redirect the agent with a list of the specific things that are still failing.
endorsed
Keep receipts from every run — this value cannot be overemphasized. For an e-commerce app, save receipts of the entire user journey: login, page hits, errors seen, catalog load time, cart actions, and cart abandonment. Every run receipt can be fed to LLMs, or you can set up a system running all night long that aggregates the latest receipts. An overnight aggregation system can identify patterns like eight failures in the caching module and issue a PR that night to fix it properly.
endorsed
If you have a web app and don't already have a custom verifier CLI, building one is one of the highest leverage things you can start with for your codebase. Also build a foundational eval that shows what good looks like in your application.
endorsed
The speakers describe that a fixed app can still lie: after reloading, the page could say 'waitlisted' while the screenshot says 'you're in, your seat is confirmed' — a mismatch that is a failure. Confirmation text that breaks the contract (not exactly one-to-one with what the model saw) causes the judge to fail even when the feature worked. They recommend avoiding tightly coupling verification to the exact text on the screen and proving things worked without that tight coupling, noting the answer is system-one models.
If your web app lacks a bespoke CLI that lets any agent fully flex the functionality you care about verifying, you'll have a much rougher time using prompts alone. Build a bespoke CLI (like their 'Snack' tool, run with pnpm snack) that lets any agent verify the functionality you care about.
Choose sources → to get a library of your own and a daily brief.