Most real applications use a collection of code evals, LLM-as-a-judge, and human evaluation together. Use code evals when the answer is deterministic (format validation, length limits, forbidden phrases, required fields), LLM judges when you need semantic understanding, and keep humans in the loop for unseen failure modes and to verify judges are judging correctly.
Before writing any eval, read your traces and categorize what went wrong. Figure out what to measure first. Categorize eval explanations into types of failures and count up the categories — this is called coding. If you see the same explanation across 50 different traces, you have a systematic problem, not a one-off edge case, which makes it the first thing you should fix.
The speaker advised reading a dozen or more traces end-to-end — starting with the ones where something went wrong — before writing a single evaluator, asking what the input was, what the output was, and what specifically is broken.
Use a code evaluator when looking for a specific string in output (no LLM needed to check whether the ticker was mentioned). Use code evals for production-critical checks like output-is-JSON, length limits, and avoiding forbidden phrases such as 'as an AI language model'.
A demonstrated workflow: connect the MCP server to your agent and create a skill that tells it when to store memories; store the conversation in a named data pack and record incident facts as searchable memories; kill the session and resume it later from a brand-new chat window without being inside a project or enabling a special memory mode; ask the same question to verify the agent pulls prior context back out of memory. When resuming, the agent searches memory first instead of creating a new conversation, and the MCP server instructs it to create a new conversation only for storing new content.
endorsed
Harald Kirschner recommends getting your codebase as agent-ready — a no-brainer that everybody should already be doing. Create agents.md files that are lightweight enough and give agents a good map of the codebase — where to look at, mapping that out. These are living documents that should evolve as the agent makes mistakes and works. Teams that invested in developer experience gain a lot because good documentation and onboarding for a codebase is read by the agent as well and helps the agent as much.
Harald Kirschner recommends creating skills that encode expert knowledge — like VS Code infusing all accessibility best practices into a skill everybody uses. Previously accessibility required one person to get pulled in or give feedback; now that skill is reviewed and maintained by the area owner of accessibility to bring that expertise into everybody's hands. Think how you can encode your expert knowledge to unlock the few users who might always be asked for this input into your skill.
Harald Kirschner recommends that when shipping an agentic product, it's important to have your own evals of the product always at hand and easy to extend. VSC bench is VS Code's own eval harness for their agentic product. When a GitHub issue is filed to add a new eval scenario, an agent picks it up from a template and bootstraps the scenario for you. Bring developer scenarios from issues and customer conversations back into your evals so you can start hill climbing against them. Confirm improvements offline first, then run online experimentation — a classic product lifecycle.
When the authentication workflow is stable and reusable across business operations on the same portal, pull that logic out from the responsibility of the model and reuse the same authentication mechanism for many business operations. Leaving authentication in the model's responsibility forces it to re-derive a never-changing workflow on every run, adding unnecessary steps and preventing reuse.
: none
Meegan recommends giving the agent an opportunity to verify its work in real time via an OCR tool that deterministically extracts entities from the document and compares them against the system of record — a deterministic mechanism so the agent knows authoritatively whether it did its job right. Deploying without such a mechanism leaves the agent unable to know whether its trajectory succeeded or failed.
: none
When a task involves programmatically interacting with the page — e.g., downloading a file and retrieving it from storage into the agent's runtime — encapsulate the operation into a single tool call that performs download and retrieval in one transaction, instead of the model making decisions on the fly. Leaving such decisions to the model increases steps, raising unit costs and reducing performance and maintainability.
: none
Meegan recommends drafting a skill, structured like a standard operating procedure for humans but for an agent, that outlines the critical path. Introducing the critical path makes the agent more likely to take the correct next step (e.g., knowing it's at step two and that step three is next) because ambiguity is removed from its decision process. Running without a skill leaves ambiguity, making the correct next step less likely.
: none
Meegan recommends measuring agent correctness on a per-transaction basis rather than per-run, permitting retries (e.g., up to four). A hypothetical agent with a 50% per-run success rate and up to four retries climbs to a 94% per-transaction success rate. Ground the metric in the customer: customers don't care how many retries you perform, only that the workflow completed successfully and reliably.
: retry count is an example; 94% figure is a hypothetical calculation
Invest in clear documentation, modular code, reliable CI, and non-flaky tests. What is good for humans is also good for AI agents. Assess platform readiness to improve token efficiency and context provision.
Stop using Salesforce as your primary database for pipeline intelligence; push all data into a data warehouse like Snowflake with a semantic layer so it is agent-accessible. Clean your CRM data before layering on AI and put governance around accounts, contacts, and opportunities so AI systems analyze reality instead of CRM slop.
Your best outbound reps might be violating your activity metrics and benchmarks, and they might be right. Inspect their unique process rather than forcing them to increase activity. If they have discovered a better playbook, scale it to the whole team.
Use AI to automatically pull discovery qualification evidence from call transcripts and surface deal gaps without requiring AEs to enter data manually. Have human leaders run live, human-led forecasts with their own narrative and point of view, then use an AI agent to analyze those forecasts and highlight weak spots before the leader conversation.
Angel Steger emphasizes the importance of understanding the 'want behind the want' and establishing the correct mental model before building. She advises solving problems in order of importance: first validate value, then address functionality, workflow, or emotional choreography, and only then focus on details like aesthetics.
Angel Steger discusses how design can manage users' emotional states, especially in high-stakes domains like finance and health. She advocates for making emotionally charged information intentionally boring and slow, using default views that show the bigger picture, and providing educational content to help users maintain a long-term perspective and avoid panic-driven decisions.
Angel Steger advises against jumping to high-fidelity prototypes too quickly, as it makes designs predictable and lacking in vitality. Instead, she recommends starting with low-fidelity methods (e.g., storyboards, sketches) to focus on the fundamental experience and value before details.
Define a rubric for what a correct extraction looks like for your documents, then use an LLM as a judge against that rubric to evaluate extraction outputs.
Set up Sales Navigator reports to surface the people most likely to help you, then build a referral prospecting list from former colleagues, your boss and boss's boss, cross-functional leaders, veteran reps in your space, and your company's current customers, working with your CSM to get connected to those customers.
The creators behind most of what cleared in Money.
11494 cleared insights match
2032 cleared insights match
1148 cleared insights match
Choose sources → to get a library of your own and a daily brief.
