Recommendation · MoneySam recommends setting up an alert for when anyone who used to work at your company's customer (even a logo you didn't personally close) changes jobs and moves into a new territory — e.g., anyone who used to work at Acme (your customer) now moving to Beta (your target account).
30 Minutes to President's Club | No-Nonsense Sales · #618: LinkedIn Sales Navigator: How I Get Warm Intros Every Week
Recommendation · MoneySam recommends setting up an 'in-network job changers' report: first-degree connections you know in real life who change jobs and move into your territory — even if they're not a buyer, text them: 'Yo, you work at a company I'm trying to sell to. Can you help me out?'
30 Minutes to President's Club | No-Nonsense Sales · #618: LinkedIn Sales Navigator: How I Get Warm Intros Every Week
Recommendation · MoneyNick recommends setting up a Sales Navigator report to flag past users of your product who changed jobs into your target territory — e.g., anyone who came from Gibson Dunn (a customer) and moved to Jones Day (a target account).
30 Minutes to President's Club | No-Nonsense Sales · #618: LinkedIn Sales Navigator: How I Get Warm Intros Every Week
Recommendation · MoneyNick recommends setting up a series of LinkedIn Sales Navigator reports to identify the people most likely to help you get warm intros: former colleagues from your current company, your company's current customers, and prospects who know you or have a vested interest in you.
30 Minutes to President's Club | No-Nonsense Sales · #618: LinkedIn Sales Navigator: How I Get Warm Intros Every Week
Recommendation · MoneySam recommends setting up a Sales Navigator alert for when anybody above a certain altitude (seniority threshold) joins your company — new executives want to make an impact quickly and are primed to help. Reach out with: 'Hey, new head of product. I know you're in New York and I'm in California, so we haven't met in the office, duh. But I'm on the Western regional team and I'm actually wondering if you might be able to help me out. You're connected with seven people that I'm actually trying to get meetings with. And if you actually know them, I'm wondering if you might be able to help me out. Could I ghostwrite something for you? Could you make an intro?'
30 Minutes to President's Club | No-Nonsense Sales · #618: LinkedIn Sales Navigator: How I Get Warm Intros Every Week
Recommendation · MoneySam recommends setting up a LinkedIn Sales Navigator alert whenever an executive on your team connects with someone in your territory — a new connection means something happened (e.g., they met on a panel), and it opens a warm referral path that 99.9% of sellers will never know about.
30 Minutes to President's Club | No-Nonsense Sales · #618: LinkedIn Sales Navigator: How I Get Warm Intros Every Week
Recommendation · MoneyWhen asking a well-connected person for intros, name the specific targets: 'these eight or these 12 or these 20 people, I'm trying to get a meeting with them. I'm wondering who you actually know. If I ghostwrote you something or if I gave you some more context, would you be open to connecting with me?'
30 Minutes to President's Club | No-Nonsense Sales · #618: LinkedIn Sales Navigator: How I Get Warm Intros Every Week
Recommendation · MoneyDoneyli advises not auto-merging to main just because the eval gate passed; let the result sit for human review. He acknowledges you could set up a full end-to-end process that creates a PR automatically and waits for someone to review it if the eval clears the bar, and notes some teams push the PR straight into main if it's clearing the bar — but he discourages pushing a PR based on vibes or on what you think the agent is good at. Instead, keep a paper trail of eval evidence documenting why you made the decision for a new agent configuration.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyDoneyli recommends building an annotation queue for the domain expert to review agent outputs and provide input. Make the process very simple — experts have day jobs and cannot spend hours looking at traces. Give them a couple of simple scores to choose from (yes/no, true/false, or a 1–10 scale) and let them add a free-text comment, e.g., noting an SME failure mode like a missing citation. Annotations should take only seconds each — around 10 to 30 seconds per trace — so an hour of their time covers a meaningful batch. Experts can even dictate comments with a tool like Whisperflow instead of typing. Once completed, the human annotation lives with the trace as evidence.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyThe most important goal of traces is getting deep insights into what your agent is actually doing. A trace should show aggregate time and costs at the root plus a breakdown of each individual step the agent takes, including generations and tool calls. Use the trace as the foundation to track cost, latency, and tokens, and as an inspectable log for humans and coding agents. You always want to know what your agent is actually happening: what steps it is taking, what inputs and outputs it is seeing, and what tools are being called.
endorsed
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyAuto-improvement definitely needs the foundation of a ground truth data set. Build ground truth data sets through expert curation by working with domain experts within the company. Use shadow runs: run the process manually continuously while the agent runs in parallel so you can compare. Shadow runs let you measure agent accuracy while also building your ground truth data set. Teams in regulated industries build ground truth data sets that they can loop on and improve over time.
endorsed
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyDoneyli recommends incorporating experiments into your CI/CD pipeline so you don't have to be in the loop. When a change hits your agent, a CI job kicks off to run the experiment against the dataset and evaluate the agent with the new configuration, giving a pass-fail gate across all the scores and evaluators — a consistent way to evaluate agent outputs.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyDoneyli recommends that after extracting the most common categories from the domain expert's corpus, you define your vectors or dimensions of quality (scores). Example dimensions include source correctness, formal register (addressing customers with a more formal tone in some languages), language matching the customer's incoming language, and must-include / non-solicited items. Once you have your quality categories, decide how to grade them: what type of score and what type of evaluator each one gets.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyDoneyli recommends versioning your system prompts in Langfuse (free of charge) and treating a prompt version change as the trigger for kicking off the fully automated eval: configure an automation that triggers on any prompt changes in the agent config, with the action set to launch a CI job in GitHub. Keeping multiple versions of prompts also lets you track how the prompt evolved over time, and you can promote a previously tested version back into production when regressions are observed in the current version. All of this can be done via the API and CLI instead of clicking through the UI; tagging the promoted version with a production tag kicks off the eval pipeline.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyDoneyli recommends performing error analysis by looking at your traces to understand how the agent is behaving. When reviewing traces, check whether the agent is taking all the steps it needs to take, calling all the tools it needs to call, and giving the output that you want.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyUse LLM-as-a-judge, i.e., another LLM checking what the LLM did, as a scalable evaluation approach. Use code-based evals for classification tasks where you can check for an exact match. Use deterministic checks for behaviors like detecting all-caps messages.
endorsed
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyDefine scores for your use case that are automatically calculated whenever a new interaction or trace is captured by the system. Score whether the information given to the customer is compliant with banking information — e.g., if the customer asks about the weather, that is not compliant. Score for failure modes and for guardrails blocking answers given back to the customer. Use scores to filter traces for specific intents, e.g., filtering for traces where customer intent was investment advice.
endorsed
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyDoneyli describes two main ways to create golden datasets: have a team maintain a spreadsheet of things to test on the agent and upload it as a CSV, or add items that are already part of your production traces. He frames the latter as 'dogfooding' the dataset — building it from production traces as long as you've determined they are good examples of desired future outputs.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyDoneyli recommends evaluating agents against specific evaluators such as source recall, language match, correctness, and non-unsolicited upsell, and defining minimum score thresholds for each evaluator: if a score doesn't clear the bar, it's not considered a pass to go into production.
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
Recommendation · MoneyEnable PII identification so any data deemed PII is automatically masked before it gets persisted into Langfuse. Financial institutions do not want to persist credit card information or social security numbers in an observability system; they want that data kept in systems of record they own. You cannot control what your users input into the system, so you must catch sensitive data (like credit card numbers in clear text) when instrumenting and building your evals. Use a masking function that immediately masks any defined PII so it is not persisted in plain text in your system.
endorsed
AI Engineer · Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse