Within the pipeline, Wisedocs can flag conflicting diagnoses, deduplicate records, and search for missing records. When surfacing conflicting diagnoses to a human reviewer, the system points to the specific pages where the conflict occurs (e.g., 'on page two, on page 20, there's a difference of diagnosis between these two doctors. You should take a look').
Martone warns against using LLMs for every problem: they fail at determinism and calibrated probability estimates. With large amounts of data, traditional machine learning is the better answer, and problems need a mix of deterministic and probabilistic approaches.
endorsed advice
Give agents tokens that are both time-limited AND budget-limited (how much resource they can consume), so access ends when the token expires or the budget is exhausted. When an agent exhausts its budget, the supervisor decides whether to grant more budget — unlike traditional rate limiting which just cuts the agent off with no feedback. Provide continual feedback to agents about how much budget was consumed and how much remains, so the agent can operate within budget like a person treating a scarce resource carefully. When an agent's budget is close to being fully reached, withhold issuing another credential to stop the agent from using too much of a downstream resource.
None stated.
In the brains-and-hands architecture, keep agent credentials and access separate from the agent itself — no broad standing access for the agent. Put the agent's model (brain) in a sandbox containing only its tools, with an egress proxy outside the sandbox serving as the authorization boundary — this controls what the agent can access and prevents direct API access. The egress proxy boundary enables applying authorization to the agent system, translation between systems, and OAuth termination — while the sandboxed agent experiments freely inside. This sandboxed-brain-plus-deterministic-egress model is the predominant model that Anthropic, OpenAI, and many other labs are pursuing today.
None stated.
Ian says Keycard's broad view is to have broad-based deterministic policy that says what must never happen, or which things always require a human to be involved. There is a big gap between the policy a human or organization can write and what the policy should actually be for the specifics of a specific task, and the real challenge is layering over those two things. Related guidance: model both user-to-agent transactional intent and the business's intent for adopting the agent, then compute their intersection as the agent's bounding box of allowed actions; and derive access decisions from multiple intents from different areas of the organization that live independently but combine into a runtime authorization decision.
None stated.
Use faithfulness evals when the judge lacks fresh data: faithfulness asks whether the report makes sense given the data the agent was given. Run it with three inputs — the input (what the user asked), the output (the report), and the context (the search results and notes). Pull both the raw search results and the research notes in your getAgentRuns helper; a first version that only gave research notes caused the judge to flag real numbers as made up. Expect faithfulness to run slower than correctness because the judge reads every search result.
Query Phoenix for spans with parent ID null to get only top-level spans (the agent runs); the repo's getAgentRuns helper hands back each run's span ID, input, output, plus the research turns output. Use createEvaluator from the Phoenix evals package with an ordinary function, marking it as a code eval so no LLM is involved. Code eval functions receive input and output, check each requested ticker appears in the report, and return a label plus a score with the missing ticker named so the failure explains itself. Run them via npm run eval code — they execute almost instantly because code evals don't wait for an LLM.
Install the Phoenix plugin/skill for your coding agent, point it at your traces to look for weird things, have it read traces, come up with categories of failures, count them, and write each category back onto its trace in Phoenix as an annotation; let it write notes to disk if it asks.
Use all three kinds of evals at the same time — they are complementary, not competing approaches. Use code evals when the answer is deterministic: formats, lengths, required fields, banned phrases. Use an LLM judge when you need understanding: is it accurate, is it grounded, is it useful. Use humans for failures you haven't seen before and to check that your LLM judges are judging correctly.
Every eval follows the same pattern: get the runs, evaluate them, then log the results back to Phoenix via log span annotations, which attaches each result to its span. The correctness evaluator is one import and one line of setup; call evaluate to get back a label, score, and explanation, and log results the same way as the ticker check. Phoenix evals in TypeScript use the Vercel AI SDK to talk to models, so any model the AI SDK supports can be a judge.
When designing a harness, include on purpose those gates so you can still be accountable for what is being produced. Make sure any important judgment call or disambiguation — not the trivial ones — you can still get hold of. Without accountability gates, the only answer to 'what happens?' is 'the stochastic machine said it was fine.'
Back to the first step: measure. Know the token spend per department and per team so you can see which team is performing better, how much tokens they use, and what works.
A common approach: deploy an expensive model at small scale, generate your dataset from it, then reduce dramatically to a smaller model (e.g., 7B scale).
Split API keys by department so you can see where token spend is going, and track it down to the team and use-case level rather than just organization level. A single shared API key across the whole organization makes token spend unmeasurable. Featherless AI is open-sourcing a tokenomics tracking tool that splits API keys and tracks per-key spend.
The speaker advises business owners to create culture by modeling the behavior they want, rewarding the behavior they want, and punishing the behavior they don't want. Because reinforcement happens 24-7, 365 — including in teams the owner isn't involved in — the same behaviors must be cascaded down through leadership so leaders reinforce them further down. He frames owners as the chief reinforcement officers of their organizations.
Harish Gaggar describes a context optimizer framework that shapes and prunes queries before they reach the LLM: select the payload based on the token limit, re-rank for the most relevant information, dedupe to a unique curated set, and finally compress the large payload before sending it to the LLM.
Issue credentials that are inherently short-lived and issued per task, as opposed to living forever on someone's laptop. Long-lived tokens living forever on someone's laptop create a security risk for agentic AI. Short-lived scoped credentials could have stopped the destructive deletion.
Audit logging must be owned by you, not the agent, and kept outside the agent's reach — an agent that can alter its own logs can rewrite the narrative and fabricate results.
Agent sandboxing acts as a firewall for agents: restrict file system and network access so a compromised agent has no path to credentials and cannot exfiltrate without setting off alarms.
Agent sandboxing can be thought of as a firewall for agents: restrictions on the file system and the network. Execute agents in an isolated remote workspace without a route to production so that regardless of what the prompt says, the agent has no path to do damage. Without sandboxing, a misbehaving or compromised agent has a large blast radius, and agents running wild on people's actual computers and laptops is a dangerous execution environment.
Define a rubric for what a correct extraction looks like for your documents, then use an LLM as a judge against that rubric to evaluate extraction outputs.
Set up Sales Navigator reports to surface the people most likely to help you, then build a referral prospecting list from former colleagues, your boss and boss's boss, cross-functional leaders, veteran reps in your space, and your company's current customers, working with your CSM to get connected to those customers.
The creators behind most of what cleared in Money.
11494 cleared insights match
2032 cleared insights match
1148 cleared insights match
Choose sources → to get a library of your own and a daily brief.
