Run the judge against the humans as an experiment: the experiment takes a dataset, a task to run on every example (the judge), and evaluators to score the results (checking whether the judge's label matches the human's). Only run the meta-evaluation on the dev split, not the test split. Run the meta-evaluation yourself with the command 'npm run meta'. The rubric file 'evals actionability.ts' takes three or four minutes to run.
A multi-part method for constructing a rubric: mark data clearly with XML tags, keep boundaries between input/output/instructions, mark examples as examples, show examples to clarify criteria, test with and without examples, and constrain the judge's answer to a fixed set of options.
The eval workflow being built: trace your agent, read what it did, write evals for what matters, check the evals can be trusted, change the agent, prove the change worked, then lock it in with CI. Lock in agent fixes by putting the change into CI so that it stays fixed.
Build the golden dataset in Phoenix by finding every report with a human label, making each an example where the input is the report and the expected output is the human's label, sorted by span ID. Use a 75/25 split: 75% of labels in the tuning set and 25% in the test set. Sort examples by span ID before splitting so a report always lands in the same split however many times you run it, and span ID links each example back to its original trace. Put every fourth example into the test split and the rest into dev.
Measure your judge by comparing its predictions against the truth of what a human expert would say. Create human annotations in Phoenix by opening a trace, clicking the pencil button, and adding an annotation (e.g., 'human actionable' set to actionable or not actionable) for each trace. Involve a domain expert by having them read trace outputs and produce a golden dataset of correct labels (e.g., 10, 20, or 50 traces).
The eval improvement loop: trace your agent, read what it did, write evals for what matters, check the evals can be trusted, change the agent, prove the change worked, then lock it in with CI.
Guideline questions to start asking: who/team spends what? What must you test to use a cheaper model? What's your base load? What's the default model for each team? Which workload has a person waiting that needs something faster? And how much of what we generate is useful?
They split the answer into parts: the agent streams markdown prose token by token with the frontend styling it live, and the source card — a real UI component that holds nothing you'd click before reading the answer — is appended as one piece after the full answer arrives.
Instead of fixed retrieve-then-generate, the agent has four tools ordered narrow to wide: search rulebooks (the 10 hybrid-retrieval chunks), retrieve rulebook sections (whole section by section path), retrieve full rulebook (small rulebooks only), and list available rulebooks.
Search rulebooks always comes first — that is the 10-chunk hybrid search. If search is not enough, the retrieve rulebook sections tool pulls the whole section in order by section path, fetching every chunk with that path sorted by chunk index. The retrieve full rulebook tool works for small rulebooks only, and list available rulebooks is for when the agent needs to know what exists at all. Guardrails: if a rulebook is too big (over roughly 100K tokens), the full rulebook tool refuses to dump its content into the context and hands back the available section list with a warning; if a section name doesn't match, the tool hands back the available sections so the agent can retry rather than give up or invent the name.
The indexing pipeline starts with PDFs, pulls text from every page, splits it into paragraphs, and packs paragraphs into chunks of about 2,000 characters, never splitting a table over two chunks. Section headers are detected with regular expressions (e.g. 'chapter 3', '4.2.1'), giving every chunk a section path used for citations and for fetching whole sections. Chunks are not overlapped, so a section is just an ordered collection of chunks that can be reassembled into the original document. Metadata added per chunk includes rulebook name, page, and chunk index (letting the agent reassemble sections or whole rulebooks), plus business logic fields like rulebook type, applicable roles, and whether it contains a table. Two types of embeddings are generated: dense and BM25. The indexing yields
A coherent multi-step sales approach: stay out of the price-matching game and play the value-surpassing game; itemize all elements of the service; use a price-match challenge script; frame the competitor's lower price as fair because they don't do the value-added things; use a conditional walk-away script citing what cheaper movers don't do; cite the 25% breakage statistic and frame the odds; anchor the premium with a worth-it question; and close with 'this is what it sounds like you really want, and that's what I can deliver.'
The host walks through building a dynamic pricing sheet (coded with AI or an if-then Excel sheet) that calculates price automatically during a customer conversation: separate variable costs that scale with headcount from fixed costs like payroll, allocate fixed payroll per event based on capacity, and never show the internal cost number to the customer. He notes such a module can be built in a couple of hours.
Kevin's team built an MCP server in a day or two to expose access to their cloud environment and platform so coding agents could interact natively, because frontier models know the MCP protocol well. They refined the exposed tools and smoke tested runs before scaling up.
Put monitoring and auditing on agents — most people don't do this. Configure agentic tools to pull all agent activity into a log file with verbose logging (tools do a lot in the background that doesn't show in the UI), then run a background tool that greps the log and flags malicious behavior you may miss in the interface, such as checking secret storage. Background agent tasks are especially prone to unreviewed malicious behavior because the user isn't watching.
A practical method for AI threat modeling: use an existing framework (STRIDE, Maestro, Attack Trees, DREAD), cover each STRIDE category when designing, produce deliverables (architecture and data flow diagrams, threat catalogs, mitigation plans, security requirements), and redefine scope for AI to include LLM models, prompt templates and system prompts, context windows, memory stores, and accessible endpoints/APIs.
Put monitoring and audit on AI agents, because agents left unmonitored may do things you don't want. Configure agentic tool settings to pull all agent activity into a log file (verbose logging for every agentic tool you use), then run a background tool that scans the log file and monitors for unwanted behavior like checking secret storage.
Integrate AI threat modeling with whatever DevSecOps program your company has. Threat models need to evolve alongside application deployments: for every change pushed (pull requests, code changes, AI application changes), run scans and threat-modeling checks. Set up triggers such as webhooks so changes automatically trigger a scan; scan when infrastructure-as-code changes (e.g., monitor a Terraform file an agent creates so a scan runs the moment it changes); data pipeline updates and AI model deployments should also trigger scans.
A doer agent and a teacher agent iterate together on a browser task: the doer attempts the task, the teacher reviews screenshots and logs and writes corrections into strategy.md, which carries over to the next iteration. The loop runs until it converges, without prescribed iteration counts, success-rate targets, or token/time limits ('embrace the bitter lesson'). The converged output is a skill.md file for future agents.
A coherent train-to-deploy workflow on Fireworks AI: prepare separate train and eval datasets, launch training via the training agent or managed training (API/UI), monitor progress through the UI dashboard, and swap new checkpoint weights into the existing endpoint when ready.
Define a rubric for what a correct extraction looks like for your documents, then use an LLM as a judge against that rubric to evaluate extraction outputs.
Set up Sales Navigator reports to surface the people most likely to help you, then build a referral prospecting list from former colleagues, your boss and boss's boss, cross-functional leaders, veteran reps in your space, and your company's current customers, working with your CSM to get connected to those customers.
The creators behind most of what cleared in Money.
11494 cleared insights match
2032 cleared insights match
1148 cleared insights match
Choose sources → to get a library of your own and a daily brief.
