37,665 results · Most relevant
Put the LLM in judge mode by telling it it is an evaluator and what it's evaluating. Give it the query and the response as inputs; for groundedness, also give it the context. Force a binary pass/fail decision, have it also give a reason, and ask for JSON output so results are parseable.
Pamela Fox walks through a repeatable workflow: check whether the model calls the right tool and passes expected arguments; treat wrong arguments as a signal of ambiguity in the description; clarify the description with explicit constraints (e.g., 'don't include me or you, just named attendees', 'first and last if provided'); then re-evaluate across all inputs.
Pamela Fox recommends using DSPy to avoid manually tweaking prompts per model: declare the program's constraints (e.g., take user input and turn it into a structured calendar event) and DSPy figures out the prompt that adheres to them. She also advises setting up checks/tests to verify whether output satisfies each constraint so DSPy can iterate, and providing the student model and reflection model so it can run the optimizer.
The speakers demonstrate a proof command using Playwright to verify end-to-end behavior: taking screenshots to verify UI state, checking the inner text of a status element to confirm the email address appears, checking storage state after an action to verify data was actually persisted (e.g., an email saved to the waitlist), and using reload and snapshot together to confirm state survives a page reload. They also note checking rendering at different widths and in dark/light mode to catch rendering bugs that affect real users.
Laurie Voss recommends giving the faithfulness judge the full context — raw search results plus the agent's research notes — not just the notes. If you only give it the research notes, it will flag numbers as made up because it doesn't know the web search actually found that data. The getAgentRuns helper pulls both.
Laurie Voss recommends pointing a coding agent at traces to look for weird things, claiming coding agents are aces at reading traces, while keeping manual inspection as a sampling technique since agents won't always find what you expected.
The simplest useful code eval checks whether the report mentions the ticker symbol of the company asked about — a binary check where using an LLM would be overkill. It found real errors (Tesla and Amazon reports missing their tickers; the Amazon one wrote only about AWS), and it costs nothing to run.
Open a branch on a tagged challenge: a separate agent tasked with reviewing the plan, reviewing the assumptions, and doing healthy challenge.
endorsed advice
Try the simplest method first, then try something a little more complicated, and check whether the performance lift justifies the additional complexity. If it does not, stay with the more interpretable method. The guest notes there is a lot of temptation to go straight to neural networks and transformers, but that is often not the right way to go.
We need something in the very near term before an NTSB for AI could be set up — could be a not-for-profit. We need a non-punitive reporting system for AI safety, best practices plus postmortem analysis, and something like US GAAP for AI: generally accepted AI deployment principles with auditors (like KPMG or Ernst & Young) you can choose from. We should also take validation of AI seriously from a number of different perspectives.
Applying the Swiss cheese model to large language models: we control the training data, the pre-train, the RLHF, the RSI, all the data being sampled, and we can take out the bad data. Percy is reportedly running a small model to look over the pre-train data to remove bad stuff so the model can't possibly learn those bad things. After cleaning training data, the next layers are: do a failure analysis on the data, estimate failures, sample some failures, estimate failure modes, get to a p fail (or one minus p fail, which is p success), then tools, and all the way to runtime monitoring. Clark adds that you need all these layers — the opening-remarks example had the chain of thought monitor turned off, which would be one of those layers.
Have the best agent do planning, a faster cheaper agent do implementation, and the best agent for review again. Faster cheaper models like JRM 5.3 Flash can do implementation tasks at the same quality level as stronger agents when a stronger agent does the planning.
Add an extra step between observing and reasoning — something like 'recall long-term memory' — that lets the agent recall memories and makes it introspective before solving the problem. The recall step goes to the vector store (or wherever the long-term memory episode from a previous conversation is recorded) and pulls from it if the agent considers it interesting. This acts like an additional abstraction between the observing and the reasoning parts of the loop.
creator's recommendation for the loop design
The host advises that if you're a vendor, you need someone on your marketing team monitoring agent recommendations weekly to make sure you stay number one, because the agents are going to change. A non-expert would just use the top agent recommendation instantly without qualifying or testing alternatives.
Enterprises should pick Claudeforce out of the box precisely because it's not connected to other apps — it provides guardrails so sellers don't automatically write everything back to Salesforce, whereas Replit/Lovable create data instantaneously and are harder to guardrail.
None; presented as direct advice.
When choosing between the platforms: if cost is important to you, maybe just use Lovable. If it's not as important, then Replit might win — especially if you need to connect the app to many different external apps, as the speakers have done with 10K connected to 20 different apps over the last year. The deciding question is how much read/write security you need — how secure you want people's read and write access back to the source system.
Hedged with 'maybe' on the cost recommendation.
If you just want to use Salesforce primarily (or HubSpot or similar), use Claudeforce — it's easier to set up, has no token expiration issues, and is out-of-the-box as good as Replit or Lovable for that single-app case. But skip Claudeforce and use Replit or Lovable as soon as you need to connect apps beyond the natively integrated one (e.g., QuickBooks, Bill, AgentForce), since Claudeforce can't incorporate external integrations. For small SMB/startup teams (roughly 3 people doing millions in revenue), Claudeforce may be too limiting.
None; presented as direct advice.
Technique guidance for the hack lift: drive the knee forward to load the quads and avoid hitting the calves, stick the chest out and knees out when setting up to get tension into the quads, and go deeper with a wave-forward motion. Driving the knee forward also means no back issues since the bar is behind you.
A coherent technique sequence for the clean and press: set a foot-width clean stance, place the crook of the elbow just outside the knee, drive the hips, reset, dip, and drive as fast as possible overhead, widening the feet slightly, keeping an open-palm dip grip with mobile lats, and bending the bar apart in the overhead position.
A locally coherent setup-and-execution method for the Jefferson deadlift: position the lead leg first with about a 90-degree angle on the other side, place the bar slightly closer to the front of the lead leg, use a grip similar to a conventional deadlift, keep foot width near your conventional deadlift stance with 90-degree foot position, put a little more load on the lead leg, set the back in a slightly rounded position, and pull as quickly as possible. Do it on both sides.
Choose sources → to get a library of your own and a daily brief.