36,594 results · Most relevant
When a user or customer reports a failure (e.g. via thumbs down), Ramakrishnan Lokanathan prescribes the classic QA flow: reproduce the issue, understand where the model is failing (including whether it is a tool call), identify what part of the eval suite failed, review it, and add it to the eval pipeline.
The speaker lays out an ordered method for turning failures into evals: every failure should have a proper workflow for it to enter your system; keep observing, using the multiple observability tools that give you automatic data; give users the option to report failures with a simple thumbs up or thumbs down and collect as much data as possible where customers tell you where there is a failure; then do classic QA on software — reproduce the issue and try to understand where the model is failing, whether it is a tool call, and what part of your eval suite failed; review the failure and add it to the eval pipeline. Keeping this feedback loop going is extremely important so you access new frontiers of things you are missing at every point in time.
Sakai lays out an ordered method for working with AI. First, sense the world with your body — go to the actual place, talk with people, and try to experience it yourself; before giving AI useful input and prompts, you need to understand what's not already written in the brief or captured in the data. Second, frame the questions — the questions you ask can change everything about the result; framing questions is where your point of view enters the work, and it shapes every question you bring to AI. Then bring the understanding to AI — explain the situation, what matters and what matters not, and generate the possibilities. Build something you can try — not only imagine it, but something you can actually try; AI can help you discover and explore options you haven't considered before, and the
hedged
Irina describes the full experimental loop used in Project Paradox: define a controlled scenario, run the simulation, collect structured traces (observations, conversations, memory writes, retrievals, belief updates), score behavior against ground truth, propose a small policy change, rerun, and keep the change only if the score improves and guardrails hold — otherwise revert. The harness, scenarios, and metrics are frozen throughout.
For the hypertrophy version of the face pull, the creator advises going slightly heavier in the 8 to 12 rep range, keeping a good deliberate pace and unshrugging the shoulders. Towards the end of a set it's okay for reps to look like struggling — give it a breath, come back and squeeze until you can't do it anymore — but never let a rep degenerate to the point where you don't recognize what you're doing; that's form failure. Training to form failure lets you build up a great back with the exercise.
The creator demonstrates the face pull: grab the rope from underneath with thumbs up (palms-down/thumbs-down is the wrong way), step back so the arms are at arm's length with the rope at face level, get the hands far enough apart to set the elbows up correctly, keep the elbows at 90 degrees, and force the shoulders down rather than shrugging, which opens the chest. At the top of the rep you should feel as if doing a double bicep flex. The 90-degree elbow position sets the back muscles up to do the work, and with an overhand grip the traps would take over.
For the neuromuscular re-education version of the face pull, the creator advises pulling back very slowly and precisely, keeping the traps down, holding the isometric position about 3 to 5 seconds to ensure the load isn't shifted elsewhere, and coming back slow — it's not a race to finish but about doing every repetition with high quality. He frames it as 15 sets of one high-quality repetition rather than one set of 12 or 15, with the rhythm 'Back, hold.' Racing through reps or shifting the load defeats the corrective purpose.
Jemima Abu lays out a sequence for getting your agent to write accessible code: first, define your accessibility requirements — every company or organization has different requirements depending on field (a web app builder has different requirements than a mobile app builder), and even if you're vibe coding you should know the accessibility standards for that app and keep them in mind when prompting. Second, audit your existing code base for accessibility faults — it's easy now because you can tell your agent to do it, and you can set up an agent that carries out audits and fixes issues. Third, implement a design system or reusable system so fixing an issue in one place fixes it across many pages — a simple component library where fixing one component fixes it across 150 pages is a really
Jesse Hall lays out a benchmarking method: take a complete working agent with a real backend as the test subject, put simulated callers with goals against it who can change their mind and add background noise, define scenarios as what the caller wants, what actions the agent should take, and what the outcomes should be, score on whether the correct outcomes were met (e.g. the correct reservation hit the database, or the correct authorized amount got charged in a financial benchmark), verify the agent took the right steps (not 10 steps when 2 would do, and no hallucinated function calls), swap the models in the same agent with the same scenarios and run the loop again, ensure every model sees identical input byte for byte, and run the loop continuously to build a personalized benchmark for
Jeff Cavaliere's method for setting up a dumbbell bench press grip: instead of gripping the dumbbell traditionally down the middle with a flat wrist, shift the grip slightly toward the end of the handle, focus the squeeze through the last two fingers (forefinger and middle finger) wrapping over the forefinger, and squeeze as hard as possible through the thumb-side end of the handle. The remaining fingers just wrap around for support — this is not a hook grip, since you are not pulling anything. He demonstrates this grip on camera.
Create two slides: slide one restates the customer's situation ('here's what I hear: you're trying to do X because of Y, you've considered options but they're lacking') and asks 'do I have that roughly right?'; slide two offers how you can help and asks if it fits. Look for 'oh my God, yes' on slide one and 'how do we move forward?' on slide two.
Snyder structures the case study factory as three steps: pipeline (how people talk to us), sales (how potential customers become customers), and delivery (how customers become weird not to churn). He applies the theory of constraints from manufacturing to identify the single bottleneck to growth and solve only that.
If you're just starting the eval process, create a blank eval and start writing your template. Turn your written criteria into a prompt — much of it can be pasted in as the bullets you wrote. Start the prompt with an intro like 'You are an eval agent, or an eval LLM, looking at the output accuracy for your application.' The speaker likes to start evals with an example of what the LLM is supposed to do and always centers it in its job. After the intro, provide the data — this is really important. Include the check-in particulars, such as the three bullet points for pass and fail. Example instruction: mark only correct if the output is consistent with the input; incorrect if there's a wrong destination or property type.
Load in the dataset of saved spans so you have inputs and outputs to test the eval against. If needed, map the dataset's inputs and outputs to different variables in the eval template. If you created an eval in the eval hub, you can add it in the playground from the dropdown and it will pop up. Run the eval in the playground (e.g., with 4o1 mini) to see the output of the eval right there. The playground is a great spot to iterate on your eval template: look at the response, agree or disagree, adjust the prompt, and continue running it over time. Save the eval once the template is in a good spot, and it becomes available to use across the platform for your demo project or any application you bring into Arize.
When jotting down eval ideas, write down the task: what you want this eval to be going off of. Define the eval task first: what the eval judges, what success looks like, and what failure looks like. Write down what the eval is actually supposed to judge. Write three bullets on what pass would look like and what fail would look like. Sit down and think about what your success criteria is when building evals for your application. The notes you write will be used to actually build the eval in the next step.
The overall first-pass eval workflow: filter the spans you're judging, look through traces, identify what span you need to evaluate, build your filter, and add those to a dataset. To evaluate input versus itinerary, save all of the root spans to a dataset. To test whether a tool call returns the right information, add all of those tool calls to your dataset. Filters are the best way to identify data: to filter for root spans, use parent ID equals null or go off the default traces; to filter for a specific tool call, filter by the tool's name. Then select spans from the span view and add them to a new dataset.
Sofia Jakovcevic describes the change workflow: evaluate production traces to get scores before implementing a change (prompt, model, tool, skill, or policy change), then test and iterate on changes with the experiments workflow before shipping. If a change passes, ship it; if it is negative, go back, test, and iterate. If a change fails, bring the failed cases back to the observability layer so new production traces that failed become new cases in the regression testing suite, building regression test suites that constantly build on themselves for a more robust method of continuously iterating and improving.
Sofia Jakovcevic lays out a five-step regression loop for agents: start with production traces, build evals, run experiments, and close it all up as an automatic gate at CI. The loop uses observability to build a robust regression process that feeds automatically into CI/CD.
Farza describes his company's standing question of 'what are we launching this week?' and working backwards from the launch. When building a feature, they discuss the launch copy — the Instagram caption and the video hook — as part of building it, not after, because how you communicate the product is just as important as what you're building.
Farza Majeed shares his consistent launch video formula: this is who I am, this is what's going on, this is what I'm working on, let me show you.
Use saved training trajectories and a controlled intervention to explain a failure, separating equilibrium theory from observed empirical stability.
Demonstrate all 3 repaired bugs and defend why a passed gradient check is necessary but not sufficient for generalization.
Use the tractable fixture and posterior predictive results to separate approximation, sampling and model errors; justify one revised modeling assumption.
Creators bringing real, checkable value to the library.
business · mind
YouTube
Extracted from 216 analysed sources.
insights
protocols
Explore intelligence by what matters to you.
Choose sources → to get a library of your own and a daily brief.
Map each implemented component to the original paper and explain 2 reduced-model failure modes without claiming full-scale reproduction.
Use calibration and shift measurements to justify a monitoring or abstention threshold, and give an explicit confounding counterexample to causal feature-attribution claims.
Defend a causal claim or refusal using the DAG, numerical estimates and a counterexample showing why predictive model fit cannot establish identification.
business · fitness · mind
YouTube
Extracted from 22 analysed sources.
insights
protocols
business · fitness · mind
Podcast
Extracted from 28 analysed sources.
insights
protocols
Money · Recommendation
Build a large expert-labeled eval set to regression-test your AI system
AI Engineer
Money · Recommendation
Get your data story right before attempting success with AI agents
AI Engineer
Money · Recommendation
Define an eval set and run evals consistently to measure system health
AI Engineer
Money · Recommendation
Get your data story right before building AI agents
AI Engineer
Money · Recommendation
Build good evals: model, harness, and data changes all shift outputs
AI Engineer
