36,594 results · Most relevant
Wisedocs' baseline preprocessing uses OCR and images to extract information from underlying records; a series of models (NLP, extraction, VLLMs, entity extraction) then structures the data; a summary layer sits on top and is human-verified. The abstraction pattern concentrates information density as you go up the stack, verified at various points with models and human intervention.
The baseline data-structuring approach uses OCR and images to extract information from the underlying records; each step of the AI data-extraction pipeline must be calibrated (build good evals, calibrate judges) because incorrect upstream data used downstream gains nothing; experts who can verify the information must work across teams; with good claim-level extraction you can flag conflicting diagnoses, deduplicate, and search for missing records; hallucinations show up when people assume something is in a medical record but it's not.
Wisedocs built the MLCR benchmark with Artificial Analysis to measure AI effectiveness on domain-specific extraction: tiered question difficulty from simple (when did the injury happen, what was the injury) to harder (how many surgeries, patient capacity); an open-source dataset for the easy tiers plus a private hosted set to stop benchmark-maxing; measurement on three axes — conciseness, accuracy, completeness — against a human expert-validated answer, with triple-judge verification; calibration methods were described in a blog post and follow-up.
Consequential creator advice requires independent verification before Today can promote it.
Wisedocs structures benchmark questions in six tiers from simple factual extraction to complex tasks, releasing an open-source dataset for the first three tiers while keeping a private hosted set (with Artificial Analysis) to stop bench-maxing. AI answers are measured on conciseness, accuracy, and completeness against a human expert-validated answer, with triple judge verification, and each step of the extraction process is kept well calibrated.
Consequential creator advice requires independent verification before Today can promote it.
Run the judge against the humans as an experiment: the experiment takes a dataset, a task to run on every example (the judge), and evaluators to score the results (checking whether the judge's label matches the human's). Only run the meta-evaluation on the dev split, not the test split. Run the meta-evaluation yourself with the command 'npm run meta'. The rubric file 'evals actionability.ts' takes three or four minutes to run.
A multi-part method for constructing a rubric: mark data clearly with XML tags, keep boundaries between input/output/instructions, mark examples as examples, show examples to clarify criteria, test with and without examples, and constrain the judge's answer to a fixed set of options.
The eval workflow being built: trace your agent, read what it did, write evals for what matters, check the evals can be trusted, change the agent, prove the change worked, then lock it in with CI. Lock in agent fixes by putting the change into CI so that it stays fixed.
Build the golden dataset in Phoenix by finding every report with a human label, making each an example where the input is the report and the expected output is the human's label, sorted by span ID. Use a 75/25 split: 75% of labels in the tuning set and 25% in the test set. Sort examples by span ID before splitting so a report always lands in the same split however many times you run it, and span ID links each example back to its original trace. Put every fourth example into the test split and the rest into dev.
Measure your judge by comparing its predictions against the truth of what a human expert would say. Create human annotations in Phoenix by opening a trace, clicking the pencil button, and adding an annotation (e.g., 'human actionable' set to actionable or not actionable) for each trace. Involve a domain expert by having them read trace outputs and produce a golden dataset of correct labels (e.g., 10, 20, or 50 traces).
The eval improvement loop: trace your agent, read what it did, write evals for what matters, check the evals can be trusted, change the agent, prove the change worked, then lock it in with CI.
Zoom out of the 162-game season to find long stretches without off days (10, 13, or 17 games in a row), map time zone, surface, and elevation changes, factor in turf vs grass and stadium dimensions (e.g., Rockies' big outfield, Fenway's short porch), then plan with baseball and performance staff how to mitigate stress through tough streaks — including skipping early work on high-distance days and managing sleep as a performance team.
Use traditional periodization for starting pitchers since they pitch once every five to six days. Work backwards from the start date: schedule the upper body lift on bullpen throwing day to marry stress, give a recovery day (spa day) after the start followed by the lower body lift, schedule sprinting the day before the start as a neuromuscular primer, and add other conditioning work in addition to lifting and sprinting.
Set up a testing profile with a global view: rotational power (Proteus), explosive countermovement jump on force plates, lower body strength (belt squat ISO, BBT force-velocity profiling), sprint assessment (1080, Velo), and site/tissue-specific measures (Nordics for hamstrings, Force Frame for groin and shoulder internal/external rotation). Collaborate with sports med, ATs, and PTs, and apply the approach to any sport.
Consequential creator advice requires independent verification before Today can promote it.
Starters pitch every 5-6 days, so work backwards from the start date: upper body lift on bullpen day, lower body lift after a recovery 'spa day' following the start, sprint the day before the start as a neuromuscular primer, plus other conditioning.
Use Lean as a theorem-proving system that checks every step and doesn't let you make a mistake — what you need in high assurance situations. Rewrite programs in a common intermediate language, compile to verification conditions in Lean, and prove Lean theorems to show the code is correct. Lean SMT can prove many Lean theorems in milliseconds on a CPU. With VerisForge, you enter a specification in English and it builds the code and does the formal proof.
When simulating safety scenarios, don't waste time on non-hazardous situations (e.g., aircraft separated by five miles); simulate critical scenarios representative of the airspace. If falsification reveals failures that give you a bad feeling, either don't deploy the system or implement mitigations to prevent those critical scenarios. Simulated inferences about safety are only as good as the simulator, so real-world testing is also needed. There is no one silver bullet — it requires a number of different kinds of approaches. Continue validation after deployment via runtime monitoring or runtime assurance: validate sensor inputs going into the decision system, monitor the internals, and model and monitor the outputs.
Guideline questions to start asking: who/team spends what? What must you test to use a cheaper model? What's your base load? What's the default model for each team? Which workload has a person waiting that needs something faster? And how much of what we generate is useful?
Bug Hunter is a very effective flow and pattern for closing loops and making systems run more autonomously, available as a template and in Claude Code. Instead of a single agent reviewing a PR, it uses a multi-phase flow: (1) code is split into slices, each handed to an individual context/agent that reviews only that slice and writes down potential problems; (2) a reviewer agent gets the slice plus the flagged problems and can agree or disagree with each; (3) a final phase picks candidates that might actually be real problems needing verification. In a prover phase, an agent can download the repo, try to reproduce the bug or verify it's a legit problem, and surface results in a report. In practice the report is added as comments to the actual PR, raising issues in line (e.g., a change to a
described/endorsed
Bug Hunter splits code under review into slices, each reviewed by an agent that writes down potential problems; a second-phase reviewer agrees or disagrees with each problem; a final phase picks candidates that might be real; a prover phase downloads the repo and tries to reproduce the bug; findings are added as inline PR comments with severity priorities.
They split the answer into parts: the agent streams markdown prose token by token with the frontend styling it live, and the source card — a real UI component that holds nothing you'd click before reading the answer — is appended as one piece after the full answer arrives.
Use saved training trajectories and a controlled intervention to explain a failure, separating equilibrium theory from observed empirical stability.
Demonstrate all 3 repaired bugs and defend why a passed gradient check is necessary but not sufficient for generalization.
Use the tractable fixture and posterior predictive results to separate approximation, sampling and model errors; justify one revised modeling assumption.
Choose sources → to get a library of your own and a daily brief.
Map each implemented component to the original paper and explain 2 reduced-model failure modes without claiming full-scale reproduction.
Use calibration and shift measurements to justify a monitoring or abstention threshold, and give an explicit confounding counterexample to causal feature-attribution claims.
Defend a causal claim or refusal using the DAG, numerical estimates and a counterexample showing why predictive model fit cannot establish identification.
