Recommendation · MoneyTo apply the hill-climbing technique to any domain you need (A) a replayable environment so loops can run, and (B) a verifiable outcome or quantifiable metric. As long as you have a verifiable outcome, you should be able to design a good loop. A replayable environment is under-discussed but extremely important.
AI Engineer · Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase
Recommendation · MoneyRun evals multiple times over the same corpus to measure consistency; measure pass^k (correct on every run) not just pass@k (ever correct) — when you change the metric, the numbers completely flip.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Recommendation · MoneyRename graph query tools to 'search' so agents use them without training; return graph results in the same format as grep/ripgrep output because that's what the model's been trained on; lean into the model's prior experience rather than fighting it, and account for how the model will query the tool when designing it.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Method · MoneyBuild tools that specifically harness the graph's capabilities to give the agent something that makes its job easier. Build a 'blast radius' tool: pass in the qualified/canonical name of something, and it walks down the graph edges to figure out exactly what depends on it and what will break if it gets changed.
Method, not a verified protocol.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Recommendation · MoneyPractice eval-driven development when building anything agentic — you need to be able to measure it; create a set of real production data engineering use cases, tasks, and workflows ranging in difficulty from easy to blind, with each task run multiple times on isolated clones of the actual data environment so there is no leakage.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Recommendation · MoneyThe speakers recommended a runtime engine — Falco, KubeArmor, or Tetragon — to watch what happens at the Linux kernel level in production. They contrasted approaches: Falco watches everything and alerts (with Falco Talon able to kill the pod where an exec happened), while KubeArmor uses Linux security modules to stop forbidden commands from ever happening. They advised preferring prevention over response — wiring into the Linux kernel on the front end — noting Tetragon uses eBPF and LSM, that security doesn't always require fancy tooling (sometimes just a kernel change), and that a fork bomb not caught within about 200 milliseconds can spawn enormous numbers of process IDs and exhaust resources.
The 200ms fork-bomb figure is an approximate field observation.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Recommendation · MoneyUse an MCP gateway or registry to provide a list of allowed tools that get pulled in, rather than pulling directly off the internet; in an enterprise you don't pull NPM or PyPI packages straight off the internet either.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Recommendation · MoneyThe speakers discouraged letting an AI agent have direct access to your cluster — a mistake you wouldn't make with a person — and recommended Kyverno as an admission controller to validate or reject deployments based on cluster-level, organization-specific policy rules: requiring certain resource requests, allowing pulls only from a trusted internal registry while blocking external registries, and supporting mutating policies in addition to validating ones.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Recommendation · MoneyWhen choosing a model, look for a very high ceiling, a very stable average, and a very high solid base. If two models have nearly the same average, pick the one with the higher solid base — a 6 percentage point difference is a noticeable gap that justifies choosing the higher-consistency model. Before picking, first decide whether you care about capability or reliability.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyPin the Harbor evaluation harness to one version to preserve comparability. Changing the harness version changes a variable and degrades score comparability even if the new version has a bug — you are at least testing all the models with the same bug. Switching to a new benchmark version breaks score comparability and forces retesting all models you already did.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Recommendation · MoneyUsers are not locked into Claude Code or Codex; any coding agent can be connected to the Arize skill. The core benefit is that the skill lets the agent pull down information from traces so fixes are grounded in where and why something failed, not just the knowledge that the output is wrong.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyThe skill provides the coding agent access to codebase information, APM tickets, APM logs, JIRA tickets, and Arize traces. The recommendation is to bring all of this data — plus evals and human annotations — into the agent so it can reason about what changes to make in the failing area, such as correcting tool calls so the agent goes down the right pathways.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Method · MoneySetting up the CLI and skills takes just a couple of steps and gives your coding agent access to a bunch of different tools. Once set up, the coding agent can pull down traces, understand what's going wrong with the agent, look at existing evaluations and annotations, do its own analysis, and come back with fix suggestions. From within the coding agent you can pull down traces, build out datasets, run experiments, and run evaluations — the full gamut of the experience.
Method, not a verified protocol.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Method · MoneyTo speed up debugging, pull in Arize traces to get all the information of what your agent did, pull in your evaluations to see where things actually went wrong, bring in annotations from humans for more information on where things could go right, bring in your actual APM traces to check whether the system is up or down, and bring in your code base so you can understand what changes to make — e.g., calling the right tools and fixing broken tools — based on where the failure occurred.
Method, not a verified protocol.
AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Recommendation · MoneyPeter Werry recommends using an allow list (deny by default) of allowed operators and potentially allowed fields when sanitizing LLM output.
AI Engineer · Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked
Recommendation · MoneyWhat is important is to match your bottleneck rather than simply going with a lever that has worked before. For this pipeline the speaker chooses async IO for fetching images and Ray Actors as the multiprocessing library for processors. Ray Actors are essentially the same as general processes but offer an API to communicate across workers, so you don't have to reinvent the wheel.
AI Engineer · From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model
Recommendation · MoneyDisaggregate the process so workers are spread across multiple nodes, doing their own image processing and storing, retrieved lazily just-in-time by DP ranks. Two flags to tune for disaggregation: zero-copy retrieval and a spread scheduling mechanism.
AI Engineer · From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model
Recommendation · MoneySpread data-processing workers across multiple nodes so they do their own image processing and storing, with results retrieved lazily just-in-time by data parallel ranks.
AI Engineer · From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model
Recommendation · MoneyUse a background thread that fills up a queue ahead of time and always stays ahead of the consumer, so the trainer is never bottlenecked on requesting data. The producer should produce examples at a multiplier of the consumer's rate; making sure your producer is always ahead of your consumer will pay great dividends.
AI Engineer · From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model
Recommendation · MoneyUse an object store (a Ray convenience method) where each sample carries an object reference (a pointer to the data) instead of the data itself. With an object store, the tensor never round trips to the head node, and Megatron DP ranks can directly call the array store to fetch data as needed. The Ray store also eliminated an extraneous put object call that the prefetcher solution had.
AI Engineer · From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model