Principle · MoneyThe speaker advises treating MCP servers and AI-generated code exactly like any other code project, because many have bugs, security bugs, and security issues — despite people treating them as unicorns or magic boxes nobody looks inside.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Principle · MoneyDakkak argues that relying on a single vendor benchmark number (like 6x) leads to mismatched expectations because results vary case by case; he recommends benchmarking against your own inference stack, reproducing claims on the exact model version, and asking for the distribution of results rather than headline numbers.
Framed as advice from the speaker, not a proven rule.
AI Engineer · Taming the AI Hardware Cambrian Explosion — Abdul Dakkak, Modular
Creator perspective · MoneyDruid explains AI agent evaluation to customers using an intern analogy: a manager at a banking customer service center hires two summer interns with the same job description. Intern A contains 80% of conversations and escalates 20% to senior bankers, while Intern B contains 96% and escalates only 4%. Giving the full-time return offer to Intern B (the higher containment intern) is the wrong decision.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Creator perspective · MoneyIn the intern scenario, a customer authenticated at 7 a.m. on a mobile app asks what it would take to pay off a mortgage early. An AI agent can answer the math by pulling the balance and running the calculation, giving a complete response contained by every definition. But the question is a signal that the customer is rethinking their mortgage — a refinance opportunity and real revenue on the table. The bank writes a rule to answer the customer and then route the context to a senior banker, and that design handoff is part of Agent A's 20% escalation rate.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Principle · MoneyEach individual component in the framework is scored on three levels: 0 points if the component is missing or does not exist, 1 point if it can be observed and measured (not necessarily automatically — measuring by hand is equally good), and 2 points if it is managed, meaning it is tracked, observed, and analyzed over time, because everything is dynamic and what happened yesterday might be different today.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
PrincipleFuad Ali explains that because agent experiments are exposed via skills and API, the agent can redeploy itself on the new PR and verify itself against a dataset of actual failure cases before notifying you or deciding it's worth human review.
AI Engineer · Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
PrincipleFuad Ali explains that harness-as-a-judge performs adaptive analysis using trace analysis to catch unknown unknowns you wouldn't have written a test case for.
AI Engineer · Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
PrincipleFuad Ali describes harness-as-a-judge as an approach where eval criteria emerge from the agent's trajectory rather than being pre-written, evaluating questions like whether the right tool was picked, arguments were valid, and context was grounded in handoffs.
AI Engineer · Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
Report · MoneyIn the second notebook, when creating the answer you add a tool alias so the model uses the Tavily search tool to find the answer on the internet instead of generating the answer itself — that's the difference from the first notebook. When the model responds, it makes a search tool call (e.g., a Tavily query) and receives URLs with titles and content for each URL. The notebook builds on notebook 1's imports, model configs, seed dataset, and sampler columns (question difficulty, question type).
AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA
Creator perspective · MoneyA naive approach — running a loop a million times prompting an LLM for a product, review, and customer profile — leads to a lack of diversity, because after 10, 12, maybe 20 tries the model starts repeating itself. It also lacks correlations between features, multimodality, steerability, and validation/verification.
AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA
Report · MoneyIn Data Designer, data is generated column by column (columns are an abstraction for features). Sampler columns inject diversity — e.g., a difficulty sampler with easy/medium/difficult coding questions — and sub-samplers can map each difficulty level to a specific problem type. There are columns for text, structured data, code, and now images (multimodal).
AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA
Principle · MoneyWhen scaling up, the main thing to focus on is incrementalism — figuring out how to rebuild the graph without regenerating all of it. Design for incremental indexing first as a first class citizen; that's the only way you'll get to a point where you have to deal with the other scaling problems. You have to deal with the problem of re-syncing information long before you have to deal with the actual scaling problem of how to store it or how to compute across it all — the speaker's team fights with incremental rebuilding at Typedef when building at larger scale across a large data architecture.
endorsed
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Principle · MoneyYoni Michael recommends making the graph feel more like the model is using graph by adjusting the interface, and accounting for how the model is going to query the tool when designing its interface. As a concrete example, he says to return the results from the graph in the same form as ripgrep, because that's what the model has been trained on and knows how to handle. He also notes that if you call a tool 'search', the agent will basically always do multi-keyword search into it with no training.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Principle · MoneyDesign with the grain as much as possible — help agents lean into what they want to do anyway, and it will help you out quite a bit.
endorsed
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Principle · MoneyYoni Michael explains that you can discover which data models relate to which business domains by examining the structure of the data itself: table-level lineage, column-level lineage, shared keys, and shared naming can be used to cluster data models into domains.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Principle · MoneyIn an ideal world, your agent gets better and better at answering the same questions. Changing the questions based on a changing corpus doesn't actually help you evaluate if your agents are getting better — the best way to evaluate if your agents are getting better is having to do the exact same task again. For the purpose of evals, your goals should be constant against a constant corpus.
endorsed
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Report · MoneyKube Armor or Falco watches the runtime — what's happening at the Linux kernel level — it doesn't know about your Kubernetes. Falco watches everything, and if something goes wrong it can send an alert if it sees suspicious commands. Falco can send alerts to Falco Talon, which takes action on the alert — for example killing the pod where the exec happened to stop it from happening again. Kube Armor uses Linux security modules, so it can stop a forbidden command from ever happening in the first place — if an LS command is attempted, it fails. With Falco, if an LS command is attempted, it goes through briefly and then Talon kills the pod or whatever you have it do based on your policy. Tetragon is a sibling of Cilium and uses eBPF and LSM to get runtime enforcement done.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Report · MoneyThe output-sanitization technology demonstrated is LLM Guard, running as a proxy that scrubs responses before they return to K-Agent and then Bedrock. LLM Guard has an output_scrub field that ensures specified words never get seen in the output. Output guardrails can be applied while streaming: you can turn them on mid-stream and catch sensitive content as it streams through your system. With input guardrails off and output guardrails on, the agent's output was redacted for sensitive content.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Report · MoneyYou can set up code-based evaluators for more deterministic checks about your application, and LLM-as-a-judge evaluators for implicit things you want to extract from conversation history. Code-based evaluators support Python and TypeScript, and can run custom code checking for specific words such as brand names that should always be written the right way (e.g., catching 'Langfuse' misspelled with a capital F).
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Creator perspective · MoneyMany variables are relevant to benchmarks: the benchmark version (e.g., drop-in-bench 2.0 vs 2.1), the agent version, the model service and route (including updates to the inference endpoint), sandbox resources, sampling settings such as temperature, thinking defaults, and timeout and retry policies. If the agent runs out of sandbox resources it will fail, and you get a bad score for the model although it wasn't the model's fault. If you don't retry and there has been an infrastructure error, you would get a bad score.
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases