Explore

Defend against prompt injections

To defend against prompt injections, sanitize input, put a good system prompt, require human approval for high-risk actions in agents, configure agents to always ask before doing anything like deleting emails or sending an email on your behalf, constrain the model behavior with a system prompt, and define and validate the output formats to ensure anything that should be treated as text and not as a command.

Contested by The Official SaaStr Podcast: SaaS | Founders | Investors

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Scan-fix-rescan loop for AI-generated code

The speaker recommends a workflow: create an agent hook so that whenever Claude generates code, a security tool scans it for vulnerabilities; if vulnerabilities are found, use another tool to fix them or have Claude generate new code, then re-scan; repeat until the new code has no security issues. This can be integrated into Claude or another AI coding tool through a system prompt.

Method, not a verified protocol.

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Audit and monitor AI agents, including background log scanning

Javier Garza recommends putting auditing and monitoring on AI agents — something he says most people don't do. Without auditing, agents may do things you don't want without your attention. He suggests logging agent activity to a file and running a background tool that scans the log and flags malicious behavior that may be missed in the user interface, especially for background agent tasks which are prone to hiding malicious behavior.

AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk

Two conditions for applying hill climbing: replayable environment and verifiable outcome

To apply the hill-climbing technique to any domain you need (A) a replayable environment so loops can run, and (B) a verifiable outcome or quantifiable metric. As long as you have a verifiable outcome, you should be able to design a good loop. A replayable environment is under-discussed but extremely important.

AI Engineer · Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase

Run evals many times and measure pass^k, not just pass@k

Run evals multiple times over the same corpus to measure consistency; measure pass^k (correct on every run) not just pass@k (ever correct) — when you change the metric, the numbers completely flip.

AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef

Design tools to match model priors: name them 'search', return ripgrep format

Rename graph query tools to 'search' so agents use them without training; return graph results in the same format as grep/ripgrep output because that's what the model's been trained on; lean into the model's prior experience rather than fighting it, and account for how the model will query the tool when designing it.

AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef

Build graph-harnessing tools, including a blast radius tool that walks dependency edges

Build tools that specifically harness the graph's capabilities to give the agent something that makes its job easier. Build a 'blast radius' tool: pass in the qualified/canonical name of something, and it walks down the graph edges to figure out exactly what depends on it and what will break if it gets changed.

Method, not a verified protocol.

AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef

Practice eval-driven development for agentic systems

Practice eval-driven development when building anything agentic — you need to be able to measure it; create a set of real production data engineering use cases, tasks, and workflows ranging in difficulty from easy to blind, with each task run multiple times on isolated clones of the actual data environment so there is no leakage.

AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef

Prefer kernel-level prevention over detect-and-respond for runtime security, using KubeArmor, Falco, or Tetragon

The speakers recommended a runtime engine — Falco, KubeArmor, or Tetragon — to watch what happens at the Linux kernel level in production. They contrasted approaches: Falco watches everything and alerts (with Falco Talon able to kill the pod where an exec happened), while KubeArmor uses Linux security modules to stop forbidden commands from ever happening. They advised preferring prevention over response — wiring into the Linux kernel on the front end — noting Tetragon uses eBPF and LSM, that security doesn't always require fancy tooling (sometimes just a kernel change), and that a fork bomb not caught within about 200 milliseconds can spawn enormous numbers of process IDs and exhaust resources.

The 200ms fork-bomb figure is an approximate field observation.

AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog

Use an MCP gateway or registry with an allowlist instead of pulling tools directly off the internet

Use an MCP gateway or registry to provide a list of allowed tools that get pulled in, rather than pulling directly off the internet; in an enterprise you don't pull NPM or PyPI packages straight off the internet either.

AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog

Don't give an AI agent direct cluster access you wouldn't give a human; enforce policy with Kyverno admission control

The speakers discouraged letting an AI agent have direct access to your cluster — a mistake you wouldn't make with a person — and recommended Kyverno as an admission controller to validate or reject deployments based on cluster-level, organization-specific policy rules: requiring certain resource requests, allowing pulls only from a trusted internal registry while blocking external registries, and supporting mutating policies in addition to validating ones.

AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog

Test extraction accuracy on a small sample before scaling to a million documents

Omar recommends running a quick check on a small sample of documents to verify extraction accuracy matches expectations before scaling parsing up to a million documents.

AI Engineer · Parsing the Epstein Files for JMail — Palak Agarwal & Omar Alhait, Reducto

Choose models by high ceiling, stable average, and high solid base

When choosing a model, look for a very high ceiling, a very stable average, and a very high solid base. If two models have nearly the same average, pick the one with the higher solid base — a 6 percentage point difference is a noticeable gap that justifies choosing the higher-consistency model. Before picking, first decide whether you care about capability or reliability.

AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases

Pin the harness and benchmark version to preserve comparability

Pin the Harbor evaluation harness to one version to preserve comparability. Changing the harness version changes a variable and degrades score comparability even if the new version has a bug — you are at least testing all the models with the same bug. Switching to a new benchmark version breaks score comparability and forces retesting all models you already did.

AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases

Use any coding agent with the Arize skill to pull trace information for fixes

Users are not locked into Claude Code or Codex; any coding agent can be connected to the Arize skill. The core benefit is that the skill lets the agent pull down information from traces so fixes are grounded in where and why something failed, not just the knowledge that the output is wrong.

AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

Feed the coding agent full context: codebase, APM tickets/logs, JIRA, traces, evals, annotations

The skill provides the coding agent access to codebase information, APM tickets, APM logs, JIRA tickets, and Arize traces. The recommendation is to bring all of this data — plus evals and human annotations — into the agent so it can reason about what changes to make in the failing area, such as correcting tool calls so the agent goes down the right pathways.

AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

Set up the CLI and skills so the coding agent can pull traces, analyze issues, and suggest fixes

Setting up the CLI and skills takes just a couple of steps and gives your coding agent access to a bunch of different tools. Once set up, the coding agent can pull down traces, understand what's going wrong with the agent, look at existing evaluations and annotations, do its own analysis, and come back with fix suggestions. From within the coding agent you can pull down traces, build out datasets, run experiments, and run evaluations — the full gamut of the experience.

Method, not a verified protocol.

AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

Pull all debugging data sources together: Arize traces, evals, human annotations, APM traces, and the code base

To speed up debugging, pull in Arize traces to get all the information of what your agent did, pull in your evaluations to see where things actually went wrong, bring in annotations from humans for more information on where things could go right, bring in your actual APM traces to check whether the system is up or down, and bring in your code base so you can understand what changes to make — e.g., calling the right tools and fixing broken tools — based on where the failure occurred.

Method, not a verified protocol.

AI Engineer · Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

Sanitize LLM output with an allow list of operators and fields

Peter Werry recommends using an allow list (deny by default) of allowed operators and potentially allowed fields when sanitizing LLM output.

AI Engineer · Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked

Attack acceleration first and use more steps in the first 5-10 meters

The host recommends attacking acceleration first when learning to sprint, since most sports rarely express max speed (only wingers covering 40-50m breaks benefit). Taking huge long strides in the first 5-10 meters prevents generating enough horizontal force; athletes need more steps (higher cadence) in the initial phase instead.

Host's endorsed heuristic with a stated exception for wingers

Garage Strength · How To Get Strong and Still Be Athletic w/ Sika Strength