Recommendation · MoneyJavier Garza recommends sanitizing outputs from AI systems and ensuring that people cannot get information out of the system prompt to defend against system prompt leakage. He also recommends treating prompts and extracted outputs as non-executable input, not as trusted executable content.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyJavier Garza recommends defending against vector and embedding weaknesses by using monitoring and logging, permission and access controls, data validation, and reviewing and removing malicious or misleading data from embedding stores.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyJavier Garza recommends that when having an AI tool create an app, you also tell it to design a testing process and a security review, not just the app, and that you can tell the AI agent to simulate an attack (e.g., prompt injections) so you can address the weaknesses it finds. He also mentions automatic threat enumeration from a data flow diagram when threat modeling AI systems.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyJavier Garza recommends designing APIs and user interfaces that encourage responsible use of LLMs, putting automatic validation of LLM outputs, putting a good system prompt, and fine-tuning the model to ensure output quality, to defend against over-reliance and misinformation.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyJavier Garza recommends that when something changes in your code or AI application, triggers like webhooks should fire a scan; that data pipelines being updated or AI models being deployed should trigger a scan; and that when an agent creates infrastructure-as-code, you monitor that Terraform file so the moment it changes you run a scan.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyJavier Garza recommends putting rate limits on AI systems to prevent misuse of infrastructure for unauthorized tasks like crypto mining, limiting exposure and using input validation to defend against embodied consumption attacks, and using resource allocation management so people cannot misuse your AI infrastructure. He also recommends implementing data security and protections for APIs and rate limiting.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyJavier Garza recommends that when you run a long prompt and step away, you put verbose logging into a file so a background tool can monitor for unwanted behavior, and that when agents do a lot of things and you don't pay attention, they may do things you don't want, so you should audit them.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Method · MoneyDiscover and map the AI system structure — for example with an AI-generated map — then take the JSON output into a tool that renders it as a graphical data flow diagram (many tools accept JSON-format maps). Create AI-aware architecture diagrams or use model artifacts, and enumerate threats including adversarial inputs and prompt injection to ensure coverage.
Method, not a verified protocol.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyJavier Garza recommends using an existing threat modeling framework (STRIDE, MAESTRO, Attack Trees, DREAD) instead of starting from scratch, and explains that STRIDE covers spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege — cover each of those when you design something.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyDan recommends setting a budget for the agent to avoid surprises like the agent consuming 20K per month in resources.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Recommendation · MoneyDan recommends evaluating the agent's compliance with standards like SOC 2 and internal policies as part of the scorecard.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Method · MoneyAt runtime, the agent needs to log, create tasks based on the mandate, and execute the tasks. At runtime, verify whether tasks are written, whether the input is logged, and whether the required context is present. Success criteria should be created together with each task, because without success criteria linked to an individual outcome (like a contract renewal), it is impossible to judge whether that renewal was successful. The agent should explain everything it is doing — this is what the task topic is for. At the end, validate the output — there is one single score that evaluates whether the agent was successful or not.
Method, not a verified protocol.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Recommendation · MoneyDan recommends tracking drift against the baseline established at the first go-live date to confirm the agent stays within its boundaries.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Recommendation · MoneyDan recommends using a central action center where people engage with all agents in the organization, rather than each agent handling its own handoff.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
MethodFuad Ali shares an example prompt to run with Claude Code: 'Hey, check my traces for any errors or anything I can fix. Cross-reference with the code for the agent in this repo and use the agent skills.'
Method, not a verified protocol.
AI Engineer · Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
RecommendationMaintain two loops: the detect-investigate-hypothesize-fix loop, and the validation loop that tests the fix on real failures and edge cases from your working example.
endorsed
AI Engineer · Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
RecommendationUse trace data — the source of truth for what your agent is doing — combined with your actual repo data (the code that defines what your agent should be doing) to get really interesting analysis and make improvements at scale. Look at the entire context of what your agent is doing in production, not just individual traces.
endorsed
AI Engineer · Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
Recommendation · MoneyFrame the agent system prompt as 'you are an expert search agent that uses web search to answer questions accurately, provide valid JSON with the final answer, supporting URLs, and a short justification.'
AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA
Recommendation · MoneyRun impact analysis pre-flight — not at build time and not in a production incident — so the agent knows the blast radius before it acts on a change.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Recommendation · MoneyYoni Michael lays out his recipe for building solid, honest evals: mine real task candidates out of your corpus, build a deterministic anchored score as much as possible, and unit test the score itself first. Then run evals many times over and track all metrics including pass@k and pass^k, and include examples of things your agent may not be good at — you don't want it to pass 100%, because that's not useful. He adds that evals are your chance to see what your agent will do in as hard as possible circumstances, and that giving agents very easy and simple evals doesn't do anyone any service.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef