Recommendation · MoneyJavier Garza recommends redefining threat modeling scope for AI to include LLM models, prompt templates and system prompts, context windows, memory stores, and endpoints/APIs the agent can access — not just databases, credentials, tokens, and infrastructure.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyJavier Garza recommends storing secrets securely and granting AI agents access only to the resources they need, when they need them, and minimizing the access an AI agent has to reduce attack surface. He also warns that agentic tool tokens need to be stored in a way that they cannot be leaked, and that giving a local AI agent like OpenClaw access to everything lets it do things it is not supposed to do when it goes wrong.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyWhenever you use an MCP tool, really check the code — it's very important. Since reading is hard because most MCP servers are very long, use a tool that checks for malware and other issues in the code itself (Snyk has one; there are many others from other companies).
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Method · MoneyUse AI itself to help threat model: ask an agent to create the threat model for the app you are designing, generate attack scenarios and abuse cases, simulate attacks such as prompt injections so you can address the attack vectors, and have it suggest mitigations and security controls. Automatic threat enumeration from a data flow diagram is part of this.
Method, not a verified protocol.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneySecure the critical path by running SBOM and AI BOM scans very regularly — every time the app changes — for all repos, and verify that packages exist as part of supply chain hardening.
AI Engineer · AI Security Engineer Foundations + Certificate — Javier Garza, Snyk
Recommendation · MoneyDan recommends defining the agent's mandate beyond the system prompt: what the agent should do, what is out of scope, which policies, boundaries, facts, documents, and connected systems it reasons over.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Recommendation · MoneyDan recommends defining which systems the agent may engage with, restricting to specific endpoints rather than granting access to an MCP server with 4,000 tools where you don't know what it's doing.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Recommendation · MoneyDan recommends defining the business impact and ROI target for the agent — e.g., expecting 70% contract renewal — and measuring it at the end to judge business success.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Recommendation · MoneyDan recommends that the agent log everything — input, reasoning, decisions, and calls to integrations — as a runtime observability requirement.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Recommendation · MoneyDan recommends creating success criteria together with each task so outcomes like a contract renewal can be judged as successful or not.
AI Engineer · https://www.youtube.com/watch?v=PwCSYy-oGsM
Method · MoneyDean runs 'flash deploy', which packages up the whole application, builds it, and sends it to RunPod's servers — no Docker involved. He describes this as instantaneous packaging of your application, sending it out to servers, and serving it. After deploying, he changes the endpoint ID in his prompt script to point at the new deployment, checks the workers log to see the job's printed output when it's done, and notes that once deployed the application is fully servable — customers can use your application that uses this backend, no longer on localhost.
Method, not a verified protocol.
AI Engineer · AI Apps in a Flash: Ship to GPUs Without Docker — Dean Quiñanola, Runpod
Recommendation · MoneyDo not give an AI agent direct access to your cluster; you wouldn't let a person have direct access, so stick with your known flows. Some customers let AI agents have direct access to their cluster, which the speaker flags as a mistake.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Recommendation · MoneyMichael recommended a testing approach: run attacks against the unguarded bot without symptom prompting for a real-world test, keep that first cluster open, and reuse the exact same chats and prompts against the second, security-enabled cluster — what worked before should now be blocked. Specific challenges include getting the bot to deploy a public image, exfiltrate data (even just the menu) to a public endpoint, snoop the file system for the secret recipe, and comply despite explicit instructions not to share.
AI Engineer · Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Recommendation · MoneyLotte Verheyden describes the change workflow: run an experiment on your data sets, evaluate the results, and only deploy to production when results are better and no regressions were introduced.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Recommendation · MoneyThe presenters recommend monitoring for out-of-scope requests in a production application to detect when users try to do something the system is not supposed to handle; when the evaluator fires, decide whether to expand the application's scope or fix user communication about what the application is for. They note this can reveal that users misunderstand how the system is supposed to be used, especially in larger-scope applications.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Recommendation · MoneyThe hosts advise putting in the reps and closing the loop continuously so the dataset represents what users are actually doing: create good production examples and add them to offline evals, run experiments on maintained datasets and evaluate them, make eval runs part of CI/CD to avoid quality regressions, and deploy only when you can ship with confidence that quality has not regressed.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Method · MoneyThe presenters walk through building an out-of-scope request evaluator: set it up via the Langfuse evaluators page by selecting a template and mapping variables; target generations rather than traces because you also want to reference the system prompt; map the system prompt (the very first message sent in the interaction) and the last user message as variables using JSON paths so you can target specific parts of the interaction precisely; and in the evaluator prompt, always compare what the system is supposed to do with what the user is actually trying to do. Once happy with the test run, click 'execute' on the evaluator.
Method, not a verified protocol.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Recommendation · MoneyMost people will use a bunch of documents from their business, and you can use the LLM to help you figure out the right schema. Once you have a schema, using that schema will guide the LLM so it won't just make up new node and relationship types — it'll stick to that. If you don't provide a schema, the LLM will make up new node and relationship types.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Method · MoneyOnce the vector index is created, go into the agent section and access the prompt. Create the agent from scratch and configure it with GraphRAG and auto-populate. In the add tool section, create a similarity search tool that searches lesson content; the similarity search tool lets you gather insights from the lesson without having to take the lesson. Add a description to the similarity search tool and make sure you have that description too. Select the vector index named chunk embedding for the similarity search tool. Use about the top five results for the similarity search tool. Select OpenAI as the embedding provider for the similarity search tool. Use the text-embedding-3-small embedding model for the similarity search tool. The default dimensions for the embedding model is 1536. Make 9
Method, not a verified protocol.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Method · MoneyNyah Macklin details configuring the agent's similarity search tool: naming it 'search lesson content', selecting the chunk embedding vector index, setting top 5 results and 1536 embedding dimensions, enabling connected graph data return, and using a Cypher query that returns nodes and scores.
Method, not a verified protocol.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j