Log everything: input, reasoning, decisions, and integration calls
Dan recommends that the agent log everything — input, reasoning, decisions, and calls to integrations — as a runtime observability requirement.
37,065 results · Most relevant
Dan recommends that the agent log everything — input, reasoning, decisions, and calls to integrations — as a runtime observability requirement.
Dan recommends creating success criteria together with each task so outcomes like a contract renewal can be judged as successful or not.
Dean runs 'flash deploy', which packages up the whole application, builds it, and sends it to RunPod's servers — no Docker involved. He describes this as instantaneous packaging of your application, sending it out to servers, and serving it. After deploying, he changes the endpoint ID in his prompt script to point at the new deployment, checks the workers log to see the job's printed output when it's done, and notes that once deployed the application is fully servable — customers can use your application that uses this backend, no longer on localhost.
Do not give an AI agent direct access to your cluster; you wouldn't let a person have direct access, so stick with your known flows. Some customers let AI agents have direct access to their cluster, which the speaker flags as a mistake.
Michael recommended a testing approach: run attacks against the unguarded bot without symptom prompting for a real-world test, keep that first cluster open, and reuse the exact same chats and prompts against the second, security-enabled cluster — what worked before should now be blocked. Specific challenges include getting the bot to deploy a public image, exfiltrate data (even just the menu) to a public endpoint, snoop the file system for the secret recipe, and comply despite explicit instructions not to share.
Lotte Verheyden describes the change workflow: run an experiment on your data sets, evaluate the results, and only deploy to production when results are better and no regressions were introduced.
The presenters recommend monitoring for out-of-scope requests in a production application to detect when users try to do something the system is not supposed to handle; when the evaluator fires, decide whether to expand the application's scope or fix user communication about what the application is for. They note this can reveal that users misunderstand how the system is supposed to be used, especially in larger-scope applications.
The hosts advise putting in the reps and closing the loop continuously so the dataset represents what users are actually doing: create good production examples and add them to offline evals, run experiments on maintained datasets and evaluate them, make eval runs part of CI/CD to avoid quality regressions, and deploy only when you can ship with confidence that quality has not regressed.
The presenters walk through building an out-of-scope request evaluator: set it up via the Langfuse evaluators page by selecting a template and mapping variables; target generations rather than traces because you also want to reference the system prompt; map the system prompt (the very first message sent in the interaction) and the last user message as variables using JSON paths so you can target specific parts of the interaction precisely; and in the evaluator prompt, always compare what the system is supposed to do with what the user is actually trying to do. Once happy with the test run, click 'execute' on the evaluator.
Most people will use a bunch of documents from their business, and you can use the LLM to help you figure out the right schema. Once you have a schema, using that schema will guide the LLM so it won't just make up new node and relationship types — it'll stick to that. If you don't provide a schema, the LLM will make up new node and relationship types.
Once the vector index is created, go into the agent section and access the prompt. Create the agent from scratch and configure it with GraphRAG and auto-populate. In the add tool section, create a similarity search tool that searches lesson content; the similarity search tool lets you gather insights from the lesson without having to take the lesson. Add a description to the similarity search tool and make sure you have that description too. Select the vector index named chunk embedding for the similarity search tool. Use about the top five results for the similarity search tool. Select OpenAI as the embedding provider for the similarity search tool. Use the text-embedding-3-small embedding model for the similarity search tool. The default dimensions for the embedding model is 1536. Make 9
Nyah Macklin details configuring the agent's similarity search tool: naming it 'search lesson content', selecting the chunk embedding vector index, setting top 5 results and 1536 embedding dimensions, enabling connected graph data return, and using a Cypher query that returns nodes and scores.
If a question can be solved with a search engine, vector-based RAG is usually sufficient. If a question requires multiple hops (at least three hops) through connected data, GraphRAG becomes imperative. GraphRAG is also helpful for tasks like source code analysis, supply chain questions, or anything requiring aggregation or concatenation of data from different documents.
Nyah Macklin explains that similarity search (vector search) must be added to the agent before creation, and walks through getting the sample 'create vector index' code from the GraphRAG for Python section, pasting it into the Aura console's Cypher query window, and running it, noting the 'already exists' message if run again.
Josh Woodward argues every company needs a set of people whose job is to be at the frontier — trialing new technology and reporting what the rest of the company should pay attention to. These members should hold subscriptions to all the major AI tools (roughly $200/month across everything), and the organization must be architected so that can exist. A frontier member who hasn't heard the latest hot thing viral on X in the last 24 hours is failing at their job.
Josh Woodward advises creating a space where weird things can grow — you can't just nestle the lab under an existing business unit. It needs to report to the CEO, or at enough organizational distance to maintain independence.
Josh Woodward describes his hiring approach: obsess over finding people who just can't stop themselves from building. Look for intellectual curiosity — reading stuff, being on X, listening to podcasts. Hire people obsessed with solving people's problems, not their own ego or glory. Hire people who add energy — the kind you'd run through a wall with. In interviews, ask how they navigate unknown, unstructured situations where it's unclear which way is forward; being energized by that is a huge credibility boost.
As a leader, Josh Woodward recommends creating an environment where people can ask uncomfortable questions and say 'hey, I don't think this is working' — and when they do, respond with 'this is great, thank you for doing this' (he replied-all exactly that when a team member said a launch shouldn't happen). When the team says it's not going, pivot and do something different.
Implement a retry loop that takes validation or execution errors and passes them back into the LLM for self-correction, up to a max number of attempts. The error retry loop is one of the most important components of the system. When validation fails, return an error that gets dumped out for the retry loop to consume.
Sanitize LLM-generated database queries using an allow list rather than a deny list, denying everything by default. Create an allow list of allowed operators and potentially an allow list of allowed fields when validating LLM query output. Allow-listing operators and fields prevents prompt injection through where or function clauses that could run JavaScript on the server.
Choose sources → to get a library of your own and a daily brief.