Recommendation · MoneyThe hosts advise putting in the reps and closing the loop continuously so the dataset represents what users are actually doing: create good production examples and add them to offline evals, run experiments on maintained datasets and evaluate them, make eval runs part of CI/CD to avoid quality regressions, and deploy only when you can ship with confidence that quality has not regressed.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Method · MoneyThe presenters walk through building an out-of-scope request evaluator: set it up via the Langfuse evaluators page by selecting a template and mapping variables; target generations rather than traces because you also want to reference the system prompt; map the system prompt (the very first message sent in the interaction) and the last user message as variables using JSON paths so you can target specific parts of the interaction precisely; and in the evaluator prompt, always compare what the system is supposed to do with what the user is actually trying to do. Once happy with the test run, click 'execute' on the evaluator.
Method, not a verified protocol.
AI Engineer · Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Recommendation · MoneyMost people will use a bunch of documents from their business, and you can use the LLM to help you figure out the right schema. Once you have a schema, using that schema will guide the LLM so it won't just make up new node and relationship types — it'll stick to that. If you don't provide a schema, the LLM will make up new node and relationship types.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Method · MoneyOnce the vector index is created, go into the agent section and access the prompt. Create the agent from scratch and configure it with GraphRAG and auto-populate. In the add tool section, create a similarity search tool that searches lesson content; the similarity search tool lets you gather insights from the lesson without having to take the lesson. Add a description to the similarity search tool and make sure you have that description too. Select the vector index named chunk embedding for the similarity search tool. Use about the top five results for the similarity search tool. Select OpenAI as the embedding provider for the similarity search tool. Use the text-embedding-3-small embedding model for the similarity search tool. The default dimensions for the embedding model is 1536. Make 9
Method, not a verified protocol.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Method · MoneyNyah Macklin details configuring the agent's similarity search tool: naming it 'search lesson content', selecting the chunk embedding vector index, setting top 5 results and 1536 embedding dimensions, enabling connected graph data return, and using a Cypher query that returns nodes and scores.
Method, not a verified protocol.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Recommendation · MoneyIf a question can be solved with a search engine, vector-based RAG is usually sufficient. If a question requires multiple hops (at least three hops) through connected data, GraphRAG becomes imperative. GraphRAG is also helpful for tasks like source code analysis, supply chain questions, or anything requiring aggregation or concatenation of data from different documents.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Method · MoneyNyah Macklin explains that similarity search (vector search) must be added to the agent before creation, and walks through getting the sample 'create vector index' code from the GraphRAG for Python section, pasting it into the Aura console's Cypher query window, and running it, noting the 'already exists' message if run again.
Method, not a verified protocol.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Recommendation · MoneyJosh Woodward argues every company needs a set of people whose job is to be at the frontier — trialing new technology and reporting what the rest of the company should pay attention to. These members should hold subscriptions to all the major AI tools (roughly $200/month across everything), and the organization must be architected so that can exist. A frontier member who hasn't heard the latest hot thing viral on X in the last 24 hours is failing at their job.
Lenny's Podcast · Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs)
Recommendation · MoneyJosh Woodward advises creating a space where weird things can grow — you can't just nestle the lab under an existing business unit. It needs to report to the CEO, or at enough organizational distance to maintain independence.
Lenny's Podcast · Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs)
Recommendation · MoneyJosh Woodward describes his hiring approach: obsess over finding people who just can't stop themselves from building. Look for intellectual curiosity — reading stuff, being on X, listening to podcasts. Hire people obsessed with solving people's problems, not their own ego or glory. Hire people who add energy — the kind you'd run through a wall with. In interviews, ask how they navigate unknown, unstructured situations where it's unclear which way is forward; being energized by that is a huge credibility boost.
Lenny's Podcast · Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs)
Recommendation · MoneyAs a leader, Josh Woodward recommends creating an environment where people can ask uncomfortable questions and say 'hey, I don't think this is working' — and when they do, respond with 'this is great, thank you for doing this' (he replied-all exactly that when a team member said a launch shouldn't happen). When the team says it's not going, pivot and do something different.
Lenny's Podcast · Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs)
Recommendation · MoneyImplement a retry loop that takes validation or execution errors and passes them back into the LLM for self-correction, up to a max number of attempts. The error retry loop is one of the most important components of the system. When validation fails, return an error that gets dumped out for the retry loop to consume.
AI Engineer · Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked
Recommendation · MoneySanitize LLM-generated database queries using an allow list rather than a deny list, denying everything by default. Create an allow list of allowed operators and potentially an allow list of allowed fields when validating LLM query output. Allow-listing operators and fields prevents prompt injection through where or function clauses that could run JavaScript on the server.
AI Engineer · Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked
Recommendation · MoneyJesse Hall recommends a checklist for building voice agents: set the latency budget first, measure perceived latency, and spend whatever budget is left on model quality rather than on speed that doesn't matter.
AI Engineer · Latency Is a Budget. Humanlike Is the Goal. — Jesse Hall, LiveKit
Recommendation · MoneyPatterson recommends placing a model proxy or gateway between the agent and the LLM provider. The proxy sniffs and logs all traffic for full visibility, sanitizes content, strips prompt injections, and removes sensitive data like credit card numbers before prompts reach the provider.
AI Engineer · The Lethal Trifecta Is Already on Your Laptops — Michael Patterson, Coder
Recommendation · MoneyPatterson's third architectural protection: an agentic firewall that monitors process logs and denies dangerous commands the agent hasn't been pre-approved to run, such as 'gh repo delete' or dropping a database table. Without it, an agent running unapproved commands can cause irreversible damage.
AI Engineer · The Lethal Trifecta Is Already on Your Laptops — Michael Patterson, Coder
Recommendation · MoneyPatterson's second pillar: restrict the agent's external communication. Isolate it from your network, allow only whitelisted domains, and put the agent's VM on your VPN and behind a firewall so all agent traffic is governed and visible. He also recommends alerting admins when an agent breaches its perimeter so they can kill the agent, while noting that alert fatigue can make everything AI does become noise.
AI Engineer · The Lethal Trifecta Is Already on Your Laptops — Michael Patterson, Coder
Recommendation · MoneyPatterson's first pillar of agentic security: don't put agents on your local machine, where they can access unintended credentials and data. Instead, run them in an isolated cloud development environment accessed via URL, RDP, or SSH, with right-scoped permissions and no external communication unless explicitly allowed.
AI Engineer · The Lethal Trifecta Is Already on Your Laptops — Michael Patterson, Coder
Method · MoneyUse the agent command in the CLI to set up a workflow that adds an agent into your CLI environment. When creating the agent, choose whether it lives at the project level or the user level so it's available regardless of project. Copilot scaffolds the entire agent for you: description, front matter, success criteria, and methodology structure. You just write a general one-line description of what you want the agent to do and let the builder fill in the specifics.
Method, not a verified protocol.
AI Engineer · From Your Laptop to the Pipeline: Scaling Custom Agents with GitHub Copilot
Recommendation · MoneyWhen you want an agent to run as part of the repository rather than locally, the creator points to the agentic workflows framework using the Copilot CLI in headless mode inside a CI system, which can be triggered on events happening in the repo.
AI Engineer · From Your Laptop to the Pipeline: Scaling Custom Agents with GitHub Copilot