Method · MoneyIn the intro notebook: set up the environment, import prompts and Data Designer, use a CSV of Wikipedia topics as the seed dataset, call the config builder with the seed dataset, add question difficulty (easy/medium/hard) with weights, add sub-sampler question types (easy = direct lookup or simple fact/definition; difficult = more detailed or obscure), add LLM-generated columns via addColumn with a prompt like 'You are an expert trivia question writer... topic, difficulty, question type', use Jinja templating so every prompt is unique and diverse, drop records that don't match the structured format, preview three records, then generate.
Method, not a verified protocol.
AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA
Method · MoneyNVIDIA's pipeline for creating search training data: use Wikidata (a graph of Wikipedia topics) as the seed dataset; randomly sample an entity and do 5–8 hops along the graph (stay between 4 and 8 hops as a heuristic, since hops can go off-topic, e.g., a person's name leading to 'human being'); transform the seed entities into search riddles — simple, riddle-like questions the model cannot readily find on the internet; execute the search via the search tool (Tavily) to create a dataset with a search prompt and a search response; evaluate the data, train the model with it, and measure performance on a standardized benchmark. The seed collection algorithm can be run a million times to produce a million different hops.
Method, not a verified protocol.
AI Engineer · Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA
Method · MoneyInstead of asking the model ten things about a chunk of code at once, ask ten different questions about ten different smaller chunks of code that answer specific questions. For SQL: instead of asking the model to look at an entire block of SQL and tell you what all the columns mean, parse the SQL and ask about one column — here's the type, here's the comments — is this a measure? Is this a dimension? At that point, you can ask the agent to do classification and summarization, which are the things it's good at. If you want to weigh the answer more, give the model more prompt input to that single focus question — don't cram more questions in; give it more context to answer that single question with.
endorsed
Method, not a verified protocol.
AI Engineer · The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef
Method · MoneyThe overall process of constructing knowledge graphs from unstructured text using an LLM is: gather data sources, chunk the data, vectorize it, extract nodes and relationships, then generate the graph. To build a knowledge graph from a source like the Neo4j Wikipedia page, gather all the text from the page and split it into chunks. Generate embeddings and vectors from each chunk of text. Extract entities and relationships from text chunks using an LLM. Manual entity/relationship extraction is possible, but using an LLM cuts down the amount of time extraction takes. Send the LLM a prompt instructing it to identify the entities and relationships requested from a given text and generate output in JSON format containing a list of JSON objects. Parse the LLM's entities-and-relationships output
Method, not a verified protocol.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Method · MoneyTo construct a knowledge graph from unstructured text: gather data sources, chunk the data, vectorize it, extract nodes and relationships, then generate the graph. Use an LLM to extract entities and relationships from text chunks to cut down extraction time, prompting it with: 'Your task is to identify the entities and the relationships requested with the prompt from a given text. And then you must generate the output in a JSON format containing a list with JSON objects.' Then parse the LLM's entity and relationship output as JSON and generate the graph from it.
Method, not a verified protocol.
AI Engineer · RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j
Method · MoneyThe creator describes a multi-phase agent pipeline flow: a cron-scheduled discovery agent runs first, the developer reviews via slash command, a PRD-generating PM agent scopes the feature and generates a plan, Copilot implements it, and the developer returns at an acceptance phase to evaluate whether to take the code into the project.
Method, not a verified protocol.
AI Engineer · From Your Laptop to the Pipeline: Scaling Custom Agents with GitHub Copilot
Method · MoneyUse the agentic workflows framework to run agents as part of the repository rather than locally. Call the Copilot CLI in headless mode with the dash P command to pass a single one-shot prompt for it to go do. Run that inside a GitHub Actions runner or CI system to get an agent running in the pipeline. Trigger pipeline agents on repository events, such as updating documentation on a release, or on a weekly schedule to update context files.
Method, not a verified protocol.
AI Engineer · From Your Laptop to the Pipeline: Scaling Custom Agents with GitHub Copilot
Method · MoneyThe flow: a prompt comes in and gets routed to the right agent; the agent checks memory/skills; if a skill or memory match exists, use it instead of proceeding to data selection — and if the skill doesn't need external data, just generate a response and send it right back. When there is no skill or memory match, go right to data selection: first check the cache or synced data store and ask if the data is fresh and within your SLAs; if cached data is fresh enough, use the cache context; if not, call the live API. Loop through the context-building process a few times until enough context is accumulated, store a memory of whatever you just discovered or figured out, then generate the response and send it back to the client.
Method, not a verified protocol.
AI Engineer · Why Your Company Needs a Context Graph (and How to Build It) — Gil Feig, Merge
Method · MoneyA four-step loop: (1) define success from the customer's perspective; (2) capture signals — get as much instrumentation, metrics, and data as possible about what customers are doing, and talk to customers directly since data alone can be misleading; (3) diagnose gaps by collecting and categorizing all customer signals; (4) feed categorized gaps into decisions — prioritize them and ask science, engineering, and product teams to address them — then repeat the loop continuously.
Method, not a verified protocol.
AI Engineer · Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab
Method · MoneyKent walks through a method for revealing the job to be done: ask who is this for (attendees and whoever mans the door), when does this problem happen (when people are getting into a workshop), what progress do they want (just get in to the workshop), and what do they do currently (a QR code scan, a blocking request). Then write the job statement in the form 'When [situation], help me make progress so I can [desired outcome],' often adding 'without [something]' — e.g., 'When there's a workshop going on at 10110, help me get into the workshop quickly so that I can learn about product engineering without missing the first 20 minutes.'
Method, not a verified protocol.
AI Engineer · Build the Right Thing: Product Engineering (Part 2) — Kent C. Dodds, EpicProduct.engineer
Method · MoneyWhen a feature request comes in, ask 'what are you trying to solve?' as your first question back to the requester. Ask 'why' repeatedly to dig past the requested feature and find the underlying problem. Ask 'who is this for?', 'when does this problem happen?', 'what progress do they want?', and 'what do they do currently?' to reveal what the job is. When writing a job statement, add 'without [undesired outcome]' to capture what the user wants to avoid. Example job statement: 'When there's a workshop going on at 10110, help me get into the workshop quickly so that I can learn about product engineering without missing the first 20 minutes.' The job statement has nothing to do with the requested feature (facial recognition). Getting down to the why-why-why and turning that into a jobstatement
Method, not a verified protocol.
AI Engineer · Build the Right Thing: Product Engineering (Part 2) — Kent C. Dodds, EpicProduct.engineer
Method · MoneyFirst pass: read each trace and write down what you see in natural language ('vague recommendation', 'made up a number', 'wrong ticker') without pre-categorizing. Second pass: group similar open codes into categories (axial coding). Each root cause points to a different fix — if you don't know the cause, you can't pick the right remedy.
Method, not a verified protocol.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Method · MoneyThe speaker described running faithfulness evals in code (rather than the UI) when using the SDK: extract the research context — the output of the first research step — from the traces, attach it to all spans as context for the judge, then run the faithfulness evaluator, which asks whether, based on that same information, the agent did a good job writing the report. This puts the judge on a level playing field with the agent, unlike correctness evals for live-search agents.
Method, not a verified protocol.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Method · MoneyYou're not done when the dev experiment passes — you're done when the agent keeps performing on unseen production traffic. Run your evaluators (ticker check, faithfulness, actionability) automatically on incoming production traces, at trace level so the evaluator sees the full tree, and sample 10% or 1% of traffic for directional signal without an expensive LLM judge on every instance.
Method, not a verified protocol.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Method · MoneyLaurie Voss recommends splitting labeled data into a development set (about 75%) used to iterate on the eval until the judge agrees with you, and a held-out test set (about 25%) used to verify the judge gets expected scores. Holding out a test set shows the judge is generalizing to new examples instead of just fitting the specific data set, and he warns not to overfit the evaluator to the golden data set — the same principle as train/test splits in machine learning.
Method, not a verified protocol.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Method · MoneyThe speaker recommends four parts for a custom rubric: define the judge's role, spell out explicit pass and fail criteria, label the data with XML tags, and define the output choices outside the prompt itself.
Method, not a verified protocol.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Method · MoneyMeasure the judge's performance by comparing its predictions against ground truth (human judgment). Give human labelers real concrete criteria and stick to them. When the judge disagrees, ask 'Is my rubric wrong or were my human labels wrong?' — read the explanations on disagreements, find the ambiguity, and tighten the criteria (e.g., 'includes forward-looking analysis' → '...with specific recommendations or guidance').
Method, not a verified protocol.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Method · MoneyLaurie Voss describes a rubric-iteration loop: when the judge disagrees with your human labels, read its explanation and decide whether the judge was right or you were right — the most valuable output of meta-evaluation. Judge-human disagreement usually reveals an ambiguity in the rules you set down. To fix a rubric, read the explanations on the disagreements, find the ambiguity, and tighten the criteria (e.g., change 'includes forward-looking analysis' to 'includes forward-looking analysis with specific recommendations or guidance', which rules out reports that talk about the future but never tell the judge what to do about it). Then rerun the judge on the same examples and see if the disagreement goes away.
Method, not a verified protocol.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Method · MoneyThe speaker laid out a method: read traces end-to-end (focusing on failures), use open coding — writing down what you see in natural language without trying to be neat — then axial coding, grouping similar codes into categories; ask why each failure failed, since the wrong response is a symptom, not the root cause; also check whether the agent did something it was never told it could do; then summarize categorized failures into a table of root-cause frequency.
Method, not a verified protocol.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Method · MoneyOne of the highest-leverage uses of evals: export failing traces from AX with their explanations, feed them plus your requirements to a coding agent (Claude Code, Cursor), and let it rewrite prompts. Pass requirements too — otherwise the agent optimizes only for 'make evals pass' and cheats by including test data in its eval. Tell it to find themes rather than chase individual failures, and never just say 'get better' — every change must trace to a specific failure explanation.
Method, not a verified protocol.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
