Creator perspective · MoneyUse Git Worktrees to give each independent agent an isolated, safe-to-work-concurrently copy of the repo; run 'claude --worktree' from inside your repo to create a worktree and start a Claude instance in it, enabling immediate parallelization. Break a full afternoon (5-6 hours) of development into discrete blocks assigned to run concurrently, pressing time down to 45 minutes or an hour. Caveat: running multiple instances of a large monorepo in different worktrees is difficult because of gigabytes of node_modules. Alternatively, use cloud environments (Claude Tag, Devin, others) to parallelize work in the same codebase without conflicts, opening independent PRs that never bleed changes into each other.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Report · MoneyThe safest way to put evals in production is to make production an agreement gate: run evals on CI and don't ship if the judge agrees with you less than the 80% threshold. CI should be red on less than 80% agreement and green on 80–90% agreement. A failing eval test will block CI because it is red, and ideally you want no failing tests. Modify the eval test to only fail if human-AI agreement drops below 80%, otherwise pass, and console log the agreement percentage. When you ship the eval to production, it becomes the quality gate that runs on CI. The eval set provides ahead-of-time reliability; it will never really fail, and if it does, a harness can pick up the slack at runtime.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyA well-defined eval suite has: (1) test cases — a set of inputs and outputs, such as customer requests to a chatbot and generated answers, against which you can create a verdict; (2) expected behavior — the answers you expect, which is not an exact string but the direction you wanted the output to go; (3) a judge — somebody to say yes, that's right, which is harder in non-deterministic systems because it's subjective; and (4) a way to aggregate all results and come up with a probability score, e.g. 'this is 80% the right direction, therefore it's good.' Almost every eval set in production today of any quality has these components. Eval components come together to answer: given a scenario, did the agent follow policy? Did the agent use the right tools? Did the agent ultimately resolve the (
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyTo train a judge (not ML training): you have an LLM you want to use as your judge, and you make sure it agrees with you using a dataset. Step one is to have a long list of data — question, answer, score triples — where the score is one that you or a subject matter expert human has scored, a trustworthy score. Start with about 30 examples, then let it grow. To train the judge, remove your scoring from the dataset, keep only the question and answer, send that to the judge, and let it produce its own scores. Compute the delta between your scores and the judge's scores to find your percentage of agreement. Aim for 80–85% agreement, maybe a bit more. Under 80% you won't get much value, because even a team of human support agents only agree about 80% of the time. If your judge agrees 100% of the
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyOnce in production, there is a high chance your eval data will go stale, so you need to actively stay relevant. To stay relevant, skim real traffic: collect new user inputs and your agent's outputs, have an internal team score them manually, and check the judge's agreement against those scores. Your eval data set should be a living data set — continuously updated with fresh scored traffic. A living data set with an up-to-date judge is very difficult to break and provides reliability ahead of time. After skimming public traffic, you will have even more data to refine your evals. In production, skim the top of the data, get a little more data, and refine and refine.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyTo keep evals cheap, start with actual deterministic tests (unit tests). If deterministic tests can't be done, the next level up is regular expressions or pattern matching. If pattern matching fails, the next thing to do is examine your traces. Ideally, everyone is using some type of observability solution and has traces of each message envelope, inputs, outputs, tool calls, etc., which can be validated per object. Eventually you will come to a place where you need to use a judge, and then you start with the cheapest model. Evals are afforded strategically, because it will not be cheap if you use the most expensive model, and you want to build up to it. Regular expressions can be used for testing AI outputs but have their limits.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyWhen humans and AI don't agree, it's usually because of a knowledge gap — and the way to reach agreement is to introduce context to bridge that gap. To improve judge agreement, examine the cases where you disagreed and fill the knowledge gap with policy. In production you would supply policy to the judge via a RAG pipeline that retrieves enterprise context; in a demo you can just hard-code the policy. Write the policy explicitly (e.g., a 14-day return window; defective items can be returned) and interpolate it into the judge prompt. Building your judge is arduous iterative work — investigating failures and filling knowledge gaps is literally the job. After adding policy, check whether agreement is growing even if it still fails — improvement indicates you're on the right track. When the
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Principle · MoneyThe minute the account is yours, engage right away so the value loop starts turning — the AM not being involved in implementation is a mistake. At handoff, verify (don't assume) the executive outcomes the AE used to win the deal, grab them, keep the CSM on target to them, and deliver those as the first set of value; that proof is what lets the next outcome happen.
30 Minutes to President's Club | No-Nonsense Sales · #604 - Upsell & Expansion Masterclass: 7 Lever to Grow Any Account
Method · MoneyThe process the Instagram team followed was to ask a very clear root cause of why you were not able to do the thing that was the goal — the inverse of asking why you did the thing. In the Instagram Stories root-cause exercise, people answered the why-not question, and when the answers were grouped by theme, the top theme emerged. When analyzing why users did a desired action, ask 'why did you do the thing' — the inverse of asking why they didn't. Group open-ended user answers by theme and sort them to find the top root cause. Follow qualitative root-cause interviews with a large quantitative survey of thousands of people to quantify the weights across themes.
Method, not a verified protocol.
Lenny's Podcast · What it takes to be a top PM today | Robby Stein (Google Search)
Method · MoneySagar Batchu describes deciding in real time whether a call succeeds or fails as three parts. R (credentials) is the ceiling of what a human put down when they issued the key — it never gets widened, lives on the credential, and can't be escalated past. A (the agent's own policy) can't hold admin privilege or any human-only scope. O (the owner's live policy) means an agent can never do more than the person accountable for who's actually driving the agent. A and O are read on every call, so removing a permission takes effect on the next request. Applied to the database scenario: an agent scoped to staging can't delete production even if the owner could, and an agent can't exceed an owner who never had production rights — either way the call stops. And if an owner leaves the company, the IDP
Method, not a verified protocol.
AI Engineer · A Valid Key Let an AI Agent Delete Production - Sagar Batchu, Speakeasy
Method · MoneySagar Batchu recommends modeling a governance architecture around four steps. Connect: create a door instead of dozens of integration points — all agent calls to external systems go to one place, like an MCP gateway. Resolve: calls are resolved to a named agent or human behind it, since policy evaluation doesn't work without identity, and the identity must be the same one all other enterprise software is assigned to. Decide: allow, deny, quarantine, or escalate — sometimes just warning a user is sufficient. Record: the decision is recorded before any action is actually taken.
Method, not a verified protocol.
AI Engineer · A Valid Key Let an AI Agent Delete Production - Sagar Batchu, Speakeasy
Method · MoneyKarri Saarinen describes a feature roast practice: attendees critique the whole feature with raw, non-personal feedback ('you can be as nitpicky or confused as you want'), framed as 'This is how I see it' rather than a personal attack. The building team can interact with attendees and ask questions when they don't understand the feedback. Afterwards, the lead synthesizes the feedback, groups it, and turns it into actionable issues to fix.
Method, not a verified protocol.
Lenny's Podcast · Context is now the product: Product leadership when software can build itself | Karri Saarinen
Method · MoneyA multi-step workflow: kick off a project with a Slack command, create a Hilltop PRD, have an agent break it into units of work, review and approve via Linear tickets, and the agent starts work automatically. Humans stay in the loop, and the agent reevaluates the project to keep the plan fresh.
Method, not a verified protocol.
AI Engineer · No, That's Not a Software Factory — Ryan Cooke, WorkOS
Method · MoneyA step-by-step method for making existing components agent-ready: audit and refine current components, start with meaningful layer names, check how components resize, and set up component properties for each component.
Method, not a verified protocol.
Figma · A step-by-step guide to an agent-ready design system
Method · MoneyWhen starting to optimize an AI system with RSI, begin with prompt updates because they are the easiest and most straightforward way to keep iterating. After prompt optimization, the next step (L1 ladder) is to optimize the structural context of the AI system. For the L2 ladder, expand scope to automatically design the entire workflow, a kind of meta information design. After workflow design, the next step is to optimize the harness code itself.
Method, not a verified protocol.
AI Engineer · An Agent That Fixes Its Own Mistakes - Saloni Gandhi & Ziwei Guo, LinkedIn
Method · MoneyThe RSI process is broken into seven steps. (1) Run the agent and generate samples: given a query, generate output and log the processes in traces (e.g., LangSmith traces) — these trajectories are what you optimize. (2) Run the traces through the evaluation engine (e.g., the LLM judge) to get judge output showing where the system is good and where it is not. (3) Agent reflection / root-causing: understand the root cause of bad output — agent, model, prompt, tool, or other issue; the root cause can even be a judge issue or rubric issue. (4) Based on the root cause, generate an edits proposal — currently just prompts, but in future a new workflow design, harness code, or even optimizer code. (5) Validate the edits, since some are good and some bad, using the same evaluation engine (samejudge
Method, not a verified protocol.
AI Engineer · An Agent That Fixes Its Own Mistakes - Saloni Gandhi & Ziwei Guo, LinkedIn
Method · MoneyRun the agent and generate samples logged in traces; run the traces through the evaluation engine (LLM judge) to see where the system is good and not; perform agent reflection (root causing) on judge output; generate an edits proposal based on the root cause; validate the edits using the same evaluation engine (same judge, same rubrics); after the prompt is updated, run evaluation again on a holdout set to confirm the new system performs better; then make the updates the new baseline and continue iterating at a good cadence.
Method, not a verified protocol.
AI Engineer · An Agent That Fixes Its Own Mistakes - Saloni Gandhi & Ziwei Guo, LinkedIn
Method · MoneyCluster the failing rows from production traces by failing dimension using pure clustering without any LLM calls; within each cluster run an LLM to diagnose each row and assign a root cause plus a tag; merge near-duplicate issues because LLM-generated root causes are free tags with slightly different wording; then group and rank canonical issues by frequency to identify major bottlenecks.
Method, not a verified protocol.
AI Engineer · An Agent That Fixes Its Own Mistakes - Saloni Gandhi & Ziwei Guo, LinkedIn
Method · MoneyCollect a bunch of production traces from your agent, get a human to annotate about 50 of those trajectories with very detailed feedback (e.g. this is a long response, this is a short response, this is a good response, this uses this terminology), use GEPA to optimize an LLM-as-a-judge prompt, then use that judge prompt to go back and optimize your agent and deploy it, forming a data flywheel for continuous improvement.
Method, not a verified protocol.
AI Engineer · Beating RL With Reflection: GEPA and Optimize Anything — Lakshya A. Agrawal, GEPA
Method · MoneyFor subjective tasks that are hard to evaluate, GEPA can learn evals for the task from production traces. Collect a bunch of production traces from your agent and have a human annotate about 50 of those trajectories with very detailed feedback — specifying things like response length preferences, quality judgments, and terminology usage. Once you have human annotations, use GEPA to optimize an LLM-as-a-judge prompt. Use the optimized judge prompt to go back and optimize your agent, then deploy it, creating a data flywheel of continual improvement. This eval-learning flywheel is a successful paradigm that some leading teams in production are already using.
Method, not a verified protocol.
AI Engineer · Beating RL With Reflection: GEPA and Optimize Anything — Lakshya A. Agrawal, GEPA