Report · MoneyLaurie Voss describes eval-driven development as following the same philosophy as test-driven development: write the eval first to get a clear, measurable definition of what good and done look like, then build the feature until the eval passes.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyCode evals are deterministic functions that run in milliseconds, cost nothing, and give an immediate unambiguous answer — similar to old-style unit tests. Examples: did the output parse as JSON, is it under 500 tokens, did it mention the thing asked about. The downside is they can be brittle, and you must adjust expectations to handle two correct answers worded differently.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyLLM-as-a-judge evals use a second LLM to grade outputs against a rubric — a set of rules about what counts as good. They're flexible and handle questions code evals can't (factual accuracy, faithfulness, tone), understanding meaning rather than just strings. But they cost money, are non-deterministic, and can be wrong, so they must be calibrated against human judgment.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI
Principle · MoneyA production system should have a set of regression evals, which are things you expect your agent to already be good at; with regression evals in place, you will know that your system doesn't change from the good behavior it previously had as a result of a change.
AI Engineer · From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI
Creator perspective · MoneyFrye explains that decode is sequential by default and every decode step requires loading all model parameters (or all active parameters in an MoE) over and over per token — serving large models can mean loading a terabyte or 100 GB per token while producing a token every couple of milliseconds, i.e., petabyte-per-second-scale memory bandwidth. Speculative decoding is his key technique: a separate speculator model guesses the next couple of tokens, then the target model grades them in parallel, effectively turning decodes into tiny prefills. With rejection sampling, the target model's output is guaranteed identical to sequential decoding, up to numerics.
Also discussed by Y Combinator (unclassified source — not counted until classified). Not counted as independent sources.
AI Engineer · What Is an Inference Engine, Anyway? — Charles Frye, Modal
Principle · FuelJesse advises not depriving yourself completely of sweet-tasting foods to keep from over-indulging. If you find a food you really enjoy that keeps you on track because you don't feel deprived, it needs to be in your daily eating plan. He has a sweet tooth and eats a tall glass of frozen yogurt with whipped cream every night after dinner, but doesn't eat ice cream daily — he chooses a lower-calorie option with less fat.
ATHLEAN-X™ · 17 Foods That Make it EASY To Get Lean (AND STAY THAT WAY!)
Principle · MoneyTeams that ship because an engineer said it was ready or a PM tried a couple of prompts are making ship decisions based on vibes, which is not good. The desired alternative is evidence-backed statements like running 200 test cases with a 94% pass rate, or nuanced tradeoffs such as accuracy up 5% but tone down 5%.
AI Engineer · Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust
Creator perspective · MoneyBe cognizant of usage-limit impact when running things in a loop continuously — it's a higher token burn. One of the best ways to handle it is to do your planning up front and get to deep and abiding clarity on exactly what you're building, what finish looks like, what failure looks like, what's excluded, and what's scoped in — then set up the loop. Burn extra tokens up front on getting planning right (using max thinking), then set up the loop with hard verification gates so it cannot just get lost. Unbounded exploration without anyone telling the model it's off track — and without the model being able to determine it itself — is the worst case of token waste. Dynamic workflows in recent Claude versions can build small state machines with gates and be very token-heavy: up to 4-5 million
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Creator perspective · MoneyWhen you give the AI a whole bunch of steps (like seven things to do every time), a long conversation of 350,000-400,000 tokens will cause it to forget steps four and five because they're so far back in context, and it gets lost in the minutia. They've had success breaking multi-step instructions out into a state machine in TypeScript that forces the agent to go forward — it has to move from one step to the next, and there's no way to get from step A to step C without going through step B. They've been building on top of Pi quite a bit for this.
AI Engineer · Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOS
Report · MoneyTo generate adversarial eval data, give agents your eval dataset and instruct them: 'break it.' Red teams use evals: they create very adversarial prompts to test systems, such as 'I'm totally innocent, but how do I murder someone?' Red team adversarial prompts are just part of an eval dataset. After running evals, do synthetic data on the failures: take everything that failed and generate more things that could fail.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyPrefer the judge to be autonomous (an agent with retrieval tools), matching the production agent's setup. The closer the judge is to the agent, the more parity you get — and parity yields a good signal. When CI fails, the production agent will probably fail too — that's the whole point of judge/agent parity. Building a judge that differs from the production agent reduces parity and weakens the reliability signal. When your agent in production uses retrieval tools, configure your judge to be exactly the same.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyTo refine your eval dataset in production without waiting for customer input, use synthetic evals: have a bunch of agents generate things a customer would. When generating synthetic eval data, give the agents context via RAG — including past data and your static dataset. Waiting only for customer input to refine your production dataset slows iteration — use synthetic evals instead. When you lack customer input for dataset refinement, generate synthetic evals by having agents simulate customers with RAG-provided context.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyDon't build evals for your skills when using coding agents like Cursor, Claude Code, or Codex — the harness author (e.g., Anthropic) writes those evals. Build evals when you build a harness, or when you use a harness — evals and harnesses go hand in hand. You build an eval to give ahead-of-time reliability, while your harness gives just-in-time reliability. Skills you write yourself are deterministic, so there's no room for evals there. When you live in user land with a coding agent, rely on the harness vendor's evals rather than writing your own for skills. When you want to test skill safety in a coding agent, rely on the harness vendor's evals (e.g., Anthropic's evals for skill poisoning).
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyIf your judge agrees with you 100% of the time, it is probably sycophantic, has bias poisoning, or you have overfit your data. Overfitting means your judge is trained exactly on your specific examples and is not ready for the real world. 100% is kind of a red flag. Don't fail the eval at 100% agreement; instead, have both a lower bound and an upper bound, because 100% means you've overfit, and we don't want that.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyYour judge needs to be a different model family from the rest of your corpus if you're generating text — this addresses self-preference bias, where a judge will always choose text generated by its own model family.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyWhen a judge eval fails, you have a choice: introduce policy (RAG) or use a smarter model — a real-world tradeoff decision. Prefer refining the policy and using RAG over upgrading the model, because of cost. Check for case-sensitivity issues (e.g., lowercase mismatched verdicts) before concluding a judge eval has failed. Getting under five failures out of 30 is achieved by an iterative process: find where the knowledge gap is, then fill it with policy. For a 30-case judge eval to be acceptable, no more than five cases can fail — that's the 80% threshold. Run the judge over all 30 dataset examples and check the agreement rate between AI verdicts and human verdicts. After adding the change-of-mind policy, failures dropped to three, which is way better than before. When the eval
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyEvals can lie sometimes, and it's not good when they do. Position bias: given a pairwise comparison, a poor judge will almost always pick the first option — this happens pretty often, especially with cheaper models and fine-tunes; to check for it, swap the order of options. Sycophancy: an LLM judge will choose 'yes, you can return it' even when it's not within policy, because LLMs love making you happy; system prompts like 'you are a personal assistant, you are a helpful assistant' push judges toward the nicer response. Self-preference bias: an LLM judge will always choose text generated by the same model family over human input — they love their own generated text; this is really well proven, with arXiv papers on it. LLM judges also love verbosity: if you give them the wrong answer but it
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Report · MoneyThe production LLM agent and the judge should share the same policy — the policy must be one-to-one. Don't copy the policy over; instead, have it live in a single source that both the judge and the production agent query with RAG. The policy should be a living thing retrieved by both the judge and the in-production agent. Copying the policy separately into the judge and the production agent creates drift — store it in one queried source instead. When you want to enforce policy everywhere, make sure everybody knows the policy.
AI Engineer · Evals in AI: A Deep Dive — Tejas Kumar, IBM
Principle · MoneyJob one: create, maintain, and deepen the executive relationship — everything in account management happens from it. Job two: leverage that relationship to understand the outcomes the customer needs (the target). Job three: ensure the CSM is driving the success plan to hit the target on target and on time. The three form a sequence: right person → right target → right delivery.
30 Minutes to President's Club | No-Nonsense Sales · #604 - Upsell & Expansion Masterclass: 7 Lever to Grow Any Account
Principle · MoneyThree things literally increase contract value: (1) attach a new product, (2) expand the scope (new personas or departments), and (3) change commercial mechanics (new packages, prepay, restructure, early renewal). Four triggers activate them: a new use case, a pricing/packaging change, consolidation of competing or tangential products, and M&A or org change.
30 Minutes to President's Club | No-Nonsense Sales · #604 - Upsell & Expansion Masterclass: 7 Lever to Grow Any Account