Sell the outcome, not the tool — the core principle of agentic SaaS
Sell the outcome, not the tool — the core principle of agentic SaaS.
42,308 results · Most relevant
Sell the outcome, not the tool — the core principle of agentic SaaS.
The main strategic move in the AI era is to become the worker — start doing the work your customers currently do with your tool, rather than just selling a tool. A large portion of B2B SaaS follows the model of selling a tool that a business owner hires an employee to operate in order to produce an outcome.
Al Chen lays out that deploying on-prem AI at an air-gapped bank involves a more complicated flow, organized around three pillars: (1) the build pipeline — how banks pull images from the vendor's software into their cluster; (2) data resilience — the plan if a region fails mid-request; and (3) ensuring no sensitive information leaks out to internal bank employees or outside the bank's VPC.
described
Al Chen identifies signals that shorten time to production — having a lot of processes set up for how you pull images down from a registry, and having GPUs sourced and ready to go for running inference. Timelines get pushed out to a few months with formal security reviews, a custom LLM gateway, or RBAC configuration, and pushed even further out when there's no approval process for pulling images down to the registry or no story around sourcing and procuring GPUs. Most signals that push timelines out are less technical and more about internal human and organizational challenges and alignment.
described
Al Chen challenges the audience to look at the system prompts for their LLM judges and check the traces to see if the traces, with the judges' reasoning, contain any PII or MNPI. A system prompt for a simple judge outputs a score and explanation of how well the agent performed; a seemingly innocuous line asking the human reviewer to be able to verify the agent's work may have allowed the judge to leak unredacted PII or MNPI. Your UI can still look good while on the back end the judge's reasoning unredacts customer information.
described
Al Chen explains that compliance velocity and CVEs go hand in hand: any serious platform has a plan for addressing common vulnerabilities and exposures as the first compliance hurdle. A single CVE can be researched on the NVD, CISA, or MITRE websites, but in reality a platform has multiple images and each image has multiple CVEs, so vendors serving multiple banks must create a system and process around CVE management.
described
In regulated spaces, define hard limits — things that definitely shouldn't happen (e.g., a financial services agent should never give investment advice) — and ingrain them in the evaluation set. Even if a violation is never fully prevented, you will at least know about it when it happens.
Define functions for LLM tool calling using JSON schema, including name, description, and typed parameters. Writing that schema — types, descriptions, parameter names — is prompt engineering: the LLM decides how to call the function based on that information.
When evaluating tool calling, check whether the model calls the right tool and passes in the arguments you expect, across all the different inputs. If a model calls a tool with wrong arguments, that means there's ambiguity in your function description — go back and clarify it with explicit constraints (e.g., 'don't include me or you, just named attendees, first and last if provided').
Build agents by having the model call tools in a loop until it reaches a goal. When evaluating models with tool calling in a loop, test their efficiency — can they accomplish the same goal with fewer tool calls? Count the number of tool calls needed to reach the final answer, and look at both the final answer and how long it took to get there, since every tool call is another network request adding latency.
Backtesting matters because if you know the existing evaluation set, you can overfit it. Overfitting is very dangerous because an overfit algorithm is unlikely to be successful in the future. You need to do backtesting with a holdout set. Because you have backtesting capability, you can check for overfitting by going back into your data and constraining the companies you know something happened to (e.g., 10 companies) against 20 companies that didn't, then train and iterate on your model, and make sure it continues to perform beyond just the publicly visible sets.
Commercial context: The creator describes this as their own product.
Larry and Jerry are obsessives and purists who wanted all the flexibility and pacing of editing a movie while also getting the benefit of a live audience telling them if the joke is landing — so they invented a novel production process to get the best of both worlds. Seinfeld episodes were not live camera switched; all the real work was done in post-production editing, which allows much tighter cuts — shorter scenes, more scenes, more situational jokes per episode — and lets you tell jokes you couldn't otherwise do, like The Contest, where the joke is cutting between characters in bed, restless or sleeping like a baby.
described
In The Busboy, Larry David introduced interweaving storylines that all intersect and resolve at the end in a grand finale; Seinfeld gave each of the four characters their own storyline, creating four concurrent storylines per episode, which forces extremely efficient storytelling — no wasted words, no wasted minutes.
described
Nick Nisi describes relying on CI and the tests running in CI to validate agent work before shipping, plus automated code review tools — his team uses Greptile, and he names CodeRabbit and Codo as alternatives. He treats a 5-out-of-5 Greptile score plus all CI checks passing as the signal that a change is safe to ship, giving him extreme confidence he won't break production. At the end of the day he reviews every lane/branch started that day and confirms each is correctly done.
Zack Proser recommends getting to the point where your changes and your PRs automatically flex that eval, so you know if changes are safe to land, whether they're improving things, making things worse, or not changing anything at all.
Nick Nisi advises thinking about how you would reproduce a given bug and, if you can break that down, encoding it for the agent. As a concrete example, for sign-on flows you verify the cookie is properly set, the expiration is set, and the claims in the JWT are correct. He also notes that error messages matter more than ever: when the app throws an error, an agent can interpret and run with that error really well, so poor error messages block self-healing — and observable state (like ball coordinates in a canvas game) should be surfaced to the model so it can debug what it cannot see with its own eyes.
Nick Nisi walks through the possible failure outcomes of the verification game: the agent names the bug with output but has no evidence, claims it works when it didn't, claims it works when it did, or blames a bug that wasn't there. He highlights the 'overpromised' case where a screenshot or judge check reveals the agent claimed a pass that didn't happen, and the case where code lies — after a page reload the app says 'waitlisted' while the screenshot shows the seat confirmed, and the mismatch is itself a failure. He also cautions that a successful screenshot is not proof: it may show a state you didn't expect, cause side effects, or take a UI path that isn't actually possible.
Wisedocs' system is set up so that a human always makes the final decision, with AI surfacing insights for human review rather than acting autonomously.
Dick explains that OAuth 2 added scopes, but before that it was just 'do anything', and it required preregistration and expected a browser with a redirect to be involved. In the agentic world, you don't know ahead of time what the agent needs to do; the authorization decision must be made over time, and different agent instances need different kinds of access. Related framing: making a payment is not a scope; it is whether you can do this one thing in this particular context.
None beyond the speaker's own framing.
Laurie Voss suggests that if you don't have users, you can make some up: get an LLM to generate lots of test queries and vary them on purpose — e.g., 'Research Tesla's financial performance', 'what's going on with Tesla stock', and 'yo, is Tesla buy right now'. Then get a human to look at the LLM-generated query list, because LLM-generated queries are a useful way of seeding your initial data but are not perfect. LLM-generated queries are relatively conservative and don't go as far afield as human users tend to actually go — human users send wildly off-domain queries, like asking a financial analyst agent 'tell me what's wrong with my child's health', and your agent has to have something to do with that.
Choose sources → to get a library of your own and a daily brief.