Roland Gavrilescu recommends optimizing for valued work per watt: first make sure the work you're generating is valuable, then make sure the economics makes sense — the difference in price is basically what people would switch away from Cloud Code to something you provide.
Roland Gavrilescu recommends that loops should be the way you distill systems into recipes: failure patterns should become judges and evals, repeated behavior should become skills and prompts, and user frustration should become extensions and memories to your harness.
Roland Gavrilescu says the calibration bit and the eval generation bit should be doable by agents to build. You need a human in the loop to confirm the approach and agree with the agent's judgment (e.g., whether to look for hidden gems rather than big tech employees) — but you don't need the human to actually build the evals; you need them to calibrate the evals. Agents should be the ones that take the taste of the maker and put it into code.
Roland Gavrilescu says you can have a good offline eval set, but the real test is when you go to prod — whether the end user agrees with your taste. He recommends building a product that emphasizes your taste, then making sure your users appreciate and value that taste, using A/B tests — or a multi-armed bandit scenario — to make sure that's the case.
Carson describes how Untangle runs its AI engineering org: humans at the top set priorities and talk to customers; manager agent sessions (named 'engineer manager') never write code and manage child builder agents; Devin is the cloud coding agent for all engineering work; the manager works from a Linear ticket board as the source of truth, with every ticket being one PR; each manager concurrently manages up to about 20 builder agents (varying 5–25), bottlenecked by your ability to talk to the manager and understand what's happening — pushing beyond ~25 builders per manager will quickly spin out of control; you never go into builder sessions directly, all management goes through the manager agent; the manager usually answers technical questions and unblocks small things; an automation (a 15
Carson describes the human-in-the-loop step where a human has to say 'ship it': the manager session lets you know a video is ready; you zoom in and understand what the PR is doing by reading a summary, then watch and engage with the video; watching the video is the part where you'll want to cut corners, but you have to watch it — it's the bottleneck you must engage in as a product owner; review videos on your laptop, not your phone, because you need to see them well enough; he recounts being on his phone before the talk, tempted to say 'ship it' without seeing the video well, and getting back to his machine instead.
Carson describes the second stage: pick the top five stack-ranked candidates and invite them to a paid project — a fixed fee for 16 hours of their time, with a seat on Devin; ask them to get a real PR ready for merge; have them sign an NDA and record all their video (using Loom, producing 16 hours of video per round); feed all the videos into an agent (Devin) and ask which candidate to pick; then pick your finalists. He notes he personally watched all 16 hours for the last round — brutal, but it gave him a good read on the people.
Carson describes the entry stage of Untangle's hiring: ask candidates to submit a ~45-minute whole-screen recording of themselves shipping a feature (a PR, typically) on an app they already maintain, with no instruction or guidance — you just want to see how good of an agent operator the person is; the submission is free (the company doesn't pay for it) and is basically how you get in the door; feed the videos and transcripts into a ranking agent that watches the video and transcript and stack ranks the people; if you have time you could watch all the videos yourself, but he finds a very intelligent model is pretty good at understanding how good a candidate is; some candidates mumble or don't think out loud so you can't tell what they're thinking, and some talk a lot, so evaluation signals
hedged
Roman Ugarte recommends giving each specific bot a name and a bot identity, and creating a bot per task or use case rather than a thread conversation or one-off job. He says the naming piece is a very important part of making the bot feel like a colleague, and discourages creating a new chat for each discrete unit of work, suggesting instead roles or swim lanes to group categories of work, the way a team groups work by role.
hedged
Roman Ugarte recommends treating AI bots like new human colleagues: give each bot its own laptop, credentials, and workspace instead of sharing yours, and give it access to the same tools you use to do your job. He argues that a bot with its own computer and the right tools can get a lot of the way there and do great, ambitious work.
Hassan describes the end-to-end pipeline they built: scrape public filings, timestamp extracted facts, filter to a pre-announcement cutoff date, use a recent model to translate the question into structured JSON, feed it into a cheap open-weight model that builds an auditable Excel financial model with explicit formulas and assumptions, run many parallel builds, discard failing runs, and evaluate predicted profit against actual profit from earnings reports.
Hassan El Mghari recommends that if you're trying to do any sort of forecasting, find a model whose training data ends before the event you are trying to forecast, so the model has an unbiased point of view.
Aamir Shakir recommends creating an embedding on the visual representation of a page (screenshot) instead of relying on OCR, which gets rid of the lossy information OCR introduces.
Jesse Hall recommends that when a caller asks a question requiring a backend lookup (transaction, payment status, dispute), the agent should acknowledge the delay verbally — saying something like 'give me a second, I'm looking that up' — and multitask by continuing the conversation while the tool runs in the background. When latency cannot be hidden, the agent should ask another question to keep the conversation moving.
Karthic Rao recommends designing each LLM call as a small, typed, schema-checked unit so the same one-line migration change applies to almost all of them.
Karthic Rao recommends pointing every model call at a privately hosted LLM as a configuration change rather than a rewrite, because each call is small, typed, and schema-checked; one line changes and the same answer comes back.
After the first batch of sales calls, if people have pull, you now have customers — time to deliver, start repeating, and figure out how to automate and systematize. Once a couple of customers say yes, do not just send them a login and pray: go deliver the product firsthand yourself. You will learn a ton from it, and passive onboarding causes you to miss critical learning opportunities.
The first step of defining an eval is defining what good means. Avoid vague criteria like 'helpful, accurate, good response' — the more specific and concrete you can be, the better your eval will be. Write concrete criteria such as 'answer that follows the days' and 'keeps totals within the stated budget'.
Sofia Jakovcevic says a regression dataset should be representative of the average workflow users push into systems, and should capture known failures (moments where the agent often errors or fails eval metrics), edge cases even though they are less common, and successful workflows you want to protect, because a fix must not break what was already working. She notes that forgetting to capture successful workflows is a common user mistake.
Sofia Jakovcevic recommends pushing the experiment into CI, for example in GitHub Actions, so each pull request runs through an experiment workflow against team-deemed-important evaluations. Merging PRs without meeting regression criteria means pushing on vibes instead of with confidence in positive performance impact.
Define a rubric for what a correct extraction looks like for your documents, then use an LLM as a judge against that rubric to evaluate extraction outputs.
Set up Sales Navigator reports to surface the people most likely to help you, then build a referral prospecting list from former colleagues, your boss and boss's boss, cross-functional leaders, veteran reps in your space, and your company's current customers, working with your CSM to get connected to those customers.
The creators behind most of what cleared in Money.
11494 cleared insights match
2032 cleared insights match
1148 cleared insights match
Choose sources → to get a library of your own and a daily brief.
