Principle · MoneyWolfram Ravenwolf describes temperature as a difficult setting for a benchmark because some model providers give a recommended temperature, some don't, some agents have a default, and some agents decide based on the model. He recommends deciding per model what the recommended temperature is, and if there is none, choosing which temperature to use. He also recommends making sure that people running the same benchmark are using the same temperature setting you chose, deciding on a top P value (whether to use 0.95, 1, or any other value), and deciding on max new tokens — how many tokens the model is allowed to generate — because there are provider recommendations and host limitations, and you usually have to configure it. He recommends setting a seed when you can, so the randomness is reduced
endorsed
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Principle · MoneyAfter choosing one part for each of the six layers — TerminalBench for evaluation, Harbor at a specific version, the sandboxes, Terminus 2 as the agent, GLM 5.2, and specific settings — you have made your evaluation configuration. Wolfram Ravenwolf says that whatever framework and model you use for evaluation, you have to make these choices, and you should be very clear about what you are using. Being clear about your configuration is important when publishing results, so others know what you have actually used and measured. He also suggests downloading the full configuration via the QR code so you don't have to copy settings from the slides.
endorsed
AI Engineer · From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Creator perspective · MoneyEngineers often ask questions like 'PRs that I merged last week' where they know the starting point but not the destination, but they do know the relationships. Using RAG for such a query can return the wrong answer: a semantically related PR merged three years ago, because the temporal/structural aspect is missed. Traditional RAG cannot answer structural questions around state or temporality correctly.
AI Engineer · Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked
Creator perspective · MoneyThorsten explains that choosing containers or VMs as the distributable unit can incur cold-start delays from several seconds up to multiple minutes, especially when scaling fast to meet unexpected demand. Over-provisioning can overcome cold starts, but keeping many unutilized containers or servers raises cloud spending and is not sustainable environmentally.
AI Engineer · Zero Cold Starts: Serverless AI Agents on Akamai Functions — Thorsten Hans
Creator perspective · MoneyPatterson frames the talk's core argument: there are three architectural ways to protect an organization from an agent going crazy — cloud development environments, a model proxy, and an agent firewall. He also warns that having all three Lethal Trifecta layers present means a security incident is a matter of when, not if.
AI Engineer · The Lethal Trifecta Is Already on Your Laptops — Michael Patterson, Coder
Report · MoneyShare agents by making a marketplace: a repository with extra JSON in one of the directories marking it as a marketplace. A marketplace gives you light package management capabilities inside the repo: simultaneous agent updates, versioning, and security vulnerability notifications. There is also a capability to sync marketplaces with the local client device and push agents in and out.
AI Engineer · From Your Laptop to the Pipeline: Scaling Custom Agents with GitHub Copilot
Principle · MoneyRay by default prefers to schedule and co-locate workers on the same nodes, which makes it easy to saturate one machine's resources (e.g., its NIC) before utilizing others.
AI Engineer · From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model
Creator perspective · MoneyZero-copy retrieval and spread scheduling don't help individually at small scale, but together they have a synergistic effect at large scale. A config can do poorly at a small scale but perform really well at a larger scale, so you should always think about how to ablate your configs across your scaling ladders.
AI Engineer · From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model
Principle · MoneyFeedback is what closes the loop for memory. When an agent retrieves a candidate memory, it will use it or reject it; if using it causes everything to go haywire and crash, that feedback needs to come back to the memory system. If a memory was harmful, its feedback score should go down; if it was useful, maybe promote it. Your memory layer needs to store whether memories were helpful, stale, irrelevant, harmful, et cetera — it's not just a repository.
AI Engineer · From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle
Principle · MoneyViren Baraiya frames the agent harness as the deterministic 'hands' that executes work, while the LLM is the non-deterministic 'brain' that reasons about what to do next. The harness is a combination of deterministic and non-deterministic parts: determinism is delivered by the harness, and non-determinism comes from the LLM's reasoning and thinking process.
AI Engineer · Brains vs Hands: How to Run AI Agents Safely in Production — Viren Baraiya
Principle · MoneyGiving someone 10 bullets of scope and a few user journeys can result in a 20-page document that looks polished but requires hours to figure out whether there's any real thinking inside it. AI-generated work in the worst sense does not clarify the problem — it expands it, handing back ambiguity as a larger artifact for other stakeholders to resolve. In code, a vague task becomes a large PR, and a small feature becomes a pile of files.
AI Engineer · Generation Is Cheap, Review Is Expensive: How to Stop Shipping AI Slop — Gabriel Martinez, G2i
Creator perspective · MoneyBecause efficiency in doing a task means it can be done more often (and translates into cost savings), the same evaluations can run at much higher frequency — at PR level and even on the developer's machine — catching problems earlier in the lifecycle.
AI Engineer · We Mapped 115 Microservices for Our Coding Agents — Kamalakannan Nandagopal, Postman
Creator perspective · MoneyOne of the standout opportunities is bringing the semantic layer of what the business does into the context layer that coding agents take as input, including reasoning about why a data model exists, why a domain maps the way it does, and why one API should be chosen over another — which improves subsequent coding decisions.
AI Engineer · We Mapped 115 Microservices for Our Coding Agents — Kamalakannan Nandagopal, Postman
Principle · MoneyTo drive AI adoption across 32,000 engineers, Cisco guaranteed employees that if they don't use AI they will lose their job (the inverse of the usual fear), told non-adopters Cisco is the wrong place while offering full tooling and support, and gave everyone unlimited tokens to start.
Y Combinator · How Jeetu Patel Runs Cisco Like the World’s Largest Startup
Report · MoneyAll internal apps are automatically protected by Okta SSO so only employees can log in, requests are logged and auditable, secrets live in Doppler rather than code, and sensitive apps with PII or customer data cannot be accessed outside the company.
AI Engineer · Let Anyone at Your Company Ship Internal Apps with AI — Garrett Galow, WorkOS
Principle · MoneyMatt Gibiec argues that letting agents do everything without human oversight would create a messy environment and that pushing fixes to production without human approval means losing control. He recommends maintaining human-in-the-loop control over agent actions and giving the human the final decision on whether to push a fix or feature to production, reviewing the full SDLC context before deciding.
AI Engineer · Your AI Agent Has No Nervous System — Matt Gibiec, Dynatrace
Creator perspective · MoneyAbhi Arya's principle: to avoid auto mode, you don't hand your agent an API — you hand it tools that carry the overall expertise, such as the right patterns, the state, and the validators. Instead of one create step to call over and over and praying the agent assembled it right, he built tools that understood how their internal dashboard or product works.
AI Engineer · Why We Deleted Our MCP Server and Rebuilt It — Abhi Arya, Reducto
Principle · MindDr. Elissa Epel advises: when you know the effort is productive, work (muscle it); when you know there's a brick wall, let go (release it). Letting go is a wise, discerning way to mitigate stress in the right situations and at the right time. You can't muscle through everything. Huberman adds that dropping the rope (letting go of a struggle) frees up energetic resources that were being consumed by a high-demand, conclusionless effort.
Andrew Huberman · Essentials: Manage Stress & Build Resilience | Dr. Elissa Epel
Principle · MoneyThere is a huge difference between getting the whole job done versus getting everything except for the last mile — the last mile can feel worse than a non-starter. If a tool is a non-starter, the user can just do it themselves or try another route, but if it gets 99% there and then fails, it's a worse unfulfilled promise. Computer use will always work — it might be slow or cost a lot of tokens, but it will get the job done all the way.
described
Lenny's Podcast · Stop planning for 2027: how OpenAI builds product 90 days at a time | Tara Sesha and Nan Yu (OpenAI)
Principle · MoneyTara Sesha says she shifted her mindset from getting it perfect a priori to shipping something and iterating as quickly and effectively as possible, because nothing compares to actual empirical evidence of seeing users try and use a product before theorizing; urgency matters more than polish when shipping AI-era products.
Lenny's Podcast · Stop planning for 2027: how OpenAI builds product 90 days at a time | Tara Sesha and Nan Yu (OpenAI)