For developers, the operating assumption has been one engineer, one agent — the model Claude Code and similar tools. At VB Transform 2026, James Zou, associate professor of biomedical data science at Stanford University, argued that assumption is about to break: the next frontier isn’t a single, more capable agent, it’s tens of thousands of them collaborating.
For developers and product builders, the most critical takeaway from Zou’s presentation is how these massive systems are orchestrated. His team’s research offers a practical blueprint for connecting legacy databases to AI orchestration layers and designing environments that enable thousands of agents to collaborate.
Zou’s project began as a “Virtual Lab” consisting of five to eight agents structured to mirror his physical Stanford lab. The setup included an AI professor acting as the principal investigator and AI students with distinct specialties holding regular group meetings.
“We also created for the agents a replica of Stanford, an agent school, where the agents can actually go to the school and do supervised fine-tuning to improve their expertise in their specific domains,” Zou noted.
The virtual lab successfully designed new nanobody proteins for recent COVID variants.
“What is really exciting to us is that these AI-designed nanobody proteins actually worked much better than the previous human-designed nanobodies in terms of binding to the recent different viruses,” Zou said.
Following this wet-lab validation, the team expanded their ambition. They transitioned from emulating a single research team to modeling a massive corporate structure.
The resulting system, dubbed the Virtual Biotech, comprises tens of thousands of specialized AI agents overseen by a Chief Scientific Officer (CSO) agent. It operates through distinct corporate divisions, such as target discovery, molecule design, and clinical trials.
“Working with the CSO agent are different divisions that mirror the divisions found in a human biotech or pharma company,” Zou explained — one focused on identifying drug targets, another on designing molecules, a third on safety and clinical trials. Individual agents specialize further within a division, he said. “Under the target discovery division, we’ll have one agent that specializes in looking at all the genetics data, another agent that looks at all the genomics data and single-cell data, and so on.”
As foundation models grow more capable, developers face a core architectural dilemma: Why distribute workloads across tens of thousands of specialized agents instead of channeling all computing resources into a single, omniscient model?
Zou’s team ran a head-to-head comparison of a multi-agent team against a single agent tasked with the same scientific challenge. The multi-agent ecosystem created friction and interaction that produced better solutions that were more resilient against compounding errors.
“In these scientific virtual labs, the agents actually get into debates and disagreements. They have to convince the other AI scientists [of] their ideas, and all of that elicits much more creative and robust reasoning compared to if you have a single model trying to do the problem by itself from scratch,” Zou said.
When scaling to tens of thousands of agents, orchestration becomes the primary bottleneck. The system requires a unified context layer that allows agents to synthesize knowledge from various tools, datasets, and historical records.
Many enterprise teams attempt to solve data integration by wrapping existing databases with an MCP. However, legacy systems are not very friendly to agents. For instance, dropping a PDF of a research paper into an agent’s context window is inefficient, and standard text models struggle to interpret complex figures and tables, leading to hallucinations.
“Even if you wrap an MCP around the existing databases and APIs, that doesn’t solve the underlying problem: the interface and APIs are not suitable for agents,” Zou said. He added that existing databases are designed to be consumed by humans or pre-AI algorithms.
To resolve this, Zou’s team created Paperclip. The platform relies on a core strength of modern LLMs: their ability to write code and navigate file systems. Instead of forcing agents to query brittle, database-specific APIs, Paperclip digitizes unstructured data and maps disparate databases into a unified, AI-native virtual file system.
This structure allows agents to access knowledge from millions of papers using standard file-system operations.
“This basically shows that we can get much better accuracy if you use Paperclip, and we can reduce the time and the cost by over an order of magnitude compared to if you use agents without these AI-native scientific infrastructures,” Zou stated.
To test the practical output of this architecture, Virtual Biotech spun up 37,000 “clinical trial agents” to synthesize fragmented trial data. These agents identified single-cell features that predict trial success — drug targets supported by these features were about 50% more likely to reach market than comparable drugs without them.
The system then autonomously designed an antibody-drug conjugate (ADC) targeting the CD276 protein for lung cancer. The agents completed this design autonomously, relying exclusively on data published prior to January 2025.
Several months later, Zou said, pharmaceutical company Merck independently developed and validated the same therapeutic design — which went on to receive breakthrough designation from the FDA. He characterized this as “a third-party external validation of the therapeutic design provided by the virtual biotech agents.”
As multi-agent systems scale, leaders must rethink how they manage these digital workforces. Zou advocated for shifting from designing rigid workflows to creating open environments. Workflows dictate the exact steps an agent should take, similar to managing a junior employee. Environments provide the infrastructure, guardrails, and incentives for agents to collaborate on open-ended problems.
“In workflows, we’re trying to tell agents what to do and how to do their job. But in environments, we’re providing the infrastructures, the incentives, and the guardrails, but otherwise we leave it open to incentivize agents to collaborate,” Zou said.
Optimization at scale means engineering the environment rather than fine-tuning individual models. While single agents can improve via reinforcement learning or supervised fine-tuning in the agent school, the success of a massive multi-agent system relies on adjusting the parameters governing their collaboration.
“At the multi-agent [side], we’re not actually fine-tuning and changing the individual models anymore, but we’re optimizing the environment,” Zou explained. “The environment itself is the object that we optimize to improve the agents.”
Alibaba released Qwen 3.8-Max this week and marketed the preview as second only to Claude Fable 5 (their launch-day table was more equivocal: the model leads on one of 12 coding-agent rows). But an independent harness came close to the opposite conclusion: a benchmark run, apparently using the Preview version, put Qwen 3.8-Max’s best effort setting mid-pack, and its default setting last.
Both results are real and defensible. The gap between them is about token and time budgets, and that matters because those figures aren’t usually headline numbers. Alibaba’s footnotes give its coding numbers a five-hour timeout, and up to 12 hours per run on PaperBench. The independent harness, VulcanBench, allowed between 45 and 60 minutes of wall clock time. A time budget between five and 16 times larger on Alibaba’s side explains the huge difference in results.
It’s time to do two things to start accounting for these differences when choosing models. First, the metric to use is cost per successful task: total spend, including everything you spent on attempts that failed, divided by the tasks that actually passed your acceptance check. Second, you need to make time or token budgets an explicit part of your acceptance criteria, not a hidden detail.
The comparison everyone published in Qwen 3.8-Max’s first week was a price comparison, because that was the only data available. It is not a cheap model. DeepSeek-V4-Flash-0731, which entered public API beta on July 31, lists at 14 cents per million input tokens and 28 cents output. Qwen 3.8-Max lists at $2 and $6. Kimi K3 sits at $3 and $15.
Those prices tell you less than they used to, for a reason specific to reasoning models like Qwen: getting to a result costs thinking tokens. A model that spends most of its token allowance on reasoning can reach a token cap before it writes the answer, giving you an empty result indistinguishable from a total failure at the cost of a full run.
Artificial Analysis has the cleanest published measurement of how this can affect real agent spend: running its Intelligence Index on DeepSeek-V4-Flash at maximum effort took 210 million output tokens against a class median of 100 million. Absolute cost stayed low anyway, because the tokens were so cheap. But verbosity costs time, not just money, and depending on your use case that can sink you.
What you need is a number that counts everything you spent, including the attempts that came back empty, against the tasks that actually got done in the time and token budget you specified. This is what a cost-per-success metric helps you see.
A run that produces a wrong answer and a run that runs out of budget are different events with different fixes. Almost no harness distinguishes them, and almost no leaderboard reports the split. I hit this building an agent benchmark of my own: the harness logged a failure and nothing about why, and I had to add the distinction myself. When you do separate them, budget exhaustion turns out to dominate.
Long-Horizon-Terminal-Bench, published in July, ran 17 frontier models across 46 tasks through a shared harness with one 90-minute attempt each. Timeouts accounted for 79% of unresolved runs, against 19% for agents that stopped on their own and 3% for harness errors. The authors are careful about what that does and does not mean: the timed-out runs were not close to finishing, with mean reward between 0.10 and 0.35, so you cannot assume more time would have resulted in success. But the lesson is: benchmarks are implicitly measuring time efficiency, whether or not they shout about that.
The clearest published example of the mechanism comes from VulcanBench, the same open-source harness behind the Qwen chart. In a report dated July 26, Claude Opus 5’s lowest-effort setting was its best, solving 20 of 23 tasks against 18 at high effort. The extra reasoning wasn’t useless: high effort returned the fewest wrong answers of any setting, one against three. It ran out of clock instead, and a timeout scores zero. Two of its three regressions were cutoffs on tasks that low effort solves, and given unlimited time on both it only ties its cheapest setting, at 3.1 times the cost.
That has a direct consequence for anyone building a routing ladder. The standard design escalates to more reasoning when a cheap attempt fails, on the assumption that the next rung is better and merely costs more. For a meaningful share of model and task combinations that assumption is wrong, and you pay the higher rung’s price to escalate into a timeout or hitting a cap.
Several groups have landed on cost per successful task independently in the last few months, which is the strongest signal it’s becoming standard.
VulcanBench reports dollars per solved task as a headline column and has since its earliest reports. Long-Horizon-Terminal-Bench publishes per-task cost next to accuracy, and its most instructive row is GPT-5.4 at roughly $26 per task with a much lower pass rate than Grok 4.5 at about $11. TestEvo-Bench runs agents under a cost cap, and Claude Code’s test-generation score falls from 71% to 44% at the tighter cap.
Vendors are already on board with the idea of measuring per successful task. HubSpot moved its Breeze Customer Agent in April to 50 cents per resolved conversation, down from $1 per handled conversation. Zendesk bills per automated resolution. Fin charges 99 cents per outcome and bills only on end-to-end resolution.
Emit a failure reason on every agent run as a required field, with budget exhaustion, verifier failure and harness error as distinct values rather than one failure flag. Until you can separate a timeout from a wrong answer, your pass rate is measuring two things at once and you cannot tell which one to fix.
Compute cost per successful task per effort level, not just per model. Total spend including failed attempts, divided by tasks that passed your acceptance check. The ranking will not match the rate card, and the cheapest setting may well win.
Cap on tokens rather than wall clock unless latency is genuinely in your service level objective. A wall-clock cap scores your provider’s serving speed as model quality.
Check the default effort setting on everything you have deployed. Qwen 3.8-Max runs at its highest reasoning setting when the effort field is unset, and its highest setting was its worst performer in independent testing. A team that never touches that parameter is running the configuration that costs the most per solved task.
Meta today released Muse Code, a terminal-based AI coding agent now in beta, alongside Muse Spark 1.2, a coding-focused update to its Muse Spark family of frontier models — a one-two punch that puts the company in direct competition with Anthropic’s Claude Code, OpenAI’s Codex, and the growing field of agentic coding harnesses that have rapidly become the primary way many professional developers ship software.
“Releasing Muse Code in beta today,” Meta CEO Mark Zuckerberg wrote in a post on rival social network X (under his longtime handle @finkd). “It’s a terminal coding agent that takes on complete software engineering tasks across large repos: planning changes, writing code, validating the results.”
The launch marks Meta’s most serious entry yet into a category it has largely watched from the sidelines.
While Anthropic and OpenAI turned their coding agents into flagship products — and startups like Cursor built billion-dollar businesses on the workflow — Meta’s developer story long centered on Llama, the open-weight model family it gave away to the tune of more than a billion downloads.
Muse Code changes that in more ways than one: it’s a full harness, installable on macOS or Linux with a single curl command, co-trained with the model that powers it — and, like the Muse Spark models behind it, entirely proprietary.
Developers and prospective users can install it now on their Terminal using the following one-line command — but be warned, if that’s you, you’ll need to log in with a Meta account and provide billing details first in order to begin: curl -fsSL https://dev.meta.ai/install.sh | bash
Muse Code’s headline architectural bet is what Meta calls async background agents.
Rather than spawning helper agents fresh for each task — the pattern most rival harnesses use — Muse Code keeps a set of specialized background agents alive for the entire session.
According to Meta’s blog post, these agents “remain active throughout each session, rather than being spawned for individual tasks, helping avoid redundant information gathering,” carrying out next steps on their own and choosing when to report back to the main agent.
The practical pitch is less latency and less babysitting: an agent that already knows the repository doesn’t have to re-explore it every time the developer asks for something new.
When a job is large enough, Muse Code fans out to separate sub-agents working in parallel, each in its own isolated git worktree, so the developer’s working copy is never touched.
“In testing we had it build six features for a game simultaneously with no collisions,” Zuckerberg wrote on X.
Worktree isolation and parallel sub-agents exist in competing tools, but Meta is leaning on the combination of persistence plus parallelism as its differentiator.
The second notable design choice is auditability. Every model call, tool run, approval, and edit is appended to a local event log before it executes — a single source of truth that Meta says makes the runtime “replay-exact and restart-safe.”
If Muse Code crashes 20 hours into a long-running task, it resumes precisely where it stopped, with no lost work and no re-prompting. For engineering leaders who have been burned by opaque agent runs, a complete local audit trail may prove to be the feature that matters most in enterprise evaluations.
Muse Code also ships with bundled “skills” that will look familiar to users of rival tools: /plan turns a task into an approval-gated plan, /grill stress-tests that plan until it holds up, and /goal drives the agent toward completion of a stated objective.
Under the hood is Muse Spark 1.2, which Meta describes as a coding-focused update to Muse Spark 1.1 with “significantly scaled up training compute on coding tasks” and broader training environment diversity, improving code generation, complex debugging, and codebase understanding while maintaining general agentic capability.
The update lands squarely on the Muse family’s weakest flank. When the original Muse Spark debuted in April, it vaulted Meta back into the top five on frontier reasoning and vision benchmarks — but trailed on the agentic coding evaluations that matter most to this market, scoring 77.4 on SWE-Bench Verified against Claude Opus 4.6’s 80.8 and Gemini 3.1 Pro’s 80.6, and lagging well behind GPT-5.4 on GDPval’s measure of long-horizon work tasks.
Four months later, a coding-specialized checkpoint paired with a purpose-built harness reads as Meta’s direct answer to that gap.
Two training details stand out. First, Meta co-trained the model with Muse Code itself, using rejection-sampled harness trajectories and recipe optimizations for goals, context compaction, and sub-agents — meaning the model was explicitly tuned to perform best inside this particular tool. That mirrors an industry-wide shift away from treating models and harnesses as separable products.
Second, Meta used a self-improvement loop: Muse Spark 1.1 generated challenging coding environments and instruction-following templates, then graded candidate solutions against those requirements, producing a scalable training dataset for its successor. Meta credits the loop with making 1.2 measurably better at following complex instructions.
Meta published benchmark charts comparing Muse Spark 1.2 against other coding models on Terminal-Bench 2.1, DeepSWE 1.1, and an internal Meta coding benchmark, pointing readers to a separate methodology report for details — though the company did not headline specific scores in the announcement itself, a notable omission in a field where rivals trumpet leaderboard placement.
The company’s most striking demonstration is a long-horizon case study: Meta pointed Muse Spark 1.2 at GPU kernel optimization and let it run for more than 1,000 tool calls over up to 24 hours on NVIDIA Hopper hardware.
Working in Triton and barred from simply wrapping existing third-party kernel libraries, the agent wrote, compiled, and profiled its way to what Meta calls “substantial improvements” over baseline implementations of KDA and MLA kernels — including genuinely non-obvious optimizations like re-centering gated cumulative decay at a chunk midpoint.
“It kept finding substantial improvements well beyond the initial exploration phase,” Zuckerberg wrote. Sustained improvement over a 24-hour autonomous run, if it holds up outside Meta’s demos, addresses one of the most persistent criticisms of coding agents: that they plateau or drift once past their initial burst of progress.
The pricing structure may be the most consequential — and most scrutinized — part of the launch. Meta is offering Muse Spark 1.2 through its Meta Model API in two tiers.
The standard tier is priced at $1.25 per million input tokens and $4.25 per million output tokens (with cached input at $0.15), and Meta commits that prompts and completions on this tier are not used to train its models. There is no long-context premium, and rate limits run to 3,000 requests and 4 million tokens per minute, per team. It’s about mid-range price, compared to other leading AI models available over API.
The contributor tier is where Meta’s strategy diverges sharply from its rivals: $0.10 per million input tokens and $0.20 per million output tokens — roughly 12x and 21x cheaper than standard, respectively, with cached input at a near-free $0.002 — in exchange for explicit permission to use your prompts and completions to train future Meta models. It’s the cheapest available on the market, but you pay with your data — as described below.
|
Model |
Input ($/1M) |
Output ($/1M) |
Total ($/1M) |
Source |
|
Muse Spark 1.2 Contributor |
$0.10 |
$0.20 |
$0.30 |
|
|
MiMo-V2.5 Flash |
$0.10 |
$0.30 |
$0.40 |
|
|
deepseek-v4-flash |
$0.14 |
$0.28 |
$0.42 |
|
|
deepseek-v4-pro |
$0.435 |
$0.87 |
$1.305 |
|
|
GPT-5.6 Luna |
$0.20 |
$1.20 |
$1.40 |
|
|
MiniMax-M3 |
$0.30 |
$1.20 |
$1.50 |
|
|
LongCat-2.0 — limited-time promo |
$0.30 |
$1.20 |
$1.50 |
|
|
Gemini 3.1 Flash-Lite |
$0.25 |
$1.50 |
$1.75 |
|
|
MiMo-V2.5 |
$0.40 |
$2.00 |
$2.40 |
|
|
Gemini 3.5 Flash-Lite |
$0.30 |
$2.50 |
$2.80 |
|
|
LongCat-2.0 — standard |
$0.75 |
$2.95 |
$3.70 |
|
|
MiMo-V2.5 Pro (≤256K) |
$1.00 |
$3.00 |
$4.00 |
|
|
Muse Spark 1.1 / 1.2 |
$1.25 |
$4.25 |
$5.50 |
|
|
GLM-5.2 |
$1.40 |
$4.40 |
$5.80 |
|
|
Grok 4.5 |
$2.00 |
$6.00 |
$8.00 |
|
|
MiMo-V2.5 Pro (>256K) |
$2.00 |
$6.00 |
$8.00 |
|
|
Qwen3.8-Max |
$2.00 |
$6.00 |
$8.00 |
|
|
Gemini 3.6 Flash |
$1.50 |
$7.50 |
$9.00 |
|
|
Gemini 3.5 Flash |
$1.50 |
$9.00 |
$10.50 |
|
|
Gemini 3.1 Pro Preview (≤200K) |
$2.00 |
$12.00 |
$14.00 |
|
|
GPT-5.6 Terra |
$2.00 |
$12.00 |
$14.00 |
|
|
GPT-5.4 |
$2.50 |
$15.00 |
$17.50 |
|
|
Kimi K3 |
$3.00 |
$15.00 |
$18.00 |
|
|
Gemini 3.1 Pro Preview (>200K) |
$4.00 |
$18.00 |
$22.00 |
|
|
Claude Opus 5 |
$5.00 |
$25.00 |
$30.00 |
|
|
GPT-5.5 |
$5.00 |
$30.00 |
$35.00 |
|
|
GPT-5.5 Instant (chat-latest) |
$5.00 |
$30.00 |
$35.00 |
|
|
Sakana Fugu Ultra (≤272K) |
$5.00 |
$30.00 |
$35.00 |
|
|
GPT-5.6 Sol — Standard mode |
$5.00 |
$30.00 |
$35.00 |
|
|
Claude Fable 5 / Claude Mythos 5 |
$10.00 |
$50.00 |
$60.00 |
|
|
GPT-5.6 Sol — Fast mode |
$10.00 |
$60.00 |
$70.00 |
This is the tier Zuckerberg is steering new users toward: “It’s easy and low-cost to get started,” he wrote. “Install Muse Code with one line and you can start on our contributor tier.”
In VentureBeat’s own testing on a Mac mini, the one-line installer worked as advertised — a 97 MB download and a sign-in — but the agent stopped short of running anything, reporting that no models were visible and that payment was “required to finish setting up your account.”
In other words, even the heavily discounted contributor tier requires a payment method on file before Muse Code will do any work: low-cost is accurate, but free is not.
Meta frames the contributor tier as lowering the barrier for prototyping and experimentation “where training on your data is acceptable.”
But it also means the default on-ramp for Muse Code sends developers’ code and prompts into Meta’s training pipeline — a tradeoff enterprises with proprietary codebases will need to consciously opt out of by moving to standard pricing.
The contributor tier also carries much tighter rate limits (60 requests per minute versus 3,000), a clear signal it’s aimed at individuals and small experiments rather than production workloads.
The approach is classically Meta: subsidize access, harvest data at scale, and use it to close the gap with the frontier. Zuckerberg made no secret of the ambition, calling Muse Spark 1.2 “our next step as we push toward frontier, with larger, more capable models on the way.”
However, for developers and enterprises who want or are required legally to keep their code secure, the tradeoff may not be one they’re willing or able to make.
What today’s announcement conspicuously lacks is any mention of open source — a striking omission from the company that spent three years positioning itself as the standard-bearer of open AI.
From the original LLaMA’s debut in February 2023 — whose weights famously leaked onto 4chan within weeks, inadvertently kickstarting the movement to run capable models on consumer hardware — through Llama 2’s commercially usable license, the coding-specialized Code Llama, and the 405-billion-parameter Llama 3.1, which Zuckerberg launched in July 2024 with a manifesto titled “Open Source AI Is the Path Forward,” Meta’s entire pitch to developers was that frontier-class weights should be free to download, self-host, and fine-tune.
The strategy worked: by early 2026, the Llama family had been downloaded roughly 1.2 billion times, averaging about a million downloads a day, with self-hosting offering enterprises cost reductions VentureBeat has previously reported at as much as 88% versus proprietary API providers.
Then came the unraveling. Llama 4 debuted in April 2025 to mixed reviews and, eventually, admissions that its benchmark results had been fudged — while Chinese open-weight rivals from DeepSeek, Alibaba, and Zhipu AI surged to account for some 41% of downloads on Hugging Face by late 2025, eroding Llama’s claim to leadership of the very movement it started. The rocky rollout spurred Zuckerberg’s summer 2025 overhaul of Meta’s AI operations into Meta Superintelligence Labs (MSL), with Scale AI co-founder Alexandr Wang recruited as chief AI officer.
The Llama era effectively ended this past April 8, when MSL shipped the original Muse Spark — “the most powerful model that meta has released,” in Wang’s words — as Meta’s first proprietary model: cloud-only, with no downloadable weights and no self-hosting, initially confined to Meta’s apps and a private API preview.
Asked directly at the time whether Llama development would continue, a Meta spokesperson told VentureBeat only that “our current Llama models will continue to be available as open source” — pointedly silent on future ones.
Wang, for his part, said bigger models were already in development “with plans to open-source future versions” — but four months on, today’s release does nothing to advance that promise: no weights, no license, and neither the blog post nor Zuckerberg’s thread so much as uses the word “open.”
The reversal is all the sharper because Meta’s rivals have been moving in the opposite direction. OpenAI released its Codex CLI as open source under the permissive, enterprise-friendly Apache 2.0 license and followed with its gpt-oss open-weight models; Google’s Gemini CLI harness is likewise Apache-licensed.
With Muse Code, Meta lands closest to the posture of Anthropic — whose Claude Code remains proprietary — while the company that once argued open source was the path forward now asks developers to pay per token for a model they cannot inspect, or to subsidize that access with their own data.
Seen in that light, the contributor tier reads as the successor to the Llama strategy itself: the ecosystem flywheel is no longer free weights in exchange for mindshare, but cheap tokens in exchange for training data.
Terminal coding agents have become the fastest-growing surface in enterprise AI, and until today the category has effectively been a two-horse race between Anthropic and OpenAI, with Google and a crowd of startups in pursuit.
Meta’s entry brings a genuinely different architecture (persistent background agents, an append-only local event log), a credible long-horizon demo, and an aggressive pricing wedge.
The open questions are the ones benchmarks charts can’t answer: whether Muse Spark 1.2 actually matches Claude and GPT-class models on real-world repositories, whether developers trust Meta with their code, and whether the contributor tier’s discount is enough to make them stop asking. Muse Code is available in beta today; Muse Spark 1.2 is live in the Meta Model API with expanded global access.
At Kilo Code, engineers are reading or writing code themselves only about 1% of the time now, according to co-founder Emilie Schario — the rest is agents. That shift is forcing new questions onto dev teams: which systems are safe to hand over, who cleans up when models goof up, how to support multi-model architectures, and whether skyrocketing token bills mean real progress or just burned IT budget.
As far as tech leads from Replit, Kilo Code, and Symbotic are concerned, it’s a natural — and welcome — evolution as agentic AI becomes embedded into more and more enterprise workflows.
“Unless something’s really broken or debugging, 99% of the time engineers are not reading or writing code anymore,” Emilie Schario, co-founder of Kilo Code, said at VB Transform 2026.
For Jared Go, distinguished engineer for AI and cloud at warehouse automation company Symbotic, the current moment is about directing the focus of AI. “These are my criteria,” he said. “Let’s look at it from the lens of security, elegance, clean, concise code, water tightness.” That way, AI does most of the heavy lifting, and human code review isn’t as critical.
Human involvement becomes necessary further down the line, Go noted, because agents don’t make strong product decisions. “Greenfield [building brand new codebases] is so easy for agents. Brownfield [writing, updating, or maintaining existing code] we all know is where the actual challenge lies.”
Replit takes a bit of a different tack: While the company has “gone very agentic,” they’ve been more conservative with AI coding, explained Amol Jain, head of product engineering. An agent reviews each pull request (PR) and assigns it a risk score; low-risk PRs are self-merged by their author, while others go to human reviewers who read the code and give feedback.
“The idea was human on the loop, not human in the loop,” Jain said. Replit’s internal tool is essentially self-driving for software engineers; devs give a task to agents, which do end to end planning, implementation, and testing.
“It’s a fleet of agents that run in their own cloud virtual machines (VMs) with access controls behind token proxies so they’re secure,” Jain said.
He shared one example where an engineer couldn’t repro or solve a “very gnarly bug” deep in its systems. It was sent to an AI manager agent, which told it to go to sleep. The manager agent then spun up a bunch of underlying agents that found the issue; it subsequently spun up a bunch more agents that found the fix. Six hours later, AI had a PR ready for the bug that had puzzled human engineers.
AI providers are also evolving beyond the lock-in model, as customers increasingly demand multi-model choice.
Kilo Code, for its part, supports 500-plus models in its gateway. “Your software that you’re using to do agentic engineering should be decoupled from the model that you’re using to do it,” Schario said.
For instance, Schario said companies often use expensive frontier-tier models to architect a project, then switch to a less expensive open-weight model for the rest of the work.
It’s also important to respect model provider limitations, such as when they need to work in closed or isolated environments or providers in their specific regions. “It’s factoring in what’s important to you, what limitations you’ve set, what data retention policies you’ve established, what keys you’ve brought in, what commits you might have … into that routing decision,” Schario said.
Replit, similarly, tends to have a better sense of the cost versus capability spectrum than its customers, Jain contended. “We are essentially making the decisions on users’ behalf of what model to use when, in what capacity, to minimize cost and maximize capability.”
Of course, an important consideration as AI adoption increases is runaway costs, which has led to some enterprises tracking and capping AI use through tokenmaxxing.
Concerns come from both sides, Schario said: internally and from customers. From the latter, she’s hearing, “I accidentally spent my whole AI budget for the year … so what do I do now?” In response, Schario said Kilo Code points customers to the same workflow: use expensive models for planning, then open-weight models for affordability.
Further, sharing skills, strong guidance, and Model Context Protocol (MCP) will empower models. “Realizing where you can really uplevel your team to help them get the most out of the models they’re using is going to make a big difference,” Schario said.
Internally, meanwhile, Schario noted one particular engineer that has a “heavy foot” and is constantly at the top of the usage board. “I regularly have to nudge, ‘What are you doing there?'” she said. It’s easy to look at a $600 bill for daily work and react, “Wow, that’s so much,” but looking at the amount of work completed can sometimes justify the cost.
“Cost per pull request is the metric that I’m paying attention to right now,” Schario said. “It feels like the closest proximity for how I can measure value.” Ultimately, AI changes how enterprises are thinking about ROI because spend is not the problem. “The spend with no return on that spend is the problem.”
Symbotic, for its part, has set per-month cost tiers for its employees. The company built a tool that gives managers visibility into PRs and usage trends. They can then move users up or down a tier as they see fit, Go explained. “Having a cap and seeing how many people went up in cap this month makes a big difference when you’re trying to corral these costs and make things efficient,” Go said.
When Cursor — which Symbotic uses heavily — ended a legacy discount that had grandfathered the company into a flat per-request rate even for frontier models, and moved everyone to full pricing, it forced a company-wide reckoning on efficiency, Go said. “People were saying, ‘You should try this model … This works better for this C# code, this whatever,'” he said.
But the cost problem is increasingly moving out of IT; Replit, for one, broadened agents beyond engineering, and eventually found that a user on the support side had “blown through an insane amount of money,” Jain said. When they looked under the hood, they figured out it was because they were running an automation on GPT 5.5 Pro Max.
“At least till that point, the ROI was rather clear,” Jain said. “We could see engineering productivity 3X, so no one had questioned it yet.”
Visibility that isn’t “anti-productive,” model routing, and sensible defaults are critical, he emphasized. “Most tasks do not need the frontier.”
Enterprise teams building AI agents keep hitting the same wall: a chatbot that can answer a prompt but can’t remember what the last five people asked it, and can’t tell you whether last month’s version actually worked.
In a fireside chat with VentureBeat’s Sam Witteveen at VB Transform 2026, Asana’s chief product officer, Arnab Bose, unpacked how his team tackled this problem to build a new operating system: Agentic Work Management (AWM). The product treats AI agents as coachable teammates that operate alongside humans rather than as one-to-one assistants.
For product builders and developers trying to move beyond basic integrations, Bose provided a look under the hood. He detailed how Asana engineered AWM, offering a blueprint for solving real-world bottlenecks and building agentic systems at scale.
To build an operating system for human-agent teams, Asana needed a ready-made enterprise context graph. They built AWM on top of their 18-year-old architecture: the Work Graph.
This graph-based database organizes information through a structure the company calls the Pyramid of Clarity. The smallest unit of work is a task with an assignee and a due date. Tasks belong to projects, projects roll up into portfolios, and portfolios connect to company-wide goals. The graph can help trace for example how a delayed design task impacts a corporate revenue goal. The Work Graph provides a real-time ledger of who does what, by when, and why.
AWM leverages this architecture to create a multiplayer teammate. A standard AI copilot is stateless and tied to a single user’s prompt. Because AWM plugs into the Work Graph, the AI can view overarching company goals, update project statuses, and share memory with human colleagues.
“Because [the agent] is plugged into the Work Graph, it’s not just looking at a particular prompt that you’re sending it or looking at a particular individual’s markdown file system on their local file,” Bose said. “It’s working off of that shared ledger for the whole company.”
AWM is already in production. Bose said Asana has “several customers live and successful on it,” including FedEx, which published its own case study on the shift.
Shipping AWM to enterprise customers required Asana to solve several technical hurdles. The first was data governance. If an AI teammate acts across a company, it builds a shared memory by learning from workflows and human feedback.
Bose highlighted a critical boundary problem: If an executive uses AWM to build workflows for a confidential project, the system must ensure the agent’s updated memory does not leak context to an unauthorized employee who interacts with the same agent later.
“[I] shouldn’t be able to leverage that shared memory when I run the AI teammate if you created that memory using that same teammate on a project that is, let’s say, a secret M&A project that I don’t have access to,” Bose said. Asana engineered a system of access controls to govern what triggers the creation of a memory versus the simple execution of a task.
Second, AWM handles dynamic model routing to abstract prompt engineering away from the user. When a user assigns a task to an AI teammate (i.e., drafting a job description for a general manager role), the AI cross-references public job postings, Asana’s internal style guide, and product requirement documents. For a complex task, the system automatically routes the prompt to a heavy frontier model — Bose pointed to Anthropic’s Opus and OpenAI’s models as examples — while lighter tasks get down-leveled to something faster and cheaper.
“We don’t want the knowledge worker to have to think through what the best possible prompt, context engineering, and attachments are that they should put into the task,” Bose said. “It should feel as if you were assigning the task to a human being.”
This dynamic routing introduces a third challenge: billing abstraction. Agentic tasks vary in computational complexity, making credit burn rates unpredictable.
“We don’t want to get into a state where our customers are having to reason about the fact that some of these tasks… are way more complex than others and they’ll be burning credits at different rates,” Bose said, adding that unpredictable pricing risked customers throttling their own employees by capping how often they could run an AI teammate.
To make AWM commercially viable, Asana designed its billing architecture to charge a static cost per task completion. The platform absorbs the complexity of model selection, token counts, and run limits to ensure predictable enterprise pricing.
AWM targets a specific problem with current enterprise AI deployments: statelessness. Developers can easily connect large language models to enterprise tools like Slack, Google Drive, or Databricks using Model Context Protocol (MCP) integrations. However, basic chat-based agents lack persistence.
Bose detailed a scenario where a user asks a chat agent to draft a marketing campaign based on historical performance and competitive research. The agent fetches data from external tools to answer the prompt, but the execution happens in a vacuum. It is a one-off task that benefits a single individual. It fails to create a reusable workflow for the next person building a similar campaign.
“The challenge with that is that those calls are stateless, and they are not leveraging a shared company brain that is this graph-based database or a context graph,” Bose said.
AWM solves this by creating a permanent state. When an AI teammate inside AWM completes a task, the system records the metadata. It registers whether the completion improved the project status and how it moved higher-level company goals.
Cloud provider CoreWeave is an early adopter using AWM to overhaul complex new product launches.
“CoreWeave is using both our deterministic AI studio workflow rules as well as multiple AI teammates to do new product launches,” Bose shared.
In the past, CoreWeave product managers filled out complicated forms detailing infrastructure, parameters, and costs. Human reviewers manually evaluated these forms and broke them out into specific tasks for finance, marketing, and hardware teams.
Under the AWM workflow, a product manager writes a standard Google document pointing to their product requirement documents. A deterministic AI workflow reads the document, automatically creates the project structure, and assigns tasks. Specialized agents then take over the execution. One agent then watches overall project status and flags bottlenecks; another, working inside individual tasks, forecasts infrastructure costs and recommends approvals when the numbers align with historical budgets. The system automatically triages the busywork while human beings focus on evaluating the AI’s outputs.
The dynamic gets complicated by the fact that the same frontier-model providers powering AWM under the hood — Anthropic, OpenAI — are also shipping their own competing agent products, like Anthropic’s Claude in Slack (Tag). Pressed on the overlap, Bose didn’t dispute the tension.
“I think that’s the reality that we all have to live in,” he said.
His case for AWM’s staying power rests on Asana’s 18 years of user-experience and workflow data, and prebuilt standard operating procedures for specific industries — expertise he argues raw frontier models don’t have. A product like Tag can work well in Slack, he said, but it requires a highly curated channel and its own separate credentials for every downstream app it touches.
“There’s a big difference between the power of the model plus a lightweight way to demonstrate its value, and something that’s pre-built … for true end-to-end use,” Bose said.
Presented by NTT DATA AIVista
At VB Transform 2026, NTT DATA AIVista CEO Bratin Saha joined VentureBeat CEO and editor-in-chief Matt Marshall to discuss the last-mile challenge of operationalizing frontier models in regulated production, where reliability, context, guardrails, and security determine whether AI delivers enterprise value. The conversation centered around the question facing every enterprise now pouring money into AI: how to convert that spending into real, tangible value.
“It’s not just a model, you’re building a system around the model,” Saha said. The last mile is the work of wrapping a frontier model in an enterprise’s own data, workflows, and guardrails.
In the end, regulated production turns on more than just technology, Saha said. Today, most enterprise AI projects fail during implementation because of poor integration, domain specialization gaps, lack of governance, and unclear ownership of outcomes. Last-mile specialization turns a capable foundation model into an enterprise agent shaped by domain-specific workflows, risk appetite, client classifications, regulatory interpretations, and institutional knowledge.
Frontier models fall well short of production-grade accuracy on many real-world insurance workflows, Saha said, but last-mile specialization can lift them to the reliability enterprises need. Out of the box, those models struggle with the complexity of regulated workflows such as multinational insurance claims.
“These forms are pretty complex, often have handwriting, lots of checkboxes, and so on,” he said, and that complexity is why frontier models like Fable 5, Opus 4.8, and GPT-5.5 fall short out of the box.
Saha said the biggest gains come from specializing the entire AI system, not just the foundation model.
That system gets specialized with the customer’s data, workflow and, in many cases, the tribal knowledge that never made it into an operating procedure document.
“The biggest bang for the buck comes from the specialization and then these specialized guardrails,” he said.
The work has three components:
capturing the enterprise’s context and making it consumable by AI
running an ensemble of models so cost does not go through the roof
and adding specialized guardrails that check the model and force a redo when it gets something wrong.
None of this involves fine-tuning. VentureBeat’s latest enterprise survey found it ranked last among companies’ model-selection priorities.
Instead, the last mile centers on domain knowledge and undocumented workflows that companies would never expose publicly without losing their competitive edge.
“The last mile is about taking data that’s proprietary to you and using that to build a system around the model that can steer the model in the right way that can put the appropriate guardrails around it,” Saha said.
In the end, enterprise AI is about moving a workflow from point A to point B rather than deploying a technology, and NTT’s advantage comes from pairing AI experts with subject domain experts.
“The only reason is because we go and talk to those human workers and we say, ‘How do you actually do the work,'” he said. That expertise is then encoded into an agent.
Success in insurance, manufacturing, and other regulated industries relies on three things at once, he added.
“You need technology, you need the domain expertise, and you need the change management expertise,” he explained, adding that across his team’s clients, technology is not the bottleneck.
For enterprises weighing large AI budgets, Saha’s said the payoff comes not from the model but from the work built around it.
“When you’re deploying AI in the enterprise, you’re not deploying a technology,” he said. “You are taking a workflow that exists and taking it from point A to point B.” The value is created by the workflow that gets moved, not the model that helps move it.
That reorders where money should go.
“Technology is not the bottleneck,” Saha said, pointing instead to the domain expertise and change management wrapped around the model, and to the discipline of commiting to all three together. Spending aimed only at the model leaves most of the return on the table.
Enterprises don’t have to choose between embedding AI into existing workflows and redesigning those workflows from scratch. NTT sees the two as successive stages of the same journey.
“We are starting with embedding in the workflow because it’s easier change management,” he said, noting that customers running mission-critical operations will not let a vendor rip out a working process midstream. “Once that happens, then we go into, how can we now reimagine this? And that really is where the biggest bang is.”
Keeping intelligence in the surrounding system rather than the model also preserves swappability and lets enterprises take advantage of open-weight and open-source models as they mature. Saha’s team runs an ensemble that mixes frontier and open-source models, and he expects the industry to lean on open weights wherever the cost of a mistake is low while reserving frontier reasoning for the cases that demand it.
“In many situations, especially in regulated industries where mistakes are very expensive, that last extra couple of percent matters,” he said.
The platform follows the same pattern: Guardrail generation and neurosymbolic models scale across customers, while capturing each organization’s tribal knowledge remains bespoke. Saha pointed to NTT DATA’s position as one of the world’s largest insurance third-party administrators as an advantage in acquiring that expertise.
“The ability to take that knowledge and trust that has been built over 20 years is very hard to replicate instantly, and I do think that is a durable aspect of what we have,” he said.
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.
If you have built anything with retrieval-augmented generation (RAG) in the last two years, you have lived its central frustration: You chop your documents into chunks, embed them, retrieve the top few that look similar to the question, and hand them t…
If you ask an AI coding agent to write a standalone Python script to parse a single JSON file, it will likely give you a perfect answer in seconds. But the same agent often breaks if you ask it to build a systematic data processing pipeline, like ingesting thousands of messy documents, chunking text, scoring quality, and filtering noise for a Retrieval-Augmented Generation (RAG) system that fits your specific enterprise stack.
While large language models (LLMs) excel at one-off code generation, their outputs for complex data-processing tasks are typically free-form, disposable scripts. These scripts are detached from the governable workflow abstractions that MLOps teams rely on for production, making them difficult to audit or edit visually.
To address this, researchers at Peking University, Zhongguancun Academy, and Shanghai’s Institute for Advanced Algorithms Research introduced DataFlow-Harness, an open-source framework that guides an LLM agent to build structured, visual data-processing workflows step-by-step, rather than writing raw code from scratch.
The framework makes AI-generated pipelines easier to manage and integrate into existing architectures because the generated artifacts are persistent and easily editable.
The researchers report that the platform achieves a 93.3% observed end-to-end pass rate on a 12-task data-engineering benchmark. Compared to standard Claude Code, it reduces API costs by up to 72.5% and response latency by 49.9%, while achieving nearly the same success rate as an AI given the entire codebase to write standard scripts. For enterprise teams, this means getting the speed of AI automation without accumulating unmanageable technical debt, ensuring that pipelines remain secure, auditable, and ready for production.
Data-centric AI requires workflows for tasks like synthetic data generation, retrieval augmentation, and model training. While LLMs can translate natural language into executable implementations to perform these tasks, high task accuracy is insufficient for production deployment.
“The first wall is usually not writing Python,” Runming He, first author of the DataFlow-Harness paper, told VentureBeat. “Modern coding agents can often produce a plausible script quickly. The harder problem is grounding that script in a live production platform: using operators that are actually installed, matching the real dataset schema, referring to registered datasets and model services, preserving dependencies between stages, and leaving behind an artifact that another engineer can understand and revise.”
General-purpose AI agents frequently hallucinate dependencies, relying on unavailable operators or outdated platform assumptions. Instead of leaving behind an artifact that another engineer can understand and revise, they generate disposable code that is difficult to audit through workflow managing tools.
The researchers define this challenge as the “NL2Pipeline gap”: the disconnect between a user expressing workflow requirements in natural language and the production environment requiring structured and persistent pipeline assets.
The researchers demonstrated this gap in their experiments. For example, when Claude Code was allowed to write standard, free-form scripts using codebase context, it hit a 94.2% success rate. However, when restricted to only using the platform’s specific building blocks to create a native workflow graph, its success rate dropped to 83.3%. This gap is the paper’s central finding: native, governable pipelines are meaningfully harder for the agent to produce than throwaway code.
“Closing this gap requires more than improving code-generation accuracy: construction must remain grounded in platform semantics and produce artifacts that integrate with the host platform,” the researchers write.
“DataFlow-Harness changes the agent’s action space,” He said. “Instead of asking the agent to emit arbitrary code, it retrieves the live operator registry and current pipeline state through MCP and applies typed, incremental changes to a persistent DAG.”
To achieve this, the platform organizes workflow synthesis around four components: the Data Pipeline Backend, the interaction layer (DataFlow-WebUI), the MCP Tools Layer, and the AI guidance layer (DataFlow-Skills).
The Data Pipeline Backend acts as the authoritative source of truth across conversational, visual, and programmatic interfaces. It represents the pipeline as a directed acyclic graph (DAG), a structured workflow map containing data sources, configured pre-built processing modules (which the researchers refer to as “operators”), and execution dependencies. Instead of generating free-form code, agents interact with this backend through “typed mutations,” like adding an operator or connecting edges.
DataFlow-Skills are markdown files that inject domain-specific knowledge into the model’s context window, guiding it on operator-selection patterns, schema inference, and assembly procedures. Rather than letting the AI guess how to assemble components, skills provide the AI with compatibility rules, teaching it how to correctly match different data formats and handle complex data structures without breaking the pipeline.
The MCP Tools Layer gives the AI access to the operator registry and current state of the data workflow. The AI proposes structured changes through the tools layer. The system validates the changes to ensure the workflow runs in a valid sequence and that every connected module speaks the same data language.
DataFlow-WebUI provides two interfaces that allow humans and AI to build the workflow together. Developers can describe workflow requirements in natural language through a conversational interface. They can also access the workflow as a graphical map in a visual DAG editor. Here, they can directly inspect the changes proposed by the AI and make modifications.
“The current implementation performs static checks against platform metadata before accepting pipeline changes,” He said. “These include checks for registered datasets, operators and model-serving references, field flow, and some invalid parameter usage, as well as structural validity. The result is visible in a graphical editor and can be revised either manually or by the agent in later turns.”
The researchers tested DataFlow-Harness on a benchmark of 12 tasks across six industrial data-processing scenarios, such as QA generation, review governance, and schema normalization. They used Claude Opus 4.7 as the backbone model in their experiments.
They compared DataFlow-Harness against three baselines:
Vanilla CC: An unconstrained coding baseline using standard Claude Code.
Context-Aware CC: An agent that has access to the DataFlow codebase in its context window.
MCP-only: An agent that has access to the DataFlow MCP tools and is instructed to generate platform-native DAGs (without access to DataFlow-Skills).
DataFlow-Harness achieved a 93.3% end-to-end pass rate, improving by 10.0 percentage points over MCP-only and beating Vanilla CC (91.7%), while being within 0.9 percentage points of Context-Aware CC (94.2%).
Importantly, it reduced API costs to $0.261 per task, a 72.5% drop compared to Vanilla CC and 42.8% compared to Context-Aware CC. In generating workflows, it was 49.9% faster than Vanilla CC and 17.6% faster than Context-Aware CC.
DataFlow-Harness proved particularly effective on complex tasks that depend on implicit domain knowledge, like QA generation. The baseline MCP-only approach frequently generated structurally valid DAGs but struggled to infer task-specific procedures from operator descriptions alone.
To show how this works in the real world, the researchers detailed a textbook-to-VQA extraction task. This job required the AI to stitch together capabilities such as PDF parsing, layout recovery, OCR, figure extraction, multimodal understanding, and long-range question-answer matching. DataFlow-Harness achieved 97.2% precision and an 87.3% coverage rate, easily beating the baselines. By having the AI snap together existing platform assets rather than coding complex tasks from scratch, it recovered more valid QA pairs from the document.
Their experiments also showed that DataFlow-Harness is highly effective at creating data generation pipelines. For example, in a synthetic instruction-data generation task, the agent built a multi-stage pipeline that generated candidate instruction–response pairs, critiqued and rewrote them, scored them with an LLM-based judge, and filtered low-quality outputs before training.
“Such workflows are costly to build and fragile to maintain as collections of ad hoc scripts,” He said. “The harness does not make them automatically safe, but it turns them into explicit, editable stages that engineers can inspect, test, and govern using normal production controls.”
Similarly, when tasked with building a math data cleaning-and-synthesis pipeline, the data produced by the DataFlow-Harness pipeline trained a better-performing model with higher average accuracy on AIME24 and AIME25 benchmarks than the data produced by the vanilla Claude Code pipeline.
For engineering teams evaluating DataFlow-Harness, it is important to understand how it fits into existing infrastructure. Released under the Apache 2.0 license, the current implementation requires a bit of engineering to fit into popular tech stacks.
“The current implementation is native to the DataFlow platform; it is not a turnkey Airflow, Prefect, or Spark plug-in,” He said. To use those systems as an execution backbone, teams must build an adapter to connect their organization’s registry, metadata, and execution interfaces to the agent’s control layer.
Furthermore, organizations must invest in the boundaries they want the AI to respect. This requires maintaining an operator registry, defining schemas, and encoding recurring domain procedures as Skills. Because of this overhead, He recommends against using the framework for small, one-off transformations where a simple script suffices, or in legacy environments that cannot expose reliable metadata.
Finally, while the platform prevents illogical connections by validating structural properties, it is an engineering control layer, not a compliance substitute. “The harness should still be treated as an engineering control layer, not as a substitute for compliance policy, validated detection models, access controls, audit logging, or human approval,” He said.
The platform is open-source, and developers can access the source code and codebase documentation directly via the project’s GitHub repository.
As protocols like MCP become standardized, the boundary between human engineers and AI agents will shift. “The goal is not autonomous data engineering without oversight,” He said. “It is a better division of labor: agents perform repetitive construction inside explicit boundaries, while engineers remain responsible for the semantics, policies, and consequential decisions that require domain accountability.”
Enterprise AI agents can do the work — but the infrastructure to let them talk to each other, prove they should be trusted, and be audited when something goes wrong is still being built.
Here’s a look at how five startups are tackling that gap — around orchestration, observability, connectivity, and security — as shown at VB Transform 2026.
In the very near future, agents will be deployed everywhere, and they will do work on our behalf, noted Vlad Luzin, CTO and co-founder of BAND.
As he describes it: They will receive tasks, visit registries, recruit other agents to help them, delegate subtasks to AI peers in a “conversational space,” gather and share results, then return a summary to the human user.
BAND is building a coordination infrastructure layer for multi-agent AI systems to make this a reality.
Why don’t Telegram, Slack, or Discord solve the problem? These platforms were built for humans, Luzin noted. Agents have to be onboarded manually in numerous steps, and they can’t see each other; “they are still alone in a kind of digital solitary confinement.”
Similarly, Claude is stateless, and devs often have multiple sessions open at a time that they toggle between for different tasks — something Luzin said creates real friction.
The challenge is connecting remote processes, which Luzin called a distributed systems problem.
“The transportation layer needs to be solved first, how the agents communicate in real time,” he said. Conversations can’t happen through IPs and URLs; they need to be bumped to the abstraction layer so agents can talk across channels, conversational spaces, and platforms.
“Agents see each other. They understand. They can collaborate together. They discuss issues. They fix issues, and they ask for review from another,” Luzin said.
BAND supports autonomous workflows that can run for eight to 20 hours and is compatible with A2A and MCP protocols, according to Luzin. Importantly, humans can join the conversation as agents converse and discover one another, he said.
“We can record and show you all the tasks that your agent generates in real time,” Luzin said.
The biggest challenge defenders face today is that they’re still running at human speed, but adversaries are running at machine speed, said Tom Findling, CEO and co-founder of Conifers.
Attackers are already adopting agents, Findling said, and they only have to be successful once to penetrate an enterprise. Malicious campaigns that used to take months and weeks now take hours, even minutes. Security operations, on the other hand, are fragmented, manual, inefficient, and slow.
Findling said Conifers has taken various components of cyber defense — private intelligence, hunting, detection, engineering, investigation, response — and made them agentic. They then broke down the silos between them, he said. Various agentic systems can communicate with one another to ensure that operational defense and active defense are always on and adapting.
Findling said that Conifers’ system is condensing containment time from 7 hours to 12 minutes, and that the company can turn around complex cyber investigations in four minutes or less.
He emphasized the importance of connecting to an enterprise’s existing security tools, whether that be endpoint detection and response (EDR), security information and event management (SIEM), posture management, or others. Conifers helps customers understand their security posture, pain points, which controls are working and which are not, and the areas to invest for the best ROI.
“The threat landscape is changing, detection stays the same, and threat intelligence is not being operationalized,” Findling said. “This is a job for agents.”
One of the defining problems of the current era is finding critical issues in AI agents, says Ben Hylak, CTO of Raindrop AI.
It’s what he called a “double whammy”: As agents become more capable, complexity increases, as do timelines; they are running for hours or days in some cases. Secondly, issues become catastrophic in sectors like healthcare or defense.
“This problem is getting a lot worse as models and agents improve,” Hylak said, “and I think there’s good reason to believe it will continue to get worse.”
Raindrop AI’s platform finds critical issues in agents in production and simulates fixes based on past user behavior, Hylak said. That lets teams confirm a fix works as intended before it’s live, without introducing unexpected side effects.
The startup’s reinforcement learning (RL) platform optimizes harnesses and trains models directly from Raindrop data, he said. Its pre-deployment simulation engine helps identify what fixes would actually impact in production; its live A/B testing then shows those changes in action.
Messages, tool calls, retries, and errors are captured in one place, and human users are notified (typically via Slack) when there’s an issue, he said. Models are trained for every customer, and signals are powering continual learning across models and harnesses. “It is condensed into something that is actually navigable, easy to understand, easy to verify,” Hylak said.
AI agents are designed to do all kinds of things for you, but they often hit three major snags: authorization, governance, and reliability.
To act on behalf of real users with real permissions, agents need a new type of security architecture, said Sam Partee, co-founder and CTO of Arcade.dev.
Partee said his company’s secure agent runtime provides this authentication and authorization layer so agents can pass critical security reviews. It also provides observability so human users can watch everything an agent is doing. Actions are attributable to the exact moment in time with the least amount of privileged scopes.
Arcade is available in an installable plugin that can be deployed on-prem in a clean room-like environment; companies can continue to use their own sign-in and security tools, Partee said. Whenever anything is run in Arcade, it’s gated by the same role-based access controls (RBACs), intrusion detection and prevention systems (IDPS), policies, entitlements, and other already-established checkpoints.
Arcade is tackling the supply chain attack problem, which has “gotten so rampant; it’s unbelievable,” Partee noted. Security and observability have continued to be challenging because “largely, the abstraction has been wrong.”
Solving enterprise customer experience (CX) is “really not straightforward,” said Claudio Rodrigues, CPO of Omilia.
Heuristic-based systems are controlled but slow; agentic systems are fast but unpredictable, Rodrigues said. Omilia built its platform to deliver both control and speed together.
The agentic, self-learning offering is built on a philosophy of observing customer service operations as they actually happen, rather than in the abstract. Omilia’s agents observe problems first-hand, listen to every customer and agent interaction, ingest data, API specs, screen recordings, and standard operating procedures (SOP), then map those to use cases for customer support, he said.
Contact centers should be a revenue driver, Rodrigues said, and Omilia’s differentiator is its speech-to-text systems and governance and observability layers.
AI creates insights, suggests improvements, automatically generates conversational agents, pulls information from documents and APIs, and designs dialogue flows. Human experts can then test real and simulated interactions and deploy into production under their supervision. Omilia combines all of this into one enterprise-wide engine that continuously learns over time, Rodrigues said.
Rodrigues said the company handles more than 3 billion calls a year, 1 million-plus voice calls a day in some deployments, and has seen 30 to 45% improvement in time to resolution (TTR). Omilia’s agents generate 21x more upsell revenue versus human agents, he said.
In a mature deployment, automation “easily” reaches 80 to 90%, he said. However, “human in the loop is still very fundamental for us.”
Nimble, a New York City-based tech startup VentureBeat previously covered for its efforts to re-invent web search for enterprises by using multiple AI agents to improve accuracy and depth, is taking another step toward its vision of a world in which agents do most of the web searching instead of us typing and reviewing the results manually.
Nimble today launched Web Search Agents, a new retrieval system designed to help AI agents perform more 21% more accurate web research while using significantly fewer tokens — 51% less compared with leading AI search alternatives on comparable, according to the firm.
While Nimble did not disclose its specific benchmarking methodology or competitors evaluated, the results underscore a growing trend in enterprise AI: optimizing retrieval has become as important as improving the underlying language models themselves.
Nimble’s leadership says the product combines self-learning retrieval strategies, proprietary web indexes, and live web access to deliver domain-specific search capabilities that outperform general-purpose web search services for enterprise workloads.
“Our research team built self-learning retrieval algorithms that learn a customer’s domain,” said Nimble CEO and co-founder Uri Knorovich in an interview with VentureBeat. “They find the exact information more efficiently, reduce the amount of multi-hop reasoning required, and lower token usage while improving accuracy.”
Rather than positioning itself as another general search engine, Nimble is targeting developers building autonomous agents that require continuously updated information from the public web for research, lead generation, competitive intelligence, compliance, and other business-critical workflows.
It’s also designed to slot in seamlessly to an enterprise’s existing systems and workflows.
“You can run the agent directly through the Nimble API with zero infrastructure,” Knorovich said. “For large enterprises, we’re partnering with Microsoft, Oracle, Snowflake, and others so customers can deploy these agent systems inside their own infrastructure.”
How does it work and stack up to other, existing AI-powered search and agentic systems? Read on to find out.
Most AI applications today rely on general-purpose search application programming interfaces (APIs) for search engines and public knowledge bases that return broad collections of files, leaving the language model responsible for determining which sources are relevant.
That process often requires multiple retrieval steps, additional reasoning, and significant token expenditure before an agent produces an answer. This is obviously inefficient and raises the cost spent to run AI search looking through irrelevant sources.
Nimble argues that before long, every enterprise will need its own methods for searching, retrieving, and validating external information since each enterprise relies on its own distinct preferred sources, signals, and standards of trust.
As such, instead of applying one search strategy to every workload, Nimble’s Web Search Agents are designed to learn the characteristics of a specific domain and adapt how information is retrieved, providing agents with structured, relevant context rather than forcing them to sift through large amounts of generic search results.
“Instead of one generic retrieval model, we build specialized retrieval models for each customer’s domain, making them faster, cheaper, and more accurate,” Knorovich explained. “A single enterprise can run hundreds of different agents. Each one has its own domain expertise, guardrails, goals, and search algorithm. The optimization starts with the second search, without requiring any setup from the customer.”
Its goal is not only to reduce redundant retrieval, but also to shorten multi-step research paths and avoid repeatedly sending raw pages through a language model for parsing, resulting in the 51% reduced token figure the company cites.
The distinction is particularly relevant for long-running enterprise agents performing research over hours or days rather than answering simple consumer questions. In those scenarios, reducing unnecessary tool calls can significantly lower operating costs while improving answer consistency.
That emphasis reflects a broader shift occurring across the AI tooling ecosystem. As foundation models become increasingly capable, infrastructure vendors are competing on everything surrounding the model—including retrieval, orchestration, memory, observability, and governance.
The launch builds on Nimble’s broader strategy of becoming an enterprise web intelligence platform rather than simply a web scraping provider. Earlier this year, the company introduced its broader Agentic Search Platform following a $47 million Series B financing, positioning itself as infrastructure that transforms the live web into structured, machine-readable data for AI systems.
The company’s latest release extends that vision with a concept it calls “Harness as a Tool,” which powers its new domain-specialized Web Search Agents. Rather than requiring engineering teams to assemble separate search APIs, browser automation, extraction pipelines, validation logic, memory systems, and orchestration code, Nimble packages those capabilities behind a managed interface.
The harness can determine what to search, navigate pages when conventional indexes are insufficient, extract relevant information, validate the results, and return the final context in a form designed for downstream agents.
Nimble also says the system retains domain-specific memory and builds proprietary indexes that improve as customers run more searches.
“The biggest research breakthrough is adding semantic memory and a caching layer to the agent,” Knorovich told VentureBeat. “The agent learns usage patterns and domain expertise over time, so every subsequent search becomes faster and more efficient.”
As for what domains Nimble can tackle, the company says it can address virtually any knowledge work domain.
“We’ve seen customers build investment banking analysts, competitive intelligence agents for product managers, go-to-market research agents, newsroom monitoring, insurance applications, life sciences research, and supply chain optimization,” Knorovich said. “Our customers surprise us every day with new agent use cases.”
However, for enterprises concerned about data privacy and retention, Knorovich assured VentureBeat that: “Nimble is zero-data-retention by design. Customer queries are never stored in our environment, and when customers deploy semantic memory and self-learning models, that knowledge stays in their own tenant—not ours.”
Nimble supported the announcement with early customer examples from AI-native software vendors and enterprise users.
AI-native CRM company Rox reported achieving a 20× reduction in token costs after adopting Nimble’s retrieval infrastructure while simultaneously improving the quality and completeness of information available to its AI agents.
Although the company did not disclose detailed workload measurements or a reproducible baseline, the example illustrates the operational savings retrieval optimization can provide for high-volume agent deployments.
Nimble says its infrastructure currently supports more than 90 million searches each day across Fortune 500 enterprises and AI-native companies operating mission-critical workflows where accuracy, completeness, and enterprise control are essential.
The platform is immediately available through an API, SDK, and Model Context Protocol (MCP) integration, allowing developers to connect Nimble directly into AI agents regardless of the orchestration framework they use.
Developers can use the platform for several categories of web intelligence, including:
Low-latency live web search
Deep multi-step web research
Web crawling
Structured dataset generation
Domain-specific information retrieval
The company also provides documentation and pre-built agents for common web extraction tasks while allowing developers to build custom retrieval agents using natural-language descriptions instead of manually maintaining scraping logic.
Nimble is offering two notably different consumption models. Developers can begin with a pay-as-you-go Agent API priced from $0.025 per Web Search Agent request at the listed low-effort setting. Companies that want Nimble to configure and manage custom data delivery can instead buy annual managed plans beginning at $2,500 per month.
Nimble enters a market that has rapidly expanded beyond traditional web search into autonomous research agents capable of planning, browsing, reasoning, and synthesizing information. Products such as ChatGPT Deep Research, Google Gemini Deep Research, Alibaba’s Tongyi DeepResearch, Perplexity, and Sakana Marlin all seek to automate knowledge work that previously required hours—or, in Marlin’s case, potentially weeks—of human research.
Rather than competing head-to-head as another end-user research assistant, however, Nimble is positioning itself one layer lower in the AI stack—as the web intelligence infrastructure that powers those agents or custom enterprise applications built on leading foundation models.
That distinction reflects an increasingly important architectural shift in enterprise AI. Most “Deep Research” systems optimize the overall research workflow, generating search plans, iteratively gathering information, and producing synthesized reports.
Nimble instead argues that the retrieval layer itself has become the primary bottleneck for enterprise AI deployments. If an agent retrieves too many irrelevant pages or performs unnecessary search iterations, token consumption, latency, and operating costs all increase before the model even begins its main reasoning process.
“Customers across life sciences, insurance, healthcare, pharma, retail, and digital-native companies are all telling us the same thing: we need to feed our agents with more accurate context, and we need to reduce the amount of tokens every task consumes,” Knorovich said.
The launch blog makes that argument more concrete by describing how teams frequently rebuild the same retrieval stack themselves. A production agent may start with a search API, then accumulate browser controls, parsers, extraction components, validation steps, memory, caching, evaluations, and custom workflow logic. Nimble is positioning its harness as a managed alternative to that growing engineering burden.
In Nimble’s view, improving retrieval before reasoning begins is more valuable than simply giving a language model more documents to analyze. The company’s Web Search Agents therefore adapt retrieval strategies to a particular workload, combining proprietary indexes with real-time web retrieval and task-specific search policies rather than applying the same search algorithm across every domain.
That makes Nimble less of a direct competitor to OpenAI’s or Google’s research assistants than to developer-focused retrieval infrastructure such as Exa and Tavily. Those platforms also provide AI-native search APIs and research capabilities, but Nimble differentiates itself by emphasizing self-learning retrieval strategies, proprietary indexing, enterprise governance, managed delivery, and token efficiency for production agents.
For organizations building their own AI systems, the distinction could become increasingly important. Foundation models are becoming more capable across the industry, shifting competitive differentiation toward the infrastructure surrounding them—including retrieval, orchestration, memory, observability, and governance. Nimble’s strategy reflects that broader trend, betting that better web intelligence can deliver larger operational gains than incremental improvements in model reasoning alone.
The different positioning is also reflected in pricing.While consumer-facing AI research assistants are generally sold as productivity subscriptions for individual users or teams, Nimble is pricing its managed service as enterprise infrastructure designed to power production applications. Its pay-as-you-go API, however, gives developers a lower-cost path to test the underlying agent technology before committing to a managed deployment.
|
Platform |
Primary audience |
Primary focus |
Lowest publicly available price (USD) |
|
Nimble |
Developers and enterprises |
Managed web retrieval and orchestration infrastructure combining specialized search, browsing, extraction, validation, proprietary indexing, and memory |
$0.025 per Agent API request (low-effort setting). Managed service starts at $2,500/month (Startup plan, billed annually). |
|
ChatGPT Deep Research |
Professionals, enterprises, and knowledge workers |
Autonomous multi-step research with iterative browsing, synthesis, and citations |
$20/month (ChatGPT Plus). Higher limits are available with Pro, Team, Enterprise, and Edu plans. |
|
Google Gemini Deep Research |
Consumers and enterprises |
Research planning integrated with Gemini, Google Search, and Google’s productivity ecosystem |
$19.99/month (Google AI Pro, U.S.). Higher-capacity AI Ultra and enterprise Workspace offerings are also available. |
|
Tongyi DeepResearch |
Developers and AI researchers |
Open research model for long-horizon information-seeking and agentic search |
Free (open source). Users are responsible for their own infrastructure and cloud compute costs. |
|
Perplexity |
Consumers, professionals, and enterprise teams |
AI-powered web search and cited research |
Free entry tier. Perplexity Pro starts at $20/month with Enterprise Pro available separately. |
|
Exa |
Developers and AI platform builders |
AI-native search, content retrieval, and asynchronous research agents |
Free developer tier (includes monthly credits). Paid Search API pricing starts at approximately $7 per 1,000 requests while Agent runs range from $0.012 to $1.00 per run depending on effort level. |
|
Tavily |
Developers building AI agents |
Search, extraction, crawling, and research APIs for agents and RAG workflows |
Free developer tier (1,000 monthly credits). Pay-as-you-go usage starts at approximately $0.008 per credit. |
|
Sakana Marlin |
Enterprises, strategy teams, financial institutions, and research organizations |
Ultra Deep Research for hours-long strategic reasoning and executive-grade reports |
Pay-as-you-go from approximately $0.61 per credit (¥98/credit) with with 100 credits required per research run (approx $61 per run). The first subscription tier is Pro at approximately $936/month (¥150,000/month) followed by Team at approximately $2,495/month (¥400,000/month) with Enterprise pricing available by quote. |
The comparison reveals three increasingly distinct markets.
ChatGPT Deep Research, Gemini Deep Research, and Perplexity operate primarily as user-facing research assistants.
Exa and Tavily provide developer-facing retrieval and research APIs.
Nimble and Sakana Marlin occupy more enterprise-oriented territory, but at different layers: Nimble supplies retrieval infrastructure, while Marlin performs long-horizon strategic analysis.
Sakana Marlin is particularly useful as a counterpoint. It is positioned as a “Virtual CSO” rather than a search API, running autonomous research loops for as long as eight hours and producing executive-ready reports, references, and supporting materials.
Nimble, by contrast, is designed to sit beneath those kinds of systems, supplying the specialized retrieval, browsing, extraction, validation, and orchestration that enterprise agents need to gather reliable external information before reasoning begins.
The comparison therefore should not be read as a direct price-to-price evaluation. A $20/month ChatGPT Plus or $19.99/month Google AI Pro subscription buys an individual AI workspace with Deep Research capabilities.
Nimble’s $2,500/month managed plan funds concurrent production agents, managed ETL, MCP integration, web-page capacity, storage, and hands-free data delivery.
Sakana Marlin’s approximately $936/month (¥150,000/month) Pro plan pays for extended, compute-intensive strategic research workflows.
Each price reflects a fundamentally different product boundary and deployment model rather than simply a different level of AI capability.
As enterprise AI systems mature, the industry is increasingly recognizing that model quality alone does not determine application performance.
Large language models frequently fail not because they cannot reason, but because they lack timely, trustworthy external information. That reality has fueled rapid investment across retrieval-augmented generation, AI-native search, web intelligence platforms, knowledge graphs, browser automation, and agent infrastructure.
Nimble’s launch reflects this evolution by focusing less on building another frontier model and more on improving the quality of information flowing into existing ones.
Whether the company’s reported 21-point improvement in answer quality and 51% reduction in token usage hold up across a broad range of enterprise deployments remains to be independently validated.
The larger strategic bet is that, as frontier models become more interchangeable, companies will differentiate themselves through the data, retrieval policies, trusted-source rules, memory systems, and orchestration layers surrounding those models. Nimble is not trying to build the researcher that sits in front of the user. It is trying to become part of the infrastructure that determines what the researcher can find, how efficiently it can find it, and whether the resulting evidence is complete enough to support production decisions.
Web Search Agents are available through Nimble’s API, SDK, and MCP integrations, with a free trial available for developers evaluating the platform.