A team routing queries across a coding specialist, a logic specialist, and a generalist model assumes each will cover the others’ blind spots. A new study evaluating 67 frontier models from 21 providers shows that assumption is mathematically flawed — and the flaw has a name: the co-failure ceiling.
The assumption works like this: as long as two models don’t usually fail on the exact same prompts, combining them is supposed to create a safety net against failures.
The real limit on orchestration is not how often models disagree, but the percentage of prompts where every model in the pool gives the wrong answer at once. By ignoring the co-failure ceiling, enterprises are building complex, expensive routing infrastructure to chase performance gains that do not exist. Fortunately, developers can use this same math to build a cost-free test that determines exactly when multi-model orchestration will actually pay off.
To orchestrate multiple language models, developers typically rely on three architectures. Model routers act as traffic cops, sending complex queries to expensive models and simple queries to cheaper ones. Cascades send every prompt to a cheap model first, only escalating to a premium model if the initial system signals low confidence. Finally, approaches like Mixture-of-Agents (MoA) fuse multiple models by asking them the same question and generating a synthesized answer from their combined outputs.
These architectures introduce a “shadow price” to inference costs. Every time a development team implements a router or a cascade, they pay a premium in added system latency, complex infrastructure maintenance, and increased governance risks across multiple API providers.
To justify these operational costs, engineers rely on “pairwise error correlation” to select their model pool. Imagine a developer has Model A, which writes excellent Python but fails at SQL, and Model B, which writes excellent SQL but fails at Python. Because they fail on different types of prompts, their pairwise error correlation is low. The developer assumes that by placing a routing layer in front of them, they have created a composite system that rarely fails at coding.
According to the study, throwing diverse models together based on low correlation can actually hurt performance if the models are not equally capable — when you vote across diverse but unequal models, the weaker ones often gang up and outvote the smartest one.
Josef Chen, author of the paper, told VentureBeat that in their experiments, “Naive majority voting across unequal models had negative mean gain (minus 10 points on our hard mix): diverse-but-weaker members outvote the strong one.” The actionable advice for developers is to “combine only models within a matched quality band.” If you cannot match quality, take the single-model baseline and spend your budget on the best model available.
The paper provides one bright spot for this approach regarding MoA architectures. When building ensembles, teams often use “Self-MoA,” where they query the same premium model multiple times to generate a synthesized answer. The researchers found that at matched quality, building a diverse ensemble of models with low pairwise correlation beats a high-correlation Self-MoA setup.
However, when teams use that same pairwise correlation metric to predict the absolute accuracy of their overall system, the math breaks down.
“So teams pay the orchestration overhead up front (latency, complexity, multi-provider operations) on the assumption that a diversity dividend arrives later,” Chen said. “Usually it doesn’t, because today’s best models agree, and, worse, they fail on the same queries … the prompt simply carries little signal about which model will be the one that’s right when the frontier disagrees.”
The core finding of the study centers on a metric called the “co-failure rate” — the formal name for the all-wrong scenario described above. No router, voting system, or cascade can ever achieve an accuracy higher than the ceiling it imposes.
The coding, logic, and generalist pool shows low pairwise correlation on routine prompts — they rarely fail together. But the co-failure ceiling represents the obscure, highly complex edge case that pushes past the limits of current AI architectures. If a prompt is so difficult that all three models hallucinate or fail, it does not matter how intelligently the router distributes the task. The entire pool wipes out at once.
The researchers tested their 67-model pool, which included GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro, on the open-ended MATH-500 math benchmark. Based on standard pairwise correlation, statistical models predicted that the entire pool would wipe out simultaneously on only 2.3% of the questions. In reality, the co-failure rate was 5.2%.
Standard correlation metrics underestimated the failure rate by roughly 2.25 times. The culprit is not just independent difficulty, but a shared failure point.
“The driver is what we call a common-mode atom: a slice of queries on which the entire market fails together, which no pairwise statistic can see,” Chen said. “Adding a 20th model to your pool doesn’t buy tail coverage. The tail is shared.”
The researchers also found that task format directly triggers co-failure. When they took graduate-level science questions from the GPQA benchmark and changed them from multiple-choice to free-response formats, the all-wrong tail expanded to 12.7%.
Developers can engineer around the ceiling, though. “The engineering implication is uncomfortable: multi-model setups buy the least exactly where teams want them most, on open-ended generation,” Chen said. “Anywhere you can convert generation into verification or constrained selection (structured outputs, checkable answers, execution tests), you reopen the ceiling.”
Ultimately, the researchers found this ceiling limits AI applications in two distinct ways, depending on the domain:
Ceiling-bound environments (e.g., open-ended math): The co-failure rate is high. The task is too hard, and all models fail simultaneously. No amount of routing can bypass the lack of underlying capability.
Realizability-bound environments (e.g., graduate-level science): The co-failure rate is near zero, meaning at least one model in the pool usually knows the answer. However, the models disagree so subtly that a routing layer cannot reliably pick the correct answer without an omniscient oracle.
Before dedicating engineering hours to building a router, teams can calculate their absolute performance ceiling for free using a mathematical formula called a Clopper-Pearson bound.
The Clopper-Pearson bound operates as a worst-case scenario calculator. If you flip a coin ten times and get eight heads, you cannot guarantee the coin will land on heads 80% of the time forever. The bound takes a small sample of test questions and outputs a mathematically guaranteed ceiling.
Applied to language models, suppose a team tests a pool of five agents on 50 sample queries and finds they all fail together on just two questions. A developer might assume their multi-agent system will achieve 96% accuracy in production. The Clopper-Pearson formula corrects this optimism. It analyzes the small sample size and provides a mathematical guarantee that the true co-failure rate could actually be as high as 12%.
To use this in practice, enterprises must build a held-out dataset. A fintech company, for example, could take 200 complex customer support tickets from the previous quarter and have human agents write perfect resolutions to serve as a benchmark. While this sounds like a heavy manual project, mature engineering teams can automate the entire ceiling calculation.
“Integration is trivial: it’s a counting job over eval logs teams already produce,” Chen notes, “so it runs in the same CI stage as the eval suite and re-triggers whenever the model pool or the workload changes.”
The engineering team then runs its candidate models against these 200 tickets once and records the results. When they want to evaluate multi-model configurations, they can use the co-failure rate measure to predict the maximum accuracy they can get from the system without running extra queries.
One important conclusion the study draws is that on tasks where answers can be definitively checked, combining models rarely beats using the single best model on the market, unless the team possesses an exceptionally strong query-level routing signal.
In an enterprise environment, a definitively checked task has an objective, zero-tolerance answer. This includes generating a SQL query that must execute without error, extracting a specific invoice total from a 50-page PDF, or formatting a JSON payload that perfectly matches a strict schema. For these tasks, enterprises are usually better off paying a premium for the smartest frontier model rather than weaving together three cheaper models and hoping a router picks the correct output. The study didn’t test subjective, ungraded tasks like drafting marketing copy — the authors note that whether these findings hold outside their verifiable benchmarks remains an open question.
Because this mathematical check is free, enterprise teams can track their own co-failure rates as new models drop.
“The measurement costs nothing, so any team can track its own co-failure rate across model generations and watch whether the tail is closing,” says Chen. Ultimately, “the lever buyers hold is failure-mode heterogeneity and market churn, not model count.”
Presented by SAP
Generating code with AI is fast, but getting that code to run reliably inside a large enterprise, integrated with live systems, governed for compliance, and maintainable over years requires foundational work that most organizations underestimate.
While 81% of all organizations have a detailed strategy, only 12–16% reach AI‑driven execution, says SAP’s Michael Ameling, CPO of SAP Business Technology Platform, and the reasons rarely come down to the quality of the generated code.
“Across industries, enterprises that have invested heavily in AI tooling are hitting a wall when generated code meets the reality of their existing environments, because generating code and operationalizing it are not the same problem,” Ameling says.
There are specific requirements for deploying AI-generated logic at enterprise scale: what data and integration readiness actually look like, how governance works when AI agents move from producing recommendations to executing workflows, and how development teams are changing their role as AI takes over more of the coding work.
The productivity gains from AI code generation are real and well-documented, but the ease of prototyping has given many organizations a misleading sense of how far along they actually are.
“Generating code is one thing,” Ameling says. “Enterprise customers, including multinationals and large organizations, need to ensure there are no compromises in compliance or security. Code that runs reliably for ten or twenty years, as it does at many of SAP’s largest customers, also has to be maintained, patched, and understood by whoever inherits it. Life cycle management, in other words, does not generate itself.”
The issue is rarely the generation quality. Teams build something compelling, then discover they lack access to the data it depends on, or the integrations it assumes, or the permissions required to run it in a real environment. The problem is essentially that AI amplifies an organization’s existing data and process maturity, but it can’t substitute for it.
This dynamic intensifies as AI moves from producing code to executing actions. Latency, cost, and system load all increase when logic runs continuously against live data rather than rendering a one-time output. The performance requirements of an autonomous agent operating across a multinational’s transaction systems are categorically different from those of a developer copilot.
The architecture challenge that most enterprise AI projects underestimate is integration. Real enterprise environments are not clean slates: they combine cloud systems, legacy on-premise infrastructure, fragmented data stores, and dozens of business applications that were never designed to talk to each other. Getting AI-generated logic to operate reliably across all of them requires a layer that unifies data access, process context, and governance, and it has to be in place before any agent starts executing. And organizations that see AI as a reason to defer infrastructure modernization are making a mistake.
“The question is not whether to modernize or not. Of course you need to modernize,” Ameling says. “But the value you get on top of this is much higher with AI. Federated data access and harmonized process layers are not alternatives to upgrading a fragmented landscape, they’re what make the upgrade worthwhile.”
At the platform level, this translates into a set of practical requirements: structured data integration, end-to-end process visibility, and the ability to discover and connect to APIs across both modern and legacy systems. SAP’s approach with the Business AI Platform draws on tools including its Joule Studio, Integration Suite, Business Data Cloud, and SAP AI Agent Hub enterprise architecture layer to provide that context. The goal is to give AI-generated logic accurate, current knowledge of what a business is doing and how, rather than just access to raw data.
AI agents handle large challenges by dividing them into smaller, autonomous tasks, with each agent responsible for a specific domain, and all coordinated toward a shared outcome. A financial close, for example, involves dozens of discrete sub-processes. Agents handling each task in parallel, within defined constraints, can compress cycle times dramatically, but only if the underlying systems they interact with are coherent and accessible.
When AI moves from assistant to operational actor, the governance questions loom large, because agents that trigger workflows, update records, and interact with live business systems need the same accountability framework that applies to human employees, i.e., identities, defined privileges, and auditable behavior.
There are two distinct models:
Principal propagation, where an agent acts on a user’s behalf, inheriting that user’s permissions and scope.
System-triggered agents, where the agent operates under its own identity and role-defined privileges, functioning more like an automated HR role than a personal assistant.
Both models require the same underlying infrastructure: an agent hub where operators can see which agents exist, what APIs they can access, and what they are authorized to do. Observability also needs to be operationalized correctly for AI, combined with both technical and business evals.
“In production, openness is very important,” Ameling says. “We use OpenTelemetry as a framework, so we can integrate with other solutions, for end-to-end observability of the tool, third-party agents and the like.”
On top of that, standard technical evals, which test whether an agent produces consistent outputs, are necessary but not enough. Business evals assess whether an agent is actually moving the performance indicators it was deployed to improve, but it has to work end-to-end.
Where the testing happens is equally important. The traditional software development cycle across dev, test, and production environments breaks down when a model produces different outputs depending on whether it is running against test data or live data. Getting to trustworthy AI in production means accepting that validation looks fundamentally different from what engineering teams have practiced for decades, with live environment testing, even A/B/C testing to ensure outcomes are reliable.
The role of the developer is not disappearing in this environment, but its center of gravity is shifting. The productivity multiplier is significant when developers can run multiple coding agents in parallel across open terminals, each working on a separate problem and each taking several minutes to complete. But it introduces a new kind of cognitive demand, because humans have to stay in the loop. That means tracking context across concurrent workstreams, evaluating outputs that range across large codebases, and making architectural judgments that no agent can be trusted to make alone.
“The more specific and complete the prompt, the less intervention is required, and developers are learning that bringing more context upfront pays dividends in reduced back-and-forth,” Ameling says. “But the output still needs to be understood, not just accepted.”
The competitive edge will remain intellectual property, not tooling. The companies that pull ahead will be those that most effectively encode their domain knowledge into the systems they build.
“A manufacturer’s process expertise, a financial institution’s risk logic, a logistics firm’s routing intelligence, these are the assets that AI can accelerate, but only if the organizations that hold them do the work to make them accessible and usable,” Ameling says. “Protect that, and apply AI to accelerate your differentiation.”
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.
Presented by Oracle NetSuite
Every major technology transition produces a set of assumptions about where the market is headed. The assumptions are often directionally correct, but they tend to underestimate the degree to which organizations adapt new technologies to their own circumstances. AI is following a similar trajectory.
Many current discussions about enterprise AI assume a future in which employees interact with business systems through a common interface. The details vary depending on the prediction, but the destination often looks similar: a conversational system that becomes the primary way people access information, complete tasks, and interact with software.
The history of enterprise technology suggests a more complicated outcome. Organizations rarely adopt new capabilities uniformly because different parts of the business operate under different constraints. A finance team responsible for reporting accuracy, controls, and approvals approaches technology differently than an analytics group exploring operational data. Both groups have different requirements than a customer service organization focused on response times and case resolution. Even when there is broad agreement that a technology is valuable, the path to adoption tends to vary across functions.
The shift to cloud software followed this pattern — some organizations moved aggressively while others spent years operating hybrid environments. Different departments often modernized on different timelines, reflecting the priorities of the work itself rather than any industry consensus about the correct pace of adoption.
AI has accelerated many aspects of technology development, but it has not changed this underlying dynamic. Organizations still evaluate new capabilities through the lens of existing processes, responsibilities, and operational requirements.
For some employees, the most useful AI capabilities may be the least visible ones. A finance manager closing the books is often less interested in a new interface than in shortening a reporting cycle. An operations leader dealing with inventory issues is usually focused on identifying problems earlier and resolving them more quickly. In these situations, the value of AI comes from reducing the amount of effort required to complete existing work.
At the same time, another group of users increasingly wants direct interaction with AI systems. Analysts, planners, and operational teams often benefit from the ability to explore information conversationally, compare scenarios, and investigate questions that do not fit neatly into predefined reports. For these users, the interface itself becomes valuable because it provides a more flexible way to work with business information.
A customer service representative handling a high volume of inquiries has different requirements than a financial analyst investigating a trend in operating expenses. One benefits from information appearing automatically within an existing process while the other may benefit from the freedom to ask follow-up questions, explore alternative explanations, and move through data more dynamically.
Many organizations are discovering that both patterns exist simultaneously, which reflects a broader reality about how businesses evolve. Operational complexity accumulates gradually, systems multiply, and processes become fragmented. Information becomes distributed across applications, reports, spreadsheets, and workflows and employees spend increasing amounts of time locating information before they can begin acting on it.
Much of the value created by enterprise software over the last several decades came from reducing that fragmentation. Bringing financials, operations, inventory, customer information, planning, and reporting into a common system created a more complete picture of how the business was operating.
AI is beginning to address a related problem. Once information exists within connected systems, employees still need to find it, interpret it, and apply it. Reporting cycles consume time. Routine questions require investigation. Managers often spend considerable effort assembling information before they can make decisions. As organizations grow, these activities become increasingly expensive because they consume attention from people whose expertise is often in short supply.
AI’s promise is to reduce the effort required to move from information to action.
At Dura Software, AI-connected workflows are helping automate portions of revenue reporting that previously required manual preparation during each reporting cycle. Sloan Session, CFO at Dura Software, described the arrangement in practical terms: “The agents handle the pull. The humans handle the judgment and the personal touch.”
That observation captures an important aspect of current AI adoption. Most organizations are not attempting to remove judgment from business processes. They are trying to reduce the amount of time spent gathering, organizing, and preparing information so that experienced employees can focus on the decisions that require expertise.
A similar pattern emerged at S&B Filters. Employees previously spent several minutes during customer interactions collecting backorder information from multiple systems. By connecting AI to operational data, the company reduced that process to seconds and eventually extended the capability directly to customers through self-service.
In both cases, the benefit comes from reducing the friction associated with finding and using information rather than introducing a new interface. The moment information becomes easier to access, questions about access itself become more important. Permissions, approval structures, and security policies exist because businesses need mechanisms for controlling access to information and managing risk. Those requirements do not disappear when employees begin interacting with data through AI systems. If anything, they become more important because AI can make information easier to access.
Berry Carter, CEO of S&B Filters, described the principle clearly. If a user cannot access specific information within NetSuite, that user should not gain access to the same information through an AI assistant. The statement sounds obvious. Implementing it consistently across systems, workflows, and models requires considerably more discipline than the statement itself suggests.
Lauren Polasek, former NetSuite administrator and board member of the Texas NetSuite User Group, recently made a related point. Connecting technology is often the easier part. Organizations still need to determine which tools should be used, who should have access to them, and how governance should evolve as adoption expands.
This is one reason predictions about a single AI interface are difficult to reconcile with how enterprises actually operate. The requirements of a finance organization closing the books are different from those of a customer service team handling thousands of interactions each day. Some AI capabilities will be embedded directly into business processes where employees may barely notice them. Others will provide more direct access to operational information through conversational systems. Many businesses will end up using both approaches because the underlying work is different.
That perspective has shaped how we think about AI at NetSuite. Some customers want AI embedded directly within operational workflows. Others want the ability to connect NetSuite data to external models and assistants so they can interact with business information through tools that are already part of their daily work. Increasingly, organizations are asking for both.
The NetSuite AI Connector Service and our support for Model Context Protocol (MCP) were designed with that reality in mind. The goal is to allow organizations to connect business information securely to the workflows and systems that make sense for them while continuing to benefit from AI capabilities built directly into NetSuite.
The history of enterprise software suggests that adoption rarely follows a straight line. As organizations adopt AI, business leaders should identify the business objective and the workflows involved so they can match the solution to the reality of the work.
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.
Five years and $27.7 billion after Salesforce acquired Slack, the two products are finally starting to function as a single system. On Tuesday, Slack launched an integration that connects Slackbot — the personal AI agent built into every workspace — to the entire Salesforce platform, including CRM data, Tableau analytics, Data 360 customer profiles, and a growing constellation of third-party applications, all through a single conversational prompt.
The mechanism behind the expansion is a set of dedicated Model Context Protocol (MCP) servers from Salesforce that connect Slackbot to the company’s Headless 360 infrastructure. In practical terms, a salesperson can now ask Slackbot for a customer’s deal history, receive a live Tableau visualization of pipeline trends, update a CRM record, and trigger a DocuSign approval — without ever switching tabs or logging into another application. According to Slack, the Salesforce IT team has already used this architecture to save its 1,500-plus engineers “thousands of custom coding hours annually.”
The timing is not accidental. Slack is making this move amid escalating competitive pressure from Microsoft Teams, which claims 320 million-plus monthly active users and has Copilot embedded across the Office suite, and from Google, which continues to weave Gemini deeper into Workspace. And just days ago, The Information reported that some smaller companies are using Anthropic’s Claude to replace Salesforce CRM entirely — one Atlanta-based property management firm with about 55 employees reportedly saved around $100,000 annually by building a custom replacement using Claude Code and Replit.
Against that backdrop, Slack CMO Ryan Gavin sat down for an exclusive interview with VentureBeat to frame the announcement and argue that the company’s future depends on an idea he calls “multiplayer AI” — and that the 25 years of customer data locked inside Salesforce is an asset no vibe-coded alternative can replicate.
Gavin’s core argument is that the enterprise AI conversation has been stuck in single-player mode for too long, and that Slack is uniquely positioned to break it open.
“So much of what we’ve seen are just these incredible tools that have largely been single-player, incredible tools for individual productivity, helping people complete tasks and write code,” Gavin told VentureBeat. “But as we’ve always known at Slack ever since our inception, work is a team sport. For AI to really take hold in the enterprise, it has to be multiplayer.”
The distinction matters commercially. Most AI assistants today — ChatGPT, Claude, Copilot — default to one-on-one conversations with a single user. A researcher queries a model, gets a response, and acts on it alone. The insight stays in a private chat window, invisible to colleagues. Gavin argues this creates a new version of the tab-switching problem that plagued pre-AI enterprise software, except now employees are also navigating dozens of individual agent interfaces on top of their existing applications.
“It’s going to benefit almost no one if every enterprise application out there spawns hundreds of agent babies, and employees end up in a worse world than they were before,” Gavin said.
Slack’s answer is to make Slackbot the orchestration layer. Because everything happens in shared channels, any action an agent takes — pulling a customer profile, flagging a deal risk, updating a Jira ticket — is visible to the entire team. A colleague can redirect, build on, or correct the agent’s work in real time.
The technical backbone of the announcement is the Model Context Protocol, an open standard originally developed by Anthropic that defines how AI models discover and invoke external tools. MCP has seen rapid adoption across the AI tooling ecosystem. By early 2026, it had been adopted by Claude Code, Cursor, GitHub Copilot, and OpenAI’s tooling, with managed hosting available from AWS, Cloudflare, and Vercel. As a DEV Community explainer puts it, MCP “is the closest thing the AI tooling ecosystem has to a standard.”
In this implementation, Salesforce exposes its platform capabilities — CRM records, Tableau visualizations, Data 360 customer profiles, Agentforce agents — as MCP servers. Slackbot operates as an MCP client, connecting to those servers and routing user queries to the appropriate back-end system. When a user asks Slackbot about a customer, the bot discovers which MCP tools are relevant, calls them, and synthesizes the results into a single response — all within the Slack conversation.
Gavin explained the architecture in simple terms: “Salesforce is extending what has always been our open platform through our Headless 360 strategy — making all of these MCP endpoints available. And then Slackbot acts as an MCP client, connecting to those MCP servers and bringing all that data in within the confines of a trusted permission platform.”
That permission layer is critical. Slackbot respects each user’s Salesforce permissions, meaning a marketing coordinator cannot accidentally access sales pipeline data they are not authorized to see. Validation rules, field-level security, and org-wide data boundary configurations carry over automatically. For admins, setup requires no custom integration code — Salesforce MCP servers can be discovered, installed, and governed from a single UI using the existing Slack-Salesforce connection.
Salesforce first introduced the Headless 360 concept at its TDX developer conference in April, positioning it as an API-driven layer that exposes the platform’s data, workflows, and governance controls so that software agents, rather than human users, can execute business processes directly. As CIO.com reported at the time, analysts viewed the move as an effort by Salesforce “to position itself as a central layer for managing agent-driven operations across different business functions.”
When asked whether Slack is making a risky bet on MCP as a protocol — given that standards in AI tooling can shift rapidly — Gavin reframed the question entirely.
“We’re not betting on MCP, per se. We’re betting on what we’ve always bet on, which is that Slack is an open platform,” Gavin told VentureBeat. “MCP happens to be the best agent-to-agent protocol that the industry is rallying around right now, but if something better came out tomorrow, you’d see the same pattern from Slack — we’re going to stay open. MCP and APIs are simply tools that facilitate that.”
That open-platform philosophy is central to Slack’s identity and, Gavin argues, its competitive differentiation. Slack already hosts more than 2,600 app integrations. The new MCP-native partner ecosystem includes Atlassian, Box, DocuSign, Canva, Lucid, Zoom, and more than 25 additional companies, each of whose agents can be added directly to shared Slack channels. MuleSoft Agent, now connected to Slackbot, helps manage integrations for the team — checking system health or surfacing critical error alerts in the same workspace where the team is already collaborating.
But MCP is not without trade-offs. The protocol requires tool discovery on every connection, and large tool libraries can consume significant context tokens. One technical analysis noted that a server exposing 300 tools could cost 5,000 to 10,000 tokens per session before the model does any useful work. For an enterprise like Salesforce with hundreds of potential tools across CRM, analytics, and service platforms, careful filtering and segmentation of MCP servers become essential design decisions — a challenge the company will need to navigate as the ecosystem scales.
Perhaps the most delicate topic in the interview concerned Slack’s relationship with Anthropic, the AI lab behind Claude — and one of Slack’s most visible power users. Just last week, Anthropic launched Claude Tag, a persistent AI teammate that works inside Slack channels, prompting confusion among Salesforce employees who worried it competes directly with Slackbot and Agentforce. The Information reported internal anxiety about whether Salesforce was welcoming a competitor into its own living room. Salesforce has financial reasons to maintain the partnership: the company reportedly expects to spend $300 million on Anthropic tokens this year and holds a stake in Anthropic.
Gavin addressed the tension head-on, framing it as a feature of Slack’s platform strategy rather than a threat.
“We’re incredibly excited and bullish about what Anthropic is bringing into Slack. Period. End of statement,” Gavin said. He noted that Anthropic “is building roughly 65% of their code with Claude in Slack,” and pointed out that ChatGPT was originally built in Slack, as was Perplexity.
“Building nowadays happens in the open, and every company is going to be building in the open with tools like this, and you need a platform to build in the open,” Gavin said.
His argument is that feature overlap between Slackbot, Claude Tag, and other third-party agents is “actually a feature, not a bug” — a sign of a healthy platform rather than a competitive vulnerability. He compared it to an ecosystem where multiple products serve similar needs but win on craftsmanship, ease of use, and integration depth.
“One of the reasons Slackbot has been the fastest-adopted feature in Salesforce history is the simplicity, the approachability — underpinned by the trust that comes from having an agent that knows me, knows my tone, knows my work, knows my people, knows my data,” Gavin said.
The distinction Slack draws is structural: Slackbot has access to a user’s full workspace context, Salesforce data, permissions, and connected applications by default. Claude Tag, by contrast, only sees the channels it is explicitly added to. For Slack’s leadership, that asymmetry is the moat.
Asked directly about competitive positioning against Microsoft Teams and Google Workspace, Gavin pointed to Slack’s open channel architecture as the differentiator no competitor can replicate.
“If you spend any time in Teams, it’s a lovely tool for chat, direct messages, and video, but it has no platform for open communication across organizations,” Gavin said. “Its SharePoint-based architecture is fundamentally limiting.”
He cited Shopify as an example, where an internal AI agent called River is deployed across approximately 4,400 channels serving 6,000 employees. He also referenced a Fortune report noting that Microsoft’s own head of AI mandated that his team run on Slack rather than Teams — a pointed detail Gavin clearly relished. “There’s a reason for that,” he said. “We’re in an era right now where openness matters, and all the other tools you mentioned, they’re still relatively closed.”
The competitive pressure is real and intensifying. Microsoft has integrated Copilot across its entire productivity suite, giving it a distribution advantage that reaches virtually every Fortune 500 company. Google has been similarly aggressive with Gemini across Workspace. And new entrants are crowding the market: a startup called Viktor, which embeds AI agents inside Slack and Teams workspaces, recently raised a $75 million Series A led by Accel — with Slack cofounders Stewart Butterfield and Cal Henderson participating as angel investors.
Box, one of the enterprise customers highlighted in the announcement, told Slack it aims to have its sellers complete 75 to 80 percent of their work inside Slack. Gavin repeated that figure as evidence that the platform is becoming the default workspace for entire organizations, not just engineering teams — a shift he believes accelerates as AI makes every employee a builder.
Gavin saved what he considers the most underappreciated element of the announcement for last: the democratization of Salesforce’s CRM.
For 25 years, Salesforce’s CRM has been used primarily by sales, service, and marketing professionals — a relatively modest percentage of a company’s total workforce. The promise of Slackbot as a conversational interface is that any employee, regardless of their role or technical fluency, can now query and act on CRM data simply by asking a question in natural language.
“What most people don’t realize is that this democratization of CRM is going to take its usage from a modest percentage of employees to the entire enterprise,” Gavin said. “When you can make systems like Data 360 or Agentforce for Sales accessible to the entire employee base — not just a percentage — think about how much more valuable those investments become.”
He cited Engine, a company that handles 800,000 customer inquiries a year, as an example. Previously, answering a customer inquiry required a specific employee with access to a specific tool to look up a customer’s history. Now, anyone in the company can ask Slackbot and see a complete customer profile, review case history, and write updates — all without being retrained or learning a new interface. Engine’s CEO Elia Wallen, in a statement sent to VentureBeat, described the integration as enabling employees to “make data-driven decisions and take action without leaving the conversation.”
The financial logic is straightforward: if Salesforce can make its platform useful to 100 percent of a customer’s workforce rather than the 20 or 30 percent who currently hold licenses, the value of the existing Salesforce investment multiplies without requiring a proportional increase in spending. That pitch becomes especially potent at a time when CIOs are scrutinizing every line of their AI budgets.
The announcement is a significant architectural evolution for Slack, but several questions remain unanswered.
First, pricing. The company did not directly address whether Slackbot’s MCP-powered Salesforce integration will require additional SKUs or license tiers. As Info-Tech Research Group analyst Scott Bickley cautioned when Headless 360 was first announced in April, “Salesforce’s MO seems to be to announce new capabilities that require SKUs. CIOs should be asking about pricing now.”
Second, performance. Routing user queries through MCP servers to Salesforce back-end systems introduces latency that could affect the conversational feel Slack prides itself on. Neither the press release nor the interview disclosed SLAs for MCP tool calls — a gap that enterprise buyers will want addressed.
Third, the competitive dynamics of the platform play. Slack’s open-platform philosophy invites powerful partners like Anthropic and OpenAI into its ecosystem, but those same partners are building their own surfaces for enterprise work. Anthropic reportedly plans to expand Claude Tag to Microsoft Teams, email, and other project management tools — meaning the partner Salesforce is paying hundreds of millions a year is building the infrastructure to be useful without Slack at all.
And fourth, the broader existential question facing all enterprise software: whether AI agents will ultimately reduce the need for CRM systems entirely. Gavin’s pitch — that Slack makes CRM more valuable by making it more accessible — is the inverse of the bear case. The market will ultimately decide which thesis prevails.
Salesforce reported record first-quarter revenue of $11.1 billion in fiscal Q1 2027, with Agentforce ARR surpassing $1 billion for the first time and combined AI and data ARR reaching $3.4 billion. Those numbers suggest the AI strategy is beginning to generate real revenue, even as the company navigates a market that remains uncertain about the long-term trajectory of legacy enterprise software.
“Slack has quickly moved from this beloved collaboration tool from the last ten years to now this multiplayer AI platform that we call a work operating system,” Gavin said.
Five years ago, Salesforce paid $27.7 billion for what was, at its core, a very good group chat application. On Wednesday, it started trying to prove that group chat was never the product — it was the foundation. In the age of AI agents, the most valuable real estate in enterprise software may not be the database where the data lives. It may be the conversation where the decisions get made.
Presented by Box
Content access, governance, and platform flexibility are emerging as the dividing lines between AI leaders and laggards, according to the new State of AI in the enterprise report from Box, which surveyed 1,640 IT decision makers across the US, UK, France, and Japan. One of the report’s major findings is the speed of the shift: the combined share of organizations describing themselves as advanced or leading edge soared from 8% to 64% just over the past year, while the share calling themselves early stage or not yet started collapsed from 53% to just 9%. Eighty percent of organizations reported a notable return on their AI investment, defined in the survey as an improvement of at least 10%, and more than half saw measurable business impact within six months of getting a project approved.
The swing is largely due to how enterprises are now organizing their AI use rather than to any single technical breakthrough, says Olivia Nottebohm, COO of Box.
“We’ve moved from standalone experimentation that lived at the individual level into systematized, integrated agentic operations, agents that are in production and can be used in a repeatable manner,” Nottebohm says. “That’s where the impact is coming from.”
The divide between tiers is a matter of execution. Significantly, half of leading-edge companies reported AI-driven ROI above 25%, compared with just 11% of early-stage companies, with the advanced (33%) and developing (16%) tiers falling steadily in between. But Nottebohm says the real differentiator was not whether companies adopted AI, but how rigorously they integrated and managed it.
“What separates the leading edge is the operating muscle they’ve built: the right teams to deploy agents, formal governance to control them, and consistency in the content layer those agents work from,” she explains. “Earlier stage companies are approaching it in a much more ad hoc, experimental way, letting people play around with it without the same intent or structured design.”
Content, rather than model quality, is the defining bottleneck of 2026. Ninety-six percent of organizations say agents need access to company-specific content, yet only 36% have connected agents to trusted content across many use cases. It’s an issue of trust rather than raw capability.
“We started this journey assuming enterprise AI was about access to the latest model,” Nottebohm says. “But the question now is whether agents have access to the right content, and whether that content is protected, because those agents are only as good as the content they can reference, and only as safe as the security around it.”
Getting that content layer right has a second benefit beyond safety, since it’s also what finally lets agents work across departments that previously operated in isolation from one another. And while roughly a quarter of organizations point to data fragmented across systems, 24% cite difficulty integrating AI into existing systems, 21% say they lack adequate permissions and access controls, and 18% describe their content as too unorganized to make accessible at all. Among the most mature organizations, 63% now treat unstructured documents, contracts, and reports as a competitive advantage rather than dead weight sitting in a digital filing cabinet.
Nearly half of all organizations say they have already experienced an AI-related data exposure incident. That figure rises to 60% among leading-edge companies, which may face greater exposure from more agents and connected systems — but may also be better equipped to detect it.
The share of organizations reporting established or advanced governance frameworks rose from 24% in 2025 to 73% this year, but real gaps remain in instrumentation: only 39% have comprehensive visibility across sanctioned and unsanctioned AI use, 34% have formal standards for how agents access company data, and 27% still describe their governance as ad hoc. But those incidents function as a forcing mechanism rather than a setback, Nottebohm says.
“Governance used to be seen as something that slowed people down, but 93% of respondents told us better governance is actually what let them move faster,” she explains. “It makes scaling AI survivable. Once content is secured and highly permissioned, you can run multiple agents across multiple processes and get a real multiplier effect.”
One practical consequence of that shift is that permission structures built for human employees are now being revisited with agents in mind, a process most enterprises are only partway through.
“The permissions enterprises set up two years ago need to be reviewed,” she explains. “Until fairly recently, people weren’t setting permissions on a document with how an agent might use it in mind, but now they’re much more deliberate about that. It leaves them with a whole corpus of unstructured data to go back through and either clean up or repermission.”
That’s part of a broader move away from governance designed for people and toward governance designed for agents from the start.
“Enterprises need to make the transition from governance that’s retrofitted from human workflows to governance that’s built specifically for agents,” Nottebohm says. “That means tracking what an agent has touched, whose permissions were applied, and which sources were used, and all of that is now shaping how governance gets applied.”
“The days of token-maxing are already gone,” Nottebohm says. “It’s now about the responsibility of delivering efficient AI. Organizations want to use the cheapest model that meets the quality bar they need, not necessarily the most expensive one, because different model families keep leapfrogging each other and companies want to preserve that choice.”
That means enterprises are avoiding lock-in more than ever. Sixty-eight percent say they’re concerned about depending on a single AI provider, the average number of officially adopted AI tools has climbed to 3.3, and 79% now consider it important or critical that agents operate headlessly, connecting directly to systems and APIs without a human interface in between.
It’s a trend similar to the shift toward multi-cloud infrastructure, and driven by a similar reluctance to hand any one vendor outsized negotiating power.
“A flexible architecture is built on platform interoperability,” Nottebohm says. “It runs on multiple models, operates headlessly, and keeps every part of the AI stack swappable, so organizations don’t have to bet on which individual tool wins, and that’s part of the broader shift away from defaulting to the biggest, most expensive model available.”
Over the next three years, businesses should prioritize organizing, classifying, and cleaning up unstructured content, actively hiring and building teams around emerging roles, and adopting a hybrid token compute budget model, where IT owns the core infrastructure and token budget while business units own the application-level spend. And right now, it’s easy to get up to speed fast.
“You don’t have to start at early maturity and slowly work your way up,” Nottebohm says. “If you build in the governance, the content layer, and the multi-model system from the start, you can enter as a leading company and capture that same outsized impact.”
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.
Anthropic on Tuesday launched Claude Cowork on mobile and web, expanding a tool that has quietly become the company’s bridge between the developer-centric world of AI coding agents and the far larger market of knowledge workers who never open a terminal.
The rollout, which begins in beta with Max subscribers before expanding to additional plans, marks a strategic inflection for Anthropic. It transforms Cowork from a desktop-only agent into a cross-device platform where tasks can start on a laptop, continue autonomously in the background, and be reviewed from a phone — even after the user closes the app entirely.
“Your work goes everywhere with you, and keeps going without you,” Anthropic writes in its announcement.
The timing is deliberate. Alongside the mobile launch, Anthropic published usage data from 1.2 million anonymized Claude Cowork sessions sampled between May 11 and May 31, drawn from more than 600,000 organizations. The data paints a striking picture: the overwhelming majority of what people do with Cowork has nothing to do with writing software.
The numbers tell a story that cuts against the dominant narrative in enterprise AI, which has fixated on coding assistants and developer productivity as the primary use case for large language models.
Business process and operations — tasks like pulling scattered updates into a single report, building onboarding checklists, and reconciling spreadsheets — accounted for 33.4% of all sampled Cowork sessions, making it the single largest category by a wide margin. Content creation and copywriting — producing drafts, slide decks, posts, and proposals — came in second at 16.4%.
Together, those two categories make up roughly half of all Claude Cowork usage. Software development, by contrast, accounted for just 8.7%. DevOps and infrastructure followed at 7%, with research and intelligence at 6.4%, data analysis and business intelligence at 5.8%, document processing and extraction at 4.1%, and sales and revenue operations at 4%.
The remaining 12 categories each represented less than 4% of usage, including personal assistance at 3.8%, education at 2.4%, and meeting intelligence at 1.8%.
Anthropic describes these dominant use cases as “the work around the work” — tasks that span nearly every role in an organization but rarely appear in anyone’s core job description. “People are using it for a variety of tasks that aren’t necessarily the hallmark of a specific role, but instead represent the connective work around a role that moves projects forward and keeps businesses running,” the company writes. “That means tasks like drafting a status update, building a slide deck, or condensing reams of research into a single report.”
That phrase — “the work around the work” — is Anthropic’s attempt to define and claim an entirely new category of AI productivity. It’s a calculated reframing: rather than positioning AI as a tool that replaces what professionals do, Anthropic is arguing that the most valuable current application is handling everything professionals do around their actual expertise.
The expansion to mobile and web introduces three concrete capabilities that reflect how Anthropic envisions Cowork fitting into daily workflows.
First, sessions now sync across devices. A user can start a task at their desk, check on its progress from a phone, and retrieve the finished output from any device. Second — and arguably more significant — Cowork can now run tasks in the background with no device online at all. Users can schedule work for a specific time, and Claude will execute it autonomously. Anthropic offers the example of setting Monday morning client prep for 6 a.m.: “Claude works through the email threads, transcripts, and recent news, builds the briefing doc, and leaves the follow-up email drafted but unsent. Review it over coffee.”
Third, when Claude encounters a decision that requires human judgment, it surfaces the question to the user’s phone. “Nothing ships until you’ve reviewed and approved it,” Anthropic states.
Desktop remains the most fully featured surface, with access to local files and the browser. But the web version also opens Cowork to users who cannot install a desktop application — a meaningful expansion in enterprise environments where IT departments control software installation.
The company also unified its interface: on web and desktop, chat and Cowork now share a single home screen, and projects and artifacts persist across both modes.
To encourage adoption, Anthropic is extending doubled Cowork usage limits through August 5.
The usage data and the mobile launch together reveal a company executing a two-track strategy. Claude Code, its terminal-based coding agent, dominates among software developers. But Cowork is designed to capture the vastly larger population of professionals whose work involves creating, organizing, and communicating information rather than writing code.
The contrast between the two products is instructive. As Anthropic notes, Claude Code “is most often used by software developers for the key parts of their role: building, debugging, and shipping code.” When developers do use Cowork, they tend to use it not for programming but for the communications-focused work that surrounds every role — status updates, documentation, and coordination.
This pattern — where AI handles the connective tissue of work rather than its core substance — aligns with what Anthropic describes as people using “Claude Cowork to assemble and structure the information they can use to act on their expertise.” The company illustrates this with three examples: a lawyer using Cowork for document formatting and filing while reserving legal judgment for themselves, a hiring manager synthesizing interview feedback while spending more time on candidate conversations, and a team lead producing a slide deck that explains a decision while focusing on actually making that decision.
The implications for Anthropic’s business model are significant. Developer-focused tools, while high-profile, serve a relatively narrow market. The Ramp AI Index published in May showed Anthropic pulling ahead of OpenAI in business adoption for the first time — with 34.4% of firms paying for Anthropic’s services compared to OpenAI’s 32.3% — and suggests the company’s enterprise push is gaining traction. Claude Code was identified as the primary driver of that shift. But Cowork targets an addressable market that is orders of magnitude larger: every knowledge worker with a laptop, a pile of spreadsheets, and a slide deck due by Friday.
The mobile launch arrives during one of Anthropic’s busiest — and most turbulent — stretches in its history.
Just last week, Anthropic launched Claude Sonnet 5, a new model that narrows the performance gap with its more expensive Opus-class models while maintaining lower pricing. The model is available at introductory pricing of $2 per million input tokens through August 31 before rising to $3 per million input tokens. Sonnet 5 serves as the engine underneath Cowork, and its improved agentic capabilities — better reasoning, tool use, and sustained task completion — directly enhance Cowork’s ability to handle complex, multi-step workflows.
Two weeks before that, Anthropic released Claude Tag, a Slack-native AI agent designed for team collaboration. Where Cowork focuses on individual task delegation, Claude Tag operates as a multiplayer tool — a single Claude identity that everyone in a Slack channel can interact with, building context from conversations over time.
According to Anthropic’s announcement, 65% of the company’s own product team’s code is created by its internal version of Claude Tag. Fortune reported that Anthropic’s head of product for Claude Code and Cowork, Cat Wu, described the distinction: “Claude Code, Cowork, and chat are very single-player, whereas Claude Tag is built to be interactive and multiplayer.”
Together, Cowork and Claude Tag represent a pincer strategy: Cowork captures individual productivity workflows across devices, while Claude Tag embeds AI into team communication channels. Both are designed to push Anthropic deeper into enterprise operations, beyond the developer seat.
The expansion also arrives against a backdrop of unresolved security concerns. On July 1, security firm Armadin — led by Mandiant founder Kevin Mandia — published research detailing what it described as a full sandbox escape in Claude Cowork on Windows, as reported by SiliconANGLE. The attack chain involved DLL sideloading against the Claude desktop executable to gain trusted access to Cowork’s virtual machine service, then exploiting undocumented parameters to achieve root access and bypass network restrictions.
Anthropic responded that the vulnerability did not qualify as a security issue because exploiting it requires an attacker to already have local code execution on the host machine. Armadin, however, raised a broader concern: that deploying local virtual machines on nontechnical users’ systems creates visibility gaps that endpoint security products struggle to monitor.
This tension takes on new dimensions as Cowork moves to mobile and web. The web and mobile versions run tasks server-side rather than in a local virtual machine, which eliminates the specific attack surface Armadin identified but introduces different questions about data handling, especially for scheduled background tasks that process email threads, calendar data, and documents without real-time user oversight.
Anthropic’s announcement states that “the decisions still come to you” and that nothing ships without review and approval. But as Cowork takes on increasingly complex autonomous workflows — processing contract folders, building client briefings from multiple data sources, drafting emails — the surface area for prompt injection and data exposure grows correspondingly.
When Cowork first launched in January, TechCrunch reported that Anthropic explicitly warned about prompt injection risks, noting in its blog post: “These risks aren’t new with Cowork, but it might be the first time you’re using a more advanced tool that moves beyond a simple conversation.”
Anthropic’s enterprise push is also colliding with geopolitical reality. CNBC reported Monday that Alibaba will ban employees from using Anthropic’s AI tools starting July 10, placing Claude Code on a high-risk software list. The move followed Anthropic’s June letter to the U.S. Senate accusing Alibaba of carrying out what it called “the largest known distillation attack” against its models.
The Alibaba ban, combined with reports that Anthropic is closing loopholes that allowed Chinese companies to access Claude through third-country entities, underscores the increasingly fraught environment for AI companies attempting to serve global enterprise customers while navigating U.S. export and security restrictions.
At the same time, Anthropic is investing massively in infrastructure. Reuters reported Monday that Anthropic signed a $19 billion, 20-year lease with TeraWulf for a data center being built in Hawesville, Kentucky, with 401 megawatts of computing power expected to become fully operational in 2028.
That kind of capital commitment only makes sense if the company expects enterprise demand — not just from developers, but from the millions of knowledge workers that Cowork targets — to grow dramatically.
Anthropic is transparent about the limitations of its usage analysis. The taxonomy classifies sessions by the type of work being performed, not by the job title of the person doing it.
There are no standalone categories for marketing, finance, or HR — functions that are likely absorbed into the dominant “business process and operations” bucket, which may partly explain why that category commands a third of all usage.
The sample is also rate-capped rather than proportional to traffic, meaning the numbers are shares of sampled sessions, not absolute volumes. Usage during peak hours is somewhat underrepresented. And roughly 5% of sampled sessions involved personal, non-work use — hobbies, personal assistance, and companionship-style conversations — meaning the data doesn’t purely reflect workplace activity.
The company also acknowledged that its labeling pipeline changed around May 11, which is why the analysis window begins on that date rather than covering a longer period.
Anthropic’s mobile launch and usage data arrive at a moment when the enterprise AI market is shifting from proof of concept to proof of value. The question facing every company deploying AI tools is no longer whether the technology works — but whether it delivers measurable productivity gains across an organization, not just within engineering teams.
The usage data suggests that the answer, at least for Cowork, is emerging in an unexpected place. It’s not in the glamorous work of building software or conducting research. It’s in the unglamorous, universal labor of turning messy information into structured outputs that move organizations forward — the status reports, the onboarding checklists, the variance memos, the client decks.
By untethering that capability from the desktop and making it available on every device, Anthropic is betting that the most valuable AI agent isn’t the one that writes code. It’s the one that handles everything else.
There’s an important distinction between AI that just works today, and AI that lasts at scale. Many companies optimize hard for the first one without ever asking whether they’re building the second.
Velocity without discipline and strategic direction is a liability, not an asset. The hardest part of building AI at scale isn’t getting a model to work once. It’s building systems that continue to work, scale beyond individual teams and use cases, and improve consistently over time.
Today’s AI systems do more than just predict and optimize. They converse, reason, and increasingly take action. An autonomous system making decisions on a traveler’s behalf creates a very different set of expectations around reliability, governance, and accountability. As AI takes on more of those roles, the principles behind how these systems operate matter more than ever.
We have spent years applying AI and machine learning (ML) across the traveler journey — from personalization, ranking, and recommendations, to fraud prevention, customer support, and, more recently, generative and agentic AI experiences. That depth of experience is what led us to develop a set of ML and AI principles to guide how we build, deploy, and evolve AI systems across our company.
The goal is simple: Make sure the systems we build create real business value, scale, and operate safely. These principles define how we measure, design, govern, and operate our systems.
Publishing principles is the easy part. The harder and more important work is turning them into operating mechanisms: Recommendations, requirements, tooling, and release processes that teams actually use.
We have begun using ‘Agentic Release’ tollgates: A set of recommended and, in some cases, required checks before launching agentic AI features. These tollgates translate principles like clear ownership, risk-based governance, evaluation, safe rollout, and monitoring into concrete expectations for teams.
Some of these recommendations and requirements are already being automated and integrated into the software development lifecycle (SDLC). Over time, the goal is for these expectations to become embedded in how we design, evaluate, approve, launch, and monitor AI systems from the start.
The first test for any model is whether it improves a business outcome and, ultimately, the traveler experience — not whether it just improves a technical metric.
Align models to metrics with business impact: Every ML effort must tie directly to a key business outcome or traveler experience metric. Technical optimizations are useful midpoints, not end goals.
Optimize for return on cost: The value a model creates has to justify what it costs to develop, train, and monitor, plus the operational complexity it adds. Favor solutions that deliver lasting impact relative to what they cost to run.
Justify complexity against strong baselines: Complexity should be earned, not assumed. Start with a strong baseline: An existing general model, a simple heuristic, an off-the-shelf solution. Reach for specialized models or more complex architectures only when simpler options genuinely can’t meet the bar.
Require both offline and online evaluation: No model goes to broad deployment on offline validation alone or jumps straight to A/B testing. Every model must perform in both offline and online evaluations. Over time, our offline evaluations should reliably predict what we see online.
Getting a model to work is one challenge. Making its value extend beyond a single team or use case is the harder one.
Build on shared foundations; specialize only when justified: Favor shared, platform-wide foundations for core capabilities, data representations, and model building blocks. Specialization should build on those foundations, not spin up isolated stacks, so when the foundation improves, the gains flow across the organization.
Treat data as a first-class product: A model’s quality is bounded by the quality of its data. We need to maintain robust pipelines, clear lineage, reproducibility, and reusable features built with documented ownership, clear schemas, and SLAs that other teams can rely on.
Prioritize generality over local optimization: When two approaches perform similarly, favor the one whose learnings, assets, and operating patterns can be reused across teams, brands, and use cases. We should optimize not just for local performance, but for how quickly improvements can diffuse across the company and compound over time.
Minimize and sunset manual business rules: Manual rules are sometimes necessary for policy, safety, or compliance, but they should be explicit and reviewed regularly, never silent patches for weak models or a source of permanent maintenance debt.
Reproducibility and traceability by default: Training data, features, configurations, evaluation results, deployment versions, and key decisions should all be documented and recoverable. That’s what lets you debug a production issue months later and hand off ownership without losing institutional knowledge.
The bar for deploying AI isn’t just “does it work?” It’s “can we stand behind it?” Trust isn’t something you add at the end; it’s earned over time and maintained across the full lifecycle of every model we ship.
Assign clear ownership and accountability: Every model needs defined ownership across its lifecycle — a business owner, a product owner, an AI owner, and an operational owner. These don’t need to be four people, but the responsibilities must be explicit. Who’s accountable for outcomes? Who responds if the model drifts? Who answers the incident at 2 a.m.? Without this in place, models become orphaned and problems surface with no one to own them.
Adhere to standards and governance: AI and ML models must use approved platforms and comply with established company standards, release gates, and governance processes. Operating outside these guardrails requires a clear, defined path to remediation or deprecation, rather than an open-ended exception.
Govern proportionally to risk: The level of review, evaluation rigor, and human oversight should scale with a model’s impact. A customer-facing model that affects pricing or availability for millions of travelers demands a far higher bar than an internal tool used by a small team. For high-impact, safety-sensitive, or highly autonomous systems, human-in-the-loop checkpoints are built in from the start.
Design for fairness, privacy, and transparency: We actively test for unintended bias, have strong data guardrails, and favor explainability when decisions meaningfully affect users. These are incorporated from the start, not added on.
Design for safe rollout, rollback, and control: Deployments are progressive, with rollback paths, fallback mechanisms, and circuit breakers ready before launch. The ability to safely undo a deployment matters as much as the ability to ship it.
Monitor continuously and adapt: Once live, teams must actively monitor quality, drift, latency, cost, and business performance and retrain or recalibrate when the data shifts. A team should always be able to explain how its model is performing now, not just how it performed when it launched.
These principles do more than define how we build. They define what we’re willing to ship and how we stand behind it. In a world where AI systems are increasingly consequential and make real decisions for real travelers and partners, these standards matter. Applied consistently, they build responsible AI that lasts.
Xavi Amatriain is Chief AI and Data Officer at Expedia Group
Xavier will share more details about Expedia’s architecture during his session at VB Transform on July 14 at 11:10 am PT. He will discuss: “Expedia’s blueprint for building autonomous agents for high-stakes transactional systems.”
Interested in attending VB Transform 2026? Register here. A select number of complimentary passes are also available to senior technology leaders. Contact us to get yours.
Presented by Microsoft and NVIDIAEvery generation of leaders has its own business transformation challenges to face. A decade ago, modernization meant cloud migration. Five years ago, it meant enabling remote and hybrid work. And just a few short years…
Most verticals aren’t clean, well-oiled SaaS databases; the reality is ugly documents, proprietary schemas, implicit workflows, and long‑running tasks that most general-purpose models struggle with.
This prompted construction project management company Trunk Tools to build a specialized, three-layer architecture — perception, semantics, agents — based on highly-detailed data to support high-accuracy, highly-relevant industry automation.
Their purpose-built stack has shrunk review cycles from months to days, prevented costly field errors, and given autonomous agents the ability to reason over millions of pages of documentation, the company says.
“We really set out to take the data from dispersed systems, pre-process it, structure it, go through our ontology into a knowledge graph, and then train AI models,” said Sarah Buchner, Trunk Tools’ founder and CEO and a former carpenter.
For builders in other verticals, the company’s approach could serve as a blueprint for transforming data chaos into agent‑ready, industry-specific workflows.
Foundation LLMs, while powerful, are optimized for breadth, not always depth.
“General-purpose LLMs are trained to be okay at everything, so they’re weak at anything niche,” said Kriti Faujdar, a senior product manager working in AI infrastructure, agentic AI, security, and LLM platforms. For instance: Rare terms, domain-specific reasoning, the unspoken context that any practitioner “just knows.”
Web, app, and software developer Sébastien De Bollivier agreed that the biggest bottleneck is reliability on data that is “jargon-dense, abbreviation-heavy, and format-specific.”
“A GPT-4-class model can understand a French legal contract, but will fumble the specific article references practitioners need to cite,” he said.
Besides, the most valuable enterprise data never made it into pretraining anyway, Faujdar pointed out. It’s sitting in internal systems and proprietary formats. “RAG helps a little,” she said. “But it’s just giving better facts to a model that still can’t reason properly in the domain.”
Pre-training on domain data is critical; enterprises should then fine-tune on good task examples and build their own evals. “A few thousand examples from real practitioners beats millions of scraped, noisy ones,” Faujdar said.
Mixture-of-experts (MoE) can provide specialization without inference costs blowing up. Pairing RAG with fine-tuning also works well; RAG handles the factual long trail while fine-tuning fixes vocabulary and reasoning.
De Bollivier pointed to the advantage of hybrid stacks: A general-purpose model for reasoning and orchestration, a smaller fine-tuned model (or dense retrieval over a curated corpus) for domain-specific extraction. He advised: “Don’t fine-tune to make the model ‘smarter’ about a domain, fine-tune to make it more reliable on the specific output format your workflow requires.”
The trades and construction are certainly industries seeing traction with these techniques, as are legal and healthcare, De Bollivier said. These verticals have “high stakes for errors plus standardized document formats, equaling clear domain-training ROI.”
One honest caveat worth mentioning, Faujdar said: Specialized models can often fall apart outside their domain, so they’re often not useful outside their expertise (unless they’re re-trained).
In highly-specialized domains like construction, “data dumps” into large language models (LLMs) don’t cut it, said Trunk Tools’ CTO Amrish Kapoor. This is because most transformers are probabilistic models: When given an image, they report back that it is “probably” a tree, or “probably” a child playing next to a tree.
This makes them insufficient for high‑precision symbolic interpretation. For instance, in construction documents, a 2-millimeter-wide symbol has a vastly different meaning depending on where it’s placed.
Further, constrained by context limits, probabilistic models struggle with long‑term project memory. “I don’t mean a context window of a few tokens,” Kapoor said. “I’m talking about long term memory that stretches across months and years, because this is how long some of these projects are.”
Instead, the company’s three-layer system breaks workflows into:
Perception (reading and extracting data from messy docs like PDFs, drawings, or scans)
A semantic/graph layer (making sense of that data and understanding their relationships).
LLMs and agents on top.
Construction drawings are typically symbolic, Buchner said. A door isn’t always labeled ‘door.’ Sometimes it’s simply an arc on a wall that a trained eye learns to read based on years of practice.
“The perception layer is what teaches AI to read that language,” she said. The semantic layer then gives that information meaning; for instance, connecting the door to the drawing that details it, the spec that governs it, and the trade that installs it. This helps answer project engineers’ critical questions: Not “is there a door here?” but “does this door create a problem down the line?”
Particularly in construction, that shift matters because the cost of a problem compounds with time. “A conflict caught in design is relatively low cost to address,” Buchner said, “whereas the same problem caught in the field might cost tens of thousands of dollars.”
At a high level, the system identifies the document type and begins extracting information based on content (drawing, schedules, paragraph text). This data is then “transformed and augmented” in the platform, which triggers agentic workflows like knowledge graph relationships and end-user workflows.
For instance, an agent might review an architecture bulletin and produce a visual overlay comparing an older version and a newer version (flagging additions and removals), then generate written narratives that describe what those changes are in simple terms. This helps users understand what’s changed and coordinate with trade partners on updated pricing and change orders.
Construction workflows are “ripe with implicit assumptions and connections between data in its myriad of sources,” Buchner said. And the amount of unstructured data is “humanly impossible” to process or make sense of.
Buchner estimated the average high-rise building generates about 3.6 million pages of corresponding documentation. “If you print it into a stack of papers it would be as high as the building itself.”
All three layers of Trunk Tools’ stack — perception, semantic, LLM — are trained on “very specific datasets” from customers with “explicit permissions” and auto‑labeling/IP, Kapoor explained. Customers who don’t want Trunk training on their data can opt out.
Data is deidentified and aggregated, and Trunk Tools also collects “tons more” labeled data through other pipelines like 3D building information modeling (BIM).
The company says it only ships agents that achieve around 95% accuracy. The team maintains continuous evaluation pipelines based on ground truth data from customers and experts. They also employ an LLMs-as-a-judge model.
“This notion of an LLM as a judge is to score how well you’re doing, both subjectively as well as objectively,” Kapoor said. Objectivity can be an easy ‘right’ or ‘not right,’ but subjectivity requires more nuance.
For instance, when creating an email or narrative or explanation, an LLM as a judge framework can create a composite score, or a numerical value that aggregates different metrics and tests a model’s performance or risk.
There can be challenges, though, particularly with latency, Buchner noted; any time the reasoning capacity of underlying models increases, the risk of latency goes up, too. Trunk Tools maintains a set of evaluation criteria to objectively measure latency whenever changes are made to underlying infrastructure, agents, and API calls.
Then, “before we release to customers, we ensure marginal changes to the end-user experience are well worth the performance enhancements,” Buchner said.
Trunk Tools’ platform powers seven AI agents purpose-built for construction, such as analyzing request for information (RFI) responses, overviewing bids, or reviewing drawings and submittals.
The submittal agent, for instance, flags missing, conflicting, or noncompliant information in product specs and RFIs. While it’s an essential step in the construction process, “it’s a super annoying workflow,” Buchner said, because human reviewers have to compare documents “with a bunch of other parts of documents.”
But the agent is able to do this in seconds, and Trunk Tools says it has reduced submittal cycles from 50 to 60 days to 10, “which has massive schedule and financial implications.”
The company is now at a place where these agents are communicating directly with each other, which is “quite exciting,” Buchner said. So, for example, one agent will review an architectural drawing for accuracy, then autonomously hand it over to agents handling RFIs and asking follow-up questions.
“If the drawings have problems, the RFI agent is taking over and is actively reaching out for clarification,” Buchner explained.
Trunk Tools says its customers report savings of 20 to 40 minutes per field question. Buchner said that users in the field know better than anyone how much of a “time suck” it is to go back and forth from office trailers, dig through project documents in scattered systems or printed PDFs, reconcile discrepancies, and return to coordinate with trade partners.
The company says its customers report these additional outcomes:
Average 8 minute time savings for single-document retrieval (status checks, location lookups, quantity queries).
Average 20 minute time savings for standard referencing (cross-referencing 2 to 3 spec sections to form an answer.
Average 40 minute time savings for multi-document research (listing and filtering queries, mapping relationships, analyzing RFIs and submittals across 4 to 6 documents).
Average 75 minute time savings for complex tasks (creating RFIs and other communication materials, deep cross-referencing across documents, change tracking).
In one instance, the company’s drawing review agent flagged that a structural beam had been moved up 8.5 inches. However, this was not documented by the architect. If the change hadn’t been caught, the project manager would likely have had to strip out and reinstall the right size beam, Buchner said. This rework would have added $10,000 or more to the budget, and “certainly there would have been implications on the schedule.”
Buchner also pointed to other examples: an agent flagged $60,000 in exaggerated pricing with no justification from landscaping subcontractors; identified a fireplace that needed to be sealed prior to drywall installation, saving around $100,000 in labor, materials, and delays; and called out that an electric door required a panel that wasn’t included in electrical drawings.
Trunk Tools’ approach to building agents is applicable to any vertical working with high volumes of unstructured, industry-specific data.
Builders working in specific verticals must understand the industry’s specific data challenges their end users face and build technical infrastructure that can transform unstructured data into something an “LLM can traverse and understand,” Buchner said.
“Only then can you build the connections between data points that ultimately feed agentic workflows.”
A lot of money is being invested in foundational models, so enterprises should build modular systems that can leverage the strengths of various models as they continue to improve, Buchner advised.
Then, “build your technical advantage where the generic models are not investing and not performing well,” she said.
Two-thirds of enterprises have hedged their AI model strategy, and the past few weeks of controversy around Anthropic’s Claude Fable 5 model showed why that posture has gone mainstream.
On June 12, a U.S. export-control order pulled Anthropic’s Claude Fable 5 — the most capable model on the market — offline for every customer, with no warning and no timeline. It returned this week wrapped in tighter safeguards, after China’s Z.ai released its open-weights GLM-5.2 into the vacuum. New VentureBeat Pulse Research, which surveyed 145 enterprises across these last few weeks, shows that two-thirds had already hedged their model strategy before the order came down: 51% blend closed frontier models with open-weight models deployed on their own infrastructure, and another 16% are moving core workflows off closed APIs entirely. The remaining third was all-in on closed ecosystems when the lights went out.
The blackout put a spotlight on vendor dependency, by showing what happens when the model you rely on disappears. But vendor dependency is only the most visible piece of a deeper problem: Most enterprises lack the monitoring to know when an AI system they’ve put into production stops working correctly.
Just 1 in 10 enterprises has automated monitoring that would catch an AI model drifting, misbehaving, or failing in production. Roughly a quarter would learn of a production failure only when end users — internal or external — report it, or lack the visibility to detect it at all. And 79% of enterprise organizations have already taken a real financial or operational hit from autonomous agents — most often shadow AI, unauthorized agentic work run by enterprises’ own employees on corporate credit cards, outside anyone’s oversight.
We call this the “Control Gap,” or the distance between how aggressively enterprises are deploying AI and how little of it they can see, own, or govern. June’s blackout turned this into a live stress test.
About this data: VentureBeat Pulse Research surveyed 145 qualified respondents at organizations with 100 or more employees in June 2026, with fielding spanning the Fable 5 blackout that began June 12. The sample is self-selected and directional: 41% work in technology/software, 20% are consultants or advisors, and the respondent base skews senior and technical — CIO/CTO/CISOs (18%), directors of engineering/IT (14%), enterprise architects (12%). More than half of the respondents were from companies with 10,000 employees or more.
While our sample is not huge, what you can trust more than the exact percentages is the pattern: Every question in the survey, independently, points the same way, with deployment running ahead of governance, visibility, and cost control.
The full methodology is in the report.
Fable 5 launched June 9 to immediate acclaim — and sticker shock, at $10 per million input tokens and $50 per million output. Three days later, the U.S. government issued an emergency export-control directive barring access by foreign nationals. Anthropic, with no way to verify nationality in real time, suspended the model for everyone.
Z.ai has continued to pick up momentum; on Wednesday it released an open agentic coding environment, called Zcode. OpenAI, meanwhile, previewed its cutting-edge GPT-5.6 line on June 26.
Enterprises had already spent the spring learning what AI dependence costs in dollars. Uber burned through its entire 2026 AI coding budget in four months after Claude Code adoption hit 84% of its roughly 5,000 engineers, Forbes reported. Microsoft canceled most internal Claude Code licenses in its Windows and Microsoft 365 division, steering engineers to its own tooling, according to The Verge.
June added the harder lesson: The model your workflows depend on can vanish overnight, by government order, through no decision of yours or your vendor’s. And Chinese companies like DeepSeek were releasing hugely disruptive, powerful models, driving down costs to a fraction of Western ones.
Brian Craig, senior director of architecture at Liberty IT, the Ireland-based engineering arm of Liberty Mutual, one of the world’s largest insurance companies, saw both lessons collide in real time. Craig is Irish, which meant the export order hit him directly as a foreign-national user.
Onstage at VentureBeat’s AI Impact event in New York on June 24, mid-blackout, I asked him about it. “Fable arrived, and immediately you saw the sticker price of using it, and you went, ‘Ooh, goodness, it better be really good,'” Craig said. “But luckily enough, we didn’t get to use it enough to get to fall in love with it.” Then it was gone.
Craig’s company was built to route around exactly this kind of disruption. Liberty IT runs what it calls an AI backbone — roughly 50 components spanning security, governance, observability, and orchestration, each independently replaceable.
“You can’t lock in right now in one vendor and even one framework,” Craig told the room. “You need to keep being able to have the flexibility with that backbone to be able to hook into different models, different vendors, depending not so much on who’s the flavor of the day, but on what you can feel confident about for the next six months.”
The survey shows Craig has plenty of company. A 51% majority of enterprises run a hybrid posture — closed frontier models for general reasoning, open-weight models deployed locally for specialized execution — and 16% are making a hard pivot, moving core workflows onto open weights running on their own hybrid or private cloud. The 32% holding a closed commitment are candid about why: The operational overhead of self-hosting still outweighs the savings for them. After June, that calculus has a new variable in it.
Defection is now the active posture, and the target may surprise you. Asked which primary AI vendor they are most likely to downsize or phase out over the next 12 months, respondents named Microsoft first at 30% — most citing cutbacks to Copilot and Azure AI frameworks in favor of direct model access — ahead of the 28% who plan to trim no vendor at all. OpenAI drew 21%, largely on pricing volatility, with Anthropic at 15% and Google at 6%. No vendor faces an exodus. But loyalty by inertia has ended: Among these enterprises, actively cutting at least one provider is now more common than expanding across all of them.
How would an enterprise know if one of its production AI models was drifting, behaving unsafely, or failing to complete tasks? We asked directly. Forty percent say they are very confident they would detect it. The question also asked what that confidence rests on, and respondents split into two camps: 30% rely on humans reviewing critical AI outputs, and just 10% — 14 of the 145 organizations — have automated monitoring and alerting running against production systems. The remaining respondents hold weaker positions still: 32% expect to catch most issues “eventually,” 19% say they would likely hear about a failure from end users first, and 8% report no systematic visibility into production AI behavior at all.
That distinction matters because the two approaches are very different. Human review may seem like the gold standard, but it only reaches the outputs someone designates as important for such a review — and it happens at the pace humans can move at, with the inconsistency any manual process carries. Automated monitoring watches everything the system produces, continuously, and flags anomalies as they happen — for the same reason enterprises stopped depending on manual checks for uptime and security a decade ago.
As agentic workloads multiply output volumes far beyond what any review team can read, the manual approach starts to fall behind. The leaders at our June 24 event in New York treat human review as a designed control with automation underneath it. “Nothing gets deployed into production unless it’s a human actually reviewing it and signing off,” Craig said of Liberty’s agentic software factory, where planning, coding, testing, critic, and librarian agents ship features from epic to production.
“It always has to be risk-based. That’s why we work for an insurance company.” Todd Johnson, the Morgan Stanley managing director who runs agentic AI across the bank’s end-of-day P&L controller process, described the same principle from finance: “One of our strong principles in our AI governance generally is that there always has to be human accountability, even if there’s a degree of automation.” VentureBeat covered Morgan Stanley’s new results around its P&L resolution agent system separately.
Liberty Mutual and Morgan Stanley chose manual sign-off deliberately, layered on top of observability, identity, and governance infrastructure. Whether the human-review camp has similar infrastructure underneath is more than a single-select question can establish. The 16% who separately named missing observability tooling as their biggest governance barrier are the ones saying outright that it hasn’t been built.
Why does the AI visibility tooling never get built? The respondents’ answers suggest it is an organizational shortcoming. The single most-cited barrier to governing AI across platforms is the absence of a single owner or accountable team, at 32%. Vendor opacity follows at 25%, missing tooling at 16% — and a lack of talent lands dead last at 5%.
The skills exist, but the organizational mandate does not: Only 38% say a central team actually governs AI behavior across their platforms today, 21% say ownership is unclear or actively contested between teams, and 17% say no role holds formal accountability at all.
The AI surface being governed makes the vacuum worse. Fully 85% of enterprises run two or more platforms each claiming to be the “primary” AI layer — ERP, ITSM, productivity suite, data platform, each with its own AI, its own controls, and its own assumptions. 36% describe an open contest between four or more. Just 8% have consolidated to one. Asked in a free-text question what one thing they would fix, respondents converged from different directions on the same answer: a single accountable owner, and a control plane that abstracts cost, drift, and model choice away from the end user.
The cost of the vacuum is showing up on corporate cards.
Asked to name the most severe financial or operational control failure they have experienced from autonomous agents, 49% of enterprises cite shadow AI — departmental teams running unauthorized agentic pipelines on corporate credit cards, bypassing central financial oversight entirely. Another 25% have been hit by an infinite-loop bill, an uncaught recursive workflow racking up thousands in token costs in a single incident, and 6% by an agent that degraded production databases with unthrottled queries. Only 21% report guarded stability, with hard token throttling and budget caps at the infrastructure layer. Add it up: 79% of these enterprises have already paid for an agent control failure in real money or real downtime.
Finally, the economics of tokens suggest the pressure will keep rising. Per-token inference costs are falling 70 to 80% a year, and agentic workloads consume 100 to 500 times the tokens of the LLM tools they replaced.
Brian Gracely, senior director of portfolio strategy at Red Hat, told our New York audience the answer starts with right-sizing: “If I’m simply trying to resolve an insurance claim, I don’t need to know about the history of Western civilization in my model. I don’t need to know soccer scores.”
Enterprises are pairing smaller, specialized models with semantic routing, he said, so the platform decides which requests genuinely need frontier-scale reasoning — and which are burning premium tokens on commodity work. (One adjacent data point from the survey underlines the appetite for pragmatism: 73% of enterprises report little or nothing to show for their custom fine-tuning investments of the past 18 months — a reckoning we’ll examine in its own report.)
The survey describes enterprises moving fast on AI with weak controls underneath. 58% are adding more AI initiatives than they retire. 85% run multiple platforms that each claim to be the primary AI layer. Three times as many enterprises rely on human review to catch a failing production model as have automated monitoring in place. And 79% have already paid for an agent control failure — most often unauthorized agent spending on corporate cards, outside IT’s oversight.
On one problem, enterprises have clearly adapted: model dependency. Two-thirds hedge their model strategy, either running open-weight models alongside closed ones (51%) or moving core workflows off closed APIs entirely (16%). The Fable 5 shutdown showed the value of that position — the hedged companies could route around a model that a government order made unavailable overnight.
The remaining problems are internal, and no purchase fixes them: 32% name the lack of a single accountable owner as their top governance barrier, and 17% say no role holds formal accountability for AI at all. Assigning an owner costs nothing and requires no vendor. It still hasn’t happened at most of these companies.
Our coming Q3 wave of research will measure whether June changed this — whether enterprises assigned owners and installed automated monitoring, or just added a second model and moved on.
Get the full Control Gap report here.
The themes in this report — agent orchestration, governance, and cost control — are the agenda at VB Transform, VentureBeat’s flagship event, July 14-15 at Hotel Nia in Menlo Park, with technical leaders from Visa, GM, Waymo, Intuit, Instacart, LangChain and others. Details and registration here.
Disclosure: VentureBeat’s June 24 AI Impact event in New York was sponsored by Red Hat and Intel. Sponsors have no input into VentureBeat Pulse Research survey design, findings, or editorial coverage.