Large language models continue to struggle with hallucinations, presenting a major roadblock for real-world enterprise applications. Reducing these errors is a messy business, forcing model developers to navigate a strict tradeoff where eliminating factual errors often suppresses valid answers.

In a new paper, Google researchers introduce the concept of “faithful uncertainty,” a metacognitive technique that aligns a model’s response with its internal confidence. This alignment allows the model to offer appropriately hedged hypotheses, such as “My best guess is,” instead of defaulting to an unhelpful “answer-or-abstain” binary.

In real-world agentic AI applications, this metacognitive awareness acts as an essential control layer. It empowers autonomous systems to accurately determine when their internal knowledge is sufficient and when they must dynamically trigger external tools or search APIs to resolve deficits.

The utility tax of current mitigation strategies

Understanding why LLMs hallucinate hinges on separating two capabilities: a model knowing facts versus knowing what is known. Historically, most factuality gains in AI have come from expanding the knowledge boundary, meaning developers simply pack more facts into the model’s parameters through larger scale and more training data.

However, expanding a model’s knowledge does not automatically improve its boundary awareness, which is its ability to distinguish the known from the unknown and recognize its own limitations.

“There are broadly two ways to improve LLM factuality,” Gal Yona, Research Scientist at Google and co-author of the paper, told VentureBeat. The first is continuing to teach the model more facts. But, Yona notes, “model capacity is finite, and the long tail of knowledge is effectively infinite.”

Once models hit this limit, the hope is they know what they don’t know and simply abstain from answering. However, this is inherently difficult for LLMs.

“This is why most practical attempts to reduce hallucinations through various interventions don’t actually make it to deployment,” Yona explains. “They do reduce hallucinations, but they also hurt utility, because the model ends up refusing to answer questions it actually does know.”

This inability to distinguish between knowns and unknowns creates what the paper’s authors call the “utility tax.” Enforcing a zero-hallucination standard requires the model to abstain whenever it is even slightly uncertain, discarding massive volumes of completely valid information. For example, the authors demonstrate that reducing an underlying 25% error rate down to a strict 5% target forces developers to discard 52% of the model’s correct answers.

Treating all errors as hallucinations forces enterprise systems to choose between trustworthiness and helpfulness. Application developers are generally unwilling to pay this massive utility tax and render their models unhelpful.

Consequently, they optimize systems to prioritize coverage, forcing models to operate in a state where they continue to generate confident hallucinations.

Reframing hallucinations as confident errors

To move past the utility tax, the researchers propose to stop treating any factual error as a hallucination. Instead, they reframe hallucinations as “confident errors”: incorrect information delivered authoritatively without appropriate qualification.

This subtle reframing dissolves the strict “answer-or-abstain” dichotomy and allows the model to express its uncertainty.

In this new framework, if a model makes a factual mistake but appropriately hedges its response (e.g., by stating, “I am not completely sure, but I think…”), it isn’t a hallucination. It is simply a hypothesis offered to the user for consideration. By expressing uncertainty, the AI preserves its utility—sharing whatever partial or likely knowledge it has—without violating the user’s trust.

However, if an AI assistant hedges all its responses with a disclaimer, the user is forced to double-check everything, defeating the purpose of the tool entirely.

The solution the researchers propose is “faithful uncertainty.” This approach requires aligning a model’s linguistic uncertainty, or the words it uses to express doubt, with its intrinsic uncertainty, which is its actual, internal statistical confidence in that specific answer. This ensures the model only hedges when its internal state genuinely reflects conflicting or low-probability information.

Faithful uncertainty forms a core component of “metacognition,” the AI’s ability to be aware of its own uncertainty and act on it. To understand this practically, consider the intuitive example of consulting a doctor. We do not trust doctors because they are all-knowing. We trust them because they reliably distinguish between a confident diagnosis (“You have a fracture”) and an educated hypothesis (“It might be a sprain, but let’s run some tests”).

Practical implications for enterprise AI

Under the new framing, errors where a model is genuinely confident but factually incorrect are categorized as “honest mistakes.” This casts knowledge expansion (training the model on more data) and faithful uncertainty as completely complementary efforts. Knowledge expansion pushes the absolute knowledge boundary outward to minimize honest mistakes, while faithful uncertainty honestly communicates wherever that boundary currently lies.

This new framing has important implications for agentic applications. The shift to agentic AI might make it seem like knowing what the model doesn’t know is redundant, since models can just search external databases. However, access to external tools actually amplifies the need for faithful uncertainty. In agentic systems, metacognition becomes the central control layer that governs the entire system.

External tools solve the storage problem because the model no longer needs to encode every fact into its parameters. However, this introduces a new control problem: managing when to retrieve information, verify facts, and orchestrate these external tools. Without faithful uncertainty, an agent is essentially flying blind and must rely on external, static heuristics or over-engineered scaffolds.

“The model might search for something it already knows confidently—wasting latency and cost for no gain. Or the opposite: it confidently answers from memory when it should have searched, producing a plausible but wrong output,” Yona said. Today’s agent harnesses try to solve this externally with query classifiers or always-search rules, but Yona notes that these are “static and brittle.” By using its intrinsic uncertainty to regulate its own behavior, the agent dynamically optimizes its tool use, choosing to invoke a search tool only when its internal confidence is genuinely low.

Beyond deciding when to search, faithful uncertainty is critical for evaluating the results of a search. If a tool returns low-quality or unexpected information, a metacognitive agent does not blindly accept whatever appears in its context window. Instead, it uses its uncertainty awareness to weigh the retrieved external signals against its own internal priors. This prevents sycophantic behavior where the system might otherwise trust external sources that conflict with its actual known knowledge.

The bootstrapping paradox: The catch to teaching uncertainty

For enterprise builders, achieving this faithful uncertainty is trickier than it sounds. It requires teaching models the syntax of uncertainty through supervised fine-tuning (SFT). Because pre-trained models are mostly fed authoritative text, they must be explicitly taught to say things like, “I’m not entirely sure, but I think VentureBeat was founded in…”

But SFT introduces a “bootstrapping paradox.” Unlike standard training datasets where the “right answer” is the same regardless of the model, the ground truth for uncertainty is the model’s own dynamic knowledge base.

“Here’s the catch: the ‘correct’ expression of uncertainty is inherently dynamic, because it depends on what this particular model knows or doesn’t know at this particular point in training,” Yona said. “If you train on a label that says ‘I don’t know X’ but the model actually does know X, you’ve taught it to hallucinate uncertainty… The training data is static, but the target is a moving one, and that’s the fundamental tension teams need to grapple with.”

The road to self-aware AI

For enterprises looking to implement these capabilities without expensive retraining, prompting serves as the most accessible entry point. “Prompt engineering is already something most engineers do today, this provides the lowest-friction path to improving metacognitive behavior today,” Yona said. Enterprise developers can explore frameworks like MetaFaith, an open-source project previously co-authored by Yona, to begin applying metacognitive prompting to off-the-shelf models.

However, Yona cautions that “there is still substantial headroom that prompting alone doesn’t solve,” meaning the industry will eventually need to rely on advanced reinforcement learning (RL) to bake metacognition deeply into model training.

Ultimately, as enterprises transition from isolated chat applications to complex, multi-agent workflows, self-awareness will become a defining prerequisite for reliable autonomy. But evaluating whether a model truly possesses this awareness remains a profound technical challenge.

“How do you actually evaluate whether a model can sense its internal states?” Yona asks. “Even in humans, it’s hard to define or separate ‘true’ self-monitoring abilities from a capable reliance on proxies. We face exactly the same challenges with LLMs: a model might learn to mimic the style of uncertainty without truly sensing its internal state. Developing evaluation frameworks that can tell the difference is one of the most important open problems in this space.”

Agent skills have become an important part of real-world AI applications, providing a mechanism — a set of instructions saved in a folder of text-based markdown (.md) files, usually — for models to adapt to specific enterprise use cases and complex workflows.

However, optimizing these skills is a slow process and faulty process, as they cannot be trained in the same way as the parameters of the underlying AI model. Instead, users typically must update them manually by retyping the instructions in each file, playing a “guessing game” as to what changes might improve agentic AI performance and reduce errors.

SkillOpt, a new, open source (MIT Licensed) framework developed by Microsoft, does one better: it introduces an optimizer designed for agent skills, turning the agent’s skill .md document as a trainable object that evolves based on performance feedback.

It uses deep-learning-style optimization to make it possible for the AI to systematically explore modifications to the document and find the best combination of instructions. Most importantly, it accomplishes this procedural adaptation without making changes to the underlying model’s weights.

On various industry benchmarks, SkillOpt outperforms existing baselines, significantly boosting accuracy for models like GPT-5.5 and Qwen. The result is a set of compact, transferable skill artifacts that allow AI agents to adapt to new domains effortlessly.

The challenge of optimizing agent skills

Agent skills package procedural knowledge into natural-language specifications, including domain heuristics, tool-use policies, output constraints, and known failure modes. These skills provide an external interface for agents to adapt to complex enterprise workflows. In practice, agent skills are stored as text documents and inserted into the agent’s context before execution.

One of the key benefits of skills is that they customize the behavior of the underlying model without changing its weights. However, the skill document itself needs to be tweaked and optimized to get the best performance out of the agent.

While deep learning relies on strict mathematical controls for stability, human prompt engineering often relies on trial and error. When attempting to automatically update a skill document based on feedback, the lack of mathematical discipline makes text highly volatile.

Yifan Yang, Senior Research SDE at Microsoft Research Asia, told VentureBeat that the problem is not making changes, but ensuring those changes are mathematically sound.

“The breaking point isn’t whether a team can change a skill, it’s that they can’t guarantee the change is an improvement,” Yang said. “Three failure modes recur: no step-size control, so skills drift; no validation, so a fix that reads as reasonable gets written in and can quietly regress performance; and no negative memory, so the same failed edit keeps coming back.”

To illustrate how easily performance can drop when edits aren’t mathematically validated, Yang noted that “an ungated rewrite pushed GPT-5.5 on SpreadsheetBench from 41.8 down to 41.1.”

According to Yang, these failure modes are amplified in multi-step workflows “because that’s where frontier models are weakest zero-shot. Not on reasoning, but on procedural discipline: format, self-verification, tool policy.”

Before SkillOpt, agent skills were primarily hand-crafted, generated in a single shot, or evolved through loosely controlled self-revision pipelines that could not reliably improve under feedback.

Prompt optimization methods like TextGrad and GEPA treat language artifacts as optimizable objects and use trajectory feedback to evolve prompts, but they focus on single-prompt configurations rather than generating persistent, reusable skill artifacts.

Meanwhile, skill evolution and discovery methods like EvoSkill and Trace2Skill convert agent execution experiences into trajectory lessons to refine skill folders, build domain-specific libraries, or perform evolutionary search.

None of them apply deep-learning-style controls, such as learning rates, validation gates, and momentum, which are necessary to continuously train a single, compact skill document.

Importing mathematical discipline to text

SkillOpt optimizes a text document through an iterative propose-and-test loop that separates the model executing the tasks from the model optimizing the skill. The process unfolds in several steps:

• SkillOpt starts with an initial skill document and a frozen target model (or harness), where the target model runs a batch of tasks to generate execution trajectories that act as the evidence for the current step.

• An offline optimizer model analyzes these trajectories, separating successes from failures into minibatches. Looking at a minibatch helps the model identify systematic procedural errors rather than one-off anomalies. Based on these patterns, the optimizer proposes structural add, delete, or replace edits to the skill document.

• The proposed edits are reviewed to filter out duplicates or contradictions, and the optimizer then ranks these candidate edits by their expected utility.

• Rather than applying all proposed changes, SkillOpt clips the list to a maximum edit budget for that step, generating a candidate skill.

• The candidate skill is evaluated on a held-out validation set using the target model. If the candidate improves the validation score, it is accepted and becomes the new current skill. If it fails, the edits are rejected and sent to a rejected-edit buffer, providing negative feedback so the optimizer knows not to repeat that mistake.

SkillOpt directly addresses the problem of treating text as a trainable object by importing mathematical concepts from deep learning. The creators note that “the deep-learning analogy is operational rather than decorative,” helping the framework avoid the instability issues associated with other optimization techniques.

The edit budget acts as a learning rate. By limiting how many edits can be applied at once, the skill version is prevented from moving too far from its previous state, preserving continuity while allowing new procedures to be acquired. 

Just like checking validation loss in deep learning, the strict held-out examples ensure that plausible-sounding text edits are only kept if they mathematically improve the agent’s actual performance on the validation split.

At the end of an epoch, SkillOpt performs a slow update by comparing tasks under the previous and current epoch’s skills. This acts like a momentum term, carrying durable, long-horizon procedural lessons forward while isolating them from the fast, step-level edits.

SkillOpt in action

To evaluate the technique in practice, researchers tested SkillOpt across different models, ranging from large-scale frontier models like GPT-5.5 to smaller closed and open models including GPT-5.4-mini and Qwen3.5-4B. They also deployed the skills within different execution harnesses, using plain chat as well as complex coding harnesses like the Codex CLI and Claude Code.

The evaluation spanned diverse industry benchmarks including single-round question-answering, multi-round code generation involving tool use, and multimodal document reasoning. SkillOpt was measured against multiple baselines ranging from a default no-skill setting to human-written skills and one-shot LLM-generated skills. It was also compared against advanced prompt-optimization and skill-evolution methods, specifically Trace2Skill, TextGrad, GEPA, and EvoSkill.

SkillOpt dominated across the board, proving highly effective on all 52 evaluated combinations of model, benchmark, and harness. It was particularly effective with frontier models, delivering an average absolute improvement of +23.5 points against the no-skill baseline on GPT-5.5. Furthermore, SkillOpt outperformed a hypothetical oracle baseline that cherry-picks the best competing method for every problem.

Small target models saw immense relative gains, proving that a compact text file can supply procedural knowledge that small models lack in their weights. For example, GPT-5.4-nano nearly doubled its score on multimodal document QA and tripled its score on embodied interaction and sequential decision-making.

These academic benchmarks map to critical enterprise pain points. Zero-shot models often hallucinate formatting or fail to use tools properly in multi-step scenarios. Yang explained that the biggest performance leaps occurred in operations that enterprises historically struggle to automate reliably.

“Document data extraction… exact figures out of contracts, invoices, and forms — AP automation, claims, compliance,” Yang said. “What improves is reliability: precise formatting, self-verification, auditable outputs. And the gains come from learning procedure, not memorizing answers.”

For enterprise practitioners, the true value of SkillOpt lies in its portability, efficiency, and compatibility with existing infrastructure. Experiments confirm that the framework is harness-agnostic. In addition to basic chat, the same optimization loop was successfully integrated into tool-backed execution environments like the Codex CLI and Claude Code with significant gains on industry benchmarks.

Developers can train a skill using one execution loop and deploy it in another. For example, a spreadsheet skill trained entirely inside the Codex loop was moved directly into Claude Code and drove a +59.7 point gain over Claude Code’s native baseline without any further changes.

SkillOpt artifacts also transfer cleanly across model scales. A skill optimized for GPT-5.4 was deployed onto the smaller GPT-5.4-mini and GPT-5.4-nano models with positive gains, proving that the learned procedures encode reusable workflows rather than just exploiting quirks of a specific model’s architecture.

Finally, the framework is highly efficient regarding token usage and context window real estate. Across all benchmarks, the final deployed skills never exceeded 2,000 tokens, with a median length of roughly 920 tokens. This results in highly readable, auditable artifacts that a human practitioner can review and manage in minutes.

Implementation strategies and the enterprise ‘catch’

For enterprise tech leaders, adopting a new framework requires understanding the overhead and limitations. While the research paper notes that training tokens can reach up to 210 million for academic benchmarks, the reality for day-to-day enterprise use cases is much lighter. The high token counts in testing were largely due to re-scoring massive held-out test sets.

“The real upfront work is the verifier and a representative held-out split. The optimizer is light; the evaluation harness is where the engineering goes,” Yang said. He added that for everyday use, “in community frameworks like GBrain, where SkillOpt updates run on Claude Sonnet, training a skill for a single task averages just $1–5.” This optimization cost is a one-time fee that amortizes completely at deployment.

However, the framework requires specific conditions to work effectively, namely a few dozen representative examples and a scorable feedback signal. Teams should avoid applying SkillOpt to open-ended or subjective tasks. “With no clean automatic scorer you have to design a human- or model-based evaluator and watch its stability,” Yang said.

SkillOpt also integrates smoothly with existing orchestration stacks, removing a major adoption hurdle. For instance, developers already using pipeline compilers can run both systems harmoniously. “DSPy is a different, complementary layer,” Yang said. “It compiles declarative LM pipelines and optimizes program structure; SkillOpt optimizes the external skill state a frozen agent loads. You can run them together.”

Looking ahead, open-source developers are already scheduling SkillOpt to run periodically over their agents’ past trajectories, creating a small ecosystem of self-optimizing code-agent plugins. This continuous feedback loop represents a significant shift in how AI systems adapt.

“The valuable version of self-improvement is an agent autonomously discovering knowledge to improve its own behavior and the user experience, under verification and audit,” Yang said. “Skills are the fastest, cheapest, most reversible first step, and the same mindset points toward agents eventually optimizing themselves, all the way down to their own weights.”

Presented by F5

Enterprise AI teams have spent years solving for compute, securing GPU allocations, negotiating cloud capacity, and benchmarking training throughput. The assumption embedded in that work is that the path between storage and compute will keep up. In production, that assumption increasingly does not hold. Real traffic introduces latency spikes, network jitter, and node degradation that controlled benchmarks fail to capture, resulting in pipelines that perform well in the lab but stall in deployment. A growing response is AI data delivery, deploying an application delivery controller (ADC) or application delivery and security platform (ADSP) in front of storage as a resilient and secure control point.

“Provisioning solves for capacity but not for delivery, and that is where the constraint now hides,” says Hunter Smit, senior manager of product marketing at F5. “Enterprises buy enough GPUs and enough storage, then assume the path between them will keep up, but AI traffic is bursty, highly concurrent, and random in its reads in ways ordinary storage networking was never built to absorb.”

The production gap benchmarks don’t show

Standard benchmark methodology compounds the problem, says Paul Pindell, principal solutions architect for technology alliances at F5.

“Benchmark testing is usually built to produce the best possible performance or security result, not the most realistic one,” he says. “With S3, latency is a known factor in degrading performance, so meaningful testing has to introduce consistent latency into the path.”

Most benchmark environments never do that, which means the performance numbers enterprises rely on for infrastructure decisions are drawn from conditions that production systems will never replicate. To test this assumption, F5 and MinIO conducted throughput testing under degraded network conditions.

“What stood out was how quickly S3 throughput falls off once you introduce latency,” Pindell says. “Even modest latency takes a real bite out of it, and as latency climbs toward long-haul distances, the degradation gets severe.”

The testing also showed latency mattered far more than jitter as a driver of throughput loss, which inverted what the team had expected going in. The upshot for enterprise architects is that S3 object storage deployments cannot be designed around clean-room assumptions; they have to be engineered for the degraded network conditions they will actually face.

The cost of fragile data paths

“In AI infrastructure, people naturally focus on GPUs because they’re the most visible and expensive resource,” says Tanu Mutreja, senior director of product management at F5. “But in production environments, GPUs generate only as much value as the data path that feeds them.”

That path runs through storage, networking, databases, security, and orchestration layers, often stitched together from multiple vendors. Customers experience none of those seams; they experience the output of the whole system.

When the data path degrades, the effects compound. GPU underutilization is the most immediate and visible symptom, but Mutreja pointed to a wider set of consequences: degraded inference performance, poor-quality AI outputs, higher egress costs from unnecessary data replication, and growing operational complexity.

“At scale, data-path efficiency becomes a strategic business lever rather than technical optimization,” she says. “When the data path is engineered well, GPUs remain productive, AI applications stay responsive and trustworthy, operations scale efficiently, and organizations maximize the return on their AI investments.”

AI workloads are structurally more exposed to these failures than traditional enterprise applications. Databases, ERP systems, and web services absorb transient storage delays through caching and buffering. AI workloads running across massively parallel GPU clusters have no equivalent protection. As Mutreja noted, even minor latency spikes or bandwidth bottlenecks can cascade across large GPU clusters, simultaneously hitting utilization, training efficiency, and the customer experience.

Treating the storage edge as a control point

For decades, storage and intelligence operated as sequential concerns in enterprise architecture: data was stored first, then analyzed downstream. Mutreja argued that this model no longer fits the demands of AI.

“Competitive advantage is determined not only by the volume of data, but also by relevance, lineage, security, and performant delivery of data,” she says. “Across the industry, from NVIDIA and AWS to enterprise storage providers, the movement is toward embedding intelligence directly into data infrastructure rather than stacking it on top.”

F5’s integration with MinIO instantiates this approach at the layer where storage and compute actually interact. As part of the F5 ADSP, BIG-IP sits in the data path, continuously monitoring the health of MinIO’s distributed storage nodes and directing requests only to those that remain available.

The operational impact of that capability becomes clear when nodes degrade, which is expected in distributed storage clusters. Without intelligent routing, clients that land on an unhealthy node must retry and may land on another degraded node, dragging down overall performance.

“F5 makes sure traffic only goes to healthy nodes, or even the least busy ones, so S3 client traffic is always processed in the most efficient way,” Pindell says.

Governance across distributed environments

The challenge grows at scale, when AI pipelines stretch across multiple locations, clouds, or edge environments.

“Once an AI pipeline crosses regions and clouds, the question stops being about performance and becomes about control,” Smit says. “You are operating under different rules in every jurisdiction, and digital sovereignty is now a design constraint. Where your data is allowed to live, who is permitted to touch it, and which borders it cannot cross now shapes the architecture before anyone talks about speed.”

That pressure is driving a visible trend of enterprises repatriating AI workloads from public cloud onto infrastructure they own and govern directly. The architecture Smit described resolves this by decoupling applications from any single storage location and placing a unified control point between them that enforces consistent policy across all of them.

“Sovereignty, resilience, and cost stop being trade-offs you manage one region at a time,” he explains. “They become a capability you run as a system.”

Storage-to-compute path as a managed control point

To solve for these issues, enterprise teams need to stop treating the storage-to-compute path as a direct connection and start treating it as a managed control point, Smit says. SecureIQLab’s independent validation of F5 BIG-IP in storage deployments has confirmed the approach delivers resilience without surrendering throughput.

“Insert a full-proxy ADC between the two, and the path becomes observable, programmable, and failure-aware, with health-based routing, quality of service, and security enforced inline,” he explains. “That single move converts data delivery from an assumption into an engineered discipline, which is what keeps GPUs fed when conditions degrade.”

Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.

Presented by Capital One Enterprises aren’t struggling to experiment with AI; they’re struggling to make it work in the real world. Moving from promising prototypes to reliable, production-scale systems is where most efforts stall.In my role within Cap…

Enterprise AI teams face a dilemma: The best models today might not be the best models a year from now. MassMutual’s answer is to stop making long-term bets — and build infrastructure that can swap models as the market shifts.

“The world of AI today is extremely dynamic,” Sears Merritt, MassMutual CIO, explained in a new VB Beyond the Pilot podcast. “We wanted to make sure we were positioned to ride that wave of dynamism.”

The strategy appears to be paying off in a big way. MassMutual has measured a roughly 30% increase in developer productivity, while AI-powered contact center workflows have reduced resolution times from 10 minutes to one and cut costs from dollars to cents.

But the broader lesson for IT leaders may be less about the results and more about how the company is thoughtfully building its AI infrastructure and keeping users at the center.

Maintaining optionality for the possibilities of tomorrow

MassMutual works with vendors at the leading edge, but keeps those relationships on a clock. “Those relationships are capped so that we maintain optionality for best-of-breed tools as things mature in this space, and at some point, settle down and stabilize,” Merritt said. 

That philosophy extends to open-source models. Merritt says his team is “100%” looking at open-source tools, and sees the technology playing a big role in how MassMutual (and similar companies) use AI. 

“We’re certainly going to need frontier models and leading edge capabilities to do what today is impossible, and tomorrow will be possible,” he said. 

Measuring outcomes from the start

MassMutual’s AI efforts fall into two broad categories.

The first focuses on enablement: Putting productivity-enhancing tools such as Copilot and virtual assistants into the hands of all employees. The second involves what Merritt describes as “deepen and focus” initiatives, where teams target a specific workflow or business process that will have a strong impact on advisors, policyholders, or employees.

Rather than focusing on adoption metrics, these projects begin with predefined success criteria. “Everything we do is measured,” Merritt said. “There’s always a success metric that we define upfront to determine whether or not we’re going to scale up some of these things.”

The company is also deliberately encouraging experimentation, giving employees access to a range of best-in-class models, “token-consumptive workflows” and other possible capabilities so they can weigh the benefits relative to “simpler, lower cost” large language models (LLMs). 

At the same time, MassMutual is collecting increasingly detailed analytics around usage patterns, developer workflows, model performance, and costs. The goal is to reduce spending while also building operational intelligence to eventually route workloads to the right model based on cost, response quality, and user experience.

Those insights will eventually drive optimization decisions around model routing, prompt selection, response times, and infrastructure design.

“We’re gaining access to analytics that let us, in a very granular way, look at usage patterns, developer workflows, and begin to make sense of who’s using what, when, and for what types of tasks,” Merritt said.

Why MassMutual sometimes chooses the more expensive model

Another interesting aspect of MassMutual’s approach is how it evaluates AI quality. Rather than focusing exclusively on benchmarks or token costs, the company uses what Merritt calls a “trust score” framework.

The process combines user feedback with operational metrics to understand how employees perceive AI-generated responses and whether those responses actually improve outcomes. 

The contact center rebuild put that framework to the test. During development, employees were given access to two different LLMs. One generated responses in near-real-time but the quality was noisier. The other more expensive option took several additional seconds to respond but consistently delivered higher-quality answers.

Conventional wisdom and the speed of business might suggest users would prefer the former; but they overwhelmingly chose quality. Merritt’s team asked users about the quality of response, their preferred model, and their overall thoughts on the experience. 

Most of the time, users said: “We want the more expensive one. We’re willing to wait, but the quality difference is so high that the two extra seconds actually is worth it to us.” 

That feedback ultimately determined which model MassMutual deployed.

“We factored that experience piece into the decision-making, and that led us to say, on a relative basis, the costs were immaterial, so we’re going to use the more complex model,” Merritt said.  

Listen to the full podcast to hear more about: 

• Why Mythos “completely changed” the cybersecurity landscape — not the type of threats, but the rate at which those threats appear; 

• How a team of AI engineers modernized MassMutual’s mainframe in 7 days (a process that previously would have taken 3 months); 

• Why MassMutual specifically avoided tokenmaxxing to rein in AI use and spending and has been going “unlimited,” to shield from cost blowups. 

• How a “multi-harness type of environment” will support agentic AI. 

You can also listen and subscribe to Beyond the Pilot on Spotify, Apple or wherever you get your podcasts.

Presented by Snowflake

As AI agents become capable of reasoning across systems and taking action, software is evolving from something employees operate into something that understands intent. Instead of navigating disparate applications and dashboards, a single system will increasingly ask: What are you trying to accomplish?

That sounds like a user experience breakthrough. It is. But the more important implication is organizational. When software no longer relies on humans to provide context, companies can no longer assume that knowledge lives in employees’ heads or is buried inside disconnected applications. The company itself has to become machine-readable.

The winners in the AI era won’t simply deploy more intelligent models. They’ll build the data foundations, semantic context, and governance frameworks that allow machines to understand how the business works and act on that understanding with confidence.

Context is becoming infrastructure

For years, companies treated context as a human layer on top of data. The data platform held the records, then the BI tool visualized them, and the analyst interpreted them. And finally, the business leader made the judgment call. Agents collapse those layers.

When an executive asks, “Why is customer churn rising in our enterprise segment?” an effective agent needs to know far more than where the customer data lives. It needs to understand how the company defines churn, which accounts count as enterprise, whether product usage data is more reliable than survey data, which renewal events matter, what the sales team has logged, what support tickets suggest, and whether the answer differs by geography or product line.

This is why semantics — the definitions, relationships, rules, and assumptions that give data meaning — are moving from a technical concern to a boardroom issue. A semantic layer used to sound like plumbing for data teams. In an agentic enterprise, it becomes the shared language between humans and machines.

If every department teaches its own agent a different version of the business, companies will get inaccuracy at scale. The organizations that pull ahead will be the ones that create a common business knowledge base: consistent definitions, governed access, documented workflows, clear lineage, and enough flexibility to evolve as the business changes. In that world, context is treated as infrastructure, rather than just a nice-to-have.

From dashboards to decisions

The first wave of enterprise AI largely gave us assistants and copilots that answer questions. Useful, but still limited. You ask a question, get a response, and then return to the work of stitching systems together yourself.

The next era of AI will be different. Agents will move beyond coordinating answers, and start getting actual work done. A sales leader starting the day will not need to open a CRM, a forecasting tool, a support dashboard, and a Slack thread to understand what changed overnight. They will simply ask an agent what needs attention. The agent will identify which accounts are at risk, explain why, summarize recent customer interactions, draft follow-up actions, and perhaps initiate the next workflow.

The dashboard does not disappear because charts become useless. It disappears because static reporting becomes too slow for how businesses need to operate. The center of gravity shifts from “show me what happened” to “help me decide what to do next.”

The new governance problem: agents that act

As long as AI is mostly answering questions, governance is about controlling what it can access. That is already difficult. Employees have different permissions, sensitive data needs protection, and answers must be traceable to trusted sources. As agents begin taking action, governance becomes even more consequential.

It’s one thing for an agent to summarize a customer complaint. It’s another for it to issue a refund, reorder inventory, or send an email to a customer. This is where many companies will be tempted to choose between two imperfect paths.

One path is to tightly constrain agents from the start: define the data sources, tools, workflows, and actions they can access. This is easier to manage and measure. It also risks limiting the creativity of employees who understand their workflows best.

The other path is to let teams experiment freely: connect agents to the tools and data they use every day, and allow new use cases to emerge organically. This can produce faster adoption and unexpected innovation. It can also create real risk: stale data, inappropriate access, duplicated workflows, runaway costs, or automated actions no one fully understands.

The right answer is not maximum control or maximum freedom. It’s to prioritize governed flexibility. Companies need architectures where governance is embedded from the beginning. An agent should know not only what it can read, but what it can do, when it needs approval, how its reasoning is inspected, and how its performance is evaluated over time. In other words, governance cannot be a review meeting after the pilot. It has to be part of the system design.

The boundary between builder and user is collapsing

One of the least appreciated consequences of agentic AI is that it will blur the line between people who use software and people who create it. When employees can describe a workflow in natural language and have an agent help build it, software development becomes less confined to engineering teams. A marketer can create a campaign analysis workflow. A finance manager can automate variance explanations. An HR leader can build a policy assistant. A support manager can design a triage process.

These employees are not becoming software engineers in the traditional sense, but they are becoming builders. That changes the talent model. Technical fluency will matter more because employees need to understand what’s possible, what’s risky, and how to evaluate an AI-generated result. Judgment becomes the most important skill.

The winners will be the people who know how to ask better questions, inspect evidence, refine workflows, and combine domain expertise with enough technical understanding to move from idea to execution.

For business leaders, this means AI adoption extends beyond an IT rollout, and is actually an organizational redesign. The distance between insight and action will shrink, and companies will need to rethink who is empowered to build, approve, and operate the workflows that run the business.

Software economics will change too

The shift from interfaces to agents will also challenge how companies buy and measure software, and change how software is priced. Per-seat licensing is giving way to consumption models, where costs reflect actual usage. For most organizations this is a better deal. You pay for value delivered, not licenses that may sit idle.

But it also changes the accountability calculus. When costs are fixed per seat, budget conversations happen once a year. When costs scale with usage, they require continuous oversight. Without visibility into how agents are used and what they produce, costs can rise quickly.

The answer is to build measurement in from the start, connecting AI usage to business outcomes, whether that is deals closed, tickets resolved, or cycle times reduced.The companies that succeed will treat AI cost management as part of operational excellence, not procurement cleanup. The question should not be, “How many tokens did we use?” It should be, “What business outcome did that intelligence produce?”

Your customers may stop using your interface

While the internal implications of agents are significant, the external ones may be even larger. Today, companies obsess over the customer experience inside their applications: the homepage, the navigation, the checkout flow, the dashboard, the mobile screen. Those things will still matter. But increasingly, customers may interact with businesses through their own agents rather than directly through a company’s app or website.

If a procurement agent compares suppliers, a travel agent books a trip, or a financial agent evaluates products, the customer may never see the interface a company spent years perfecting. The agent will care less about visual design and more about whether the company’s data, policies, pricing, inventory, documentation, and transaction systems are accessible, structured, trustworthy, and machine-readable.

That means the competitive surface area changes. A company’s brand may still be emotional, but its operational interface will increasingly be data. Businesses that expose confusing, inconsistent, or poorly governed information will be harder for agents to work with. Businesses with clean semantics, reliable APIs, governed data, and clear policies will become easier to choose, easier to transact with, and easier to trust.

The interface does not vanish only inside the enterprise. It may vanish between enterprises, too.

The real AI readiness test

Most executives know they need an AI strategy, but fewer have internalized what that really requires. AI readiness is not the number of pilots launched, the number of models tested, or the number of employees with access to a chatbot. It is whether the organization’s knowledge, data, permissions, workflows, and decision logic are ready for machines to reason over them safely.

For decades, enterprise software forced humans to become translators between business intent and machine logic. AI is reversing that relationship. Machines are beginning to adapt to human intent. But they can only do that if the enterprise has done the work to make its own context legible.

The future of software is not another screen. It is a system that understands the business well enough to help run it. And that means the next great interface will not look like an interface at all.

Baris Gultekin is VP of AI at Snowflake.

Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.

A joint research collaboration between researchers at the University of Illinois at Urbana-Champaign (UIUC), UC Berkeley, and the open source AI-native vector database platform Chroma unveiled Harness-1, a 20-billion parameter open-source search agent built atop OpenAI’s gpt-oss-20B open source model that fundamentally redesigns how AI executes complex retrieval tasks.

Harness-1 achieves a massive leap in performance, scoring 73% average on its ability to recall relevant information correctly from a curated dataset, outperforming even GPT-5.4 (70.9%) and the next, most accurate open source search agent, Tongyi DeepResearch 30B, by 11.4 percentage points. (While GPT-5.5 has also been out for more than a month, the researchers didn’t test against this model as it wasn’t available when they were building theirs.)

Crucially for developers, the model and its environment are available immediately under the highly permissive Apache 2.0 license and model code/weights on Hugging Face.

Harness-1 also serves as proof-of-efficacy of another effort, Tinker, the distributed, web-based AI model training and fine-tuning API developed by Thinking Machines. Tinker was used specifically to train and run inference for Harness-1, highlighting how interactive infrastructure is actively enabling the next generation of autonomous models.

So how did the researchers do it?

Benchmarks Decoded (and Why Harness-1 Could Help Enterprises Tremendously)

To actually put these models to the test, the researchers evaluated Harness-1 and its competitors across eight highly complex search benchmarks. Rather than asking simple trivia questions, these tests required the AI to act like a real researcher sifting through diverse, dense data sources.

The benchmarks spanned several different domains, including open web searches, complex financial filings from the SEC, technical patent databases from the USPTO, and “multi-hop” question-answering tasks where the AI had to logically piece together scattered clues from multiple different documents to arrive at the correct answer.

When the results came in, Harness-1 dominated the open-source competition in its ability to successfully find and curate the right facts. Even more impressively, this relatively small 20-billion parameter model went toe-to-toe with massive, expensive proprietary AI systems. It actually outperformed heavyweights like GPT-5.4, Sonnet-4.6, and Kimi-K2.5 — thought to be the hundreds of billions or trillions of parameters. Only one giant frontier model—Opus-4.6 — managed to narrowly edge it out in overall average performance.

Harness-1 achieves its performance gains by offloading the exhaustive “bookkeeping” of a search session out of the model’s working memory and into a structured software environment.

As enterprise use cases grow more sophisticated, demanding that models autonomously sift through thousands of corporate documents or financial filings, these systems frequently succumb to “search amnesia”—forgetting their original queries, looping over rejected documents, or losing track of the specific claims they are trying to verify.

Until now, the prevailing solution to this amnesia has been brute force. Engineers typically force models to constantly reread an ever-expanding, append-only transcript of their own actions, piling every search, read, and thought back into a massive context window.

Harness-1 introduces a paradigm shift away from this method, proving that the bottleneck for true artificial autonomy isn’t necessarily the size of the model, but how efficiently its working environment manages state. It highlights once more, as Anthropic’s Claude Code has also done, that the raw model is arguably less important than the harness — or set of conditions — through which it runs.

Technology: Doing the Paperwork in the Environment

To understand the technical leap of Harness-1, consider a real-world analogy.

Imagine hiring a brilliant research assistant and placing them in an empty room without a desk, notepads, or filing cabinets. You ask them to write a comprehensive report on a highly complex topic, which requires them to read dozens of books while keeping every single quote, citation, and dead-end search perfectly memorized in their own head. Eventually, no matter how intelligent the assistant is, their cognitive load will max out, and they will start dropping facts or losing the thread of the assignment.

This is exactly how traditional search agents operate today. They are trained as policies over growing transcripts, meaning the model searches, reads, searches again, and appends everything into its own context window.

As lead researcher Patrick (Pengcheng) Jiang of the University of Illinois noted on X: “At some point the model is not just ‘searching’ anymore. It is also being asked to be a memory system, a note taker, a verifier, and a librarian.”

Harness-1 solves this by giving the AI a desk and a filing cabinet—what the research team calls a “state-externalizing harness.”

This harness is an active, surrounding environment that takes over the routine bookkeeping, maintaining a recoverable working memory that includes a candidate pool of documents, an importance-tagged curated evidence set, compact evidence links, and verification records.

By separating semantic choices from structural state management, the AI is freed up to do what it does best.

The policy still decides what to search, determines which documents to keep, and knows when to stop, while the environment simply holds the state.

Here is a subsection breaking down the training methodology and how it differs from prior agentic search models:

Training Harness-1: A Masterclass in Data Efficiency

The training pipeline for Harness-1 represents a fundamental shift in how the AI industry approaches agentic learning.

Historically, developers have treated search agents as policies operating over massive, ever-growing transcripts, forcing reinforcement learning (RL) algorithms to simultaneously optimize both semantic reasoning and the raw memorization of a search state.

Harness-1’s creators took a radically different approach: because their custom “harness” handles all the routine bookkeeping—like maintaining evidence links, candidate pools, and verification records—the training process only needed to teach the model how to operate this structured interface.

This division of labor drastically simplified what the underlying 20-billion parameter model actually needed to learn.

The process began with a remarkably narrow Supervised Fine-Tuning (SFT) stage. Rather than scraping petabytes of new behavioral data, the team generated just 899 filtered trajectories using a GPT-5.4 teacher agent that was plugged into the exact same harness environment the student model would eventually use.

The goal of this SFT phase was not to inject vast amounts of domain knowledge into the model, but simply to teach it the mechanical rhythms of a good researcher: how to format tool calls, how to tag documents by importance, and the discipline of verifying a claim before promoting it to the final curated set.

Following SFT, the model underwent Reinforcement Learning (RL) using an algorithm called CISPO, applied over full search episodes capping at 40 turns.

The team designed a highly specific terminal reward function that explicitly separated discovery from selection. The model was rewarded not just for finding a relevant document, but for successfully promoting it into the final answer set, while being penalized if it found the answer but failed to curate it.

The researchers also instituted a “tool diversity” bonus; without this specific incentive, they found the policy would quickly collapse into a lazy, search-heavy strategy where it spammed queries but bypassed the harder work of reading and verifying the text.

What makes Harness-1 truly innovative compared to prior work is its unprecedented data efficiency. The entire model was trained on roughly 4,400 unique items—899 SFT trajectories and 3,453 RL queries.

In stark contrast, competing open-source models required vastly larger datasets to achieve worse results: Context-1 utilized over 17,200 training items, while Search-R1 relied on a staggering 221,300 items to learn search behaviors.

By proving that a smarter external cognitive architecture can replace brute-force data scaling, Harness-1 suggests that the future of agentic AI lies in building better environments for models to work within, rather than just training larger models on more data.

Product: Enterprise Applicability and Generalization

From a product perspective, Harness-1 is delivered as a highly capable 20B agent merged into the openai/gpt-oss-20b base architecture.

For enterprise tech stacks, the applicability is massive because businesses need AI to execute multi-step research across proprietary databases without hallucinating or running up exorbitant compute bills.

Harness-1 manages its frontier-level performance at what the creators describe as “Context-1-level cost and latency.” Because the context window is strictly managed by the budget-aware harness rather than continuously expanding, enterprises can deploy this agent autonomously without incurring the exponential token costs typically associated with long-horizon AI tasks.

Even more impressively, Harness-1 proves it can generalize well beyond its training data. According to the research team, it was incredibly cheap to train, utilizing just 899 filtered supervised fine-tuning (SFT) trajectories and a mere 3,453 reinforcement learning (RL) queries.

“Instead of training the model to survive a giant append-only transcript, we train it to use a structured search interface: search, curate, revisit, verify, and submit,” Jiang explained.

This leanness proves a critical point for the AI industry: developers do not necessarily need petabytes of new behavioral data if they build a better cognitive framework for the model to operate within.

Licensing: The Power of Apache 2.0

One of the most significant aspects of the Harness-1 release is its licensing. In plain language, Apache 2.0 is a highly permissive, enterprise-friendly software license that fundamentally enables commercialization.

Unlike “copyleft” licenses (such as the GPL) that can force companies to open-source their own proprietary software if they integrate the code, or “research-only” licenses that ban commercial use entirely, Apache 2.0 gives businesses the green light to freely build, modify, and monetize the technology.

For developers and startups, this means Harness-1 can be seamlessly integrated into commercial enterprise search products, internal data retrieval tools, or customer-facing AI applications without fear of legal reprisal.

The only major requirement is that users must include the original copyright notice and explicitly state any significant modifications they make to the source code, positioning Harness-1 as a highly viable foundational building block for the enterprise.

Community Reactions: A Resounding Validation

The announcement has clearly struck a nerve within the developer community, validating the very real pain points engineers face when building agentic systems. Jiang’s multi-part announcement thread on X quickly garnered massive traction, pulling in over 256.1K views, 3.7K likes, 2.9K bookmarks, and nearly 300 reposts within a matter of days.

This high engagement underscores a growing consensus in the AI space that brute-forcing context windows is a losing battle.

When Jiang posted on X, “I’ve been wondering: maybe search agents are bad at search partly because we make them do all the paperwork in their head,” the resonance was immediate.

For developers who have spent the last year wrestling with AI agents that confidently forget their primary instructions halfway through a database search, the Harness-1 approach feels like a desperately needed course correction.

Ultimately, the community sentiment highlights a shift in industry priorities. Developers are moving away from asking how large an AI model’s context window can get, and instead asking how efficiently an AI model’s environment can manage that context for it. By offloading the paperwork, Harness-1 is proving that smaller, smarter systems can outmaneuver the giants—provided they have the right desk to work at.

Microsoft used its Build 2026 conference this week to push a clear message: agents are rapidly moving into production throughout enterprise systems, and the winning platform will be the one that gives them reliable context, governance, identity, memory…