Enterprise AI has largely been built around context engineering. Teams connect enterprise systems, generate chunks and embeddings, build retrieval pipelines, and assemble the context needed by individual AI applications. While this approach works well …

For much of the past two years, the general belief in enterprise AI has been that more autonomy equals better performance. Build agents that can plan, decide, and act across multi-step workflows, and give them as much room to run as possible. That assumption is now being tested at scale, in real production environments — and in a lot of deployments it’s failing. The companies that end up benefiting from agentic AI won’t necessarily be the ones that have given their agents the most flexibility. They’re the ones who create AI agents with specific responsibilities and make sure they operate within clear rules.

Two numbers tell you almost everything about where agentic AI stands in mid-2026. By Gartner’s own forecast, more than 40% of the agentic AI projects running today won’t survive to see 2028. Not because the models fall short; because of escalating costs, unclear business value, and inadequate risk controls. McKinsey’s 2026 AI Trust Maturity Survey fits right alongside that prediction: Agentic AI deployment is accelerating across every industry, but average responsible-AI maturity sits at just 2.3 out of 4. Only about 30% of organizations have reached a maturity level of three or higher in governance and agentic AI controls specifically.  

Put those two numbers side by side, and the story tells itself. Capability is outrunning control.  

That shift is changing the competitive framing, too. The 2024-to-2025 race was about who could deploy the most autonomous agent the fastest. The 2026-to-2027 race is a trust race.  

It’s not about who can build the most capable agent. It’s about who can get an agent approved for production by risk, legal, and compliance teams, and keep it approved once it’s live. This is a different kind of engineering challenge than most enterprises are prepared for.

Why full autonomy breaks down in production  

Gartner lays out the failure pattern as specific and repeatable. Projects launch with ambitious, broadly autonomous workflows. They hit integration complexity within weeks. Then they stall, with no defensible path to production ROI. Part of the problem is vendor noise. Gartner’s own count puts it starkly: Out of the thousands of products being sold under the ‘agentic AI’ label, only around 130 actually have real autonomous capability behind them. The rest are largely automation or chatbots repackaged for the moment.

But even genuinely agentic systems run into a structural problem that has nothing to do with hype. Autonomy and accountability move in opposite directions.  

An agent capable of independently planning and executing a multi-step task is also an agent whose individual decisions get harder to trace after the fact. Let’s say something breaks a few steps into an autonomous chain. Figuring out why the agent made that decision and who is responsible can be a complicated process, not a simple lookup.

In areas like financial reconciliations, compliance processes, manufacturing quality checks, or clinical documentation, this lack of transparency can be the difference between a manageable mistake and a serious regulatory breach. It’s the reason legal, risk, and compliance teams block agentic projects from reaching production, regardless of how capable the underlying model is.  

Integration complexity keeps showing up as a leading cause of project cancellation. Bolting an autonomous agent onto a legacy workflow takes more than technical connective tissue. The workflow’s existing decision points, approval chains, and audit trails all need to be rebuilt around a system that can now act without waiting for a human. Enterprises that treat this as a pure integration problem, solvable with more engineering hours, tend to be the ones that stall.  

This isn’t a hypothetical risk. McKinsey’s research shows how exposed most enterprises currently are. Across nearly every category of AI risk, from data privacy to intellectual property exposure, the gap between the risks organizations say they’re aware of and the risks they’re actually mitigating remains wide.    

Awareness has surpassed action. This gap is reflected in the businesses that report it as an obstacle to further scaling of agentic AI. Nearly two-thirds now say security and risk issues are the greatest challenge for them, surpassing regulatory uncertainty and technical barriers.    

What governed orchestration actually looks like  

The enterprises that are leading the way are not halting their AI plans. They’re restructuring how autonomy is being distributed within the system. The four patterns that stand out in organizations that are governance-mature are:  

• Narrow-scope agents over general-purpose ones. Decompose end-to-end workflows into single-responsibility agents with tightly bounded mandates. A smaller scope of work results in a smaller scope of failure, and a smaller scope of failure is much easier to audit.  

• Human checkpoints at decision boundaries, before the outcome, not after it. Review agent decisions before high-stakes actions execute, not after the fact. That means checkpoints before sensitive data moves, a transaction posts, or an external system is triggered. McKinsey’s framework calls for real-time, data-driven monitoring built into the agent pipeline itself, with humans retaining final accountability specifically for high-stakes decisions.  

• Decision traceability as a design requirement. A full action log and decision lineage should be available on demand for any agent, any decision. It shouldn’t need to be reconstructed under pressure during an audit. Regulators are pushing the same way. The EU AI Act’s human oversight requirements for high-risk systems are still coming, even though this year’s Digital Omnibus agreement pushed the compliance deadline out to December 2027. Enterprises building agent systems now are effectively building toward that requirement, whether or not it’s technically enforceable yet.  

• Data sovereignty does active governance work, not passive paperwork. Where an agent’s data sits, and who has access to it, decides how contained a failure can be. On-premise or controlled-environment deployment limit the blast radius of a misbehaving agent and simplifies exactly the kind of audit trail regulators and boards are starting to expect.  

The risk runs in both directions, of course. Agentic AI is supposed to cut friction. An agent that needs a human to sign off on every minor task hasn’t cut anything; it’s just automation wearing a manual process as a costume. That quietly undercuts the whole case for building the agent in the first place. The goal isn’t maximum control. It’s calibrated control, concentrated where the cost of an error is actually high.  

Agent deployment is scaling roughly 8x faster than governance maturity is improving.  

A practical framework for evaluating your agent stack  

If enterprise architects are reviewing an existing agent for deployment or are considering deploying an agent, they can begin by asking four questions.  

1. Can you reconstruct, six months from now, exactly why a specific agent took a specific action?  
If the honest answer requires digging through raw logs or guessing, decision lineage isn’t a design feature of the system. It’s an afterthought. And it will show up as a gap in the next audit.  

2. Does every agent in the stack have one clearly bounded responsibility, or is at least one agent authorized to “figure it out” across a broad task?  
Broad, open-ended mandates are exactly where compounding errors and untraceable decisions originate.  

3. Are human checkpoints placed at defined decision boundaries, or only as a final review after the agent has already acted?  
A review after the fact catches consequences. A checkpoint before the fact prevents them.  

4. If an agent were compromised or malfunctioning right now, how much data and how many downstream systems could it touch before anyone noticed?  
This is where data sovereignty and access scoping stop being compliance line items and start functioning as containment strategy.  

These are all questions that don’t need to slow down the adoption of agentic AI. They need direction on how and where autonomy is of value to their organization and how to open up to exposure. This suggests building the orchestration layer on that separation, rather than adding governance after a production incident forces the question.  

The real competitive advantage  

Gartner’s 40% cancellation forecast isn’t really a warning about AI capability. It’s a forecast about organizational discipline. Right now, agentic AI sits at what Gartner defines as the “peak of inflated expectations,” and there’s a fairly straightforward explanation. Enterprises spent 2024 and 2025 optimizing almost entirely for autonomy. Now they’re paying down the governance debt that approach accumulated.  

The winning position by 2027 won’t belong to whoever deployed the most autonomous agents fastest. It will belong to whoever built agent systems trustworthy enough that risk, compliance, and legal teams stopped being the bottleneck. The architecture answered their questions before anyone had to ask them.  

That’s a different design brief than most agentic AI roadmaps were written against. It’s about making scoped autonomy, checkpointed decisions, full traceability, and data sovereignty integral to the architecture from the start, not add-ons after a pilot project has been a success.

Midhula Mariyam Jeevan is a content writer specializing in AI, enterprise technology, software engineering, and SEO.

Most teams building retrieval augmented generation (RAG) systems for high stakes classification make the same architectural bet: Route every ambiguous case straight to the language model and trust the retrieved context to sort it out. This works fine in a demo. It falls apart the moment the system has to survive an audit, a regulator, or a compliance officer asking why a specific decision was made six months ago.

I have spent the last year building RAG based classification systems in regulated enterprise settings, where the cost of a wrong answer is not a bad chatbot reply. A decision has to hold up to scrutiny long after the model produced it. This environment forces a different design philosophy than most AI engineering content assumes.

Here is what changes when you cannot afford to be probabilistic about everything, and how a cascade architecture solves it.

The invisible cost of an all LLM pipeline

The appeal of routing everything through a large language model (LLM) is obvious: Fewer moving parts, faster iteration, the model handles unanticipated edge cases. The problem shows up later, in three places.

First, auditability. “The model decided based on retrieved context” is not an acceptable answer. You need a decision path a human can reconstruct without rerunning inference and hoping for the same output.

Second, cost at scale. If your system processes tens of thousands of cases a day and every one hits an LLM call with several retrieved documents in context, your inference bill and latency both scale with volume in a way that rule based logic does not.

Third, and least discussed, model drift on the easy cases. LLMs are excellent at nuanced judgment calls. They are inconsistent, in ways that are hard to detect, on cases that should have a deterministic answer. A clear structured match against known criteria should never depend on a language model’s mood.

The cascade approach

The fix: Stop treating the LLM as the front line and start treating it as the escalation path. In practice this means a three stage pipeline.

Stage one is deterministic. Exact matches, structured field comparisons, and anything with a clear rule get resolved here with no model call at all. This stage should clear the majority of volume, often more than half depending on your data quality, and every decision is fully explainable because it is a lookup, not an inference.

Stage two is where retrieval earns its keep. For cases that survive stage one — and I mean survive as in they were not clearly resolved — you build a retrieval layer that pulls the specific evidence relevant to the ambiguity: Prior reviewer decisions on similar cases, contextual documents that explain an apparent conflict, or historical precedent that clarifies an edge case. The retrieval step matters more than the generation step here. If you retrieve the wrong context, even the best language model in the world will produce a confident, well reasoned, wrong answer.

Stage three is the LLM call, and it should only see the residue that stages one and two could not resolve. This is the part people skip when they design their first version, and it is the single biggest lever for both cost and quality. In one system I worked on, routing only the genuinely ambiguous 10 to 15% of cases to the LLM cut inference cost by roughly 6X compared to an all LLM baseline, while improving consistency on the deterministic majority to effectively perfect.

Designing the prompt for asymmetric risk

Once a case reaches the LLM stage, most teams default to a neutral prompt: “Assess whether this case should be approved or flagged.” That framing is wrong for high stakes classification because the cost of the two error types is not symmetric. Missing something that genuinely needed attention can mean real harm downstream. Incorrectly flagging something that was fine costs a reviewer’s time and a delay. Those two outcomes are rarely equally bad, yet a neutral prompt asks the model to treat them as if they were.

An asymmetric risk prompt makes that tradeoff explicit to the model rather than letting it guess at your risk tolerance. Concretely, this means instructing the model to treat uncertainty as a reason to escalate rather than clear, providing calibrated examples of both error types with their consequences spelled out, and asking for a confidence score alongside the classification rather than a binary answer. The confidence score becomes your second cascade point: Anything below a certain threshold goes to a human reviewer instead of being auto resolved, no matter what the model’s classification says.

This sounds like a small prompt engineering detail. In practice it is the difference between a system that reduces reviewer workload and one that quietly increases risk while looking like it is working.

Evaluating a system like this properly

Standard RAG evaluation metrics were not built with this use case in mind, and using them without adaptation will give you a false sense of confidence. A few adjustments that matter.

Retrieval quality needs to be measured separately from final classification accuracy. A system can have excellent retrieval ranking scores and still make bad final decisions if the generation step misweights the evidence. Track them independently.

Your evaluation set needs deliberate oversampling of the cases that reach stage three, since that is where your system’s judgment actually gets tested. If your eval set mirrors your production distribution, it will be dominated by the deterministic cases your cascade already handles well, and you will be blind to exactly the failures that matter most.

LLM as judge evaluation works for this domain but only if the judge prompt encodes the same asymmetric risk framing as your production prompt. A judge that treats both error types equally will systematically favor the wrong tradeoff when you are tuning your system.

Finally, build a feedback loop from confirmed outcomes back into your retrieval corpus. When a human reviewer overturns a model decision, that case and its correct resolution should become retrievable context for future similar cases. Without this, your system’s handling of ambiguous cases never improves, it just keeps making the same category of mistake at the same rate.

The broader lesson

The instinct to reach for the most capable model for every decision is understandable, but in domains where wrong answers have real consequences, the more valuable engineering work is deciding what should never touch the model at all. Cascade architecture is not a workaround for LLM limitations. It is what a mature RAG system looks like once you have actually had to defend its decisions to someone whose job is to find the flaw in your logic.

If you are building AI systems for any regulated or high stakes domain, the question worth asking before you write a single prompt is not “How do I get the model to handle this well.” It is “Which parts of this decision should never have been the model’s job in the first place.”

Vineet Vijay is a Lead AI and machine learning engineer.

There is a step in the development process for large language model (LLM)-assisted tooling that most teams skip because it’s tedious, time-consuming, and doesn’t produce results visible to end users: Verifying that what the model is saying is actually correct. Not fluent, not coherent, not topically relevant — correct in the sense of accurately identifying the right answer to the specific problem the tool was built to solve.

The gap between “this output sounds right to me” and “this output is verifiably correct” is where most LLM-assisted enterprise tools fail quietly. They pass internal review because the output sounds right. They fail in production because those people weren’t reviewing against ground truth — they were reviewing against their intuition about what a good answer looks like.

This distinction matters more as LLM-assisted tools move from productivity accessories to components that influence real business decisions. If your AI-assisted tool is shaping how an analyst investigates a data quality issue, how a compliance reviewer decides whether to escalate a flagged record, or how an operations team triages a validation failure — the accuracy of its output has real consequences. “Seems reasonable” is not an adequate evaluation standard for that.

What qualitative evaluation actually catches

The standard evaluation approach for LLM output in enterprise tooling is qualitative: A sample of outputs is reviewed by someone with domain knowledge, judged against a mental model of what a good answer looks like, and the prompt is adjusted if too many outputs seem off.

This catches a specific class of problems: Outputs that are obviously wrong, poorly formatted, or off-topic. These are real issues worth catching. They’re also the easy ones.

What qualitative evaluation consistently misses is the class of outputs that are wrong in ways that are difficult to see without checking against something external. An explanation that confidently identifies the wrong root cause, in language that sounds authoritative, based on reasoning that sounds plausible — this passes qualitative review. It fails the moment someone with the right context checks it against what actually happened.

In a system whose value proposition depends on accuracy, “sounds plausible” is not the same as “correct.” The two can diverge significantly, and qualitative review won’t tell you when they have.

What an actual eval harness looks like

The alternative is building an evaluation harness that scores model output against labeled ground truth — a set of cases where the correct answer is known, against which you can measure accuracy rather than coherence.

I built this while developing a root-cause explainer for data migration drift: A tool that takes a detected drift event and generates a ranked explanation of what most likely caused it. The first prototype produced fluent, specific-sounding explanations that passed qualitative review. When I tested it against cases where I already knew the root cause, the explanation was wrong often enough to matter.

The eval harness I built works in three parts.

First, a synthetic ground truth dataset: Cases where the correct answer is known by construction. This meant introducing specific, controlled causes into a test pipeline — schema changes, transformation logic bugs, source system behavioral shifts — recording exactly what I introduced, and running the model against the resulting drift events. The correct answer for each case was the cause I had deliberately introduced.

Getting the synthetic scenarios realistic enough to be useful required more care than I expected. Early versions were too clean — the drift signal was obvious in ways that real production drift events aren’t. Adding realistic noise, overlapping signals, and cases where multiple plausible causes were present simultaneously was what made the synthetic set actually predictive of real-world performance.

Second, a scoring function that evaluates ranked output. Binary correct/incorrect isn’t sufficient when the model produces a ranked list of likely causes rather than a single answer. An explanation that correctly identifies the root cause as the third most likely candidate is meaningfully different from one that identifies it as the most likely. The scoring function evaluated two dimensions: Presence — did the correct answer appear in the output at all — and rank — how prominently was it featured relative to incorrect candidates. These were combined into a weighted score that rewarded both finding the right answer and ranking it appropriately.

Third, systematic evaluation across the full synthetic dataset rather than spot-checking. Running the harness across the complete set reveals patterns that spot-checking misses: Which categories of problem the model handles reliably, which it consistently gets wrong, and which combinations of signals produce the highest rate of confident incorrect explanations.

What the evaluation revealed

The results were more informative than any qualitative review could have been.

Schema change scenarios scored well — the model was reliable at identifying upstream schema changes when the evidence was present and distinctive. Transformation logic bugs were harder — the model consistently identified the right general category but misattributed the specific change that caused the problem, particularly when multiple changes had been made close together. Overlapping-signal scenarios were the hardest — cases where two different causes occurred close in time produced the highest rate of confidently wrong explanations.

That last finding is the one that qualitative review would never have surfaced. The model’s expressed confidence didn’t correlate with its accuracy — it was most confident in the cases where it was most wrong. Without the eval harness measuring against ground truth, that pattern would have been invisible.

The practical implication for enterprise AI deployment

For teams deploying LLM-assisted tools in enterprise contexts — particularly tools that influence how people investigate problems, triage alerts, or make routing decisions — the eval harness question to answer before production deployment is: Have we measured accuracy against cases where we know the right answer, or have we only reviewed whether the outputs seem reasonable?

If the answer is the latter, the tool has been tested for fluency and coherence but not for correctness. Those are different properties. For tools that shape business decisions, correctness is the one that matters.

Building the synthetic ground truth dataset is the hard part and the part most worth investing in. It forces you to define precisely what “correct” means for your specific use case — which turns out to be a useful exercise independent of the evaluation itself. The scoring function and the harness infrastructure are relatively straightforward once you have that definition. Without it, you’re measuring something other than what you’re trying to guarantee.

Arun Mishra is an enterprise architect.

If you have built anything with retrieval-augmented generation (RAG) in the last two years, you have lived its central frustration: You chop your documents into chunks, embed them, retrieve the top few that look similar to the question, and hand them t…

The enterprise technology ecosystem is caught in a costly cycle. Over the past two years, millions of dollars have been funneled into generative AI pilots, yet many of these initiatives stall out before ever reaching a live production environment.

When a project fails, the immediate instinct of technical leadership is often to blame the model: The context window was too restrictive, the latency was too high, or the reasoning capabilities simply were not there.

But as data engineers building the scaffolding for these systems, we often see a different reality: The model receives the blame, but the pipeline usually contains the root cause. Production gen AI rarely fails because of model limitations alone. More often, it fails because the enterprise data foundation underneath it is fundamentally unready.

This is what I call the ‘Cleanup Trap’: The false belief that an organization can pipe fragmented, inconsistent, and ungoverned legacy data into a large language model (LLM) orchestrator and simply “clean it up” or patch it at the retrieval layer.

The mirage of the retrieval layer

In a standard retrieval-augmented generation (RAG) architecture, the retrieval layer is tasked with pulling relevant business context to ground the model’s responses. Because modern frameworks make it simple to stand up a vector database and a basic embedding pipeline, leadership often assumes that the data engineering problem is solved.

It is not.

When an embedding model receives raw, unvalidated data directly from operational silos, the resulting vector space inherits the structural noise, duplicate records, and conflicting states present in the source systems.

If the core data pipeline suffers from silent degradation — schema drift, missing fields, delayed change-data-capture (CDC) synchronization — that degradation cascades directly into the vector store. An AI model cannot accurately synthesize customer intelligence if the data pipeline behind it is serving stale, contradictory profiles across disparate storage layers.

No amount of prompt engineering, semantic reranking, or vector hyperparameter tuning can compensate for a broken ingestion pipeline. If the foundation is compromised, the downstream application will hallucinate, expose unauthorized context, or fail to deliver deterministic value.

Shifting from ad-hoc patching to programmatic guardrails

To break out of the ‘Cleanup Trap,’ enterprise data teams must stop treating data quality as a post-processing step. They need to treat data readiness for AI with the same rigor they bring to traditional transaction processing.

This requires a deliberate architectural shift toward zero-trust data ingestion, structured validation frameworks, and automated anomaly detection before data ever reaches an AI orchestration layer.

1. Harden the ingestion pipeline

Data quality checks cannot exist as a nightly batch afterthought. If an enterprise AI application relies on real-time data to assist users, validation must happen inline.

Teams should implement explicit schema validation checks at the earliest ingestion point, such as the streaming ingress layer or the bronze landing layer of a medallion architecture. If an upstream operational database mutates a schema without warning, the pipeline should quarantine anomalous payloads rather than allowing corrupted metadata to pollute downstream AI contexts.

2. Use multi-tiered algorithmic validation

Static row-count validation rules are insufficient for AI readiness. True data health requires a multi-tiered approach.

This means pairing structural verification — null checks, type conformance, and schema validation — with statistical profiling to monitor for data drift. Tracking metric deviations across feature distributions helps ensure that historical context remains stable over time.

If a pipeline suddenly processes an unexpected spike in empty string variables or structurally deviant fields, automated alerts should trigger an immediate pause before vector database updates continue.

3. Decouple security and compliancemfrom the model

An LLM should never be the arbiter of data access control. Trying to enforce row-level security or personal data filtering through system prompts is a compliance risk.

Security must be managed within the data infrastructure tier. Enterprise data foundations should enforce strict access controls, tokenization of sensitive identifiers, and rigorous lineage tracing before information is indexed into vector stores or passed into an agent’s context window.

Technical alignment: A pragmatic blueprint

For technology leaders mapping their infrastructure roadmaps, AI readiness requires evaluating data pipelines against a strict operational checklist.

• Can you trace a flawed AI response back to the exact pipeline execution, source record, and transformation step that produced it?

• Does your data lake architecture have a programmatic mechanism to segment and quarantine corrupted or non-compliant data before it reaches production feature stores?

• Are your operational systems and AI-facing vector databases tightly synchronized, or are your agents making automated decisions based on outdated snapshots?

These questions matter because production AI is not just a model deployment problem. It is a data reliability problem.

Building for the production era

The honeymoon phase of gen AI experimentation is ending. Enterprise leaders are demanding measurable, predictable, and secure business outcomes from their AI investments.

If an organization wants to transition from isolated, impressive-looking demos to resilient, production-grade AI systems, it must redirect its focus. Stop looking exclusively at the model tier.

The real competitive differentiator is not only the LLM an organization chooses. It is the engineering discipline, data governance, and pipeline resilience of the infrastructure built to feed it.

In the production era of AI, data engineering is no longer a backend function. It is the control plane for enterprise intelligence.

Naveen Ayalla is a senior data engineer.

DeepSeek’s recent decision to drastically cut pricing on its V4-Pro model by 75% should have been unequivocally good news for enterprise AI vendors and developers. Instead, many are discovering that cheaper models don’t automatically translate into healthier margins.

The reason is simple: While inference costs plummet, agent systems are voraciously consuming tokens faster than prices are declining. For the last 2 decades, software economics was dictated by the same rule. Infra became cheaper every year whereas applications became more capable. AI was initially hypothesized to follow the same pattern. As frontier models improved and token prices dropped, many assumed inference would become a negligible operating expense.That assumption has begun crumbling exponentially. 

A chatbot usually turns one user question into one model call. An agent turns it into a chain of planning, retrieval, tool use, verification, summarization, and follow-up decisions. The user sees one answer. The vendor pays for the loop. That is the 100x problem: The same user-visible request can cost a lot  more to serve as an agentic workflow than as a chatbot or retrieval-augmented generation (RAG) response. In longer-running workflows, the multiplier is higher. Falling model prices help, but they do not fix a product architecture that turns one prompt into dozens of billable operations.

The scale of what is now at stake is clear in how model providers themselves are pricing developer relationships. OpenAI’s proposed program to give every Y Combinator startup $2 million in API credits — a number that would have funded an entire seed round in any prior tech cycle, and when the same cohort got by on a few thousand dollars of AWS credits — is less a recruiting perk than an admission of what it now costs to run an AI-native company through its first year of product. For established enterprises retrofitting agents into existing product lines, the absolute numbers are larger still.

What token amplification is

In a single-turn chatbot, one user message produces roughly one model call. Input-to-billed ratio is about 1:5.

In a multi-step agent rolled out across customer support, sales operations, finance, legal review, and engineering, that ratio routinely lands at 1:700 or higher. Every loop iteration carries forward the cumulative conversation, tool outputs, and reasoning traces. Each step appends; nothing is dropped.

A “simple” agent query like “What did our top customer ask about last week?” typically touches seven priced operations before returning an answer:

1. User prompt (~50 tokens)

2. System prompt and tool definitions (~3,000 tokens, repeated on every call)

3. Retrieval (~5,000 tokens of context)

4. Model call #1 — tool selection (8,000 in / 200 out)

5. Tool execution (~4,000 tokens returned)

6. Model call #2 — summarization (12,000 in / 400 out)

7. Model call #3 — follow-up decision (12,400 in / 100 out)

One sentence in, roughly 35,000 input tokens billed. Somewhere between $0.10 and $0.40 per query on a frontier model. Multiply that by a million queries a month — the table-stakes volume for any enterprise B2B feature — and the line item is six figures.

Why this breaks the existing AI business model

The dominant pricing story for enterprise AI has been seat-based SaaS: Pay per-user per-month, deliver agent capability, capture margin. That model assumes a reasonably bounded cost-per-user.

Token amplification breaks the assumption. A power user running 50 agent invocations a day on a $40/seat plan can cost more in inference than the plan charges. Token amplification shatters the traditional SaaS pricing model. When a power user’s daily agent activity costs more in inference than their monthly subscription fee, vendor gross margins turn negative, a paradox that compounds as customers deepen their agent adoption, the very usage curve vendors are selling to their boards. Several vendors are now privately reporting negative gross margins on heavy users, mirroring recent cloud expenditure reports from the Bessemer ‘Supernova’ cohort, where the correlation between AI-agent adoption and gross margin contraction has moved from a theoretical risk to a primary P&L headwind.

The visible symptoms have started leaking into public coverage. Bloomberg this week documented a widening gap between Salesforce’s Agentforce marketing demos and the capabilities actually shipping to customers. This is the kind of gap that opens predictably when promised functionality is technically possible but uneconomical to serve at the price the seat plan implies. Salesforce is the most-watched case, not a unique one.

“For my team, the cost of compute is far beyond the costs of the employees.” — Bryan Catanzaro, VP of Applied Deep Learning, Nvidia

The strategic implication is not “AI is expensive.” It is that the dominant business model assumed by most AI-native company plans does not survive contact with agentic workloads.

A simple example

Consider an enterprise software vendor charging $40 per-user per-month for an AI-enabled support assistant. A traditional chatbot might cost only a few cents per user per day in inference, leaving healthy gross margins.

Now replace that chatbot with a fully agentic workflow capable of investigating tickets, querying internal systems, drafting responses, validating outputs, and escalating exceptions. If a heavy user executes 50 to 100 agent requests per day, inference consumption can increase by an order of magnitude. What was once a negligible infrastructure cost becomes a material operating expense.

This creates an unusual dynamic: The customers receiving the most value from the product are often the customers generating the highest inference costs. In extreme cases, vendors can find themselves with their most engaged users contributing the least profit. The result is a growing realization across enterprise software that agent adoption and margin expansion are no longer automatically aligned.

Agent orchestration is the new moat

The technical responses are known and converging. They are not novel, but they are critical for survival

• Cost-aware routing: This technique involves a small classifier model that decides which tier (Haiku, Sonnet, Opus equivalents) handles each query. Well-tuned routers cut inference bills by around 60% without any degradation in quality

• Prompt caching: Anthropic, OpenAI, and Google now offer 75 to 90% discounts on cached prefixes. 

• Context discipline: You can truncate tool outputs, prune reasoning traces, and cap tool depth to prevent your agent from going down a rabbit hole

• Speculative decoding: for self-hosted deployments, this technique guarantees 2 to 3X effective throughput on the same GPUs.

“Organizations using orchestration-led governance report stronger productivity gains — a holistic orchestration layer is associated with six times greater productivity impact than compliance‑only approaches” — IBM

The companies building this layer well are starting to look less like microservice operators and more like financial trading systems: Every routing decision priced, every path with its own P&L, every tenant on a metered budget.

What enterprise leaders should actually do

Four moves separate the companies that will still have margin in 24 months from the ones that won’t:

1. Make inference cost a first-class metric. Track it per-feature, per-tenant, per-query class the same way cloud cost was tracked starting in the mid-2010s.

2. Budget like a media buyer. Set cost-per-thousand-queries ceilings per feature. Cap them. Alert on overruns. Engineering will not enforce this on its own.

3. Treat the router as core infrastructure, not an optimization. It is the new load balancer.

4. Audit prompts quarterly. A 4,000-token system prompt that grew organically over six months is a six-figure bill in slow motion. Most teams have never read their own production prompts end to end.

5. Negotiate volume commits early. Frontier-model vendors now offer reserved-instance-style prepaid commits at substantial discounts. List price is the worst price any enterprise will ever pay.

The next 24 months

The structural shift underneath agentic AI is not that it is expensive. As DeepSeek’s price cut today underscores, frontier inference unit costs are dropping roughly 3X per year, and the curve is not slowing.

The shift is that amplification is outrunning the price cuts. Cutting per-token costs 75% does not help a company whose agents are doing 700X more tokens per user query than its pricing model assumed. For the first time since the cloud era began, architecture decisions are again financial decisions in real time. A prompt redesign is a margin event. A poorly bound agent loop is an outage with a credit card attached.

The companies that survive the next 24 months of AI infrastructure pricing will not be the ones running the cheapest model. They will be the ones whose agents are smart and know what they cost to think.

That is the 100X problem. And it is arriving faster than the price cuts can hide it.

Maitreyi Chatterjee is a senior software engineer at a big tech company.

Devansh Agarwal works as an ML engineer at a leading tech company.