Most AI pilots stall because nobody owns the decision to run them in production, the evaluation covers a demo rather than live work, the data and run costs were never priced, and no one agreed in advance when to stop. The model is rarely the cause. Moving an AI pilot to production means making six operating decisions before scale: a named owner, evaluation before scale, data access, run cost, change management, and kill criteria.
The gap is in the results, not the adoption
Adoption is no longer the problem. Stanford HAI's 2026 AI Index reports that organizational adoption of AI reached 88 percent. McKinsey's survey The state of AI in 2026, based on 1,719 respondents and published in August 2026, reports that 37 percent attribute at least some EBIT impact to AI, about the same share as the year before.
The same McKinsey survey defines high performers as organizations that attribute at least 5 percent of EBIT to AI and describe the impact as significant. About 6 percent of respondents meet that bar, and the share has not moved. Among organizations with more than $1 billion in revenue, 40 percent say they are scaling AI agents, up from 27 percent a year earlier. Smaller organizations stayed flat at 22 percent.
Read together, these numbers describe a long middle: many organizations use AI, a growing number scale it, and few can show the profit. Surveys cannot say why any one pilot stalls. They do tell you where to look, which is the organization around the model.
Why pilots stall
A pilot is built to prove that something can work. Production has to prove that it keeps working, at a cost the business accepts, under the control of someone who answers for it. Those are different tests, and most pilots are designed for the first one only.
Four patterns commonly explain a stalled pilot. The pilot team is a temporary group with no budget line after the demo. The test set was chosen by the people who built the system. The data access that worked for a small trial, often a manual export, has no production equivalent. And the cost per task was never measured at real volume.
None of these is a technical failure. Each is a missing decision. The six decisions below turn them into explicit choices that someone makes before the pilot ends.
Decision 1: Name one owner with authority over the outcome
Every production AI system needs one person who answers for its business result and can change the process around it. That is not the data science lead, who owns the model, and not the sponsor, who owns the idea. It is the leader of the work the system touches: the head of claims, the controller, the service desk manager.
NIST's AI Risk Management Framework 1.0, published in January 2023, makes the same point in its Govern function. Subcategory GOVERN 2.1 asks that roles, responsibilities and lines of communication for mapping, measuring and managing AI risks are documented and clear. GOVERN 2.3 asks that executive leadership takes responsibility for decisions about risks from AI development and deployment.
Put the owner's name in the pilot charter on day one, with three things attached: the metric they will answer for, the budget they control after launch, and the right to stop the system. A pilot without that owner can finish successfully and still have nowhere to go.
Decision 2: Evaluate on live work before you scale
A demo that works on ten hand-picked examples says little about month three. NIST's Generative AI Profile, AI 600-1, published in July 2024, suggests under MEASURE 2.5 that organizations avoid extrapolating performance from narrow, non-systematic and anecdotal assessments. Under MEASURE 4.2 it suggests evaluating performance in real-world scenarios to reveal issues that do not surface in controlled testing.
In practice, build the evaluation set from real historical cases, including the awkward ones, and have someone other than the builders label them. Agree the pass threshold before you see the results. Include the cases where the right answer is to refuse or hand off to a person.
The 2026 AI Index shows why this matters for agents. It reports that agents rose from 12 percent to about 66 percent task success on the OSWorld benchmark, and that they still fail roughly 1 in 3 attempts on structured benchmarks. A system that fails one task in three may be acceptable for drafting and unacceptable for payments. Only your own evaluation set, weighted by the cost of each kind of error, can tell you which one you have.
Decision 3: Settle data access before the pilot ends
Pilots often run on data someone exported by hand. Production needs a governed, repeatable feed with a named data owner, an access policy, and a plan for what happens when the source schema changes.
Ask four questions about every data source the system touches. Who owns it, and have they agreed to this use? Does the system see only what the requesting user may see? How fresh must it be, and what happens when it is stale? Who is told when the format changes?
If the answer to any of these is "we will sort it out at rollout," the pilot has not ended. It has been postponed. Treat data access as a gate with the same standing as the evaluation threshold.
Decision 4: Price the run, not just the build
A pilot's bill is small because its volume is small. Production cost depends on volume, prompt length, how many model calls each task makes, and how often people retry. McKinsey's 2026 survey reports that about one in five respondents say AI-related operating costs, including token costs, constrained their organization's use of AI.
Anthropic's documentation on defining success criteria lists price alongside task fidelity, consistency and latency as criteria to set, and asks what your budget is for running the model. Treat it the same way. Before launch, measure the average and the 95th percentile cost per completed task on the evaluation set, then multiply by expected monthly volume.
Compare that figure with the value of the task done by the current process. If the margin is thin, decide now whether to use a smaller model for easy cases, cache repeated context, shorten prompts, or limit the scope.
Cost also needs a ceiling. The OWASP Top 10 for LLM Applications 2025 lists unbounded consumption as risk LLM10, and its mitigations include rate limits, user quotas, timeouts and monitoring of resource use. Those controls protect your budget from abuse and from your own runaway loops.
Decision 5: Plan the change, not just the launch
A system that works but is not used has failed. The people whose work changes need to know what the system does, what it does not do, how to correct it, and what their role is when it is wrong.
NIST AI RMF MANAGE 4.1 asks for post-deployment monitoring plans that include mechanisms for capturing input from users, appeal and override, incident response, and change management. AI 600-1 adds, in MS-4.2-004, that organizations monitor and document instances where human operators override the system's decisions, and evaluate them to understand whether the overrides are justified.
Treat overrides as data. A high override rate in one team may mean the system is wrong for that work, or that the team has not been trained. Either way, you only learn it if you recorded it. Set up a short feedback route, a named person who reads it weekly, and a rule for how corrections reach the evaluation set.
Redesign the workflow, not just the tool. Placing a model inside an unchanged process can move the bottleneck rather than remove it, so decide which steps, approvals and handoffs change on launch day.
Decision 6: Write the kill criteria before you launch
Teams rarely stop an AI project, because stopping feels like admitting failure and the sunk cost is visible. Kill criteria remove that pressure by making the decision in advance, when nobody is attached to the result.
NIST AI RMF MANAGE 1.1 asks for a determination of whether the system achieves its intended purposes and stated objectives and whether development or deployment should proceed. GOVERN 1.7 asks for processes to decommission and phase out AI systems safely. Write both into the charter.
Good kill criteria are numeric and dated. For example: if after 90 days the system has not reached its agreed accuracy threshold on live cases, or the cost per task exceeds the agreed ceiling, or the override rate stays above the agreed limit, the owner either stops the system or presents a revised case to the sponsor. Consider a team running a document-review assistant. If it agrees these three triggers at the start, the review at day 90 is a short meeting and not a debate.
Also decide what stopping means: who switches the system off, how the work reverts, and how the data and logs are kept. Keep an inventory of AI systems, as GOVERN 1.6 asks, so that retired systems do not linger in the stack.
Where to start
You do not need a new program. You need one page per pilot that answers the six decisions, signed by the owner. Start with that page, because it shows quickly which pilots have a path and which are demos.
- Name the business owner and list the metric, the budget and the stop authority attached to the role.
- Build an evaluation set from real cases, have people outside the build team label it, and fix the threshold in writing.
- List every data source, its owner, its access rule and its refresh pattern. Mark any that rely on manual exports.
- Measure cost per completed task on the evaluation set and compare it with the cost of the current process.
- Name the people whose work changes, the override and feedback route, and the person who reads it.
- Write three numeric kill criteria with a review date, and decide how the system is switched off.
Run this page for every pilot in your portfolio. Fund the ones that can fill it in, and stop the rest without waiting for the next budget cycle.
References
- McKinsey & Company: The state of AI in 2026, On the road to ROI
- Stanford HAI: The 2026 AI Index Report
- NIST: Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1
- NIST: AI RMF Generative AI Profile, NIST AI 600-1
- OWASP: LLM10:2025 Unbounded Consumption
- Anthropic: Define success criteria and build evaluations





