Skip to content

Why AI Pilots Never End, and How to Set Success Criteria on Day One

Why AI Pilots Never End, and How to Set Success Criteria on Day One
Published

05 Oct 2026

Author
Nikesh Maharjan

Nikesh Maharjan

Why AI Pilots Never End, and How to Set Success Criteria on Day One
5:28
 
Table of Contents
Most stalled AI pilots aren't stuck on the technology. They're stuck because nobody agreed what result would fund production, so engineers, data access and budget stay parked with no date to release them. This article gives you a one-page sheet to fill in on day one, so your next pilot ends in a clear yes or a clean stop.
 
lightbulb-filament

Why we wrote this

We wrote this because of one question we ask early in almost every AI conversation: what result would make your pilot a yes? The answer is usually a good description of what the model can do, and rarely a number, a date or a name.
 
That gap matters more now that off-the-shelf models make pilots quick to start and easy to leave running. This is for the person between the build team and the budget, who has to get both sides to agree on what finished means. If you're exploring purely to learn, we cover that case too.
 

Month seven of a three-month pilot

The steering meeting is on Friday, and the AI pilot is on the agenda again. It was scoped for twelve weeks. This is month seven.

The model scores well on the test set and the team likes it. The update slide says "promising early results", the same words it used three months ago. Nobody can say what result would make it a yes.
Nobody wants to be the person who calls it a no, either. So the pilot gets another month and a bigger sample. The same slide goes into next month's pack.
 

The short answer: A pilot ends when the decision is written down before it starts. On day one, agree what result means yes, what result means stop, the date you'll decide, who decides, and whether production money exists if it works. Without those five, a pilot measures progress forever and never produces a decision.

Why pilots drift instead of finishing

In July 2024, Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025. It pointed to poor data quality, inadequate risk controls, escalating costs or unclear business value. In June 2025 it predicted that over 40% of agentic AI projects will be cancelled by the end of 2027, citing much the same reasons.

Abandoned is at least a decision. The costlier outcome is the pilot that is neither stopped nor funded, quietly holding a team for months. An open-ended pilot is a decision someone has politely deferred.
Vendors rarely push for a stop date either. An extended pilot is extended work, and that includes vendors like us, which is why we write the stop line first.
 
RAND's 2024 study of why AI projects fail, built on 65 interviews with data scientists and engineers, found that misunderstanding a project's purpose causes more failures than any other factor. A pilot with no written yes is that misunderstanding, given a budget and a steering committee.
 

Fixing it has a cost. Firm stop criteria will occasionally end a pilot that one more month would have rescued. We think that's a fair trade, but it means an uncomfortable meeting on day one that most teams would rather skip.

Five reasons pilots drift

1. Success was never written down

The pilot started with a goal like "see if AI can help with claims". That's a direction, not a test, so every result reads as progress. 

  • What it looks like: updates that say "promising" or "encouraging", with no number attached.

  • What good looks like: a measurable yes, agreed before the build, that two people would score the same way.

2. Nobody owns the decision

Pilots are often run by the technical team and sponsored by someone senior who isn't in the room. The builders can't approve production, and the sponsor never agreed to decide.

  • What it looks like: the steering meeting ends with "let's keep going", because nobody present can say stop.

  • What good looks like: one named person who will fund and run the production version, with the decision date in their diary.

3. The pilot tests the model, not the job

A model can score well on a test set and still fail in the workflow. Production depends on edge cases, the systems it must connect to, and the time a person spends checking its work. 

4. Production was never costed

Pilots run in a sandbox on exported data. Production needs security review, data residency, monitoring and support, which usually cost more than the pilot itself.

The gap shows up in the numbers. In a Gartner survey of 644 respondents, on average only 48% of AI projects made it into production. Getting from prototype to production took eight months on average.
 

  • What it looks like: a successful pilot followed by a new business case and a six-month gap.

  • What good looks like: a production budget and owner agreed in principle before the pilot starts.

5. Stopping feels like failure

Nobody's career suffers from a pilot that's still running. Calling stop feels like admitting the money was wasted, so the safe move is always another month.

  • What it looks like: extensions justified by "we've come this far".

  • What good looks like: stop treated as a result, recorded with what was learnt, and announced as calmly as a yes.

The day-one decision sheet

Each drift cause has a matching fix, and all five fit on one page. Fill it in before the build starts, with the person who will make the call. 

  1. Write the yes criteria. Name the measurable result, taken in the real workflow, that means you'll fund production.
  2. Write the stop criteria. Name the result or blocker that ends the pilot. Include at least one condition that isn't about accuracy, such as data residency or checking time.
  3. Set the decide-by date. Use a calendar date, not "end of the pilot". Agree now what a borderline result triggers.
  4. Name who decides. One person, who will fund and own the production version. Everyone else advises.
  5. Agree the production budget question. Confirm that a production budget and owner exist in principle, so a yes turns into a build.

Yes if

Stop if

Decide by

Decided by

Production budget agreed?

The measured result, in the real workflow, that funds production

The result or blocker that ends it, including one non-accuracy condition

A calendar date, plus the rule for borderline results

One named person who will own production

Yes or no, with the budget owner named

When an open-ended pilot is fine

Not every experiment needs a stop line. If your team is exploring what a language model can do with your documents, purely to build understanding, there may be no decision to make yet.

That's legitimate work. Call it research, give it a fixed time and a learning budget, and don't report it as a pilot.
 

The trouble starts when research gets a steering committee and a slide titled "pilot". If no decision depends on the work, the sheet above won't help.

A filled-in example

Here's an illustrative sheet for a mid-market insurer piloting AI triage of motor claims. The AI reads each new claim and suggests a category, which an assessor accepts or changes.
 

Yes if

Stop if

Decide by

Decided by

Production budget agreed?

85% of 300 real claims accepted unchanged, and checking takes under half the manual time

Under 70% after two tuning rounds, or claim data must leave approved Australian hosting

Friday of week 8. A result of 70% to 85% earns one four-week extension, then a final call

Head of Claims. The CTO signs off security

Yes, in principle. Owner named in week 1, sized from the integrations

Look at what the yes line measures. It isn't accuracy on a test set. It's whether assessors trust the output in their real queue, and whether checking beats doing.

The stop line includes a data condition, not just a performance one. In insurance, residency is a production blocker, so finding it in week three is cheap and finding it in month nine isn't.
 

The last column does the most work. Because a production budget was agreed before the build, a yes in week 8 starts a build, not a new business case.

What drives the effort

This isn't free, and the pilot is rarely the expensive part. Four things move the effort most: 

robot-1

Pilot length

Every extra month adds team time.
icons8-user-groups-1-1

Number of stakeholders

Each person who must agree adds a review cycle, so keep the decision group small.
icons-learning-1

Data access and sensitivity

Health, financial and personal data need approvals and controlled hosting before real records can be used.
icons-complex-workflow-2

Whether production is scoped up front

Integrations, monitoring and review effort set the production cost. Scoping them in week one takes days; discovering them after a yes takes months.
 

How we approach it

icons-salesforce

NEXUS Foundation

The pilot is one fixed-scope 8-week proof of concept. The yes and stop criteria, decision date and decision owner are agreed in week one, before any build starts.

The eight weeks produce a Working AI Solution tested on your data. They also produce a Stakeholder Investment Case, which sets the result against the criteria for the person funding production.

The Production Roadmap covers what it takes to run the solution for real: integrations, controls, review effort and sequencing. A stop result gets the same rigour, so you know why it stopped and what you learnt.
 

Check your own pilot

Answer these three questions about the pilot you're running or planning:

  1. Can you name, in one sentence, the result that means yes?
  2. Can you name the result that means stop?
  3. Is there a calendar date and one named person for the decision?

 

If any answer is no, fill in the sheet before the next steering meeting.

If any of this sounds familiar, just reply. I'll read it myself.

 

Frequently Asked Questions

What are good success criteria for an AI pilot?

Good criteria name a measurable result taken in the real workflow, such as how often staff accept the output unchanged, rather than accuracy on a test set. They also include a stop condition, a decision date and one named decision owner. Two people should be able to score the result and reach the same answer.

What is AI pilot purgatory?

AI pilot purgatory is when a pilot is neither stopped nor funded for production, so it keeps running on extensions. It usually happens because nobody agreed what result means yes, or nobody in the room can approve the next step. The cost is a team held for months without a decision.

Is an AI pilot the same as a proof of concept?

They overlap, and many teams use the terms interchangeably. A proof of concept usually tests whether an idea can work at all, while a pilot tests it with real staff, real data and the real workflow. Both need written yes and stop criteria, or both drift the same way.

How long should an AI pilot run?

Long enough to test the job on real data, and no longer. For one well-scoped use case, around eight weeks is usually enough to reach a decision. If it isn't, the scope is often too wide, not the timeline too short.

What should we do with borderline results?

Treat borderline as its own outcome, defined on day one. Allow one extension with a narrower question, a new date and the same decision owner. If the second round is still borderline, stop and record what you learnt.

Who should own the go or stop decision?

The person who will fund and run the production version, usually the business owner of the process. The technical lead advises on feasibility and risk. If the decision owner can't approve a production budget, the pilot is reporting to the wrong person.

Why do AI projects fail after a promising pilot?

Usually because the pilot proved the model, not the path to production. Security review, data residency, integrations, monitoring and support were never costed, so a yes turns into a new business case and a long wait. Agreeing a production budget and owner in principle before the pilot starts closes that gap.

What drives the cost of taking an AI pilot into production?

The main drivers are how many systems it must connect to, how sensitive the data is, where that data must be hosted and how much human review the output needs. Monitoring, support and security review add to that. Together these often cost more than the pilot, so scope them before the build starts.

Can we add stop criteria to a pilot that's already running?

Yes, and the next steering meeting is a good time to do it. Write down what result would now mean yes and what would mean stop, then set a calendar date and one decision owner. If nobody can agree what yes looks like, pause the build and redefine the question first.
 
 
Topics:

Want a second opinion before you decide?

Book a free 30-minute Fit Call. Whether it’s an AI idea, a new build or a system that’s slowing down, we’ll tell you what’s worth doing, what we’d drop, and whether we’re the right fit at all. No commitment. No pitch deck.
Book a Free Fit Call