Your AI Portfolio Has Twenty Pilots. None of Them Has a Date to Be Switched Off.

Ask the head of data at a mid-sized Australian insurer how many AI pilots are running and you will usually get a pause, then a number, then a correction. Somewhere between fifteen and thirty is common. There is a claims triage assistant, a broker email summariser, two competing document extraction tools, a fraud scoring experiment, a chatbot for the intranet, and a handful of things that started as a hackathon project and never quite stopped.

Ask a second question, which of these will be switched off and when, and the room goes quiet. Every pilot has a start date. Almost none of them has an end date, a success threshold, or a named person who is allowed to say stop.

That is not a technology problem. It is a portfolio problem, and it is quietly eating AI budgets across the country.

Why pilots never die

Traditional IT projects end because the money runs out or the system goes live. AI pilots are different. They are cheap to start, they run on a credit card or a shared cloud account, and they often produce something that looks impressive in a demo. A pilot that is sort of working is the hardest thing in the world to kill, because nobody wants to be the person who switched off the clever AI thing.

So pilots drift into a grey zone. They are not in production, so they do not get production controls. They are not finished, so they still consume people, tokens and attention. They keep a small group of enthusiastic users who would complain loudly if they disappeared, which makes them look like they have demand. Twelve months later the organisation has a long tail of half systems, each with its own data access, its own prompts, its own vendor contract and its own risk that nobody has assessed properly.

The costs show up in three places:

• Money. Model usage, vector database hosting, licences and the contractor who built the thing all keep billing. Individually small, together often a meaningful slice of the AI budget.
• People. Your best engineers get stuck maintaining pilots instead of hardening the two or three use cases that actually matter.
• Risk. Every lingering pilot holds credentials, copies of data and integrations. Under the Privacy Act and the reforms coming into effect over the next year, each one is something you need to be able to explain. A pilot nobody owns is a pilot nobody can explain.

We have written before about what happens when a pilot succeeds and the operating model does not notice. The opposite problem is just as common and far less discussed: pilots that never clearly succeed or fail, and so never end.

The missing piece is a kill criterion

The fix is not complicated, but it has to be decided before the pilot starts, not after. Every AI pilot should launch with a one page charter that answers four questions.

1. What does success look like, in numbers? Not "improve efficiency". Something like "reduce average claims triage time from 22 minutes to under 12 minutes with no increase in rework, measured on at least 500 real claims".
2. What would make us stop? A threshold on the other side. If accuracy on the evaluation set is below 85 percent after six weeks, or if reviewers are overriding more than one in four outputs, the pilot ends.
3. When do we decide? A fixed review date, usually eight to twelve weeks out. Not "when it is ready".
4. Who decides? One named person with the authority to scale it, extend it once, or shut it down. Not a committee, and not the team that built it.

That fourth point matters more than people expect. Builders are always optimistic about their own pilot. They can see the next improvement, the next prompt tweak, the next model release that will fix it. That optimism is useful during the build and terrible during the decision.

Three outcomes, not two

Most organisations treat pilot reviews as pass or fail. In practice there are three useful outcomes, and naming them makes the conversation far less political.

• Scale. The pilot met its success criteria. It moves into a funded delivery stream with production controls, proper monitoring, a support model and a business owner.
• Extend once. The results are promising but the evidence is thin. The pilot gets one more fixed period with a sharper hypothesis. One extension only. A second extension is a decision to stop disguised as patience.
• Stop and harvest. The pilot ends. Credentials are revoked, data copies are deleted, contracts are cancelled, and the team writes down what it learned. The reusable parts, such as evaluation sets, prompts, integration code and data cleaning work, go into a shared library.

"Stop and harvest" is the outcome that changes the culture. When stopping a pilot produces something useful and is treated as a normal result, people stop fighting to keep zombie projects alive.

What this looks like in practice

Consider a hypothetical state government agency running eighteen AI pilots across service delivery, compliance and corporate functions. A portfolio review using simple charters might land somewhere like this: three pilots scale, four get one extension with tighter goals, and eleven are stopped. Of the eleven, several turn out to be solving the same problem with different tools, which nobody had noticed because they sat in different branches.

The money freed up from the stopped pilots funds proper production engineering for the three that scaled. The evaluation sets built by the stopped document extraction pilots become the test bed for the one that survived. The agency's privacy team, which had been nervous about how many copies of citizen data were floating around in experiments, now has a much shorter list to worry about.

None of that requires new technology. It requires someone to write down, before the work starts, what "not good enough" means.

Building kill criteria into your AI roadmap

If you already have a long tail of pilots, start with a one off amnesty. List every AI pilot, experiment and proof of concept, including the ones running on someone's personal account. For each, record the owner, the monthly cost, the data it touches and the last time anyone measured whether it works. That list alone is usually confronting enough to drive action.

Then put the structure in place so the tail does not grow back:

• No charter, no compute. New pilots get cloud accounts, API keys and data access only after the one page charter is signed.
• Time boxed access. Pilot credentials and data access expire on the review date by default. Renewing them requires a decision, not just inertia.
• A quarterly portfolio review. One meeting where every active pilot is shown on a single page with its criteria and its current numbers. The goal is to make stopping as visible and routine as starting.
• A harvest checklist. A short, standard list for shutting a pilot down cleanly and capturing what is reusable.
• Report the kills. Tell the executive team and the board how many pilots were stopped and what was learned, not just how many were launched. A portfolio with no stopped pilots is not a sign of success. It is a sign nobody is deciding.

How TruFyre helps

At TruFyre we help Australian organisations turn a scattered collection of AI experiments into a managed portfolio. That usually starts with a rapid inventory of what is actually running, followed by charters and kill criteria for each active pilot, and a lightweight governance rhythm that fits alongside existing investment and risk processes. We then help the use cases that earn their place move into production with the architecture, evaluation and controls they need.

If your AI roadmap lists everything you have started and nothing you have stopped, talk to us. The fastest way to get more value from AI is often to do less of it, on purpose.