Your Model Is Ready. Your Release Path Is Still a Ritual

The model card looks fine. Staging accuracy is up. The pilot users said the answers felt useful. Then someone asks the only question that matters: how do we put this in production without a war room every time the weights change?

That is when the room goes quiet. There is a notebook, a shared drive of evaluation spreadsheets, and a release process that lives in someone's head. You do not have a model problem. You have a release path problem.

This piece is about treating GenAI and classical ML the same way you treat any other change that can hurt customers: versioned artefacts, a promotion path, monitoring that someone actually watches, and a rollback that is boring on purpose.

What "ready" usually means (and why it is not enough)

Teams say a model is production ready when it clears a workshop bar. That bar is usually:

• A held out set where the numbers look respectable
• A few happy path prompts or cases demonstrated live
• Sign off from a sponsor who liked the demo

None of that answers how you ship the next version on a Tuesday afternoon when half the team is on leave. None of it says what happens when retrieval quality drifts, a prompt change lands, or a vendor model version flips underneath you.

Demo readiness is a product conversation. Release readiness is an operations conversation. Confusing the two is how you get "we will just push it and watch Slack".

The ritual release

If your promotion path looks like this, you are still in ritual mode:

1. Someone exports weights, a prompt file, or a config from a laptop
2. Someone else copies it into a bucket or a secret store by hand
3. A third person restarts a service and pings the channel
4. If it breaks, the same three people reconstruct what changed from chat history

Rituals scale with heroics. They do not scale with change volume. The more often you tune prompts, refresh corpora, or swap embedding models, the more the ritual becomes the bottleneck and the incident.

Australian enterprises feel this when AI work sits across data science, platform, and a business owner who only sees outcomes. Without a shared release contract, each group assumes the others own the hard parts. Nobody owns the rollback.

What a thin MLOps path actually needs

You do not need a full ML platform programme to stop shipping on hope. You need a thin path that makes promotion and rollback mechanical.

1. Versioned artefacts. Model weights or hosted model id, prompt template, retrieval config, feature or corpus snapshot id, and evaluation report. Treat them as a single release bundle with one version number.
2. A promotion gate. Staging runs the same golden set (or shadow traffic sample) you trust. Fail the gate and the bundle does not move. Pass and a known identity promotes it. No anonymous uploads.
3. Environment parity that matters. Same identity model, same tool allowlist shape, same logging fields. You do not need identical hardware. You need identical failure modes you can reason about.
4. Live signals with owners. Latency, error rate, cost per task, groundedness or task success on a sample, and a human readable alert that goes to a named rota, not a dead channel.
5. A boring rollback. One action restores the previous bundle. Practise it. If rollback needs a war room, you do not have rollback. You have hope with a runbook.

This is the same discipline as the lean architecture argument in When Your AI Architecture Becomes a Liability: only keep the components that serve a real production caller. Here the caller is "ship the next safe version without theatre".

Prompts and RAG are deployable too

GenAI teams often version the model and forget the rest. That is how you get silent regressions.

• Prompts are code. Change them through the same review and promote path as application code.
• Retrieval configs (chunker, filters, top k, corpus snapshot) belong in the bundle. If production retrieval is still guessing, fix measurement before you automate promotion. We covered that gap in Your RAG Demo Worked. Your Production Retrieval Is Still Guessing.
• Tool allowlists for agents are part of the release. Expanding tools mid flight without a gate is how blank cheque access returns. See Your Agents Need an Allowlist, Not a Blank Cheque.

If you cannot say which prompt and which corpus produced yesterday's answers, you cannot investigate, and you cannot roll back cleanly.

Security is part of the path, not a later review

DevSecOps for AI is mostly ordinary release security with a few sharp edges:

• Secrets never live in notebooks that get promoted
• Training or fine tuning data is classified the same way as any other sensitive store
• Inference identities are least privilege, rotatable, and attributable (the same principal thinking you need for agents)
• Supply chain: which base model, which package versions, which prompt file hash landed in prod

Shadow experiments will keep happening. Your approved lane has to be the faster way to ship a safe change, or people will bypass it. That is the same dynamic as bringing shadow AI under control: convenience beats policy when the official path is slow.

A ninety day path to a boring release

Days 1 to 14. Pick one production (or near production) model path. Write down the current ritual in one page: who touches what, where artefacts live, how you would undo a bad change today. Name an owner for the release path, not only for the model.

Days 15 to 45. Bundle artefacts under one version. Put the golden set (or a fixed eval job) on a schedule. Block promotion when it fails. Log bundle id on every production request. Draft the rollback steps and run them once on purpose in staging.

Days 46 to 90. Add the minimum live monitors with a rota. Practise a production rollback during a quiet window. Only then automate more of the promote step. Do not buy a platform until two use cases share the same painful release gap.

By day 90 you should answer four questions without archaeology: which bundle is live, who promoted it, what gate it passed, and how long rollback takes when nobody heroic is in the room.

Frequently asked questions

Is this only for teams training their own models? No. Hosted model ids, prompt versions, and retrieval configs still need a release path. Vendor models change underneath you. Your bundle should record what you pinned and how you validate after a provider update.

Do we need Kubeflow or a full ML platform first? Not for the first path. Start with object storage, CI, your existing identity and logging, and a promotion checklist that is enforced. Add platform pieces when two teams share the same bottleneck.

What if evaluation is expensive? Run a small hard golden set on every change and a larger sample on a schedule. Perfect coverage is not the bar. A gate you actually run beats a perfect suite nobody triggers.

How is this different from normal DevOps? Same shape, different artefacts. Models, prompts, and corpora drift in ways static binaries do not. Your gates and monitors have to include quality and cost, not only uptime.