AI pilot to production: why projects stall and how to scope the first one

What changes between a pilot and production

A pilot answers one question: can the model do this task well enough with our data. Production answers a different one: can the business rely on it every day. The gap between the two is engineering and organisation, not prompt writing.

Five reasons pilots stall

1. The use case was chosen for novelty. A chatbot nobody asked for gets polite feedback and no daily use. Projects that survive start from a task people already find slow or tedious.
2. Integration was postponed. The pilot ran on copied data. Connecting to the real CRM, ERP or document store turns out to be the largest piece of work, and it was never budgeted.
3. Nobody defined good enough. Without a pass mark, every review turns into a debate about single examples.
4. Security and legal joined late. Questions about data processing, hosting region and retention arrive after the build and reopen design decisions.
5. No owner after launch. The partner leaves, a model version changes, quality drifts and no one is responsible for noticing.

How to scope the first project

Scope the first project as a small version of production, not as a demo. That means fewer features and more of the dull parts.

1. Pick one process with clear volume, such as answering supplier queries or drafting quotes.
2. Name the two or three systems it touches and confirm you can get access to them.
3. Decide where a person stays in the loop and what they approve.
4. Collect 50 to 200 real examples with correct answers to use as a test set.
5. Set one target number and one budget ceiling.
6. Bring in the person responsible for data protection in the first week.
7. Name the owner who will run the system after launch.

Keep a person in the loop at the start

The fastest route to production is often a tool that drafts and a person who approves. Drafted replies, suggested classifications and pre-filled forms create value on day one and carry little risk, because a mistake is caught before it leaves the building.

Every approval or correction is also a data point. After a few weeks you know how often the system is right, and for which cases. That record is the evidence you need to automate the safe cases fully and keep review for the rest.

When AI is the wrong tool

Some stalled pilots should never have started. If the rules of a task can be written down exactly, ordinary automation is cheaper, faster and easier to audit than a language model. If the input data is missing or unreliable, a model will not repair it. If a task happens ten times a month, the build may cost more than it ever saves.

A good partner raises these points during the assessment. Boutique studios such as BlackCard make that advice part of the first conversation, and larger consultancies usually cover it in a paid discovery phase. Either way, ask for the list of ideas that were rejected along with the one that was chosen.

Plan the run phase before launch

Production systems need upkeep. Models are updated or retired by their vendors, source systems change, and users find new edge cases. Agree before launch who watches quality scores, who updates prompts and test sets, how quickly faults are fixed and what the monthly budget is. Rerun the test set after every model or prompt change. A project with a funded run phase is far more likely to still be in use a year later than one handed over and forgotten.