AI POC → Production Playbook

Define a decision, not a demo

Choose one workflow and a named owner. Describe the current task, the people doing it, and the cost of doing nothing. State which decision the pilot will inform: continue, change scope, or stop.

Measure a baseline before adding AI. Track task completion, reviewer effort, time and errors on representative examples. A fluent answer is not evidence that a workflow improved.

Workflow:
Owner:
Current baseline:
Success threshold:
Failure that must never occur:
Pilot duration:
Decision date:

Build the smallest evaluable system

Start with real examples you are permitted to use, including ambiguous inputs, missing context and requests outside scope. Keep an evaluation set separate from the examples used to tune the system.

Compare model output to a simple baseline. Record the model and prompt version, input, output, latency and reviewer judgment. Test failure handling as part of the workflow: timeouts, tool failures and low-confidence answers should have a clear path back to a person.

• Assign ownership for input quality and evaluation.
• Check permission before connecting a data source.
• Use least-privilege tools; confirm consequential actions.
• Evaluate ordinary cases and costly failure cases.
• Measure the complete workflow, including review effort.

Ship a reversible pilot

Roll out to a small group with clear feedback channels. Explain what the system can do, when a person should review it and how to report a mistake. Set a usage budget and an escalation owner.

Define rollback before launch. If quality, cost or completion rates move outside your agreed limits, return to the previous workflow. Expand only after the pilot produces repeatable evidence, rather than a single successful demo.

Launch owner:
Pilot users:
Quality and cost limits:
Monitoring cadence:
Escalation channel:
Rollback procedure:
Evidence required to expand: