The AI pilot that actually changed something

The question

Most AI pilots inside large organizations follow the same arc. A tool is licensed, a group of enthusiastic people try it, somebody writes a summary, and nothing downstream changes. The pilot was real. The adoption was not.

The interesting question is not which tool to pick. It is what separates a pilot that changes how work is done from one that produces a slide.

What is actually going on

Two things usually go wrong, and neither is technical.

The wrong thing gets measured. Enthusiasm is easy to collect and tells you almost nothing. People are polite in surveys and generous about tools they have used twice. An adoption percentage is worse, because logging in is not using.

Counting hours returned is harder and much more useful. It forces a specific claim: this task took this long, it now takes this long, here is who does it and here is what they did with the time. That claim can be wrong, which is exactly why it is worth making.

The learning has nowhere to go. In a pilot, the useful knowledge is not "the tool works". It is the accumulated set of small findings about where it fails, which prompts are worth keeping, and which tasks look suitable but are not. That knowledge lives in individuals unless something collects it.

A community of practice is the cheapest structure that does this. A standing group across functions, comparing what they tried, including what failed. It looks like overhead rather than delivery, which is why it is usually the first thing cut.

A pilot that cannot say what it gave back is a demonstration, not a pilot.

What this means for you

Before the next pilot starts, agree three things in writing.

What task. Not a department, a task. Named, with someone who currently does it.

What you will count. Hours on that task, measured by the people doing it, before and after. Accept that the baseline will be rough.

Who owns it afterwards. The single most common failure is that a pilot succeeds, the group disbands, and the next team starts from zero.

If the answer to any of these is "we will work that out as we go", you are running a demonstration. That may be fine, but call it what it is so nobody expects a business case at the end.

Where I can help

I led a cross-functional AI pilot and community of practice that produced 1,150 hours of measured productivity gain, and the following year the focus moved to scaling it across the workforce. The pilot itself was straightforward. The part worth paying attention to was the structure around it.

The work I do here is usually the adoption plan rather than the tooling decision: which tasks to start with, what to measure, how the community runs, and what has to change in how managers are held accountable so the gain does not evaporate.