The project starts long before you choose a vendor.

“Understanding AI” series, note 6 of 12. The first five explained the technology. From here on the series is about your company: how it is integrated, what it costs, why it fails, what the law requires and what the people in charge need to know.

Friday, noon, a private clinic with three sites in the canton of Zurich. In January the head of IT bought licences for an AI assistant for the sixty people in administration. There was a one-hour training session. It is now October: four people use it, two of them to draft emails in French. The chief executive concludes at the management meeting that “AI has not worked here” and shelves next year’s budget. Meanwhile, at reception, an administrative assistant pastes patient reports every day into a free account with another assistant, from her phone, to summarise them faster.

The scene is imaginary. Both halves happen in real companies, and often in the same one. The first half is what happens when the tool is bought before the process is chosen. The second is what happens when nothing is chosen: people choose on their own. This note is the order I use so that neither happens. There are nine steps. The first three cost no money, and they are the ones almost everyone skips.

AI is not implemented. A process is implemented, and AI is one of its parts. Where there is no process, there are unused licences.

Step 1. A process, not a tool.

Everything starts with the map from note 4. From the forty-two proposals one is chosen, and it is chosen with three criteria: it repeats often, the error can be detected and the result can be measured in hours or in money. Supplier invoices meet all three. “Improving internal communication” meets none. A process with a name, with a manager who suffers it and with a number that is bad today. That is a project. Everything else is a licence.

There is one figure worth having in front of you when choosing. In August 2025 a report by MIT’s NANDA project, based on 150 interviews with executives, 350 employee surveys and an analysis of 300 public deployments, concluded that 95% of generative AI pilots in companies had produced no measurable impact on the profit and loss account. The same report observed two things that explain part of that number: budgets went mostly to sales and marketing, while the clearest returns appeared in the back office, in administration and operations; and tools bought from specialised vendors reached production roughly twice as often as those built internally from scratch, around two in three against one in three. These are figures from a report, not from a controlled experiment, but they match what I see: the boring, well-delimited process is the one that ends up working.

Step 2. Look at your data before the model does.

The model is going to read your documents, and it is going to answer from them. If the documents are old, contradictory or in seven different formats, the system will be fast and wrong. In September 2017 the Harvard Business Review published a study in which 75 managers from different companies reviewed a sample of the data they worked with every day: only 3% of the data sets met basic quality standards, and almost half of the new records had at least one critical error. That was before generative AI. Generative AI does not fix it: it amplifies it, because it writes with the same fluency from a good fact as from a bad one.

The work in this step is modest and decisive. Gather the documents the process needs, decide which is the right version of each, give them a date and an owner and withdraw the rest. For invoices: the clean supplier list, the current payment terms, the open orders. This is the archive I spoke about in note 5. Without it, the archivist brings up to the office whatever it finds.

Step 3. The fifty gold cases.

Before seeing a single demo, fifty real cases from the process are gathered, with the correct answer next to each: fifty invoices with how they were booked, fifty emails with how they were classified and what was replied, fifty calls with their approved summary. The people who do the work choose them, and they deliberately include the odd cases: the invoice in Italian, the email that mixes two orders, the customer who writes without punctuation.

This set is the most valuable piece of the project, and it will still be when the project ends. It serves to compare two vendors with the same yardstick, to decide whether the system is ready, to check that it has not got worse when the model version changes and to argue with numbers instead of impressions. If a company did only one thing from this note, it should be this. It costs nothing but one afternoon of the right people.

Step 4. The simplest thing that solves the case.

With the process chosen, the documents clean and the fifty cases in hand, you decide what to build or buy, and the rule from note 5 applies: an augmented model, with retrieval and one or two permissions, in a workflow of few steps. No agents until a workflow has proved it is not enough. It is built in weeks, not months, and tested against the fifty cases before anyone is shown it. The first number management should see is not a demo: it is “it gets 46 out of 50 right, and these are the four it misses”.

Step 5. Shadow mode.

Here is the step that saves the most projects. For two to four weeks, the system works alongside the people but does not act: it classifies, extracts, proposes, and everything is logged next to what the people actually did. Nothing goes out, nothing is booked, no customer receives anything. At the end the two columns are compared, case by case.

Shadow mode teaches three things no demo teaches: how the system fails with the real flow, how many cases in every hundred need a person and whether the team trusts the proposals or corrects them as a matter of course. And it resolves, along the way, the objection of the head of quality from note 4: “and who checks what it does?”. During the shadow, everyone.

Step 6. Switch on with review in proportion to risk.

When going into production, not everything is switched on. It is switched on in tiers, according to the risk of error. For invoices: those that match the order and are below a certain amount are booked with a sample review; those that do not match go to a person with the system’s proposal in front of them; those from a new supplier always go to a person. The rule is the one from note 2: checking is proportional to what a mistake costs, and where it costs, someone with a name signs.

Step 7. Measure every week what was measured on day one.

The number that was bad in step 1 is now measured every week: hours, days to collect, errors, cases passed to people. And the fifty gold cases are run again every time something changes: the model provider releases a new version, a document is added, a step is touched. Providers retire versions of their models fairly regularly; Anthropic, for example, documents a lifecycle for its models and commits to giving at least sixty days’ notice before retiring one. Sixty days is plenty of time to run the fifty cases again. If nobody runs them, the system changes its behaviour one night and nobody knows until a customer says so.

Step 8. Train the people who use it, because it is also compulsory.

The clinic’s one-hour session was not training: it was a presentation. Training means teaching each person what the system does, what it does not do, how it fails and what they must check before accepting a proposal. It is the centaur and the cyborg from note 4, brought down to the specific job. And in the European Union, since 2 February 2025, it is also a legal obligation: Article 4 of the Artificial Intelligence Act requires those who use AI systems in their activity to ensure their staff have a sufficient level of AI literacy, taking into account their training and the context in which the systems will be used. It sets no number of hours and no certificate. It sets a responsibility. I develop this in the note on law and data.

Step 9. Give a channel to what already happens.

Back to the assistant at reception. Hers is not an isolated breach; it is the norm. The Microsoft and LinkedIn survey of May 2024 that I cited in note 4 found that 78% of the people using AI at work did so with their own tools, and that 52% were reluctant to admit it for fear of seeming replaceable. The answer is not to ban it, which only hides the use. It is to provide a company tool, with the data protected, a two-line rule on what is never pasted and a place to propose new uses. The best proposals for the second process usually come from there.

Buy or build.

The question appears in step 4 and deserves its own paragraph. Buying a tool made for a specific use, invoices, customer service, transcription, is usually faster and, according to the MIT report I cited, reaches production more often. Building to measure makes sense when the process is the heart of what makes you different, when there is no tool for your case or when the data cannot leave. In practice almost everything is a mix: the model is bought from a large provider, a ready-made tool is used for the standard part and the connection to your systems and your rules is built to measure. What does not pay is building what already exists or buying what does not fit. And in either case, the fifty gold cases are yours.

A study sponsored by Microsoft and carried out by IDC in November 2024 among more than 4,000 executives gave friendly figures: companies took under eight months to deploy and around thirteen to see value, with a declared return of $3.70 for every dollar. I cite it because it is cited a great deal, and with a warning: a vendor commissioned it and it measures what executives say, not what the accounts show. Put it next to MIT’s 95% and draw your own conclusion: the return exists, and it does not come from buying, but from doing the nine steps.

1. This week

   Choose a process with the three criteria

   Repeated, with a detectable error and with a number that is bad today. Write the number down.

2. This week

   Gather the documents and decide which version is the right one

   Date and owner on each. Withdraw the rest.

3. One afternoon

   Assemble the fifty gold cases

   With the people who do the work, including the odd cases. Keep them as the asset they are.

4. Before any demo

   Insist on seeing the system against your fifty cases

   The number of hits and the misses one by one. It is the only demo that counts.

5. Two to four weeks

   Shadow mode

   The system proposes, the people do, the two columns are compared. Nothing goes out.

6. At switch-on

   Tiers by risk and a weekly number

   The safe with sampling, the doubtful with a person, the new always with a person. And the fifty cases again with every change.

How I approach it.

My Automation Assessment is, in essence, steps 1 to 3 done with the client in a week: the chosen process with its numbers, the inventory of documents and the fifty cases. With that, the report says what to build, what to buy and what it costs, with fixed prices. After that, shadow mode always, and nothing is switched on without tiered review. It is not excessive caution. It is the difference between the 95% and the other 5%.

A question for the next leadership meeting, before anyone mentions suppliers: who in your business could write the fifty gold cases for the process you want to automate today? Not who should. Who could, with what they know and the time they have.

If no name comes to mind, the project has not started yet, even if it already has a budget. And if one does, ask yourself whether that person knows the project depends on them.

Next note: what AI really costs, item by item, and where the return comes from. With the public prices of the models, the price nobody puts in the proposal and the sum you should do before signing.