What nobody puts in the presentation.

“Understanding AI” series, note 11 of 12. The first ten explained how the technology works, how it is integrated, what it costs, why it fails, what the law requires and how it is protected. This one gathers what is left out of all those conversations.

Saturday, 10 am, the living room of the chief operating officer of an engineering firm in Baden. The AI system that reads and classifies customer requests has been in production for eleven months. It worked. The first quarter’s figures were the ones in the proposal. Since then nobody has looked at them again, because there were no complaints. This morning, over a coffee, she opens the log out of curiosity and finds that, since one night in June, the system has been classifying as “general enquiry” a third of what it used to classify as “urgent”. Nobody changed anything. Or so they believe.

The scene is imaginary. Silent degradation is not, and it is the first of the ten blind spots in this note. I have gathered them because they are the things that, in meetings, are considered unimportant details, and because in projects they are almost always what decides the outcome after a year. Each one carries its source, the symptom by which it is recognised and what I do about it. I start with a text from 1983 that anyone about to automate anything should read.

Automating does not eliminate human work. It exchanges it for another kind, harder, less frequent and worse paid: watching.

1. The ironies of automation, 1983.

In 1983 the psychologist Lisanne Bainbridge published in the journal Automatica a six-page article titled “Ironies of Automation”, about industrial control rooms. She described two ironies that today read as if they were about AI. The first: whoever designs an automatic system usually regards the human operator as a source of errors, and so removes them; but leaves in their hands precisely the tasks that could not be automated, which are the hardest. The second: a person who only watches a system that almost always works loses practice, gets bored, stops paying attention, and it is precisely that out-of-practice person who has to step in when the system fails, at the worst moment and without context.

Translate that to the system in Baden. The team that used to read every request now only sees the 15% the system cannot classify: the odd cases, the ambiguous ones, the ones that require judgement. It is harder than before, not easier, and it is done with less practice, because the easy cases, the ones that taught, no longer pass through their hands. After a year, the new person on the team has never seen an easy case. Remedy: rotate, every week, someone from the team through a sample of the cases the system solves on its own, and treat review as a post with its own training, not as what is left over.

2. Silent degradation.

An AI system does not break: it drifts. The provider releases a new version of the model, a document in the archive changes, the kind of requests coming in shifts with the season, and the system keeps answering with the same confidence and a different quality. Nobody complains because classification errors do not produce complaints; they produce delays that are attributed to something else. Anthropic documents that it retires versions of its models with at least sixty days’ notice, and that new ones do not behave the same as old ones. Symptom: nobody can say when the fifty gold cases were last run. Remedy: run them every month and with every notice from the provider, and put the figure in the management’s monthly report, next to the sales figure.

3. The same question, two answers.

I told it in note 2 and I repeat it because it is the blind spot that saves the most demonstrations and sinks the most productions: a language model does not always answer the same input the same way, not even with the temperature at the minimum; Anthropic’s own documentation warns of it, and in its most recent models the temperature parameter is no longer even accepted. A process that needs the output to be exactly the same every time, a calculation, a validation, a legal format, is not done with a model: it is done with software, and the model is used beforehand, to order the input. Symptom: a demonstration with three examples. Remedy: fifty cases, each run three times, and a rule on what variation is acceptable at each step.

4. The middle of the document.

In July 2023 researchers from Stanford and Berkeley measured precisely something anyone who has used an assistant suspects: when a model is given many documents, it uses the information at the beginning and the end better, and the information in the middle worse, even in models designed for long texts. The paper is titled “Lost in the Middle” and its practical consequence is that a system which pastes twenty contracts into a single question answers, confidently, from the first and the last. Symptom: the provider boasts that the model “reads your whole archive in one go”. Remedy: retrieval with few, well-chosen documents per question, and a gold case that hides the right fact on page 14.

5. Language costs.

A 2023 study from the University of Washington compared how many tokens different languages need to say the same thing, and found that many need considerably more than English. For a Swiss or Spanish company, that means the same process costs more in German, French or Spanish than in English, and that, as the model has seen less text in those languages, it gets somewhat less right. And there is a finer layer: Swiss written German, with its own forms, its ss instead of ß and its figures with an apostrophe, and the Spanish of a client in Buenos Aires, are not the German or the Spanish of the manuals. Symptom: the fifty cases are all in the same language. Remedy: gold cases in every language the company serves in, in the real proportion, and the cost per case measured by language.

6. It agrees with you.

Anthropic’s October 2023 study on sycophancy, which I cited in note 2, showed that the five front-line assistants it tested tended to tell the user what they wanted to hear, because that is how they are trained. In a company this takes a concrete and dangerous form: the person who uses the assistant to “validate” a decision already taken, and always finds it in agreement. A risk report drafted with AI by someone who had already decided the risk was low will come out well written and will say the risk is low. Symptom: the analyses that reach the committee never contradict whoever signs them. Remedy: the rule of the counter-argument, in writing: every analysis done with AI includes a section in which the model has been asked to refute it.

7. The data that was not right before.

The Harvard Business Review’s 3% that I cited in note 6 is from 2017, and most companies have not cleaned their data since then; they have added more. A language model is the most effective way ever devised of turning a bad piece of data into a convincing text. The blind spot is not data quality, which everyone admits; it is believing that AI improves it. Symptom: “the system will learn with use”. A system with retrieval learns nothing: it reads what is there. Remedy: a named owner for every document in the archive, and an expiry date on each one.

8. What your team already pastes somewhere else.

The Microsoft and LinkedIn survey of 2024 that I cited in notes 4 and 6 put it in figures: three out of four knowledge workers were using AI, and 78% with their own tools. The blind spot is thinking that the company’s AI policy is the one that is written down. The real policy is what people do at 7 pm with the report that has to be delivered tomorrow. Symptom: the company “does not use AI” and the wording of the emails has mysteriously improved. Remedy: an official tool with the data protected, a short rule on what is never pasted and an amnesty: that telling what is used costs nothing.

9. Who can switch it off, and what happens then.

When an automatic system has been running for a year, the organisation adapts to it: the team that did the work is smaller, the knowledge of the process lives in the system’s archive and nobody remembers how it was done by hand. If the provider raises the price, retires the model or closes down, the cost of changing is far higher than the licence. It is item 7 from note 7, the one almost never negotiated. Symptom: nobody can answer “what would we do on Monday if this stopped working?”. Remedy: a manual procedure written down and tested once a year, connections on open standards and the ownership of the fifty cases, the documents and the logs always in the company’s hands, in a format anyone can read.

10. The cost of attention.

The last blind spot is the one that ties all the others together. Each of the previous nine things requires a person who looks: who runs the cases, reads the log, rotates through the sample, asks for the counter-argument, updates the archive. That attention is a small, continuous, unglamorous cost, and it is what separates the 5% of projects with a return from the 95% without one. When it is cut, nothing happens for months. Then one night in June happens. Symptom: the system has a technical owner and not one hour reserved in anyone’s diary. Remedy: two hours a week, with a name, in the diary of someone from the business, with a list of five checks. It is the cheapest insurance I know.

1. This week, 30 minutes

   Open the log

   Of the system that has been in production longest. Compare this month’s distribution with the first month’s. Whatever has changed without anyone deciding it is your blind spot number two.

2. This week

   Ask when the gold cases were last run

   If the answer is a date more than a month ago, or silence, run them this week.

3. This month

   Do the Monday drill

   What the company would do if the system did not start. Write it on one page. If it does not fit on one page, there is too much dependence.

4. This month

   Put two named hours in a diary

   From the business, not from IT. With the five checks written down. And look at it yourself once a quarter.

How I approach it.

Every system I deliver comes with a watch book: the five weekly checks, the calendar of the gold cases, the list of documents with owner and date, the Monday manual procedure and the rule of the counter-argument. It is a four-page document and it is the part of the project that keeps the most value after a year. I write it because Bainbridge was right in 1983 and still is: automating does not remove human work; it exchanges it for watching, and watching has to be designed.

Of the ten blind spots, which one made you most uncomfortable to read? Perhaps that is no coincidence. Write it on a piece of paper and leave it in sight for a week.

And a question for Friday: if your AI system stopped working for a whole day tomorrow, would anyone in your business know how to do that work by hand? And how long has it been since anyone did?

Next and last note: AI for those in charge. What whoever runs a company must know, must decide and must not delegate, with the seven questions for the committee, the reference framework regulators use and an agenda for the next twelve months.