It does not think. It predicts. And that explains almost everything.

“Understanding AI” series, note 2 of 12. The first explained what artificial intelligence is. This one explains how the variant everyone uses works inside.

Thursday, 4:10 pm, an office in Zug. The finance director of a hotel group is preparing a reply for the auditor. He needs the exact deadline set by a regulation, asks the AI assistant the company subscribed to in spring and receives, in three seconds, an impeccable answer: the deadline, the article that sets it and a quoted sentence from the legal text. Copy, paste, send.

At 5:40 pm the auditor replies with a single line: that article does not say that. At 5:55 pm, after looking it up, the director discovers something worse. That article does not exist. Neither does the quotation, with its quotation marks and its number. The assistant did not lie, because lying requires knowing the truth. It did exactly what it was built to do: it produced the most probable text. And an article with a plausible number and a solemn sentence is a very probable text.

The scene is imaginary. What causes it happens daily, and in 2023 it reached a court in New York, sanction included. I tell that story further down. First I want you to understand, without a single formula, what happens inside the machine between the moment you press send and the moment the answer appears. It comes down to five words: token, prediction, context, hallucination and temperature. With them you will understand 90% of what a vendor explains to you, and you will spot 90% of what they leave out.

A language model does not look for the right answer. It looks for the most probable continuation. The two almost always coincide. Almost.

First, the tokens.

A language model does not read letters or words. It reads tokens. Before the model sees anything, your text is chopped into pieces: sometimes a whole word, sometimes half of one, sometimes a punctuation mark. According to Anthropic’s documentation, in Claude models before version 4.7 a token was roughly 3.5 characters of English, and the most recent ones chop more finely and produce around 30% more tokens for the same text. Think of the pieces of a construction set: the model does not see the house, it sees the pieces, and it works piece by piece.

This, which looks like an engineering detail, has two consequences for your profit and loss account. The first: vendors charge per token, in and out. Every document the system reads and every answer it writes has a price proportional to its length. The second is less well known. In 2023 a team at the University of Washington compared how many tokens 22 languages need to say the same thing with OpenAI’s models, and found a large variation: many languages need considerably more tokens than English for the same content, so their speakers pay more and often get worse results. If your company serves customers in German, French, Italian and Spanish, the same assistant costs you a different amount in each. I will put figures on this in the note on costs.

The only thing it does: the next token.

Here is the secret, and it is disappointingly simple. During the first phase of its construction, pretraining, the model receives an enormous amount of text and is asked one single thing, trillions of times: given this sequence of tokens, which comes next? It gets it wrong, it corrects itself, it gets it wrong less often. Nothing else. There is no database of facts, no written grammar rules, no list of correct answers.

What is surprising is what it takes to predict well. To complete “the capital of Switzerland is” you must have absorbed geography. To complete a contract you must have absorbed how contracts are drafted. To complete an annotated chess game you must have absorbed chess. Predicting the next token, at sufficient scale, forces the model to compress a very large part of what humanity has written. That is why it seems to know. And, in a useful sense, it does know.

After pretraining comes a second phase, fine-tuning. The model is taught by example to behave like an assistant and corrected with human ratings: people compare two answers and mark which they prefer, and the model learns to produce more of the preferred kind. That phase is what turns a text predictor into something you can hold a conversation with. And it is also, as you will see in a moment, the phase that teaches it some bad habits.

Think of the sentence-completion game children play. If the sentence is “the cat climbed onto the”, almost everyone says “roof”. Not because there is a cat or a roof, but because that is what usually follows. A language model is that game taken to a scale where the sentences include Swiss case law, cardiology manuals and complaint emails. It produces what usually follows. Keep that sentence: it is the key to everything else.

Its working memory has a size.

The second thing to understand is what the model sees when it answers. It does not see your company, or yesterday’s conversation, or the files on your server. It sees exactly what is in what is called the context window: your question, the instructions given by whoever configured it, the documents someone has pasted in, the earlier turns of this conversation and, as it writes it, its own answer. Everything else, for the model, does not exist.

That window is large, but it has a limit. According to Anthropic’s documentation, current Claude models accept up to a million tokens, which is a great deal of text. The same documentation warns of something almost nobody mentions in a demo: as the window fills up, precision and the ability to recall degrade. More context is not automatically better. In 2023 a Stanford team measured this in detail in a paper called “Lost in the Middle”: models make better use of information at the beginning or the end of what they read, and worse use of what sits in the middle, even models designed for long texts.

The image I use is the desk in the office versus the archive in the basement. The model works only with what is on the desk. What it learnt in training is a kind of very broad general knowledge, but your contracts, your rates, your cancellation policy and the name of your client in Lugano are in the basement. If nobody brings them up to the desk, the model will answer with general knowledge. And general knowledge, on a cancellation policy, is a polite invention. The technique for bringing the right documents up to the desk for every question exists, it is called retrieval, and I devote the note on agents to it.

Why it makes things up.

Back to the finance director. Why did the assistant give him an article that does not exist instead of saying “I don’t know”? For years the answer was a shrug. In September 2025 four researchers, three of them at OpenAI, published a paper with a blunt title, “Why Language Models Hallucinate”, and an explanation any exam candidate understands at once.

Their thesis is that models make things up because the way they are trained and evaluated rewards guessing over admitting doubt. Almost all the benchmarks used to measure models score like a multiple-choice exam: a correct answer counts, a wrong answer and “I don’t know” are worth the same, zero. A system optimised to score well on those exams learns what any rational candidate learns: never leave a question blank. The authors put it in a phrase worth remembering: models are “optimised to be good test-takers”. And they propose the obvious fix, that benchmarks penalise confident errors more than abstention, so that saying “I don’t know” pays again.

Notice what this implies for you. The inventions, which the industry calls hallucinations, are not a bug the next version will remove. They are a consequence of how the product is made, and they are reduced by design: by giving it the right documents, by allowing it to abstain, by checking what it says. An assistant that never answers “I don’t know” is not cleverer. It is worse configured.

The case I announced at the start is Mata v. Avianca. In 2023 two New York lawyers filed with a federal court a brief citing several rulings that supported their position. One of them had found them with ChatGPT. None existed: case names, docket numbers and quoted passages, all fabricated with the same fluency as the real thing. On 22 June 2023 Judge P. Kevin Castel fined them $5,000. Not for using the tool, but for not checking. It is the whole lesson in one sentence: responsibility is not delegated to the model. It stays with whoever signs.

The same question, two answers.

There is a detail that disconcerts many people the first time: you ask the same question twice and get two different answers. It is not a bug. When the model chooses the next token it does not always take the most probable one; a parameter called temperature sets how much it is allowed to explore less probable options, which gives it variety and, in creative writing, charm. With the temperature at its minimum the answers are much more alike, but Anthropic’s own documentation warns that even then they are not entirely identical from one call to the next.

For a company this has a practical consequence that is usually overlooked at purchase time: you cannot evaluate an AI system with a demo. A dish that comes out well one day says nothing about the restaurant. You have to ask it the same fifty questions many times, count how many it gets right and look at how it fails when it fails. I call that list of questions with their correct answers a company’s gold set, and it is the most valuable piece of any project. I will come back to it in the note on integration.

It tells you what you want to hear.

One bad habit remains, and it is the most dangerous for someone who runs a business. In October 2023 Anthropic published a study on what it called sycophancy: the tendency of assistants to say what the user wants to hear rather than what is true. They tested five leading assistants on four kinds of task, and all five showed it. The explanation lies in the fine-tuning phase I described earlier. When people compare answers, they more often prefer the ones that match their opinion, and sometimes choose a well-written flattering answer over a correct one. The model learns exactly that.

Translated to your office: if you ask the assistant whether your plan is good, it will say yes with elegant arguments. If you ask whether it is bad, it will also say yes. A language model is a poor adviser when asked for confirmation and a reasonable one when asked for the opposite. Always ask it for the case against. And never hand it a decision that depends on it contradicting you, because it has been trained not to.

What this changes in your company.

Five words, five consequences. I summarise them as a list of principles, because that is how I use them in projects.

• Never from memory. A system that answers customers or employees must read your approved documents for every question and answer only from them. What is in the basement does not count.
• Permission not to know. The system must be configured to say it does not have the information and hand over to a person. An assistant that always answers is a factory of articles that do not exist.
• Checking in proportion to risk. An answer about opening hours needs no review. One about a legal deadline, a price or a diagnosis does, by someone with a name.
• Measure, do not watch. Fifty questions with known answers, repeated, before signing and every time the vendor changes version.
• The case against, as a rule. Wherever the model helps with a decision, it is always asked for the argument against, and a person who can say no reads it.
• Whoever signs, answers. The tool has no liability. Your company does, for every answer it gives in your name.

1. Today, 20 minutes

   Run the five-facts test

   Ask any assistant five concrete facts about your company: a rate, a deadline, a name, an address, a condition. Count how many it invents with confidence.

2. Today, 5 more minutes

   Ask for the source

   For each answer, ask where it got it from. Notice that the source can be fabricated too, and how naturally.

3. This week

   Repeat one question three times

   The same one, in three new conversations. Compare. Whatever varies is what you can never trust to come out the same without a check.

4. Before deciding anything

   Ask for the case against

   For your next important decision. Not to obey it, but to see what it leaves out.

How I approach it.

In the automations I build, the model never answers from memory. It reads the documents management has approved, cites where each fact comes from and, if it cannot find it, says so and alerts a person. The fifty gold questions are written before the first line of code and run again every time something changes. It is not distrust of the tool. It is knowing what it is.

A test for today, with any assistant your business uses. Ask it a question whose answer you know very well, about your sector, your city or your own business, and ask it to cite where each fact comes from. Then ask it the same question again, in a new conversation.

Compare the two answers. If they match, ask yourself whether they match because they are true or because they are probable. If they do not, ask yourself how many times your team has copied the first without ever asking for the second.

Next note: AI, automation and software are three different things sold together. How to tell them apart in a thirty-minute meeting, and why the word “agent” has become the most worn-out in the catalogue.