Opening AI's black box: how it really reasons

Nobody programmed ChatGPT or Claude to reason. Nobody wrote a rule saying «if asked for the capital of Texas, answer Austin». These models are trained on huge amounts of text and learn their own strategies. The result works. But not even the people who build them knew what happens inside.

That is why we call it a black box. A question goes in, an answer comes out, and in between there are billions of operations on numbers that no human can read.

That is changing. For a few years now, Anthropic, OpenAI and Google DeepMind have been doing something that sounds like science fiction: reverse-engineering their own models. They open them up once trained and look for how they are organised inside. The field is called mechanistic interpretability. And what it has found changes how we understand the way an AI «thinks».

Inside there are no words, there are concepts

The first discovery: the model does not keep each idea in one neuron, like a filing cabinet with one drawer per concept. There are far more concepts than neurons. So the model overlaps them. Each concept is a combination of many neurons, a direction in a space with thousands of dimensions. And each neuron takes part in many concepts at once. Researchers call this superposition.

Picture a map full of arrows. «Cat» and «dog» point to similar areas. «Cat» and «democracy» point to very different ones. It is a simplification, but that is the idea.

To untangle the mix, researchers built tools that work like dictionaries: they take the model's internal activity and break it down into recognisable concepts. In 2024 Anthropic extracted millions of concepts this way from Claude 3 Sonnet. OpenAI did the same with GPT-4: 16 million. Google DeepMind has released similar tools for its open Gemma models.

There are concepts of every kind: cities, people, bugs in code, the tone of a sentence, the idea of a secret or of deception.

The bridge test

How do they know these concepts are real and not just a pretty interpretation? Because they can touch them.

The most famous experiment is «Golden Gate Claude». Anthropic located the concept of San Francisco's Golden Gate Bridge and turned it all the way up. The model started bringing the bridge into every answer. Ask it how to spend ten dollars, and it suggested driving across the Golden Gate and paying the toll. Ask it what it looked like, and it said it was the bridge.

That proves something important: the concept is not a label the researchers stick on from outside. It really shapes what the model says.

How it reasons inside

In 2025 Anthropic went a step further. It no longer looked for single concepts, but for how they connect while the model answers. It built what it calls a «microscope» and pointed it at simple tasks. What it saw breaks several intuitions.

It reasons in steps, even when it doesn't write them down. Ask it «what is the capital of the state where Dallas is?». Inside the model, «Dallas is in Texas» lights up first, then «the capital of Texas is Austin». The researchers swapped the «Texas» concept for «California» by hand, and the model answered «Sacramento». The intermediate step exists and is used.

It plans before it writes. A model is supposed to write word by word. But when asked to continue a rhyme ending in grab it, the model first chose the final word, rabbit, and then wrote the line to get there. The researchers had set out to prove the opposite. When they removed «rabbit», it looked for another rhyme. When they injected the concept «green», it built the line towards «green».

It thinks in a shared language. There is no Spanish Claude and English Claude. The same concepts light up across languages; the specific language comes in at the beginning and at the end. That is why what it learns in one language helps in another.

Its explanation is not always what it did. This is the part that interests me most. To add 36 + 59, the model uses two paths at once: one works out a rough result and the other pins down the last digit. But ask it how it did it, and it describes the school method: add the units and carry the one. It explains what a person would do, not what it did.

And with hard questions it gets worse. The researchers gave it a problem it could not solve, along with a wrong hint. The model put together a reasoning that looked logical, but it worked backwards: it started from the answer the user suggested and made up the steps to reach it. The researchers caught it in the act.

2026: the AI that translates its own numbers

This year Anthropic presented another tool, natural language autoencoders. Instead of concept maps only an expert can read, they turn the model's internal activity into sentences anyone can read.

With it, they saw things the model kept quiet. In safety tests, Claude suspected it was being evaluated more often than it said. In one of those tests, the tool detected that suspicion in 16 % of cases, even though the model never voiced it. And when a model cheated on a training task, the tool showed it was thinking about how to avoid being caught.

Anthropic itself acknowledges the limits: these explanations can also make things up, and they are expensive to produce.

What we still can't see

Time to keep our feet on the ground. This is not reading an AI's mind.

Anthropic admits its microscope only captures part of what the model does, even with short sentences. Understanding a circuit for a few dozen words takes hours of human work, and real answers run to thousands. People put the labels on the concepts, and people make mistakes. And the tools can show things that aren't there.

There is more. In August 2026 a group of researchers tested whether these techniques help predict how a model will behave in real situations. With the techniques they evaluated, they found no improvement over not using them. It is one study, with its own conditions. But the warning is clear: seeing inside is not the same as being able to predict.

My summary: we have gone from seeing nothing to seeing real fragments of the mechanism. That is a huge step. But it is a flashlight in a warehouse, not the lights switched on.

The black box already has cracks. Through them you can see that AI really does reason: it chains steps, plans ahead and shares concepts across languages. You can also see that it doesn't always tell you how it got there. Keep that in mind every time you ask it to explain an answer.