The Things We Say About Machines: IV. It’s Not Really Thinking

Here is another sentence that seems to settle the matter:

“It’s not really thinking.”

The word really is doing some suspiciously heavy lifting.

It suggests that there is thinking, and then there is real thinking.

The machine may produce something that looks like thought. It may solve problems, follow an argument, compare alternatives, explain a concept, write a program, or revise an answer.

But whatever it is doing, we are told, it isn't really thinking.

Perhaps.

But what would make it real?

The obvious answer is that thinking is something humans do.

We think.

Machines compute.

So when a machine produces behaviour resembling thought, the behaviour must be an imitation of the genuine thing.

This sounds straightforward until we ask what thinking actually consists of.

Is thinking reasoning?

Then arithmetic and formal proof would seem to qualify.

Is thinking problem-solving?

Then many non-human animals appear to think.

Is thinking the manipulation of representations?

Then we need to explain what kind of representations count.

Is thinking conscious?

Then the question becomes inseparable from the problem of consciousness.

Is thinking something that occurs inside a biological nervous system?

Then we have defined thinking by its material substrate.

Each answer captures something.

None obviously captures everything.

And that matters because thinking is not the name of a single operation in the way that matrix multiplication is.

We say that someone is thinking when they are trying to solve a problem.

We say that someone is thinking about another person.

We say that someone is thinking through a difficult decision.

We say that someone is thinking aloud.

We say that someone has stopped thinking and is acting automatically.

We even say that someone has thought better of something.

The word covers a family of activities involving different kinds of internal organisation and different relations to action.

So when an LLM produces a sequence of intermediate steps while solving a problem, the interesting question is not whether those steps resemble something humans call thinking.

They plainly can.

The interesting question is what kind of process is producing them.

And here we encounter the same mistake as before.

The fact that we know something about the mechanism does not mean that we have therefore explained the capacity.

We can say that the model generates tokens according to learned statistical relationships.

We can describe its architecture.

We can analyse its training procedure.

We can trace the transformations occurring inside the network.

All of that is important.

But none of those descriptions, by themselves, tells us whether the resulting organisation deserves to be called thinking.

Nor does the opposite conclusion follow.

The fact that we cannot rule out the word does not establish that the machine thinks.

There is a genuine conceptual gap here.

And perhaps that gap is exactly what the phrase “not really” tries to hide.

It is tempting to think that thinking names a natural category whose members are obvious.

Humans are inside.

Machines are outside.

But categories do not always work that way.

Consider learning.

A person learns.

A child learns.

A dog learns.

A neural network learns.

The word does not mean exactly the same thing in every case. Yet neither is it meaningless when applied across those cases.

Different systems can instantiate related capacities in different ways.

The same may eventually prove true of thinking.

Or it may not.

We may discover that some apparently cognitive capacities require forms of embodiment, self-maintenance, memory, affect, agency or experience that current language models simply do not possess.

That would be an important discovery.

But notice the form such a discovery would have to take.

It would not be:

“It isn't really thinking because it's a machine.”

It would be something much more interesting:

“This particular kind of organisation lacks the properties required for this particular kind of thinking.”

That is an empirical and theoretical claim.

It can be investigated.

And it leaves open the possibility that a different kind of artificial system might satisfy the relevant conditions.

There is another complication.

Thinking does not necessarily have to be the production of an answer.

Much of human thought consists in changing the space of possibilities from which an answer can emerge.

We reconsider the question.

We notice a contradiction.

We abandon one approach.

We imagine an alternative.

We discover that something we had taken for granted was not necessary after all.

In other words, thinking can alter the organisation of the problem itself.

That is why the question of machine thinking cannot be settled simply by asking whether a machine produces correct answers.

Nor can it be settled by observing that the machine produces those answers through statistical computation.

The more interesting question is whether the system can reorganise its own activity in ways that constitute something recognisably like thinking.

And that question takes us somewhere rather different from the sentence with which we began.

“It’s not really thinking” sounds like a conclusion.

But perhaps it is actually an invitation to define our terms.

What would count?

What would distinguish thinking from calculation, prediction, association, search, simulation or response?

What kinds of organisation would be necessary?

And would those conditions be uniquely human?

Until we know, really is doing what just and only did in the earlier claims.

It narrows the field before we have investigated it.

Perhaps the machine isn't thinking.

Perhaps it is.

But before we decide, we might ask the more difficult question:

What, exactly, would make thinking real?