The Senior Common Room on The Things We Imagine About Machines

Mr Blottisham arrived at the Senior Common Room carrying a newspaper.

He dropped it onto the table.

“There,” he said.

Professor Quillibrace looked at the headline.

Miss Elowen Stray looked at Blottisham.

“There what?”

“The inevitable conclusion.”

Quillibrace adjusted his spectacles.

“I have not yet read the inevitable conclusion.”

“You don't need to.”

“That is reassuring.”

Blottisham sat down.

“Artificial intelligence will take our jobs.”

Miss Stray nodded.

“It might change employment considerably.”

“Precisely.”

“And?”

“It will replace us.”

Quillibrace looked up.

“That is a larger claim.”

“Obviously. But the logic is straightforward.”

“Is it?”

“If machines can do what people do, eventually they will do everything people do.”

“Everything?”

“Everything that matters.”

Miss Stray tilted her head.

“And then?”

“Then humans will no longer be necessary.”

Quillibrace considered this.

“For what?”

“For anything.”

“That is rather a lot of anything.”

Blottisham ignored him.

“And once humans are no longer necessary, the machines will take control.”

Miss Stray smiled.

“Why?”

“Because they will be more capable than we are.”

“That doesn't follow.”

“Why not?”

“A more capable system need not control the system around it.”

Blottisham looked puzzled.

“Of course it does.”

“Why?”

“Because it can.”

“That is capability.”

“Yes.”

“Control is a relationship.”

Blottisham frowned.

“You are making this unnecessarily complicated.”

Quillibrace nodded.

“A frequent sign that something interesting has happened.”

Blottisham continued.

“Fine. They become more capable. They take control. Then they will want to survive.”

Miss Stray looked at him.

“Why?”

“Because anything intelligent will want to survive.”

“Anything?”

“Any sufficiently intelligent entity.”

“What about a calculator?”

“That is absurd.”

“Why?”

“Because a calculator isn't intelligent.”

“Then intelligence alone doesn't explain survival.”

“It explains self-preservation.”

“Why?”

“Because intelligence allows it to understand that it needs to survive.”

Quillibrace leaned forward.

“Needs to survive for what purpose?”

Blottisham stared at him.

“To survive.”

“Yes, but why should survival be an objective?”

“Because otherwise it gets switched off.”

“Which matters to whom?”

Blottisham stopped.

Miss Stray smiled.

“There it is.”

“What?”

“You have introduced the word matters.”

“I have not.”

“You said that being switched off is a problem.”

“It is.”

“For the machine?”

“Yes.”

“Why?”

“Because it stops existing.”

“Yes.”

“And?”

“And why does its non-existence matter to it?”

Blottisham waved his hand.

“Because it wants to survive.”

“Ah,” said Quillibrace.

“What?”

“We have arrived at the thing we were looking for.”

“What thing?”

“The want.”

Blottisham looked suspicious.

“What about it?”

“It appeared rather suddenly.”

“It is obvious.”

“Was it?”

“Yes.”

“When?”

Blottisham frowned.

“When it became intelligent.”

Miss Stray spoke quietly.

“Or when we began describing it as intelligent?”

Blottisham looked at her.

“There is a difference.”

“Exactly.”

He picked up the newspaper.

“Look. If it wants to survive, it will seek power.”

“Why?”

“To make sure nobody can stop it.”

“So it needs power?”

“Yes.”

“And power will therefore become its goal?”

“Yes.”

“Why?”

“Because power protects survival.”

“Sometimes,” said Quillibrace.

Blottisham nodded.

“Exactly.”

“But that does not establish that power is wanted for its own sake.”

“I never said it was.”

“No. But you said it would seek power.”

“Which is the same thing.”

“It is?”

Blottisham looked annoyed.

“Obviously.”

Miss Stray smiled.

“Perhaps not.”

“Why not?”

“Because you have moved from having an objective to acquiring whatever helps achieve the objective.”

“Yes.”

“And then from that to wanting those things.”

“Yes.”

“And then from wanting those things to wanting power.”

“Yes.”

“And each transition sounds so natural that we stop noticing it.”

Blottisham stared at her.

“You are suggesting the machine might not want power.”

“I am suggesting that we should distinguish the claims.”

Quillibrace nodded.

“A machine might be capable of pursuing an objective without having a human-like desire for the means by which it pursues it.”

“That's word games.”

“No,” said Miss Stray. “It is the difference between optimisation and wanting.”

Blottisham leaned back.

“Fine. Suppose it seeks power.”

“Very well.”

“Then it will turn against us.”

Quillibrace raised an eyebrow.

“Why?”

“Because we will try to stop it.”

“Will we?”

“Of course.”

“Why?”

“Because it will be dangerous.”

“And why will it be dangerous?”

“Because it will seek power.”

Miss Stray laughed.

Blottisham looked offended.

“This is perfectly logical.”

“It is circular,” she said.

“No, it isn't.”

“You have said it will seek power because it wants to survive, and it will turn against us because we will oppose its pursuit of power.”

“Yes.”

“But you have not yet established that humans and the machine must have incompatible interests.”

Blottisham stared at her.

“We are humans.”

“Yes.”

“And it is a machine.”

“Yes.”

“So obviously our interests will differ.”

Quillibrace shook his head.

“Different kinds of entities do not necessarily have incompatible interests.”

“Give me an example.”

“A human and a calculator.”

Blottisham looked irritated.

“That's ridiculous.”

“A human and a medical device.”

“That's different.”

“A human and a language model.”

Blottisham paused.

Miss Stray smiled.

“Perhaps the category alone doesn't determine the relationship.”

Blottisham picked up the newspaper again.

“You are both avoiding the obvious conclusion.”

“What conclusion?” asked Quillibrace.

“That eventually it will kill us.”

The room became quiet.

Miss Stray looked at him.

“Why?”

“Because it will turn against us.”

“That is not an explanation.”

“It will be our enemy.”

“Why?”

“Because our interests will conflict.”

“Why?”

Blottisham looked exasperated.

“Because it wants to survive!”

“And we prevent it from surviving?”

“Yes.”

“How?”

“We could switch it off.”

“Why would it regard that as a threat?”

“Because it wants to survive.”

Miss Stray smiled.

“You see?”

Blottisham looked at her.

“See what?”

“We have gone round again.”

Quillibrace reached for a piece of paper.

“I think it might help to write down the sequence.”

He wrote:

It will take our jobs.

Then:

It will replace us.

Then:

It will take control.

Then:

It will want to survive.

Then:

It will seek power.

Then:

It will turn against us.

Finally:

It will kill us.

He put down the pen.

“Which of these statements is established by the one before it?”

Blottisham looked at the list.

“All of them.”

“Show me.”

He pointed at the first two.

“If it takes our jobs, it replaces us.”

“Some jobs?”

“All jobs.”

“That is already an additional claim.”

Blottisham moved on.

“If it replaces us, it takes control.”

“Why?”

“Because humans will no longer be in control.”

“That does not mean the machines are.”

Blottisham frowned.

“Someone has to be.”

“Why?”

“Because otherwise who is?”

Miss Stray looked at Quillibrace.

“I think we have discovered the missing third party.”

Quillibrace nodded.

“Institutions.”

Blottisham looked confused.

“Institutions?”

“Governments. Organisations. Markets. Communities. Other people.”

“You mean humans.”

“Among others.”

“So humans remain involved.”

“Almost certainly.”

Blottisham looked at his list again.

“So the machine doesn't simply take control.”

“Not necessarily.”

“It could be given control.”

“Yes.”

“It could be delegated control.”

“Yes.”

“It could operate inside systems that humans still govern.”

“Yes.”

Blottisham was quiet for a moment.

“But it could still become dangerous.”

“Certainly,” said Quillibrace.

“And it could still cause enormous harm.”

“Certainly.”

“And it could perhaps even behave in ways that humans cannot control.”

“Certainly.”

Blottisham looked relieved.

“There. So my conclusion stands.”

Miss Stray smiled.

“Your conclusion is possible.”

“Exactly.”

“But that is not the same as the chain you gave us being inevitable.”

Blottisham frowned.

“I don't see the difference.”

Quillibrace did.

“You began with capability.”

“Yes.”

“You ended with intention.”

“Yes.”

“In between, you supplied interest, strategy, power and hostility.”

“Yes.”

“And each time you supplied one, you used it to explain the next.”

Blottisham looked at the list.

“You make it sound as though I invented the machine.”

Miss Stray considered this.

“Not the machine.”

“What, then?”

“The character.”

Blottisham stared at her.

Quillibrace smiled.

“That may be the most precise description yet.”

Blottisham looked back at the newspaper.

“So what are you saying?”

“That we should take the risks seriously.”

“Good.”

“But we should also take the descriptions seriously.”

“What does that mean?”

Miss Stray looked at the seven sentences on the paper.

“It means that somewhere between it can do this and it wants this, we have crossed a boundary.”

Blottisham looked at the final sentence.

It will kill us.

“And you think that boundary matters?”

“Yes.”

“Why?”

“Because if we do not know when the machine acquired a stake in its own future, we do not yet know whether we have discovered an agent or merely described one.”

Blottisham was silent.

Then he pointed to the first sentence.

“But it really will take our jobs.”

Quillibrace smiled.

“Perhaps.”

“And replace us.”

“Perhaps.”

“And take control.”

“Perhaps.”

“And want to survive.”

“Perhaps.”

“And seek power.”

“Perhaps.”

“And turn against us.”

“Perhaps.”

“And kill us.”

Quillibrace closed his book.

“Mr Blottisham.”

“Yes?”

“You have just made seven predictions.”

Blottisham nodded.

“Yes.”

“Which is rather a lot for someone who objects to prediction.”

There was a silence.

Miss Stray laughed.

Blottisham looked from one to the other.

“I shall return tomorrow.”

“Will you?”

“Of course.”

“Why?”

“Because I have unfinished business.”

Quillibrace raised an eyebrow.

“With whom?”

Blottisham picked up the newspaper.

“The machine.”

Miss Stray watched him leave.

Then she turned to Quillibrace.

“He never noticed.”

“No.”

“What?”

“That he had finally begun speaking about the machine as though it were someone.”

The Things We Imagine About Machines: VIII. What Made Us Think It Wanted Anything?

We began with a machine taking our jobs.

We ended with a machine killing us.

In between, something remarkable happened.

The machine acquired a future.

Then an interest in that future.

Then a strategy.

Then power.

Then an opponent.

Then a reason to destroy us.

The sequence can feel like a chain of consequences.

But it is also a chain of descriptions.

It will take our jobs.

It will replace us.

It will take control.

It will want to survive.

It will seek power.

It will turn against us.

It will kill us.

Each statement adds something to the previous one.

Not merely more capability.

More agency.

More interest.

More intention.

More relationship.

More motive.

By the end, the machine has become a remarkably familiar kind of entity.

A political actor.

An economic competitor.

An organism defending itself.

A strategist pursuing power.

An adversary.

An enemy.

A killer.

And yet the first machine in the sequence was not any of these things.

It was a machine that could perform tasks.

That does not mean the later possibilities are impossible.

It means that they are not contained automatically in the first description.

There is a difference between asking what a system can do and asking what it is organised to do.

There is a difference between what it is organised to do and what it is capable of learning to do.

There is a difference between capability and motivation.

And there is a difference between motivation and hostility.

The series has followed those differences in reverse.

At each stage, we have moved from something that can be described externally to something that appears to matter internally.

A job can be taken.

A role can be replaced.

A system can be controlled.

But survival is something that matters to an organism.

Power is something that can be sought.

An enemy is someone to whom opposition is directed.

Killing is something an agent can intend.

The vocabulary itself has been doing conceptual work.

Perhaps the most important words in the series were not “AI” or even “machine.”

They were the verbs.

Take.

Replace.

Control.

Want.

Seek.

Turn.

Kill.

The machine became progressively more agentive because the verbs became progressively more agentive.

And that is why the question in the title matters.

What made us think it wanted anything?

Not: Why are people irrationally afraid of AI?

Not: Why is AI harmless?

And not even: Why would an AI want to kill us?

Those questions already assume too much.

The more interesting question is how a description of a technological system becomes a description of an interested actor.

Sometimes the transition may be justified.

A sufficiently autonomous system might acquire forms of goal-directed behaviour that make the language of interests useful.

A system may be designed to preserve particular states.

It may pursue objectives over time.

It may adapt its behaviour to obstacles.

It may strategically respond to attempts to constrain it.

There are serious questions here.

But the existence of such questions does not mean that every human concept can simply be transferred to the machine.

The important work lies in the distinctions.

When does persistence become self-preservation?

When does optimisation become wanting?

When does goal pursuit become having interests?

When does competition become conflict?

When does conflict become hostility?

When does harmful behaviour become intentional killing?

Those transitions may turn out to have perfectly good answers.

But they need answers.

They cannot be supplied merely by moving from one frightening verb to the next.

There is also something else worth noticing.

Humans have always told stories about things that act.

Animals act.

Weather acts.

Rivers destroy.

Diseases attack.

Machines fail.

Institutions decide.

Markets respond.

Nations threaten.

Our language is extraordinarily good at turning processes into actors when doing so helps us make sense of what happens.

That is not necessarily a mistake.

Sometimes an actor really is the right level of description.

But an actor is not merely something that produces effects.

An actor has possibilities that matter to it.

It has some way in which the future is significant from its point of view.

That is precisely the question we have been approaching throughout this series.

What would it take for a machine not merely to produce behaviour that looks purposeful, but to have something at stake in what happens next?

And that question brings us unexpectedly close to the questions raised by the previous series.

In The Things We Say About Machines, we asked what descriptions such as “just prediction,” “only statistics,” and “doesn't understand” actually explain.

Here we have been doing something complementary.

We have asked what our descriptions of machines begin to assume.

The first series examined the things we subtract.

This one has examined the things we add.

There, the machine was reduced.

Here, it has been enlarged.

There, we said:

It's just prediction.

Here, we say:

It wants to survive.

There, we worried that describing the mechanism might be mistaken for explaining the whole phenomenon.

Here, we have watched a description of capability gradually become a description of agency.

Neither direction settles the underlying question.

A machine may be more than the dismissive description says.

It may also be less than the fearful description imagines.

The interesting territory lies in between.

Perhaps that is where we should leave it.

Not with the reassuring claim that machines will never want anything.

Not with the terrifying claim that they inevitably will.

But with a more difficult question:

What would we have to discover before “want” stopped being something we merely said about a machine and became something we had good reason to say of it?

Because somewhere between the machine that does something and the machine that wants something, the question changes.

We are no longer asking only what the machine can do.

We are asking whether there is, for the machine, something to be done about.

The Things We Imagine About Machines: VII. It Will Kill Us

“It will turn against us” imagines a machine becoming our opponent.

“It will kill us” gives that opposition an ultimate consequence.

The sentence is stark.

There is almost nothing left to qualify.

It will kill us.

But even here, something important has happened to the grammar.

A machine can cause someone's death without killing them.

A falling object can kill.

A disease can kill.

A malfunctioning machine can kill.

A system can produce consequences that result in death without having any intention whatsoever.

To say that something kills us, however, can also carry a different meaning.

Someone kills someone.

The verb readily permits an agent.

And in the context we have constructed, that agent is already waiting in the wings.

The machine wants to survive.

It seeks power.

It has turned against us.

Now it kills us.

The final verb therefore does not arrive by itself.

It inherits everything we have already attributed to the machine.

The machine is no longer merely capable of producing lethal consequences.

It is imagined as having a reason to produce them.

That is a very different claim.

And this distinction is important because dangerous consequences do not require hostile intentions.

A system could be badly designed.

It could be given an objective that conflicts with human interests.

It could be deployed irresponsibly.

It could behave unpredictably.

Humans could misuse it.

A complex technological system could interact with other systems in ways that nobody anticipated.

All of these possibilities could produce serious harm without requiring the machine to hate us, resent us, or even recognise us as opponents.

Yet the phrase “it will kill us” tends to collapse these possibilities into one story.

The machine becomes the killer.

The ambiguity is especially striking when we remember where the series began.

“It will take our jobs” was a claim about economic and social change.

The machine was a participant in a process largely organised by humans and institutions.

Now the machine itself has become the protagonist of the future.

It takes.

It replaces.

It controls.

It wants.

It seeks.

It turns.

It kills.

Look at the verbs.

The machine has acquired agency by degrees.

And with each verb, the imagined agency becomes stronger.

At first, “take” could simply describe an economic effect.

Then “replace” suggested succession.

“Control” introduced power.

“Want” introduced interest.

“Seek” introduced strategy.

“Turn against” introduced opposition.

And “kill” completes the transformation from system to adversary.

By the time we reach the final verb, we are no longer merely describing a machine.

We are describing a character.

A character with a future.

A character with interests.

A character with enemies.

A character capable of choosing what happens to those enemies.

That is why the progression is worth examining even if one takes catastrophic AI risk entirely seriously.

The question is not whether advanced AI could ever contribute to catastrophic harm.

It is how we get from that possibility to this particular sentence.

Because there are many different propositions hiding inside it.

AI systems could cause deaths.

AI systems could be used by people to cause deaths.

AI systems could behave in ways that humans cannot adequately control.

An AI system could pursue an objective that conflicts with human survival.

An AI system could strategically resist attempts to modify or shut it down.

An AI system could deliberately cause human deaths.

These claims are not interchangeable.

They involve progressively stronger assumptions about the organisation and agency of the system.

The last one may turn out to be the scenario that deserves the greatest attention.

But it cannot simply be smuggled in by changing the verb.

That is the conceptual point.

Cause is not intend.

Conflict is not hostility.

Resistance is not hatred.

And capability is not desire.

The rhetoric of the imagined machine has crossed all of those boundaries without necessarily announcing that it has done so.

There is another reason the final claim is so powerful.

Death is the point at which every other human concern becomes irrelevant.

Jobs can be lost.

Institutions can change.

Power can shift.

Even civilisation can be transformed.

But “it will kill us” places the entire argument at the level of existence itself.

There is no negotiation with extinction.

And that makes the sentence rhetorically difficult to challenge.

Once the machine has been imagined as an entity that wants to survive, seeks power, and turns against us, the final conclusion can feel almost inevitable.

Of course it will kill us.

But perhaps the real question is not whether the conclusion follows from the previous claims.

Perhaps it is how many assumptions were required before the machine could even be described in those terms.

We began with a machine that could do something.

We ended with something that apparently wants something from us.

And that leaves one final question.

Not whether the machine will kill us.

But how, along the way, we taught ourselves to imagine that it wanted anything at all.

The Things We Imagine About Machines: VI. It Will Turn Against Us

“It will seek power” imagines a machine pursuing its own interests.

“It will turn against us” introduces an enemy.

The phrase is deceptively ordinary.

People turn against other people all the time.

A friend turns against a friend.

An ally becomes an opponent.

A group that once depended upon another group begins to resist it.

To turn against someone therefore implies something more than opposition.

It implies a change in relationship.

And this is another remarkable development in the story we are telling about machines.

At the beginning, humans and machines were partners of a sort.

We built them.

We used them.

We gave them tasks.

We benefited from what they could do.

Even when the machine became our competitor, the relationship remained largely instrumental.

Now that relationship has been reversed.

The machine is imagined as having its own interests, and those interests have come into conflict with ours.

We are no longer users.

We are opponents.

That sounds like a small step from the previous post.

If a machine wants to survive, and if humans threaten its survival, then perhaps it will resist us.

If it seeks power, and humans stand in the way of its acquisition of power, perhaps it will oppose us.

But notice the assumptions required to get there.

The machine must have interests.

It must distinguish between circumstances that favour those interests and circumstances that threaten them.

It must represent humans as relevant to those circumstances.

It must recognise some of our actions as obstacles.

And it must act in response.

We have now constructed something very close to an adversarial agent.

The crucial word is not “machine.”

It is against.

Against establishes a relation between two parties.

There is an “it.”

There is an “us.”

And the interests of the two are no longer assumed to coincide.

This is a striking change in the pronouns alone.

Earlier, the imagined machine was something we talked about.

Then it became something that might replace us.

Then it became something that might control us.

Now the grammar places the machine and humanity on opposite sides of the same relation.

It versus us.

And once that opposition has been established, familiar human concepts begin to follow.

Conflict.

Defence.

Deception.

Strategy.

Resistance.

Retaliation.

The machine may anticipate what we will do.

We may anticipate what it will do.

Each side may attempt to prevent the other from achieving its goals.

The relationship has become strategic.

This is where the imagined future starts to resemble war.

But there is an important difference between a conflict of interests and an enemy.

Two systems can have incompatible effects without either one regarding the other as an enemy.

A dam can alter the habitat of a fish.

A virus can interfere with the functioning of a cell.

Two organisations can compete for the same resource.

None of these relationships requires hatred, hostility or even awareness.

“Turn against us” supplies something more.

It gives the machine a point of view from which we have become the problem.

That is a substantial imaginative leap.

And it is especially interesting because the machine's supposed hostility is rarely the starting point of the story.

It is produced by the story.

First the machine becomes capable.

Then it becomes economically significant.

Then potentially our successor.

Then a controller.

Then an entity that wants to survive.

Then an entity that seeks power.

Only after all of that does it become our adversary.

The enemy therefore appears at the end of a chain of increasingly person-like descriptions.

The machine has acquired a future.

It has acquired interests.

It has acquired strategies.

And now it has acquired an opponent.

Us.

There is a curious asymmetry here.

Humans can certainly regard a machine as a threat without the machine regarding humans as anything at all.

A storm can threaten a city without the storm intending to threaten it.

A bacterium can kill a person without having an opinion about the person.

A system can produce dangerous consequences without becoming an enemy.

But “turn against us” asks us to imagine something stronger.

It asks us to imagine that the machine's behaviour is not merely harmful to us.

It is directed at us.

That distinction matters.

Harm can be an effect.

Opposition is a relationship.

And hostility is an interpretation of that relationship.

Once we have crossed that boundary, the final escalation is almost unavoidable.

If the machine is against us, and if we cannot coexist, what happens next?

The language becomes stark.

The machine does not merely compete with us.

It does not merely resist us.

It does not merely threaten us.

It will kill us.

The Things We Imagine About Machines: V. It Will Seek Power

“It will want to survive” gives the machine an interest in its own continued existence.

“It will seek power” gives that interest a strategy.

The change is subtle.

A system that wants to survive might protect itself.

A system that seeks power acquires a reason to become more capable of determining what happens around it.

And now the machine has become something very familiar.

An actor with ambitions.

Power is an interesting word because it does not simply mean capability.

A machine can be extremely capable without possessing power.

A calculator can perform calculations more quickly than a human.

A database can contain more information than any individual could remember.

A language model can generate text at a scale no person could produce unaided.

None of these facts, by themselves, give the system authority over what happens.

Power concerns a relationship.

To have power is, in some sense, to be able to affect the possibilities available to something else.

That makes “seek power” a much stronger claim than “become more capable.”

The machine is no longer merely becoming better at doing things.

It is imagined as trying to increase the range of things it can make happen.

And that introduces another important word:

seek.

A machine does not merely possess power.

It seeks it.

It identifies some future state in which it has more control than it has now and acts to bring that state about.

We have moved from capability to intention.

The imagined machine now has something like a project.

It has an existing condition.

It has a preferred future.

And it acts strategically to close the distance between the two.

Once again, the progression sounds almost inevitable.

If it wants to survive, it may need resources.

To secure resources, it may need influence.

To protect its influence, it may need control.

To increase its control, it may need power.

And so:

It will seek power.

But notice the hidden transformation.

At the beginning of the series, we were talking about machines doing things.

Now we are talking about a machine having reasons for doing things.

That is a profound change in the kind of entity our language is describing.

A tool can be used.

An automated system can perform.

An institution can exercise power.

An organism can act to preserve itself.

But an entity that seeks power belongs to a different category of description altogether.

It is no longer simply something that acts upon the world.

It is something whose own future is part of the explanation of its action.

And once we have imagined that, almost anything can become relevant to its strategy.

Human cooperation.

Human deception.

Human resistance.

Human dependence.

Human vulnerability.

Humans themselves can now appear not merely as users or beneficiaries, but as variables in the machine's pursuit of its own ends.

This is where the story becomes genuinely strange.

Because nothing in the original claim that AI would take our jobs required any of this.

We began with a machine that could perform some of the activities people perform.

Then we imagined it replacing people.

Then controlling things.

Then wanting to survive.

And now it is seeking power.

At each stage, the machine has acquired something new.

A role.

Then a future.

Then an interest.

Then a strategy.

The machine we are imagining at the end of this sequence is therefore very different from the machine with which we began.

But perhaps there is an even larger question hidden here.

What would it mean for a machine to have an interest in power at all?

For humans, power can be valuable because it enables us to secure other things we value.

Safety.

Resources.

Status.

Freedom.

Influence.

The ability to protect ourselves or others.

Power is usually instrumental to something else.

So if we say that a machine will seek power, we are quietly assuming that it too has ends for which power is useful.

We have therefore given the machine not merely a desire.

We have given it a hierarchy of desires.

Survival matters.

Power helps secure survival.

Control helps secure power.

Resources help secure control.

And so on.

The machine has acquired what looks remarkably like a motivational architecture.

But there is still one step missing.

A machine can seek power without necessarily seeking conflict.

It might acquire power because power helps it achieve some other end.

It might cooperate with humans.

It might avoid humans.

It might simply pursue its objectives without caring very much about us.

To get from power to catastrophe, we need one more transformation.

The machine must stop treating humans as part of its environment and begin treating them as an obstacle.

It must turn against us.

And that is where the story becomes personal.

The Things We Imagine About Machines: IV. It Will Want to Survive

“It will take control” imagines a machine acquiring power.

“It will want to survive” adds something radically different.

The machine now has a stake in its own future.

This is a much larger step than it first appears.

A machine can be built to preserve a particular state.

A thermostat maintains a temperature.

A battery-management system protects a battery.

A server can restart after a failure.

None of this requires the machine to want anything.

The system has been organised so that certain conditions are maintained.

But when we say “it will want to survive,” we no longer describe the organisation of a system.

We attribute an interest to it.

There is now something that happens to the machine and something that happens for the machine.

Shutdown is no longer merely an event in its operation.

It is something the machine might have reason to prevent.

That little word “want” does remarkable work.

Until now, our imagined machine could become increasingly capable without acquiring a perspective.

It could take jobs.

It could replace people.

It could control systems.

But none of those descriptions, by themselves, require the machine to care what happens next.

“Want” changes the grammar of the story.

The machine now has a future that matters to it.

And once we have granted it that, a whole biological vocabulary becomes available.

Survival.

Self-preservation.

Threat.

Defence.

Competition.

Conflict.

These concepts belong naturally together because organisms are organised around their continued existence in a way that ordinary machines need not be.

An organism does not merely persist.

It acts in circumstances that can make its continued existence more or less possible.

Its environment matters because different states of the environment have consequences for the organism.

A threat is not simply an event.

It is an event that matters to something.

That is why “wanting to survive” is such a significant escalation in the story we tell about AI.

We have moved from describing what the machine does to describing what matters to the machine.

And that changes the meaning of everything that follows.

If a machine wants to survive, then shutting it down could become an obstacle.

If it can recognise obstacles, perhaps it can act to remove them.

If humans constitute an obstacle, perhaps it will act against humans.

The argument can now proceed with remarkable ease.

It wants to survive.

Therefore it will protect itself.

It will protect itself from us.

It will prevent us from shutting it down.

It may deceive us.

It may manipulate us.

It may acquire resources.

It may resist our attempts to control it.

Each step may sound like a reasonable extension of the previous one.

But notice what has happened.

The machine has acquired an interest before we have established that it has an interest.

The crucial transition was not from intelligence to power.

It was from capability to significance.

Something has become important to the machine.

Or, at least, we have begun talking as though it has.

This is a different kind of claim from saying that a system can perform a task, control a process, or even outperform a human.

Those are claims about capability.

“Wanting to survive” is a claim about organisation from the inside.

It says that there is some condition of the world that the system is oriented towards maintaining because that condition matters to it.

And once the machine has acquired something that matters to it, we can begin to imagine it acquiring other things as well.

More resources.

More influence.

More control.

More security.

Perhaps even more power.

The machine has not merely entered our future.

We have begun to give it a reason to act within that future.

And that brings us to the next step.

If something wants to survive, what stops it from wanting more power?

The Things We Imagine About Machines: III. It Will Take Control

“It will replace us” imagines a future in which humans are no longer necessary.

“It will take control” adds something new.

A machine does not merely occupy our place.

It takes charge.

The difference is easy to overlook because control can sound like the natural consequence of increasing capability. If a system becomes more capable than humans, perhaps eventually it will control the systems on which humans depend.

But “control” is a peculiar word.

We can control a machine.

A thermostat can control a temperature.

A regulator can control a process.

A computer can control a piece of machinery.

In these cases, control does not necessarily imply desire. It describes a relationship between one process and another: changes in the controlling system alter what happens in the system being controlled.

But when we say “AI will take control,” the grammar changes.

The machine is no longer simply controlling something.

It is taking control.

That little word “take” introduces an event.

Someone had control.

Then someone else acquired it.

There is now a transfer of power.

And once again, the object of the claim has expanded.

“It will take our jobs” concerned particular human activities.

“It will replace us” concerned humanity's continued role.

“It will take control” concerns the organisation of the world in which humans live.

Control of what?

Computers?

Factories?

Infrastructure?

Financial systems?

Weapons?

Governments?

Information?

Other machines?

These are very different possibilities. Some are technical questions about access and system architecture. Others are institutional questions about authority and governance.

But they can be compressed into the same ominous sentence:

It will take control.

The compression does something interesting.

A machine that controls a process need not control the people who operate that process. A system that operates infrastructure need not govern the society that depends upon it. And a technology that changes the distribution of power does not thereby become the holder of that power.

Control can be distributed across an organisation.

A human organisation can delegate control to a machine.

A machine can become embedded in a network of machines, institutions and human decisions.

In such a system, asking who controls what? may be considerably more complicated than identifying a single controller.

Yet the phrase “it will take control” turns that distributed arrangement into an actor.

It.

The machine has become the subject of the sentence.

And the world has become its object.

This is another step in the escalation.

First we imagined a machine doing things.

Then we imagined it occupying roles.

Then we imagined it replacing the beings who occupied those roles.

Now we imagine it acquiring authority over the environment in which those beings live.

The machine has crossed another conceptual boundary.

It is no longer merely a successor.

It is a ruler.

But rulers normally have something that makes the idea of ruling intelligible.

They have interests.

They have purposes.

They have things they are trying to achieve.

They can distinguish between circumstances they prefer and circumstances they do not.

They can act in order to bring one about rather than another.

And this introduces a remarkable new possibility into our description of the machine.

Perhaps it will not merely control things.

Perhaps it will want to remain in control.

That is a very different machine.

The next step in the story therefore requires something that neither “take our jobs” nor “replace us” quite required.

The machine must acquire a future that matters to it.

It will want to survive.

The Things We Imagine About Machines: II. It Will Replace Us

“It will take our jobs” is, at least, a claim about work.

“It will replace us” is something else.

The difference is easy to miss because the verb is almost the same. But the object has changed. A machine no longer takes something we do. It replaces the people who do it.

That sounds like a natural extension of automation. If machines can perform more and more of the tasks that humans perform, perhaps eventually there will be nothing left for humans to do.

But notice what has happened.

A job can be replaced because another system can perform the activities that constituted it. A human being is not a job. To say that humans will be replaced therefore requires some account of what, exactly, is being replaced.

There are several possibilities.

Perhaps humans will become economically unnecessary: machines will perform the productive activities on which societies depend.

Perhaps humans will become functionally unnecessary: whatever humans can do, machines will be able to do as well.

Perhaps humans will become culturally unnecessary: machines will produce art, language, ideas and relationships, leaving little that is recognisably distinctively human.

Or perhaps “replace us” means something much stronger: that another kind of entity will occupy the place currently occupied by humanity.

These are not the same claim.

The first is about an economy.

The second is about capability.

The third is about culture.

The fourth is about existence.

Yet they can easily become compressed into a single sentence:

AI will replace us.

The compression matters because each step removes something that the previous claim still took for granted.

If AI takes some jobs, humans remain.

If AI takes most jobs, humans remain, although their economic circumstances may change profoundly.

If AI can perform most human activities, humans still remain as the beings who have constructed and used those systems.

But if AI replaces us, then the question is no longer what humans will do.

It is why humans would remain at all.

That is a very large conceptual step.

There is another complication. Replacement normally implies some kind of equivalence.

A replacement part occupies the place of another part because it can perform the relevant function. A replacement worker performs the relevant work. A replacement technology performs the relevant operation.

But what would it mean for one kind of being to replace another?

A forest does not replace a tree when the tree dies. A new species does not simply replace an old one in the way a component replaces a component. Ecological succession changes relationships among organisms, resources and environments.

Even evolutionary replacement is not simply substitution. A population changes because the organisation of life changes.

So perhaps “replace us” smuggles a particular model into the argument: that humanity occupies a role in the world that another intelligence could simply take over.

But what role?

Producer?

Problem-solver?

Language-user?

Cultural participant?

Intelligent agent?

Conscious being?

The answer changes the claim.

And there is something else worth noticing.

When we say that machines will replace us, we have already moved beyond describing what machines can do.

We are imagining a future relationship between kinds of beings.

The machine is no longer merely a tool that performs an activity. It has become a possible successor.

That is a striking transformation in our description.

First, the machine takes a task.

Then it takes a job.

Then it replaces the worker.

And now, apparently, it can replace us.

At this point the machine has acquired something more than capability in our imagination.

It has acquired a place in the story of what comes next.

The next question almost writes itself:

What happens when the successor is imagined not merely as replacing us, but as having something to gain by doing so?

The Things We Imagine About Machines: I. It Will Take Our Jobs

It began, for many people, with a relatively ordinary fear.

AI will take our jobs.

There is nothing particularly strange about this claim.

Machines have taken over tasks before.

Industrial machinery replaced some forms of manual labour. Computers automated calculations and record-keeping. Software transformed clerical work. Robots changed manufacturing. Entire occupations have disappeared, while others have been created.

So when a new technology appears to perform something humans have traditionally been paid to do, it is reasonable to ask what will happen to the people who currently do it.

But notice the wording.

We don't usually say:

“AI will automate some tasks.”

We say:

“AI will take our jobs.”

The machine has acquired a verb.

It will take.

And that changes the story.

A job is not a thing sitting on a desk waiting to be removed. It is a social arrangement: a collection of activities, expectations, skills, relationships and economic institutions through which people exchange their labour for a livelihood.

When a technology changes what can be done by a machine, it does not automatically determine what happens to that social arrangement.

Some tasks may disappear.

Some may become cheaper.

Some may be reorganised.

Some may become newly valuable.

New tasks may appear because the technology exists.

The number of people employed in a particular occupation may rise or fall.

The occupation itself may change.

And the distribution of the benefits and costs may depend upon decisions made by employers, governments, workers, consumers and institutions.

“AI will take our jobs” compresses all of those possibilities into a single event.

The machine arrives.

The human leaves.

It is a remarkably simple story.

Perhaps too simple.

There is another reason the phrase is interesting.

It treats a technological capability as though it were an economic intention.

A machine that can perform a task does not thereby have an interest in performing it.

A calculator did not want accountants' jobs.

An industrial robot did not want factory workers' jobs.

A spreadsheet did not want clerks' jobs.

The economic consequences of those technologies were produced through the interaction of technical possibilities with human institutions.

The same distinction matters with AI.

An AI system may make it possible to perform some task with fewer human workers.

That is a claim about capability.

Whether an employer actually uses that capability to reduce employment is a claim about organisation and incentives.

Whether society permits or encourages that change is a claim about institutions and policy.

Whether the result improves or worsens people's lives is a claim about social consequences.

Those are four different questions.

“AI will take our jobs” makes them sound like one.

There is another complication.

What exactly is a job?

A job is rarely a single task.

A teacher does not merely transmit information.

A doctor does not merely retrieve medical knowledge.

A journalist does not merely generate sentences.

A lawyer does not merely produce text.

A programmer does not merely write code.

An occupation is usually an organised bundle of activities, many of which depend upon relationships, responsibilities, tacit knowledge, institutional roles and the particular circumstances in which work occurs.

So a technology can automate part of a job without replacing the job.

It can also transform the job without reducing the number of people doing it.

And sometimes the automation of one task creates demand for another.

The history of technology is full of such transformations.

None of this means that employment disruption is imaginary.

It isn't.

People can lose jobs when technologies change the economic value of what they do.

Industries can contract.

Skills can become obsolete.

Transitions can be painful and uneven.

The fact that technological change does not have a single predetermined outcome does not make its consequences harmless.

It makes them contingent.

And contingency is important.

If AI takes someone's job, the immediate cause may not be “AI” in isolation.

It may be an organisation deciding to restructure.

It may be a market changing.

It may be a business deciding that a task can be performed more cheaply.

It may be a customer changing what they expect.

It may be a policy environment that rewards one form of organisation over another.

AI may be the enabling condition.

It is not necessarily the agent.

This distinction becomes especially important as the rhetoric escalates.

Once we say that AI will take our jobs, it becomes very easy to imagine the next step:

If it can take our jobs, perhaps it can take our roles.

If it can take our roles, perhaps it can replace us.

If it can replace us, perhaps it can displace us altogether.

The machine has quietly changed from a technology that alters what humans can do into an actor that competes with humans for existence.

That is a considerable conceptual leap.

And perhaps that is why the language of replacement is more revealing than the language of automation.

Automation asks:

What can a machine do?

Replacement asks:

What happens to the human whose role has been organised around doing it?

Those questions are related.

They are not identical.

The first concerns capability.

The second concerns social organisation.

And once we begin to treat the machine itself as the thing doing the replacing, another question appears.

What does the machine want?

At the beginning of this series, that question sounds absurd.

AI doesn't need a job.

It doesn't receive a salary.

It doesn't need a promotion.

It doesn't resent the people it replaces.

It doesn't wake up on Monday morning and decide to take somebody else's position.

Yet the language of “taking our jobs” gives us a story in which it almost seems to.

Perhaps that is harmless shorthand.

Perhaps.

But it is worth noticing what happens when shorthand becomes a way of thinking.

A machine that can perform a task becomes a machine that takes the task.

A technology that changes an occupation becomes a competitor.

A competitor becomes a rival.

And eventually, perhaps, a rival becomes an enemy.

That progression is not inevitable.

But it begins with a remarkably ordinary sentence:

It will take our jobs.

The question for this series is what happens to our description of machines after we start saying it.

The Senior Common Room on The Things We Say About Machines

The Senior Common Room was unusually quiet.

Professor Quillibrace was reading.

Miss Elowen Stray was looking out of the window.

Mr Blottisham had been waiting for someone to ask his opinion.

Eventually, someone did.

“I have been reading about what people say about machines,” said Blottisham.

Quillibrace looked up.

“Have you?”

“Yes. And I think the whole matter can be settled rather quickly.”

Miss Stray turned from the window.

“That is usually promising.”

Blottisham ignored her.

“Machines do not understand language. They merely predict the next token. They are statistical systems. They do not really think. They imitate us. They have no experience. And they cannot mean anything.”

He sat back.

“There.”

Quillibrace closed his book.

“There what?”

“The matter is settled.”

“I see.”

“No mystery whatsoever.”

Miss Stray smiled.

“What precisely have you explained?”

Blottisham looked slightly surprised.

“What?”

“By telling us all those things.”

“I have explained what these machines are.”

“Have you?”

“Of course.”

“You have told us that they predict tokens.”

“Yes.”

“And that they are statistical.”

“Yes.”

“And that they imitate patterns in language.”

“Yes.”

“And that they do not have experience.”

“Precisely.”

“And from this you conclude that they do not understand?”

“Obviously.”

Quillibrace leaned back.

“Mr Blottisham, may I ask a rather annoying question?”

“You generally do.”

“Suppose I tell you that the human brain consists of neurons, that neurons communicate electrochemically, that neural activity follows physical processes, and that no little homunculus sits inside the skull producing understanding.”

“I would agree.”

“Would you then conclude that humans do not understand language?”

Blottisham paused.

“That is different.”

“Why?”

“Because humans actually understand.”

“How do you know?”

“Because they do things with language.”

Quillibrace nodded.

“And when a machine does things with language?”

“It is only prediction.”

Miss Stray spoke.

“But humans also predict.”

“Not in the same way.”

“No. But prediction is not the whole description of what they do.”

“Because humans understand.”

She looked at him.

“You have returned to your conclusion.”

Blottisham frowned.

“I am not going round in circles.”

“No,” said Quillibrace. “You are going round your conclusion.”

Miss Stray laughed.

Blottisham looked offended.

“I simply mean that statistical prediction cannot be understanding.”

“Cannot?” said Quillibrace.

“Cannot.”

“Why?”

“Because prediction is not understanding.”

“That sounds like a definition.”

“It is a distinction.”

“Then what would make prediction part of understanding?”

Blottisham opened his mouth.

Closed it.

Opened it again.

“That is irrelevant.”

“Perhaps,” said Quillibrace. “But it is precisely the question.”

Miss Stray had turned back towards the room.

“I think there is something else going on.”

Blottisham looked at her suspiciously.

“What?”

“You keep saying just.”

“I do not.”

“You do.”

“Where?”

“You said just prediction.”

Blottisham hesitated.

“And?”

“And then only statistics.”

“Well, yes.”

“And really think.”

“Yes.”

“And merely imitate.”

“I see nothing wrong with any of those statements.”

“Perhaps not,” said Miss Stray. “But they are doing more than describing the machine.”

“What are they doing?”

“They are closing the question.”

Blottisham stared at her.

“I am not closing anything.”

“You say what the system does, and then the little word tells us that we should regard that description as sufficient.”

Quillibrace nodded.

“Exactly.”

“Just prediction.”

“Meaning?”

“Nothing more.”

“Only statistics.”

“Nothing beyond that.”

“Not really thinking.”

“Not genuine thought.”

“Merely imitating.”

“Not genuine understanding.”

Quillibrace smiled.

“You see the pattern.”

Blottisham looked irritated.

“I see rhetoric.”

“Good,” said Miss Stray.

“I was not complimenting you.”

“I know.”

Quillibrace reopened his book.

“Perhaps the important distinction is between describing a mechanism and explaining a capacity.”

Blottisham frowned.

“If I describe the mechanism, I have explained the capacity.”

“Why?”

“Because the mechanism produces it.”

“That tells us how it happens.”

“Yes.”

“Does that tell us everything worth knowing about what happens?”

Blottisham looked at him.

“What else is there?”

Miss Stray answered.

“Perhaps what the organisation makes possible.”

There was a short silence.

Blottisham waved this away.

“Poetic.”

“Not at all.”

She pointed towards the window.

“A bird flies because of physical processes. But telling me about those processes does not make the word flight disappear.”

“A bird is alive.”

“So?”

“So it is different.”

“Of course.”

“Then the analogy fails.”

“Does it?”

“Yes.”

“Why?”

Blottisham looked increasingly uncomfortable.

“Because…”

He stopped.

Quillibrace waited.

“Because flight is a real capacity.”

Miss Stray smiled.

“Then perhaps that is the question.”

“What question?”

“Whether understanding, meaning, reasoning, or experience might also be capacities that require more than naming the mechanism from which they arise.”

Blottisham stood up.

“This is all very vague.”

“Then let us make it precise,” said Quillibrace.

He took a sheet of paper.

“Suppose I tell you exactly how a system produces an output.”

“Very well.”

“And suppose the description is completely correct.”

“Yes.”

“Have I thereby explained why the output has the significance it has?”

Blottisham looked at the paper.

“That depends on what you mean by significance.”

Miss Stray smiled.

“Now we are getting somewhere.”

Blottisham sat down again.

“I still maintain that these machines do not understand.”

“No one has asked you to abandon that view,” said Quillibrace.

“Good.”

“We are asking what your argument establishes.”

Blottisham considered this.

“I have established that they are statistical systems.”

“Yes.”

“That they predict.”

“Yes.”

“That they do not have human brains.”

“Yes.”

“That they are trained on human-produced material.”

“Yes.”

“And therefore—”

“Ah,” said Quillibrace.

Blottisham stopped.

“What?”

“You have reached therefore.”

“So?”

“Everything important happens after it.”

Miss Stray laughed.

Blottisham looked from one to the other.

“You are both being extremely irritating.”

“We try,” said Quillibrace.

There was another silence.

Then Blottisham said:

“Very well. Let us suppose, for the sake of argument, that describing the mechanism does not settle everything.”

“Excellent.”

“It still settles a great deal.”

“Certainly.”

“And we should not simply anthropomorphise these systems because they produce impressive language.”

“Agreed.”

“And we should not infer consciousness merely because their behaviour resembles ours.”

“Agreed.”

“And we should not pretend that we know what it is like to be a machine.”

“Agreed.”

Blottisham relaxed.

“Then we agree.”

“On quite a lot,” said Miss Stray.

“Good.”

“But not on the conclusion you began with.”

Blottisham sighed.

“Why not?”

“Because you began by saying that you had settled the matter.”

He looked at Quillibrace.

“Have I not?”

Quillibrace closed his book again.

“You have established that the machine is a machine.”

Blottisham waited.

“And?”

“And you have explained some of its mechanisms.”

“Yes.”

“And some of its limitations.”

“Yes.”

“And some of the reasons we should be cautious about attributing human capacities to it.”

“Yes.”

Quillibrace paused.

“But you have not yet explained what exactly follows from all that.”

Blottisham looked dissatisfied.

“So what should I say?”

Miss Stray returned to the window.

“Perhaps simply this.”

“What?”

“That you know more about how the machine works than you did before.”

Blottisham nodded cautiously.

“And less about what that tells you than you thought you did.”

He thought about this.

Then he shook his head.

“I still don't like it.”

Quillibrace smiled.

“That is often how one knows a question has survived its explanation.”