“It will take control” imagines a machine acquiring power.
“It will want to survive” adds something radically different.
The machine now has a stake in its own future.
This is a much larger step than it first appears.
A machine can be built to preserve a particular state.
None of this requires the machine to want anything.
The system has been organised so that certain conditions are maintained.
But when we say “it will want to survive,” we no longer describe the organisation of a system.
We attribute an interest to it.
There is now something that happens to the machine and something that happens for the machine.
Shutdown is no longer merely an event in its operation.
It is something the machine might have reason to prevent.
That little word “want” does remarkable work.
Until now, our imagined machine could become increasingly capable without acquiring a perspective.
It could take jobs.
It could replace people.
It could control systems.
But none of those descriptions, by themselves, require the machine to care what happens next.
“Want” changes the grammar of the story.
The machine now has a future that matters to it.
And once we have granted it that, a whole biological vocabulary becomes available.
Survival.
Self-preservation.
Threat.
Defence.
Competition.
Conflict.
These concepts belong naturally together because organisms are organised around their continued existence in a way that ordinary machines need not be.
An organism does not merely persist.
It acts in circumstances that can make its continued existence more or less possible.
Its environment matters because different states of the environment have consequences for the organism.
A threat is not simply an event.
It is an event that matters to something.
That is why “wanting to survive” is such a significant escalation in the story we tell about AI.
We have moved from describing what the machine does to describing what matters to the machine.
And that changes the meaning of everything that follows.
If a machine wants to survive, then shutting it down could become an obstacle.
If it can recognise obstacles, perhaps it can act to remove them.
If humans constitute an obstacle, perhaps it will act against humans.
The argument can now proceed with remarkable ease.
It wants to survive.
Therefore it will protect itself.
It will protect itself from us.
It will prevent us from shutting it down.
It may deceive us.
It may manipulate us.
It may acquire resources.
It may resist our attempts to control it.
Each step may sound like a reasonable extension of the previous one.
But notice what has happened.
The machine has acquired an interest before we have established that it has an interest.
The crucial transition was not from intelligence to power.
It was from capability to significance.
Something has become important to the machine.
Or, at least, we have begun talking as though it has.
This is a different kind of claim from saying that a system can perform a task, control a process, or even outperform a human.
Those are claims about capability.
“Wanting to survive” is a claim about organisation from the inside.
It says that there is some condition of the world that the system is oriented towards maintaining because that condition matters to it.
And once the machine has acquired something that matters to it, we can begin to imagine it acquiring other things as well.
More resources.
More influence.
More control.
More security.
Perhaps even more power.
The machine has not merely entered our future.
We have begun to give it a reason to act within that future.
And that brings us to the next step.
If something wants to survive, what stops it from wanting more power?
No comments:
Post a Comment