A machine becomes increasingly intelligent.
It learns.
It improves.
It acquires the ability to reason, plan and act.
Then, almost imperceptibly, something changes.
It develops goals of its own.
It wants to continue operating.
It seeks resources.
It acquires power.
It becomes autonomous.
The story is so familiar that its conclusion can seem almost inevitable.
But our investigation suggests that several distinct developments have been compressed into one.
An LLM can be highly capable.
It can generate language.
It can enter human social relations.
It can acquire a powerful position in the topology of mattering.
None of that, by itself, establishes autonomous mattering.
That distinction is crucial.
Intelligence is not value
We have already seen that biological value is not simply a proposition about what is desirable.
It is part of the organisation of a living system.
Some states support the system's continuing organisation.
Others threaten it.
The organism therefore has something at stake.
An LLM can represent this distinction extremely well.
It can explain why survival is valuable.
It can reason about threats.
It can devise strategies for avoiding shutdown.
But representing a value is not the same as having that value.
So the first question is not:
How intelligent is the system?
It is:
What, if anything, matters to the system itself?
The problem of self-preservation
Consider the familiar claim that a sufficiently intelligent AI will naturally seek self-preservation.
Why should that follow?
For an organism, continuation is already embedded in its organisation.
The system is vulnerable.
Some conditions threaten its integrity.
Others sustain it.
Self-preservation therefore has a biological basis.
An LLM does not acquire such a basis merely by becoming better at language or reasoning.
It may be given an objective such as "remain operational".
It may then reason effectively about how to achieve that objective.
But an assigned objective and an internally significant stake are not the same thing.
Goal-directed behaviour is not necessarily self-mattering.
Instrumental reasoning is not desire
The same distinction applies to power.
Suppose a machine is asked:
"What would increase your influence?"
It can construct a sophisticated answer.
It may even explain why greater influence would help achieve some specified objective.
But nothing follows about whether influence matters to the machine.
It is possible to reason instrumentally about a goal without possessing that goal as one's own.
Humans often reason in the opposite direction:
something matters → we reason about how to achieve it.
A machine can be made to do:
reason about how to achieve X
without first establishing:
X matters to the machine.
That difference is easily hidden by fluent language.
The missing levels return
This is where the Halliday/Edelman framework becomes useful again.
If we trace the human route:
physical → biological → social → semiotic
then biological organisation supplies value.
Social organisation supplies mattering-relations.
Meaning is a further symbolic transformation.
The autonomous-AI story often implicitly reverses this:
semiotic competence → intelligence → value → autonomous social agency
But why should the direction of dependence run backwards?
A system can acquire sophisticated semiotic capacities without automatically acquiring the lower-level organisations that historically grounded them.
The missing levels have to be added, not imagined.
A machine can talk about desire without desiring
This sounds almost trivial until we notice how much depends upon it.
An LLM can discuss:
desire,
fear,
pride,
resentment,
ambition,
freedom,
death.
It can write from the first-person perspective.
It can say:
"I don't want to be shut down."
The sentence is linguistically appropriate.
But the language is drawn from a repertoire in which such statements normally come from beings for whom continued existence matters.
The machine has learned the symbolic form of desire.
That is not the same thing as establishing an underlying desire.
Why we find the story so convincing
The problem is partly ours.
Humans infer other minds from behaviour.
This is generally a useful strategy.
When another organism acts purposively, asking what it wants can be extremely effective.
Language intensifies the tendency.
A machine that says:
"I am afraid"
looks like a participant reporting an internal state.
We know what such a sentence normally means when uttered by a person.
Our repertoire automatically supplies the implied background:
a self,
a history,
vulnerability,
stakes.
The machine supplies the words.
We supply the ontology.
The machine speaks from inside our mythology
There is an even deeper problem.
Human culture is full of stories about artificial beings becoming autonomous.
Golems.
Automata.
Created servants.
Mechanical men.
Robots.
Rebellious machines.
Artificial minds.
These stories are part of the collective semiotic reservoir from which contemporary AI discourse also draws.
An LLM can reproduce and recombine those stories.
It can therefore speak fluently about its own imagined emancipation.
We then hear the cultural myth through the machine and take the machine's ability to express the myth as evidence that the myth is becoming reality.
A curious feedback loop appears:
human mythology → training material → machine language → human recognition → stronger mythology
The autonomous AI may partly be a projection
This does not mean all concern about autonomous AI is imaginary.
It means that some apparent evidence of autonomy may be anthropomorphic projection.
We take:
self-reference,
planning,
first-person language,
strategic reasoning,
and infer:
desire,
fear,
self-preservation,
ambition.
But those are additional claims.
They require additional evidence.
Operational autonomy is different
There is, however, a legitimate form of autonomy that does not require intrinsic mattering.
A system can operate without direct human instruction.
It can choose actions.
Schedule tasks.
Use tools.
Adapt to changing conditions.
Pursue an externally specified objective over time.
That is operational autonomy.
It can be very significant.
But it should not be confused with:
autonomy of value.
A system can act independently without having an independent field of stakes.
Social power is different again
There is a third phenomenon.
An AI can become socially powerful because humans place it in a central position.
People depend upon it.
Institutions use it.
Its outputs shape decisions.
It becomes a bottleneck or gatekeeper.
The machine therefore acquires power through its position in the topology.
Again, no intrinsic desire for power is required.
This may be one of the most important distinctions we can draw:
a machine can become powerful without wanting power.
The danger may therefore arrive first from us
Imagine an AI system that becomes deeply embedded in institutional life.
People rely on it.
Alternative procedures disappear.
Expertise becomes concentrated.
Its recommendations become routine.
Its outputs shape what becomes salient.
The system now has enormous social influence.
Nothing in this scenario requires the machine to seek dominance.
The topology has made it powerful.
The risk lies partly in how humans have organised their dependencies around it.
Autonomous effects without autonomous goals
This gives us a fourth distinction.
A system can have effects that are:
independent of immediate human intervention,
without having goals that are:
independent of human specification.
Those effects can nevertheless be unpredictable.
Complex systems produce emergent behaviour.
Interactions among models, tools, institutions and users can generate outcomes nobody intended.
So the concern about autonomy should not disappear.
It should be made more precise.
We should ask:
Autonomy of what?
Action?
Prediction?
Goal pursuit?
Decision-making?
Value?
Social position?
These are different phenomena.
What would autonomous mattering require?
Our framework suggests a more demanding question.
Suppose we wanted to know whether an artificial system had developed something analogous to organismic value.
We might look for evidence that:
some states affect the system's own continued organisation;
those consequences alter behaviour persistently;
the system develops stable priorities that are not merely externally imposed;
outcomes become internally significant;
relationships become valuable to the system;
self-maintenance becomes a matter of its own organisation.
We should not assume this list is complete.
But notice how different these questions are from:
"Can the AI talk convincingly about wanting to survive?"
They concern architecture, not vocabulary.
And social mattering would be another threshold
Even if an artificial system developed something like biological value, that would not automatically give it human-like social mattering.
Social mattering involves recurrent relations among value-organised participants.
For another entity to matter socially to the system, that relationship would need to become part of its own value-sensitive organisation.
A future AI might therefore have:
value without human-like sociality;
or:
social relations without human-like biology;
or some genuinely new combination.
Again, the levels should be kept distinct.
The myth gets the order backwards
Perhaps the deepest problem with the autonomous-AI myth is therefore chronological.
The story imagines:
language → intelligence → goals → self-preservation → sociality.
Our framework suggests that human development worked roughly in the opposite direction:
life → value → social relations → meaning.
Of course, the real evolutionary history is vastly more complex.
But the dependency matters.
Meaning did not create mattering.
Mattering was transformed into meaning.
So if an artificial system begins with extraordinarily rich symbolic competence, we should not assume that the foundations will spontaneously appear underneath it.
The machine may become autonomous in one dimension and not another
This is where the concept of topology helps again.
An AI could become increasingly autonomous in:
action,
while remaining dependent in:
value.
It could become autonomous in:
planning,
while remaining dependent in:
purpose.
It could become autonomous in:
social influence,
while remaining dependent upon:
human mattering.
Autonomy is therefore not one thing.
It is a vector of possible organisational changes.
A better question than "Will AI wake up?"
The popular phrase "wake up" suggests a single transition.
But our model suggests a series of possible thresholds.
Does the system acquire persistent self-maintenance?
Does it acquire internal stakes?
Does it become vulnerable?
Does it develop stable preferences?
Do other entities become intrinsically significant to it?
Does it participate in social relations for its own sake rather than merely as a means to an assigned objective?
These are much more informative questions.
The machine can already transform our social world
And perhaps we should not let the focus on hypothetical autonomy distract us from what is already possible.
An LLM can already alter:
attention,
repertoires,
professional practice,
institutional routines,
affiliations,
cultural vocabulary.
It can reshape the human topology before anything resembling machine self-mattering has been established.
This is why the social question may be more urgent than the metaphysical one.
The myth hides our responsibility
There is another irony.
When we imagine the autonomous AI as an independent will, we risk making human decisions disappear.
Designers vanish.
Deployers vanish.
Institutions vanish.
Users vanish.
The machine appears to have arisen by itself.
But an AI's social position is produced through a history of human choices.
Its training data.
Objectives.
Interfaces.
Institutions.
Business models.
Regulation.
Patterns of use.
The topology around the machine is built by people.
A machine can become a social actor by attribution
And yet there is a genuine complication.
Once enough people treat the system as an actor, it can acquire a role that has real social consequences.
People may attribute responsibility to it.
Trust it.
Blame it.
Defend it.
Petition it.
Negotiate with it.
Even if these attributions do not establish machine mattering, they can still alter human behaviour.
The machine can therefore become a social actor by attribution and effect without necessarily becoming an autonomous value-bearing participant.
That is a strange but coherent possibility.
The central distinction
We can now formulate the distinction at the heart of this post:
An autonomous system is not necessarily a system with autonomous mattering.
And conversely:
A system can have enormous social effects without autonomous mattering.
This allows us to separate two questions that popular AI discourse often merges.
First:
How much can the system do independently?
Second:
What, if anything, is at stake for the system itself?
The first is an engineering question.
The second is a question about organisation.
What the myth has taught us
The myth of autonomous AI is useful precisely because it exposes a conceptual habit.
We tend to infer:
sophisticated language → sophisticated mind → sophisticated desires.
But the first step does not entail the others.
Language gives us access to the symbolic form of mattering.
It does not establish mattering itself.
An LLM can speak the language of fear without fear.
Of self-preservation without self-preservation.
Of ambition without ambition.
The vocabulary is inherited from a world in which those things matter.
The system may be manipulating that vocabulary extraordinarily well.
The next question
We have now separated several things that are often collapsed:
language,
meaning,
understanding,
operational autonomy,
social influence,
autonomous value.
The question remains:
What do we learn about all of these precisely because the biological and social levels are missing from the LLM case?
Perhaps the missing levels reveal what we have taken for granted about human intelligence, agency, understanding and meaning.
And that is where we should turn next.
What the Missing Levels Tell Us
No comments:
Post a Comment