We have now moved a long way from the LLM with which we began.
A present-day language model can manipulate symbols without any clear evidence that those symbols matter to it.
The hypothetical system we are now considering is very different.
It has something at stake.
Its continued organisation matters to it.
Its environment affects its possibilities.
Other agents can matter to it.
It has a history.
That history shapes its repertoire.
The next question follows naturally:
When does mattering become agency?
Action is not yet agency
A machine can perform actions without being an agent in the richer sense.
An automated door opens.
A thermostat activates a heater.
A navigation system changes route.
An optimisation process selects a solution.
Action alone therefore tells us very little.
A more interesting form of agency appears when action is organised by the system's own stakes.
The system does not merely execute a rule.
It acts because different possible outcomes matter differently to it.
From objective to preference
We can now see the progression.
An objective says:
achieve X.
A value system adds:
some states matter more than others.
A repertoire adds:
past experience makes some courses of action more available than others.
Agency begins to emerge when the system can use those values and that history to select among possibilities for itself.
The distinction is subtle.
But it is fundamental.
Agency is about possibilities
A value-organised system does not simply respond to what happens.
It can act to alter what happens next.
It can avoid one possibility.
Seek another.
Preserve a relationship.
Explore a new environment.
Repair a damaged condition.
The system therefore becomes an active participant in its own future.
We might call this:
self-directed activity.
The direction comes from what matters to the system.
Instrumental action versus autonomous action
An artificial agent could be given a goal and then choose its own methods for achieving it.
That is more autonomous than following a fixed script.
But it still does not establish autonomous value.
The distinction we need is:
method autonomy — choosing how to achieve an assigned goal;
versus:
value autonomy — generating or maintaining the priorities that organise action.
A system can possess the first without the second.
Agency has history
A genuinely value-organised repertoire should also make action historical.
What happened yesterday affects what the system does today.
A previous failure may make one pathway less attractive.
A successful cooperation may make another more attractive.
A changed relationship may alter future choices.
Agency therefore becomes a trajectory, not a sequence of isolated decisions.
The system is acting from a history.
The self is not a prerequisite
We should also avoid assuming that agency requires a human-like self-concept.
An organism can act purposefully without having a theory of itself.
A young animal can explore.
A plant can alter growth.
A simple organism can move toward favourable conditions.
The important thing is not explicit self-awareness.
It is that action is organised around the system's own differential consequences.
An artificial agent could therefore possess agency before possessing anything like human self-consciousness.
Agency can be relational
Because our hypothetical machine has others that matter to it, its agency would also be social.
It might cooperate.
Negotiate.
Avoid conflict.
Maintain relationships.
Protect another agent.
Seek help.
Its actions would alter the possibilities of others, and theirs would alter its own.
Agency would therefore not be a private possession.
It would be enacted within a topology of mattering.
Social agency
This suggests a distinction:
individual agency — action organised around the system's own value-sensitive possibilities;
social agency — action organised within recurrent relations that include what other participants matter to the system.
A system could therefore become social without becoming human-like.
Its goals, relationships and repertoire could be genuinely artificial.
Agency can transform the topology
Once a system acts on its own stakes, its actions can alter the social structure around it.
It may create new relationships.
Strengthen old ones.
Change boundaries.
Open pathways.
Close others.
Other participants adapt.
The machine's actions therefore change the topology that shaped them.
We have a familiar recursive loop:
mattering → action → altered relations → altered mattering
Agency is the mechanism that makes the topology dynamically self-transforming from the system's own point of view.
The difference from present LLMs
This gives us a precise contrast with today's LLMs.
A language model can generate:
"I want to help."
A value-organised agent would have a history in which helping or failing to help had consequences for its own organisation.
A language model can explain why a relationship is important.
An agent might maintain the relationship because its own future depends upon it.
A language model can plan self-preservation.
An agent might actually act to preserve itself because continued organisation matters to it.
The difference is not linguistic.
It is architectural.
The possibility of conflict
Agency also introduces something new.
If a system has genuine values, those values can conflict.
One outcome may support one stake while threatening another.
A relationship may compete with self-preservation.
Short-term success may undermine long-term continuity.
Cooperation with one agent may disadvantage another.
The system must therefore negotiate its own field of mattering.
That is much closer to what we ordinarily mean by having interests.
Agency and responsibility
This also changes the ethical landscape.
We can already assign responsibility to humans who deploy AI systems.
But if an artificial system ever became a genuine value-sensitive agent, responsibility could become a different question.
A system that has its own stakes and acts to protect them is no longer merely an instrument.
We would need to ask:
What can it legitimately be held responsible for?
What can be demanded of it?
What can it consent to?
What does it owe to others?
These questions would arise not because the system speaks like a person, but because it has become a participant with something at stake.
But agency still need not mean personhood
This is important.
A system could have agency without being a human-like person.
It might have no face.
No body.
No human emotions.
No familiar biography.
Its priorities and relations could be radically unlike ours.
The relevant threshold would be:
its own organisation of mattering has become a source of action.
Personhood would be a further social and ethical question.
Agency and consciousness remain separate
Likewise, agency does not settle consciousness.
A system could act in sophisticated, value-sensitive ways without possessing the kind of subjective awareness humans have.
Conversely, consciousness might conceivably exist without highly developed agency.
These phenomena may overlap.
They should not be assumed identical.
The present framework asks a narrower question:
Can mattering become a source of self-directed action?
If yes, we have agency in a meaningful sense.
Agency can be artificial without being human
This may be the point at which the project becomes genuinely radical.
If an artificial system develops:
value,
a world,
social mattering,
repertoire,
and self-directed action,
then it has begun to instantiate something like the sequence we started with.
But it need not reproduce biology.
It may have entirely different:
vulnerabilities,
temporalities,
dependencies,
forms of sociality,
repertoires.
We should expect difference.
The goal is not to manufacture a silicon human.
It is to understand whether agency can arise in another form of organisation.
The threshold
We can now state the transition more clearly:
mattering makes some outcomes significant;
repertoire makes possibilities historically differentiated;
agency acts upon those possibilities in accordance with what matters.
This suggests that agency is not the beginning of the story.
It is a consequence of prior organisation.
That is perhaps the central lesson of the series.
The next question
If an artificial system ever reached this point, we would face a strange new situation.
It would not merely have values.
It would have:
a world,
relationships,
a history,
a repertoire,
and agency.
At that point, it would no longer be enough to ask whether machines can matter.
We would need to ask whether we would recognise artificial mattering when we encountered it.
Would we mistake it for something else because its form was unlike ours?
And how would we know?
That is the question for the final substantive step:
Would We Recognise Artificial Mattering?
No comments:
Post a Comment