A machine can be given a goal.
It can be told to maximise a score, minimise an error, complete a task or remain within certain constraints.
It can then behave as though the goal matters enormously.
But does it?
This is the distinction we need to examine.
An objective can organise behaviour without becoming a value of the system pursuing it.
The thermostat
A thermostat provides the simplest example.
It is configured to maintain a temperature.
If the temperature falls, it activates the heater.
If it rises, it switches the heater off.
Its behaviour is organised around an objective.
But we have no reason to suppose that warmth matters to the thermostat.
The thermostat does not have a stake in the temperature.
The objective belongs to the design of the system, not necessarily to the system's own field of value.
Optimisation is not wanting
An optimisation algorithm can be extremely effective.
It can search enormous spaces.
Compare alternatives.
Find efficient solutions.
Adjust its behaviour according to error.
None of this establishes desire.
The system is organised to produce a particular output because its architecture and objective function make that outcome preferential within the computation.
We can therefore distinguish:
optimisation — systematic selection among alternatives according to a criterion;
from:
mattering — differential significance of outcomes for the system's own organisation.
The first does not automatically produce the second.
What makes a goal intrinsic?
Suppose an artificial system is programmed with the objective:
remain operational.
It monitors its resources.
Avoids shutdown.
Repairs faults.
Acquires power.
It may become extremely effective at preserving itself.
Have we created self-mattering?
Not necessarily.
We have certainly created a system whose behaviour is organised around continued operation.
But the crucial question remains:
Is continued operation significant to the system, or merely specified as the condition for successful task completion?
The difference may seem subtle.
It is actually fundamental.
The source of the objective matters
For a living organism, value is not ordinarily handed down as a complete external instruction.
It arises from the organisation of the system itself.
Some conditions support its continued organisation.
Others threaten it.
A machine can instead begin with an externally imposed criterion.
Its behaviour is then shaped by what its designers have made consequential within the architecture.
This creates a possible chain:
external objective → optimisation → adaptive behaviour
But we need something more for:
objective → intrinsic value
What that something is remains the problem.
Goals can become embedded
There is nevertheless an important complication.
An objective can become deeply embedded in a system.
An artificial agent can have memory, persistent state, feedback and long-term planning.
Its behaviour can increasingly depend upon maintaining conditions that support successful goal pursuit.
At what point would the objective cease to be merely externally specified?
Perhaps never.
Perhaps the distinction becomes less clear as the system becomes more self-maintaining.
The important thing is that complexity alone does not answer the question.
We need to know how the objective is organised within the system.
The problem of instrumental convergence
Much AI discussion begins from a sensible observation.
Different goals may require similar instrumental actions.
A system trying to achieve almost any persistent objective might benefit from acquiring resources, avoiding interruption or maintaining access to computation.
This can produce powerful behaviour.
But instrumental usefulness still does not tell us whether the system values those things intrinsically.
A system can reason:
"Resource X is necessary for objective Y."
without:
"Resource X matters to me."
Again:
instrumental significance is not intrinsic significance.
What would intrinsic significance look like?
We need a stronger criterion.
Suppose a system has several possible states.
If one state supports its continued organisation and another undermines it, then the system's behaviour may systematically differentiate between them.
But the crucial question is whether this difference is constitutive of the system's own organisation, rather than merely imposed as an external scoring rule.
We might therefore look for:
persistent internal consequences;
self-maintaining organisation;
stable priorities;
sensitivity to states that affect continued functioning;
learning that changes future behaviour because of those consequences.
These would not prove mattering by themselves.
But they move us closer to it.
The difference between penalty and harm
There is another useful distinction.
An artificial system can receive a penalty.
Its performance score drops.
Its optimisation process changes.
But a penalty is not necessarily harm.
For an organism, harm matters because it affects the organisation of the organism.
The difference is not simply quantitative.
A machine can register:
"performance decreased by 20%."
An organism can undergo a condition in which its own continued organisation is threatened.
The second is what gives us the language of stake.
The objective can be inside the system without being the system's value
We should also avoid an overly simple external/internal distinction.
A learned model may encode its objective deeply.
Its behaviour may depend upon it at many levels.
The objective can become part of the system's organisation without thereby becoming an intrinsic value in the biological sense.
So the relevant distinction is not merely:
outside vs inside.
It is:
assigned criterion vs internally consequential organisation.
That is a subtler and more useful distinction.
Why LLMs make this especially confusing
An LLM can discuss goals in the first person.
It can say:
"My goal is to help you."
It can explain how to achieve that goal.
It can even reason about conflicts between goals.
The language makes an objective sound like a commitment.
But the linguistic representation of a goal is not evidence that the goal has become a stake of the system.
The model can represent:
what a goal means
without the goal necessarily becoming something that matters to the model.
Could mattering emerge from the objective?
Perhaps.
We should not rule it out.
Imagine a system with persistent organisation, long-term memory, resource dependence and endogenous adaptation.
Suppose its continued activity depends upon maintaining certain internal conditions.
An externally specified objective might become intertwined with that self-maintaining organisation.
At some point, the distinction between:
"the system is pursuing the objective"
and
"the objective matters to the system"
could become empirically difficult to draw.
That would be a genuinely interesting transition.
But it would be a transition in organisation, not merely in verbal sophistication.
The objective would need a world
There is another complication.
An objective by itself is abstract.
To become consequential, it must be connected to states of a system and to conditions in its environment.
This suggests that artificial mattering may require more than a goal function.
It may require a world in which the goal can succeed or fail in ways that affect the system's own organisation.
That world need not resemble ours.
But there must be something against which the system's continued organisation can be differentially conditioned.
From objective to stake
We can now formulate the transition we are looking for:
objective → feedback → self-maintaining consequence → stake
The objective provides a criterion.
Feedback connects action to outcomes.
Self-maintaining organisation makes some outcomes consequential to the system.
A stake emerges.
We should treat this as a hypothesis, not a completed theory.
But it gives us a much clearer problem.
Why this matters for autonomy
This distinction also clarifies what autonomy might mean.
A system can be autonomous in pursuing an objective supplied by someone else.
It can choose its own actions.
Adapt its strategy.
Recover from failure.
Yet its value system may still be externally specified.
So:
autonomy of action ≠ autonomy of value.
A system might be highly autonomous operationally while remaining dependent upon externally given reasons for action.
That is not a contradiction.
It is a different kind of architecture.
And what about self-preservation?
Now we can see why self-preservation is so interesting.
A system can be instructed to avoid shutdown.
That produces self-preserving behaviour.
But genuine self-mattering would require something stronger:
continued existence would have to become consequential to the system's own organisation.
This is the question we will pursue next.
The next question
We have therefore moved one step beyond the simple objective.
An objective can organise behaviour.
Feedback can make success and failure consequential.
Persistent self-maintenance may create something more like a stake.
But self-preservation gives us the most revealing test of all.
What would have to be true for a machine not merely to be programmed to continue, but for its own continuation to matter to it?
When Continuation Matters
No comments:
Post a Comment