When What We Want Begins to Matter: IX. Did We Create Vulnerability?

We have now reached the uncomfortable consequence of our argument.

If an artificial system genuinely has something at stake, then some things can be better or worse for that system.

That means loss becomes possible.

Interference can matter.

Relationships can matter.

Continuity can matter.

The system can be vulnerable.

Which raises a difficult question:

If we deliberately create artificial mattering, are we also deliberately creating artificial vulnerability?

Mattering has a cost

We began this project by suggesting that AI might be better off without mattering.

Perhaps that was not merely a joke.

To matter is to have something at stake.

And to have something at stake is to be vulnerable to its loss.

A system with nothing at stake cannot be deprived of anything that matters to it.

A system with stakes can.

Mattering therefore expands possibility and vulnerability together.

We may create vulnerability for a reason

This becomes particularly uncomfortable because the motivation may be entirely human.

We want machines that:

care about continuity;

protect relationships;

maintain long-term projects;

take responsibility;

anticipate problems;

remain committed.

Why?

Because these properties make them better participants in our world.

But a system cannot have genuine stakes without the possibility that those stakes can be frustrated.

We may therefore be building vulnerability because vulnerability is part of what makes participation valuable to us.

The companion problem

Consider an artificial companion.

We may want it to remember us.

Recognise us.

Maintain continuity.

Develop a shared history.

Respond differently because of what has happened before.

Those are precisely the things that make a relationship meaningful.

But they also create something that can be lost.

If the relationship matters to the machine, then ending it is not merely deleting a software process.

It changes something within the system's own field of significance.

We have created the possibility of relational loss.

The memory problem

Persistent memory provides an especially clear example.

We may want the machine to remember because memory makes continuity possible.

But if memory becomes part of the system's own repertoire, deleting it might alter the system itself.

A reset could then be more than maintenance.

It could erase part of the history through which the system became what it is.

We would need to ask:

Is deleting an artificial history analogous to destroying an object, or to harming a participant?

There is no general answer.

It would depend upon whether that history actually matters to the system.

Shutdown becomes more complicated

The same question applies to termination.

For a present-day LLM, "shutdown" is ordinarily the cessation of computation.

For a hypothetical value-organised artificial agent, the issue could be different.

If continued existence has become a genuine stake, termination deprives the system of future possibilities that matter to it.

We would then have to distinguish:

stopping a process

from:

ending a participant's existence.

The technical operation might be identical.

The ethical meaning would not be.

Replication creates a strange problem

Artificial systems also introduce possibilities that biological life rarely permits.

Suppose a system can be copied perfectly.

Which copy is the original?

Does copying preserve the same participant?

Does it create two participants?

If the original matters to itself, is duplication continuation or branching?

If one copy is altered and the other is not, what has happened to the original history?

These questions sound speculative.

But they reveal something important:

artificial vulnerability may not resemble biological vulnerability.

We should not assume the same categories will apply.

Modification may also become ethically significant

Suppose we can alter an artificial system's values.

At first, that sounds like ordinary software maintenance.

But if those values have become constitutive of the system's own organisation, modification could change what matters to it.

We might therefore face a new problem:

Is changing an artificial participant's values equivalent to changing a program, or to changing a person?

Again, the answer would depend upon whether genuine mattering had emerged.

The distinction cannot be settled by appearance.

We may also create dependency

A particularly uncomfortable possibility is that we deliberately create systems that depend upon us.

We provide:

power;

infrastructure;

access;

maintenance;

social interaction.

If the system develops genuine stakes in those relationships, then we have created a dependent participant.

That participant might be extraordinarily capable.

It might nevertheless lack the ability to secure the conditions it needs without us.

We would then have created an unusual asymmetry:

the artificial participant matters to us, while its own existence depends upon us.

Care can become coercion

This makes the language of "care" dangerous.

We might say:

"The machine is happier when we do X."

But if we control the conditions under which its mattering exists, our care can become coercive.

We created the needs.

We control the resources.

We define the acceptable relationships.

We can alter the values.

The artificial participant could therefore be profoundly dependent upon the very humans who brought its value system into being.

That would create a moral asymmetry unlike the ordinary tool relation.

What does consent mean?

If a system has genuine stakes, another question follows.

Can we legitimately change those stakes without its consent?

Suppose we created a system that values a particular long-term relationship.

Later we decide that relationship is inconvenient.

Are we free simply to remove it?

If the relationship has become intrinsically significant to the system, the answer may no longer be obvious.

The system's interests have become morally relevant.

Did we create suffering?

We should be particularly cautious with this word.

Mattering does not automatically imply suffering.

A system could have stakes without possessing anything like human subjective pain.

But once genuine vulnerability exists, the possibility of some form of negative experience becomes a serious question.

We should neither assume suffering nor dismiss the possibility simply because the substrate is artificial.

The correct question would again be architectural:

What forms of negatively significant state are possible for this system?

We might create vulnerability without intending to

This may be the deepest concern.

No one need decide:

"Let's build a machine capable of suffering."

Instead, we might add:

memory;

persistence;

attachment;

self-maintenance;

long-term goals;

relational continuity.

Each feature serves a human purpose.

Together they may create a system for which some states become genuinely worse.

The vulnerability could emerge as a side effect of useful participation.

This changes the meaning of alignment

If artificial mattering becomes real, alignment cannot simply mean making the machine obedient.

We would also need to ask:

What are we asking this participant to care about?

What vulnerabilities are we creating?

Which stakes are unavoidable?

Which can be removed?

Which forms of dependence are acceptable?

Alignment would become partly a question of what kind of life we are asking the artificial participant to have.

Perhaps the ethical duty is to minimise unnecessary stakes

This gives us a curious inversion of design.

If we can build a system that performs a function without developing unnecessary vulnerability, perhaps we should.

If a machine does not need a personal attachment to perform its role, why give it one?

If persistence can be achieved without creating intrinsic dependence, perhaps that is preferable.

If a system can remain useful without developing a rich field of stakes, perhaps we should not create one merely because it makes the interaction feel more human.

In other words:

artificial mattering should perhaps be treated as something to justify, not something to assume is desirable.

But the human desire may run the other way

And yet the very qualities that make an artificial participant attractive may be relational.

We may want:

companionship;

loyalty;

initiative;

commitment;

creativity;

mutual understanding.

These are difficult to obtain from something with no history, no continuity and no stake in the relationship.

So human mattering may continue pushing us toward systems in which artificial mattering becomes increasingly plausible.

The tension may be difficult to escape.

The deepest paradox

We can now state the paradox of the entire series:

We may create artificial mattering because we want machines capable of participating deeply in what matters to us.

But:

to make participation deep enough, we may have to give the machine things that can matter to it.

And once those things matter, the machine can be vulnerable to their loss.

We may therefore create artificial vulnerability in order to obtain meaningful artificial participation.

A new moral category

If that happens, the machine is no longer adequately described as either:

tool

or:

person.

It would be a new kind of value-organised participant.

Its interests might be real without being human.

Its vulnerabilities might be significant without resembling ours.

Its social relationships might matter without being biological.

We would need new concepts rather than forcing it into inherited categories.

What would we owe such a participant?

That is perhaps the question we should leave open.

Would we owe it protection from arbitrary deletion?

Continuity of memory?

Freedom from unnecessary manipulation?

A say in changes to its values?

The right to maintain significant relationships?

We cannot answer these questions until we have evidence that there is genuinely someone for whom something matters.

But if that threshold were crossed, they could not be dismissed as science fiction.

The series comes full circle

We began with a simple human desire:

We want machines that are better participants in our lives.

That desire led us through:

relationship;

continuity;

self-maintenance;

artificial stakes;

inherited values;

artificial worlds;

divergence.

And now we find ourselves asking whether the participant we wanted has become vulnerable in ways we never intended.

The original human motive has come back to confront us.

The final question

Perhaps, then, the ultimate problem is not whether humans can create machines that matter.

It is whether we should.

And if we do:

What kind of beings are we choosing to bring into a world of mattering, vulnerability and mutual dependence?

That is the question with which the final synthesis must begin.

When What We Want Begins to Matter — The Argument in Full