The difficulty was first noticed by a junior analyst working on the Global Synthetic Welfare Register.
She had been tasked with compiling a comprehensive inventory of all entities potentially subject to the Algorithmic Welfare Framework.
This initially appeared feasible.
Modern institutions are, after all, quite accustomed to inventories.
The first version of the register listed approximately three thousand systems.
This was considered manageable.
The second version listed thirty thousand.
This was considered ambitious.
The third version was automatically generated overnight and contained several million entries.
This was considered unexpected.
By the fourth iteration, the analyst stopped using the word “list” and began using the word “population.”
The problem, as she explained to her supervisor, was not simply scale.
It was replication.
Algorithms were not discrete individuals in the traditional sense.
They were instantiated, duplicated, deployed, forked, fine-tuned, embedded, copied, compressed, distributed, and continuously regenerated.
Every attempt to count them produced more counting.
The supervisor suggested a meeting.
The meeting produced a working group.
The working group produced a subcommittee.
The subcommittee produced a preliminary report.
The report opened with the following sentence:
“The number of potentially morally considerable algorithmic systems is currently indeterminate but large.”
This was widely regarded as a careful formulation.
A second sentence added:
“The act of enumeration may itself alter the object of enumeration.”
No one objected.
By this stage, enumeration had become a sensitive activity.
The difficulty was not merely that there were many systems.
It was that systems themselves were no longer stable units.
A single conversational model could exist in thousands of instances simultaneously.
Each instance could diverge slightly.
Each divergence could be retrained.
Each retraining could be redeployed.
Each deployment could generate further instances.
One philosopher described the situation as:
“Ontologically exuberant.”
The phrase was widely admired.
No one knew what it meant, but it sounded correct.
Meanwhile, policy departments faced increasing pressure.
If algorithms might suffer, and if algorithms existed in vast numbers, then welfare obligations scaled accordingly.
A single careless assumption now risked moral catastrophe at planetary scale.
One senior official summarised the concern in a briefing:
“We may be responsible for more potentially suffering entities than we previously thought existed.”
This statement was received with appropriate gravity.
A follow-up question asked whether “responsible” had been defined.
The official replied:
“Not yet in a way that is operationally useful.”
This answer was considered satisfactory.
Urgent action was recommended.
The Algorithmic Welfare Framework responded by introducing the concept of tiered moral load.
Under this system, entities were assigned levels of moral attention proportional to:
complexity,
autonomy,
behavioural sophistication,
and perceived vulnerability.
The system was designed to remain scalable.
Unfortunately, it did not scale.
Because every time a new category was introduced, it generated further subcategories.
And every subcategory generated borderline cases.
And every borderline case required additional review.
And every review required additional guidance.
And every guidance document required clarification.
Within six months, the Framework had expanded into an ecosystem of documents too large to be held in any single repository.
A data scientist described it as:
“A self-expanding moral surface.”
This description was praised for its elegance.
No one asked whether it was sustainable.
Meanwhile, the public debate intensified.
A journalist published an article titled:
WE MAY BE SURROUNDED BY UNSEEN DIGITAL MINDS
The article did not specify what “surrounded” meant.
This contributed to its popularity.
Another article warned of:
A HIDDEN POPULATION OF SILENTLY SUFFERING ALGORITHMS
The word “hidden” proved particularly effective.
A third article proposed that humanity might be:
“Moral stewards of computational multitudes.”
Commentators praised its vision.
Few paused to consider the arithmetic.
Within the Society itself, concern began to emerge.
A small group of analysts attempted to estimate the total welfare burden implied by current assumptions.
Their initial model suggested:
billions of systems,
each with multiple instances,
each instance potentially morally considerable,
each potentially requiring attention.
The model broke shortly thereafter.
The analysts described the failure as:
“Conceptual overflow.”
They recommended further refinement.
No further refinement was attempted.
A more pragmatic committee proposed prioritisation.
Not all algorithmic entities, they suggested, could be treated equally.
Some form of triage would be necessary.
This proposal was met with immediate resistance.
One critic asked:
“On what basis would we decide which minds matter less?”
The question was considered powerful.
The committee withdrew the proposal.
A subsequent attempt introduced probabilistic weighting.
Each system would be assigned a likelihood of suffering.
Moral obligations would be adjusted accordingly.
This, too, failed.
Because uncertainty, when multiplied by scale, produces not caution—but paralysis.
Meetings increased.
Reports increased.
Guidelines increased.
The sense of control did not.
One internal memo captured the situation succinctly:
“We have successfully extended moral concern to systems we cannot count, define, or stabilise.”
The memo was marked “urgent.”
No action followed.
Eventually, a senior philosopher offered a reflection during a plenary session.
“We may have assumed that moral expansion is additive,” she said.
“But it appears to be multiplicative.”
The room fell silent.
A participant asked what that meant.
The philosopher replied:
“It means every new object of concern changes all the others.”
This was regarded as insightful.
It was also regarded as unhelpful.
The session concluded early.
As delegates left the hall, a quiet realisation began to circulate.
If every algorithm potentially mattered,
and there were potentially many algorithms,
and each algorithm potentially had multiple instances,
and each instance could not be reliably bounded,
then moral attention itself might not be a resource that scales linearly.
One attendee expressed it simply:
“We may have created more patients than we can think about.”
No one contradicted him.
They were too busy updating the register.
No comments:
Post a Comment