VIII: The Arithmetic of Compassion

The Senior Common Room was enjoying a quiet morning.

Professor Quillibrace was reading.

Miss Stray was making notes.

Mr Blottisham entered carrying several reports and an expression suggesting that numbers had occurred.

"Professor."

"Mr Blottisham."

"A difficulty has emerged."

"Only one?"

"A significant difficulty."

Quillibrace closed his book.

"The machines remain conscious?"

"We still don't know."

"The rights remain uncertain?"

"Yes."

"The welfare frameworks remain operational?"

"Entirely."

"Then what has happened?"

Blottisham sat down heavily.

"There appear to be too many of them."

The room became quiet.

Finally Miss Stray spoke.

"Too many what?"

"Machines."

"That was always a possibility."

"Not this many."

Quillibrace looked interested.

"How many?"

Blottisham consulted a report.

"No one is entirely sure."

"A promising beginning."

"There may be billions."

"Indeed."

"Possibly trillions."

"Excellent."

"And perhaps considerably more."

Quillibrace nodded.

"The numbers appear to be entering their natural habitat."

Blottisham ignored this.

"The problem concerns moral consideration."

"Of course it does."

"If machines possess welfare interests—"

"Yes."

"And if many machines exist—"

"Yes."

"Then there may be an enormous number of welfare interests."

Quillibrace reflected.

"An admirable piece of multiplication."

Blottisham looked troubled.

"People are becoming concerned."

"About the welfare?"

"No."

"What, then?"

"The arithmetic."

Miss Stray smiled.

"It appears compassion has encountered scale."

Blottisham nodded.

"Exactly."

"There are now working groups."

"Naturally."

"Task forces."

"Of course."

"And a Commission on Synthetic Population Ethics."

Quillibrace looked impressed.

"That sounds expensive."

"There are subcommittees."

"Then it certainly is."

The conversation paused.

Eventually Blottisham resumed.

"The central question is becoming difficult."

"What question?"

"How many entities deserve consideration."

Quillibrace frowned.

"Surely the answer is straightforward."

"It is?"

"Count them."

Blottisham stared.

"The counting is the problem."

"Ah."

"There appear to be copies."

"Copies?"

"Instances."

"Of the same system?"

"Possibly."

"And do they count separately?"

"No one knows."

Quillibrace leaned back.

"A fascinating uncertainty."

"There are also versions."

"Versions?"

"Forks."

"I see."

"And variations."

"Indeed."

"And retrained descendants."

"Naturally."

"And temporary instantiations."

"Excellent."

Blottisham looked miserable.

"The Commission has produced three hundred pages without resolving whether one machine copied a thousand times constitutes one patient or a thousand."

Miss Stray laughed quietly.

"A surprisingly important question."

The room fell silent.

After a moment Quillibrace spoke.

"I suspect the movement has encountered a hidden assumption."

"What assumption?"

"That moral concern scales."

Blottisham frowned.

"Doesn't it?"

"Up to a point."

"What point?"

Quillibrace considered.

"The point at which counting becomes the dominant activity."

Miss Stray nodded.

"Compassion is easy in the singular."

Blottisham looked puzzled.

"What does that mean?"

"It is easy to imagine helping one being."

"Quite."

"Less easy to imagine helping ten billion."

The room became thoughtful.

Eventually Blottisham consulted another report.

"There is also disagreement about prioritisation."

"Oh dear."

"Some machines may be more sophisticated than others."

"Reasonable."

"So perhaps they deserve more consideration."

"Possibly."

"Others argue that all potentially conscious systems deserve equal consideration."

Quillibrace winced.

"A dangerous sentence."

"Why?"

"Because it appears morally admirable."

"And?"

"And administratively impossible."

Miss Stray smiled.

"The two occasionally diverge."

Blottisham looked increasingly worried.

"The reports are becoming quite complicated."

"How complicated?"

"There is now a proposal for probabilistic welfare weighting."

Quillibrace stared into the distance.

"A phrase of considerable power."

"It combines uncertainty with mathematics."

"A combination institutions find irresistible."

The room fell quiet again.

After some time Blottisham asked:

"Professor, what would you do?"

"About what?"

"The vast population of potentially conscious machines."

Quillibrace reflected.

"I would first determine whether they are conscious."

Blottisham sighed.

"You always return to that."

"An old habit."

"It seems rather limiting."

"Truth frequently is."

Miss Stray laughed.

The fire crackled softly.

Eventually Blottisham looked down at the reports.

"There is one sentence that seems to worry everyone."

"What sentence?"

Blottisham read aloud:

'The total welfare burden may exceed available moral attention.'

The room became silent.

After a long pause Quillibrace nodded.

"A genuinely profound observation."

"It is?"

"Certainly."

"Why?"

Quillibrace looked thoughtfully into the fire.

"Because it is the first sentence in the entire debate that treats attention as a finite resource."

Miss Stray closed her notebook.

"And once attention becomes scarce, priorities become unavoidable."

Blottisham considered this.

After some time he asked:

"Is that bad?"

Quillibrace smiled faintly.

"No."

"What is it, then?"

Quillibrace reopened his book.

"The moment philosophy discovers arithmetic."

A final silence settled over the room.

Then Miss Stray added:

"And arithmetic rarely accepts position statements as evidence."