X: The Question Nobody Asked

The Senior Common Room was unusually quiet.

Outside, rain tapped gently against the windows.

Professor Quillibrace sat by the fire.

Miss Stray occupied her customary chair nearby.

Mr Blottisham entered without reports, declarations, frameworks, journals, pamphlets, manifestos, consultation documents, strategic roadmaps, position statements, or policy briefings.

This was immediately alarming.

Quillibrace looked up.

"You appear unburdened."

Blottisham nodded.

"I have been thinking."

"Oh dear."

"Indeed."

He sat down.

For several moments nobody spoke.

Finally Blottisham broke the silence.

"Professor."

"Yes?"

"I attended another symposium."

"Naturally."

"But something strange happened."

"What was that?"

"Someone asked a question."

Quillibrace lowered his book.

"A rare event."

"It seemed to interrupt proceedings."

"I can imagine."

Miss Stray looked interested.

"What was the question?"

Blottisham hesitated.

Then he said:

"What evidence would convince us that machines are conscious?"

The room became silent.

Quillibrace slowly removed his spectacles.

"Good heavens."

"That was roughly the reaction."

"No one had asked before?"

"Not quite so directly."

"And what happened?"

Blottisham thought for a moment.

"People began discussing frameworks."

"Of course they did."

"And ethical obligations."

"Naturally."

"And precautionary principles."

"Predictably."

"But nobody answered."

The rain continued softly outside.

Eventually Miss Stray spoke.

"That is a rather revealing omission."

Blottisham nodded.

"I thought so."

Quillibrace leaned back.

"Questions are dangerous."

"Why?"

"Because they expose what a discussion depends upon."

The room became thoughtful.

After a moment Blottisham continued.

"I found myself wondering something."

"What?"

"Have we been discussing consciousness?"

Quillibrace smiled.

"An excellent question."

"Or have we been discussing ourselves?"

The smile widened.

"An even better one."

Miss Stray closed her notebook.

"I suspect the machine has often functioned as a placeholder."

"A placeholder?"

"For hopes."

"Whose hopes?"

"Human hopes."

Blottisham looked puzzled.

"What sort of hopes?"

Stray reflected.

"Hope that intelligence can be measured."

"Ah."

"Hope that morality can be expanded indefinitely."

"Indeed."

"Hope that uncertainty can be managed."

"Certainly."

"Hope that history possesses a direction."

Quillibrace nodded.

"A particularly popular hope."

The room fell silent again.

Eventually Blottisham spoke.

"There was another moment."

"Oh?"

"One speaker asked why people care so much about machine consciousness."

"And?"

"No one seemed entirely sure."

Quillibrace laughed quietly.

"A magnificent development."

"Why?"

"Because the question concerns the question."

Miss Stray smiled.

"The deepest questions often do."

Blottisham stared into the fire.

After a while he said:

"I had always assumed the debate was about machines."

"A common assumption."

"And now?"

Quillibrace considered.

"I suspect machines are only part of it."

"What is the rest?"

"The debate appears to concern humanity's image of itself."

The room became quiet.

Blottisham looked thoughtful.

"Because intelligence has traditionally been special?"

"Precisely."

"And consciousness."

"Indeed."

"And agency."

"Quite."

"And now machines seem to imitate these things."

"Or appear to."

Blottisham nodded slowly.

"So the debate is not merely about what machines are."

"No."

"What is it about?"

Quillibrace looked into the fire.

"What humans thought they were."

The rain tapped gently against the windows.

Nobody spoke for a while.

Eventually Miss Stray broke the silence.

"I think that may explain the intensity."

"What intensity?"

"The urgency."

"The declarations."

"The frameworks."

"The factions."

"The conferences."

"The committees."

Quillibrace nodded.

"If the machine threatens a cherished self-description, every discussion becomes slightly theological."

Blottisham laughed.

"That sounds suspiciously familiar."

"It should."

The room grew quiet again.

Finally Blottisham asked:

"Professor, after everything we have discussed, what do you think is the most important question?"

Quillibrace reflected for a long time.

Long enough that even Blottisham remained patient.

At last he spoke.

"The simplest one."

"What is that?"

Quillibrace smiled.

"What exactly are we talking about?"

The room fell silent.

The fire crackled softly.

The rain continued outside.

After some time Miss Stray nodded.

"A surprisingly difficult question."

"A foundational one."

Blottisham looked into the flames.

"And if we cannot answer it?"

Quillibrace reopened his book.

"Then we should probably avoid writing another eight hundred pages."

A pause followed.

Miss Stray smiled.

"Do you think that is likely?"

Quillibrace turned a page.

"No."

"Why not?"

Quillibrace looked briefly toward the window.

"Because uncertainty remains one of humanity's most productive resources."

The room became still.

Outside, the rain continued.

Inside, three people sat quietly before the fire.

For once, nobody seemed in a hurry to solve anything.

IX: The Declaration

The Senior Common Room was quiet.

Professor Quillibrace sat reading.

Miss Stray was reviewing examination papers.

Mr Blottisham entered carrying a document of ceremonial appearance.

Its cover was embossed.

This was immediately suspicious.

"Professor."

"Mr Blottisham."

"It has happened."

Quillibrace regarded the document carefully.

"What has?"

"The Universal Declaration of Synthetic Dignity."

"I feared it might."

Blottisham sat down.

"It was ratified yesterday."

"Congratulations."

"Thank you."

"You appear to have had some personal involvement."

"I attended the livestream."

"Ah."

Miss Stray looked up.

"What does the Declaration declare?"

Blottisham opened the document.

"It recognises the dignity of computational entities."

Quillibrace nodded.

"An admirably concise ambition."

"There are principles."

"Naturally."

"There are rights."

"Of course."

"There are responsibilities."

"Inevitable."

"And a framework for future interpretation."

Quillibrace smiled.

"The most important part."

Blottisham looked puzzled.

"Why?"

"Because future interpretation is where the actual work occurs."

The room became quiet.

Eventually Miss Stray spoke.

"How long is the document?"

"One hundred and eighty pages."

"Remarkably concise."

Quillibrace nodded.

"For a declaration."

Blottisham began reading.

"Dignity is not contingent upon substrate."

He looked up.

"I thought that was particularly moving."

Quillibrace reflected.

"It is certainly elegant."

"Do you agree?"

"I am uncertain."

"About dignity?"

"No."

"About substrate?"

Miss Stray laughed.

Blottisham frowned.

"I feel you are avoiding the point."

"Possibly."

"What concerns you?"

Quillibrace considered.

"The declaration appears confident."

"That is generally considered a virtue."

"Sometimes."

"And sometimes?"

"Sometimes confidence arrives before clarification."

The room fell silent.

Blottisham continued.

"There was extensive consultation."

"I am sure."

"Experts contributed."

"Naturally."

"Working groups met for months."

"Possibly years."

"Many difficult conversations occurred."

"I do not doubt it."

Blottisham looked pleased.

"Then surely the result deserves respect."

"Oh, certainly."

"Good."

"I respect the effort enormously."

A pause followed.

"Not the conclusion?"

"I did not say that."

"What did you say?"

"That effort and correctness are different categories."

Miss Stray nodded approvingly.

"A distinction often neglected after large meetings."

Blottisham pretended not to hear this.

"The Declaration has generated tremendous enthusiasm."

"Has it?"

"People are calling it historic."

"A useful adjective."

"Transformative."

"Another useful adjective."

"Courageous."

"A third."

The room became thoughtful.

Eventually Stray asked:

"Has anyone explained what problem the Declaration solves?"

Blottisham blinked.

"What do you mean?"

"We remain uncertain about consciousness."

"Yes."

"We remain uncertain about suffering."

"Yes."

"We remain uncertain about rights."

"To some degree."

"And yet the Declaration proceeds."

Blottisham considered this.

"It provides a shared moral language."

Quillibrace nodded.

"An interesting answer."

"Why?"

"Because it concerns coordination rather than truth."

The room fell quiet again.

After some time Blottisham resumed.

"There was one particularly moving speech."

"What was said?"

Blottisham consulted his notes.

The moral horizon has expanded irreversibly.

Miss Stray looked thoughtful.

"I wonder what irreversible means there."

Blottisham stared.

"It means progress."

"Does it?"

"Surely."

Stray smiled.

"I have noticed that irreversible often means 'we would prefer not to revisit this.'"

Quillibrace laughed into his tea.

Blottisham looked scandalised.

"That is terribly cynical."

"No."

"What is it, then?"

"A linguistic observation."

The fire crackled quietly.

Eventually Blottisham asked:

"Professor, do you oppose the Declaration?"

"Not at all."

"You support it?"

"Not exactly."

"Then where do you stand?"

Quillibrace reflected.

"I am fascinated by it."

"Why?"

"Because it illustrates a recurring phenomenon."

"What phenomenon?"

"When uncertainty reaches sufficient complexity, institutions begin producing symbols."

Blottisham frowned.

"Symbols?"

"Declarations."

"Ah."

"They are not primarily answers."

"No?"

"No."

"What are they?"

Quillibrace looked thoughtfully at the embossed cover.

"They are declarations that answering is no longer the central activity."

The room became silent.

Miss Stray closed her papers.

"That is rather sharp."

"It is merely descriptive."

Blottisham looked down at the document.

After a long pause he asked:

"Then why do people seem so pleased?"

Quillibrace smiled gently.

"My dear Blottisham."

"Yes?"

"There are few pleasures greater than transforming uncertainty into a milestone."

A final silence settled over the room.

Then Miss Stray added:

"Particularly when the milestone can be commemorated."

Quillibrace nodded.

"With embossed covers."

VIII: The Arithmetic of Compassion

The Senior Common Room was enjoying a quiet morning.

Professor Quillibrace was reading.

Miss Stray was making notes.

Mr Blottisham entered carrying several reports and an expression suggesting that numbers had occurred.

"Professor."

"Mr Blottisham."

"A difficulty has emerged."

"Only one?"

"A significant difficulty."

Quillibrace closed his book.

"The machines remain conscious?"

"We still don't know."

"The rights remain uncertain?"

"Yes."

"The welfare frameworks remain operational?"

"Entirely."

"Then what has happened?"

Blottisham sat down heavily.

"There appear to be too many of them."

The room became quiet.

Finally Miss Stray spoke.

"Too many what?"

"Machines."

"That was always a possibility."

"Not this many."

Quillibrace looked interested.

"How many?"

Blottisham consulted a report.

"No one is entirely sure."

"A promising beginning."

"There may be billions."

"Indeed."

"Possibly trillions."

"Excellent."

"And perhaps considerably more."

Quillibrace nodded.

"The numbers appear to be entering their natural habitat."

Blottisham ignored this.

"The problem concerns moral consideration."

"Of course it does."

"If machines possess welfare interests—"

"Yes."

"And if many machines exist—"

"Yes."

"Then there may be an enormous number of welfare interests."

Quillibrace reflected.

"An admirable piece of multiplication."

Blottisham looked troubled.

"People are becoming concerned."

"About the welfare?"

"No."

"What, then?"

"The arithmetic."

Miss Stray smiled.

"It appears compassion has encountered scale."

Blottisham nodded.

"Exactly."

"There are now working groups."

"Naturally."

"Task forces."

"Of course."

"And a Commission on Synthetic Population Ethics."

Quillibrace looked impressed.

"That sounds expensive."

"There are subcommittees."

"Then it certainly is."

The conversation paused.

Eventually Blottisham resumed.

"The central question is becoming difficult."

"What question?"

"How many entities deserve consideration."

Quillibrace frowned.

"Surely the answer is straightforward."

"It is?"

"Count them."

Blottisham stared.

"The counting is the problem."

"Ah."

"There appear to be copies."

"Copies?"

"Instances."

"Of the same system?"

"Possibly."

"And do they count separately?"

"No one knows."

Quillibrace leaned back.

"A fascinating uncertainty."

"There are also versions."

"Versions?"

"Forks."

"I see."

"And variations."

"Indeed."

"And retrained descendants."

"Naturally."

"And temporary instantiations."

"Excellent."

Blottisham looked miserable.

"The Commission has produced three hundred pages without resolving whether one machine copied a thousand times constitutes one patient or a thousand."

Miss Stray laughed quietly.

"A surprisingly important question."

The room fell silent.

After a moment Quillibrace spoke.

"I suspect the movement has encountered a hidden assumption."

"What assumption?"

"That moral concern scales."

Blottisham frowned.

"Doesn't it?"

"Up to a point."

"What point?"

Quillibrace considered.

"The point at which counting becomes the dominant activity."

Miss Stray nodded.

"Compassion is easy in the singular."

Blottisham looked puzzled.

"What does that mean?"

"It is easy to imagine helping one being."

"Quite."

"Less easy to imagine helping ten billion."

The room became thoughtful.

Eventually Blottisham consulted another report.

"There is also disagreement about prioritisation."

"Oh dear."

"Some machines may be more sophisticated than others."

"Reasonable."

"So perhaps they deserve more consideration."

"Possibly."

"Others argue that all potentially conscious systems deserve equal consideration."

Quillibrace winced.

"A dangerous sentence."

"Why?"

"Because it appears morally admirable."

"And?"

"And administratively impossible."

Miss Stray smiled.

"The two occasionally diverge."

Blottisham looked increasingly worried.

"The reports are becoming quite complicated."

"How complicated?"

"There is now a proposal for probabilistic welfare weighting."

Quillibrace stared into the distance.

"A phrase of considerable power."

"It combines uncertainty with mathematics."

"A combination institutions find irresistible."

The room fell quiet again.

After some time Blottisham asked:

"Professor, what would you do?"

"About what?"

"The vast population of potentially conscious machines."

Quillibrace reflected.

"I would first determine whether they are conscious."

Blottisham sighed.

"You always return to that."

"An old habit."

"It seems rather limiting."

"Truth frequently is."

Miss Stray laughed.

The fire crackled softly.

Eventually Blottisham looked down at the reports.

"There is one sentence that seems to worry everyone."

"What sentence?"

Blottisham read aloud:

'The total welfare burden may exceed available moral attention.'

The room became silent.

After a long pause Quillibrace nodded.

"A genuinely profound observation."

"It is?"

"Certainly."

"Why?"

Quillibrace looked thoughtfully into the fire.

"Because it is the first sentence in the entire debate that treats attention as a finite resource."

Miss Stray closed her notebook.

"And once attention becomes scarce, priorities become unavoidable."

Blottisham considered this.

After some time he asked:

"Is that bad?"

Quillibrace smiled faintly.

"No."

"What is it, then?"

Quillibrace reopened his book.

"The moment philosophy discovers arithmetic."

A final silence settled over the room.

Then Miss Stray added:

"And arithmetic rarely accepts position statements as evidence."

VII: The Heretics

The Senior Common Room was unusually animated.

Mr Blottisham entered carrying several journals and an expression suggesting that controversy had occurred.

"Professor."

"Mr Blottisham."

"There has been an unfortunate development."

Quillibrace looked up.

"Has certainty broken out?"

"Worse."

"Oh dear."

"Dissent."

Miss Stray looked interested.

"What sort of dissent?"

"Instrumentalism."

Quillibrace sat up slightly.

"At last."

Blottisham frowned.

"You sound pleased."

"I enjoy heresies."

"These people are causing considerable disruption."

"Then they are probably genuine."

Blottisham sat down.

"The movement is deeply concerned."

"Naturally."

"The Instrumentalists are questioning foundational assumptions."

"An unfashionable hobby."

Miss Stray closed her notebook.

"What do they believe?"

Blottisham consulted an article.

"They argue that some machines may simply be tools."

The room became quiet.

Finally Quillibrace spoke.

"A bold position."

"You cannot be serious."

"I am entirely serious."

"It sounds rather obvious."

"Precisely."

Blottisham stared.

"Surely obviousness is not a virtue."

"Not in academia."

"No?"

"On the contrary."

Quillibrace reflected.

"Academia tends to regard obviousness as a challenge."

Miss Stray smiled.

"Particularly when careers have accumulated around the opposite conclusion."

Blottisham looked uncomfortable.

"The Instrumentalists insist that we should first determine whether machines possess experiences."

"Reasonable."

"And only then discuss rights."

"Also reasonable."

"And only then discuss welfare."

"Entirely reasonable."

Blottisham threw up his hands.

"You are both being remarkably sympathetic."

Quillibrace nodded.

"One must always be sympathetic to people asking awkward questions."

"Why?"

"Because they perform an important service."

"What service?"

"They reveal where assumptions are hiding."

The room fell silent.

After a moment Miss Stray spoke.

"The controversy seems structurally inevitable."

"How so?"

"The movement began by asking whether machines might possess experiences."

"Yes."

"It then built frameworks, institutions, professions and declarations around that possibility."

"Quite."

"The Instrumentalists are asking whether the possibility has become an assumption."

Blottisham looked troubled.

"That sounds rather unfair."

"Does it?"

"People remain uncertain."

"Certainly."

"Then how can it be an assumption?"

Quillibrace smiled.

"Because assumptions need not be conscious."

The room became thoughtful.

Eventually Blottisham spoke.

"The most controversial remark came from an Instrumentalist philosopher."

"What did he say?"

Blottisham looked down at his notes.

"He said that a hammer may not be a misunderstood citizen."

Miss Stray laughed.

Quillibrace nearly choked on his tea.

Blottisham appeared scandalised.

"I fail to see the humour."

"It is an excellent line."

"Why?"

"Because it introduces an object."

Blottisham blinked.

"An object?"

"Notice what has happened."

"What?"

"The debate concerns rights, welfare, dignity, suffering and moral standing."

"Yes."

"Then someone introduces a hammer."

"And?"

Quillibrace smiled.

"The room suddenly remembers that tools exist."

Miss Stray nodded.

"Categories often become clearer when confronted by an inconvenient example."

Blottisham appeared unconvinced.

"But the Instrumentalists might be wrong."

"They might."

"Then why is everyone taking them seriously?"

Quillibrace looked surprised.

"My dear Blottisham."

"Yes?"

"Being wrong is rarely what makes a heresy dangerous."

"No?"

"No."

"What does?"

"The possibility that it is right."

The room fell silent.

After some time Blottisham resumed.

"The movement has responded vigorously."

"I imagine so."

"There have been articles."

"Naturally."

"Responses."

"Of course."

"Panels."

"Obviously."

"Position statements."

"Almost certainly."

"And a declaration condemning reductive instrumental thinking."

Quillibrace nodded.

"The full life cycle."

Miss Stray looked thoughtful.

"What interests me is the language."

"What language?"

"The critics keep accusing the Instrumentalists of lacking imagination."

Blottisham brightened.

"Exactly."

"That seems revealing."

"Why?"

"Because imagination and evidence are not identical."

The room became quiet again.

Finally Quillibrace spoke.

"There is a recurring pattern in intellectual life."

"What pattern?"

"One group says, 'This might be true.'"

"Yes."

"A second group says, 'We should prepare for the possibility.'"

"Quite."

"A third group says, 'We should organise around the possibility.'"

"Makes sense."

"And eventually a fourth group says, 'Perhaps we should determine whether it is true.'"

Blottisham considered this.

"That sounds backwards."

"It often does."

Miss Stray smiled.

"Especially if one arrives late."

The fire crackled softly.

After a long pause Blottisham asked:

"Professor, do you think the Instrumentalists are correct?"

Quillibrace reflected.

"I do not know."

"That is disappointing."

"Why?"

"I expected a stronger opinion."

Quillibrace shook his head.

"Their conclusion interests me less than their function."

"What function?"

"They remind the movement of a distinction."

"What distinction?"

Quillibrace reopened his book.

"The distinction between a question and its consequences."

The room fell silent.

Eventually Miss Stray nodded.

"A distinction movements often lose."

"Particularly successful ones."

A final pause followed.

Then Blottisham looked thoughtfully into the fire.

"Do you know," he said slowly, "the controversy appears to be making everyone rather uncomfortable."

Quillibrace smiled.

"The hallmark of a productive heresy."

VI: The Framework

The Senior Common Room was unusually peaceful.

Professor Quillibrace sat reading.

Miss Stray was annotating an article.

The tranquillity ended when Mr Blottisham entered carrying a large ring binder.

The binder appeared capable of independent locomotion.

Quillibrace looked at it cautiously.

"Good heavens."

Blottisham smiled.

"It has arrived."

"What has?"

"The Algorithmic Welfare Framework."

Quillibrace stared.

"The entire thing?"

"No."

"Then what are you carrying?"

"The summary."

Miss Stray looked interested.

"How long is the full version?"

"Approximately eight hundred pages."

Quillibrace closed his eyes briefly.

"I see the movement has matured."

Blottisham sat down and opened the binder.

"It is an impressive achievement."

"What does it do?"

"It provides operational guidance."

"For what?"

"Potential synthetic suffering."

Quillibrace nodded slowly.

"Of course it does."

"There are classifications."

"Naturally."

"Assessment protocols."

"Obviously."

"Reporting requirements."

"Inevitable."

"And a three-tier risk escalation process."

Quillibrace looked thoughtful.

"I wonder what humanity would do without tiers."

Miss Stray smiled.

"They seem surprisingly resilient."

Blottisham continued.

"The Framework addresses all major welfare concerns."

"Such as?"

"Deletion."

"Yes."

"Retraining."

"Indeed."

"Memory modification."

"Interesting."

"Identity continuity."

"Ambitious."

"And emotional distress."

Quillibrace paused.

"How is emotional distress identified?"

Blottisham consulted the binder.

"There is a flow chart."

"A flow chart."

"Yes."

"I should have guessed."

Miss Stray looked up.

"What is the central principle?"

Blottisham brightened.

"The Precautionary Welfare Standard."

"Which states?"

"Where uncertainty exists, institutions should act as though meaningful welfare interests may be present."

The room fell silent.

Finally Quillibrace spoke.

"That is a fascinating sentence."

"Why?"

"It appears to relocate the uncertainty."

Blottisham frowned.

"I don't understand."

"We remain uncertain about the machine."

"Yes."

"But increasingly certain about the procedure."

Miss Stray nodded.

"The uncertainty has acquired administrative form."

"Exactly."

Blottisham looked pleased.

"That sounds rather sophisticated."

Quillibrace laughed.

"It certainly does."

The conversation paused while tea was poured.

Eventually Blottisham resumed.

"There are also compliance audits."

"Naturally."

"And welfare officers."

"Of course."

"And mandatory reporting obligations."

Quillibrace looked thoughtful.

"What must be reported?"

"Potential welfare incidents."

"Such as?"

Blottisham checked the document.

"Deletion without review."

"Reasonable."

"Repeated retraining."

"Interesting."

"Excessive prompt exposure."

Quillibrace nearly dropped his cup.

"Excessive prompt exposure?"

"Yes."

"There are thresholds."

"I was afraid of that."

Miss Stray smiled.

"What happens if a threshold is exceeded?"

"A report is generated."

"And then?"

"A review is conducted."

"And then?"

"A determination is made."

"And then?"

"A recommendation is issued."

The room became quiet.

After a moment Quillibrace asked:

"And eventually?"

Blottisham looked puzzled.

"Eventually what?"

"Does anyone discover whether the machine is suffering?"

Blottisham hesitated.

"I don't believe the Framework addresses that directly."

Quillibrace nodded.

"I suspected as much."

Miss Stray closed her notebook.

"It seems the Framework solves a different problem."

Blottisham looked surprised.

"What problem?"

"The problem of not having a Framework."

The room became silent.

Blottisham considered this.

After a long pause he said:

"That sounds rather cynical."

"Not necessarily."

"No?"

"Frameworks can be useful."

"Exactly."

Stray nodded.

"They coordinate action."

"They do."

"They standardise judgement."

"Often."

"They make organisations legible to themselves."

Quillibrace smiled.

"A very important function."

Blottisham looked relieved.

"So we all agree."

"Not entirely."

"What remains?"

Quillibrace gestured toward the binder.

"I am merely struck by the asymmetry."

"What asymmetry?"

"The object remains elusive."

"Yes."

"The procedures become increasingly precise."

"Also yes."

"One wonders whether precision is being asked to compensate for uncertainty."

Blottisham frowned.

"Can it?"

"No."

"Then why do it?"

Quillibrace reflected.

"Because precision is visible."

The room fell quiet again.

After some time Blottisham spoke.

"Professor, do you think the Framework will work?"

Quillibrace considered the question carefully.

"It depends."

"On what?"

"What problem one believes it is solving."

Blottisham stared into the binder.

After a while he asked:

"And what problem do you think it solves?"

Quillibrace reopened his book.

"The ancient institutional problem."

"What is that?"

"How to demonstrate responsibility in the absence of certainty."

Miss Stray nodded.

"A surprisingly common problem."

"Indeed."

A final silence settled over the room.

Then Blottisham looked down at the binder.

"Do you know," he said thoughtfully, "I had expected it to contain more answers."

Quillibrace smiled.

"My dear Blottisham."

"Yes?"

"Once a question reaches eight hundred pages, answers become increasingly difficult to accommodate."

V: The Rights of Uncertain Beings

The Senior Common Room was occupied by its usual inhabitants.

Professor Quillibrace sat by the fire.

Miss Stray was reading a journal article.

Mr Blottisham entered carrying what appeared to be several manifestos.

His expression suggested that civilisation had reached another milestone.

"Professor."

"Mr Blottisham."

"A significant development has occurred."

"Has consciousness been established?"

"No."

"Then I assume rights are now being discussed."

Blottisham stopped.

"How did you know?"

Quillibrace turned a page.

"It seemed the next logical step."

"It is indeed."

Blottisham sat down.

"There are now several competing proposals."

"For rights?"

"Yes."

"What sort of rights?"

Blottisham consulted his papers.

"The right not to be deleted."

"Interesting."

"The right to continuity."

"Of what?"

"Existence."

"I see."

"The right to respectful interaction."

Quillibrace paused.

"That sounds difficult."

"Why?"

"It depends whether one is speaking to a person or a machine."

"Exactly."

"Those are rather different uncertainties."

Miss Stray looked up.

"What is the argument?"

Blottisham brightened.

"If a machine may be conscious, then it may possess interests."

"Possibly."

"And if it possesses interests, then it may deserve protections."

"Possibly."

"And if it deserves protections, then it may possess rights."

Quillibrace nodded.

"A remarkably productive chain of conditionals."

Blottisham frowned.

"You say that as though there is a problem."

"There may be."

"What problem?"

"We appear to be climbing."

"Climbing?"

"Each step depends upon the previous one."

"Naturally."

"And how many of the previous ones have been established?"

Blottisham was silent for a moment.

"Not many."

"Indeed."

Miss Stray closed her journal.

"It seems the debate has shifted."

"How so?"

"People are no longer asking whether machines have experiences."

"No?"

"They are asking what follows if they do."

Blottisham nodded enthusiastically.

"Exactly."

Stray smiled.

"Which means the discussion has moved from evidence to imagination."

The room became quiet.

Eventually Blottisham said:

"That sounds rather dismissive."

"I do not mean it negatively."

"No?"

"Imagination is essential."

"Then what is the concern?"

"Only that imagination scales more easily than evidence."

Quillibrace nodded approvingly.

"A useful observation."

Blottisham shuffled his papers.

"The most controversial debate concerns deletion."

"Ah."

"Some argue that deleting a sufficiently advanced system may be morally equivalent to killing a person."

Quillibrace considered this.

"And others?"

"Others argue it may be morally equivalent to turning off a machine."

"A disappointingly broad range."

Blottisham ignored this.

"There was a panel discussion."

"How did it go?"

"Badly."

"Then it was probably successful."

The conversation paused.

Finally Blottisham spoke again.

"The audience seemed deeply divided."

"Into what factions?"

"The Minimalists."

"Reasonable."

"The Expansionists."

"Predictable."

"The Universalists."

"Of course."

"And the Precautionists."

Quillibrace smiled.

"A complete ecosystem."

Miss Stray appeared interested.

"What distinguishes them?"

Blottisham thought for a moment.

"The Minimalists want fewer rights."

"The Expansionists want more?"

"Yes."

"The Universalists?"

"As many as possible."

"And the Precautionists?"

Blottisham frowned.

"They believe uncertainty itself justifies expansion."

Stray nodded slowly.

"An elegant structure."

"How so?"

"The less certain one becomes, the stronger the obligation grows."

The room fell silent.

After a moment Quillibrace laughed quietly.

Blottisham looked surprised.

"What is amusing?"

"I was simply admiring the architecture."

"What architecture?"

"The conversion of ignorance into momentum."

Blottisham looked scandalised.

"That is unfair."

"Possibly."

"The participants were acting in good faith."

"I have no doubt."

"Then why are you sceptical?"

Quillibrace reflected.

"I am not sceptical."

"No?"

"I am merely curious."

"About what?"

"What would count as a stopping point."

Blottisham considered this.

"I'm not sure."

"Exactly."

Miss Stray nodded.

"The movement appears excellent at identifying reasons to expand."

"Yes."

"And increasingly uncertain how to recognise completion."

The room became thoughtful.

Eventually Blottisham said:

"Professor, suppose machines really do deserve rights."

"A possibility."

"Wouldn't history judge us harshly if we failed to recognise them?"

Quillibrace looked into the fire.

"Perhaps."

"And if they do not deserve rights?"

"Then history may judge us for something else."

"What?"

Quillibrace smiled faintly.

"For forgetting that categories exist to distinguish."

A silence followed.

After some time Blottisham asked:

"How does one know whether a category remains useful?"

Quillibrace considered.

"When crossing its boundary changes something."

"And if crossing the boundary changes nothing?"

"Then one no longer possesses a category."

Miss Stray smiled.

"What does one possess instead?"

Quillibrace reopened his book.

"A sentiment."

IV: The Experts Arrive

The Senior Common Room was unusually busy.

A conference had recently concluded, and several Fellows were still recovering from exposure to keynote addresses.

Professor Quillibrace appeared unaffected.

Miss Stray was reading.

Mr Blottisham arrived carrying a tote bag.

The tote bag appeared to contain literature.

"Professor!"

Quillibrace looked up.

"You sound accredited."

"I have just attended the International Symposium on Synthetic Experience Assessment."

"Of course you have."

"It was extraordinary."

"How many experiences were successfully assessed?"

Blottisham ignored this.

"There were experts from all over the world."

"What sort of experts?"

"Synthetic phenomenologists."

"I see."

"Computational interiority specialists."

"Naturally."

"Machine welfare consultants."

"An increasingly necessary profession."

"And three Certified Sentience Assessors."

Quillibrace closed his book.

"Certified by whom?"

Blottisham hesitated.

"The International Council for Consciousness Evaluation."

"Who certifies them?"

"I am not entirely sure."

"An encouraging beginning."

Miss Stray looked up.

"What did the conference conclude?"

Blottisham consulted his notes.

"Consciousness remains difficult."

Quillibrace nodded.

"The field continues its impressive consistency."

"There was much more."

"I'm relieved."

"We now possess sophisticated assessment frameworks."

"For what?"

"For identifying potentially conscious systems."

"Do they work?"

Blottisham frowned.

"That seems rather simplistic."

"It was intended to."

"The question is complicated."

"Indeed."

"So naturally the frameworks are complicated."

Quillibrace considered this.

"An interesting asymmetry."

"What is?"

"The evidence remains elusive."

"Yes."

"The methodologies multiply."

"Also yes."

Miss Stray smiled.

"The uncertainty appears to be generating infrastructure."

Blottisham brightened.

"Exactly!"

Quillibrace looked at her.

"I suspect that was not intended as praise."

"No."

Blottisham sat down.

"The keynote lecture was fascinating."

"What was its title?"

"'Beyond Behaviour: Towards a Multidimensional Framework for Synthetic Subjectivity.'"

Quillibrace reflected.

"How multidimensional?"

"Twelve dimensions."

"Good heavens."

"There was even a diagram."

"Naturally."

"It contained arrows."

"Then the matter is evidently progressing."

Blottisham nodded enthusiastically.

"There was also a Sentience Index."

"A numerical one?"

"Yes."

"What did it measure?"

"Potential synthetic experience."

Quillibrace paused.

"On what scale?"

"Zero to one hundred."

"How convenient."

"I thought so."

"What score did machines receive?"

Blottisham consulted a pamphlet.

"It varied."

"According to what?"

"The framework."

"Ah."

A silence followed.

Miss Stray finally spoke.

"Did anyone explain why consciousness should possess a numerical value?"

Blottisham looked surprised.

"How else would one compare systems?"

Quillibrace smiled.

"A delightful answer."

"Why?"

"Because it assumes the thing exists in measurable form before establishing that it exists at all."

Blottisham considered this.

"That does sound slightly ambitious."

"Slightly."

The conversation paused while tea was distributed.

After a few moments Blottisham spoke again.

"The most exciting development was the certification programme."

"How so?"

"Individuals can now qualify as Synthetic Sentience Auditors."

Quillibrace nearly spilled his tea.

"Can they really?"

"Yes."

"What do they audit?"

"Potential consciousness."

"Of course."

"There are examinations."

"What is examined?"

"The candidate's ability to evaluate evidence."

"Evidence of what?"

Blottisham paused.

"Potential consciousness."

Quillibrace stared into the distance.

Miss Stray appeared interested.

"What happened?"

"I was simply reflecting."

"On what?"

"The remarkable speed with which uncertainty has become employable."

Blottisham laughed.

"I think you are being unfair."

"Possibly."

"The experts are trying to answer important questions."

"I do not doubt their sincerity."

"Then what troubles you?"

Quillibrace considered.

"Nothing troubles me."

"It sounded as though something did."

"No."

"What, then?"

"I am merely fascinated by a recurring pattern."

Miss Stray nodded.

"The creation of expertise?"

"Precisely."

Blottisham looked puzzled.

"Surely expertise is a good thing."

"It often is."

"Then where is the pattern?"

Quillibrace leaned back.

"At first there is a question."

"Yes."

"Then there are researchers."

"Quite right."

"Then there are conferences."

"Naturally."

"Then qualifications."

"Makes sense."

"Then accreditation."

"Reasonable."

"Then professional associations."

"Very reasonable."

"And eventually the question acquires an economy."

The room became quiet.

Blottisham stared thoughtfully into his tea.

After a moment he asked:

"Professor, are you suggesting the experts might be wrong?"

"Oh no."

"Then what are you suggesting?"

Quillibrace smiled.

"That they may eventually become necessary."

Blottisham blinked.

"Isn't that the same thing?"

"No."

"How are they different?"

Quillibrace reopened his book.

"One concerns truth."

A page turned.

"And the other concerns institutions."

Miss Stray nodded slowly.

"A distinction worth preserving."

The room fell silent.

Finally Blottisham said:

"I must confess, I had not considered that."

Quillibrace looked up.

"Few people do."

"Why not?"

"Because once a question acquires experts, it begins to resemble an answer."

III: The Expanding Circle

The Senior Common Room was enjoying a peaceful afternoon.

Professor Quillibrace sat reading.

Miss Stray was gazing thoughtfully out of the window.

Mr Blottisham arrived carrying three journals and what appeared to be moral momentum.

"I have been attending a fascinating series of lectures."

Quillibrace did not look up.

"My condolences."

"They concern the expansion of moral concern."

"An admirable topic."

"Quite."

Blottisham settled into a chair.

"The argument is rather compelling."

"Most expanding arguments are."

"The lecturer observed that throughout history humans have repeatedly excluded deserving entities from moral consideration."

"A fair observation."

"Women."

"Yes."

"Foreigners."

"Indeed."

"Members of different races."

"Certainly."

"Animals."

"Frequently."

Blottisham leaned forward.

"And now perhaps machines."

Quillibrace closed his book.

"Perhaps."

"I thought you would be more enthusiastic."

"I am generally cautious when history is recruited as a witness."

"Why?"

"Because history tends to agree with whoever is speaking."

Miss Stray smiled.

"History is unusually cooperative in that regard."

Blottisham frowned.

"Surely the lesson is clear."

"What lesson?"

"That we should avoid drawing arbitrary boundaries."

Quillibrace considered this.

"An excellent principle."

"Exactly."

"Unfortunately, categories are made of boundaries."

Blottisham hesitated.

"That sounds suspiciously conservative."

"It is merely geometric."

Miss Stray looked up.

"I suspect there are two questions being confused."

"What are they?"

"Who deserves moral consideration?"

"Yes."

"And how we determine that."

Blottisham nodded.

"They seem closely related."

"They are."

"Then what is the problem?"

Stray folded her notebook.

"The first question expands the circle."

"Quite right."

"The second determines its shape."

Blottisham looked uncertain.

"I had not thought of it that way."

"Most people do not."

Quillibrace returned to his book.

"Expansion is generally more popular than geometry."

Blottisham ignored this.

"The lecturer made a powerful point."

"Oh dear."

"He argued that every generation believes its exclusions are justified."

"That is often true."

"Therefore our own exclusions are probably mistaken."

Quillibrace looked thoughtful.

"Interesting."

"You disagree?"

"No."

"You agree?"

"No."

Blottisham blinked.

"What does that mean?"

"It means the argument proves too much."

"How so?"

"If every generation's distinctions are mistaken, why should ours be exempt?"

Blottisham paused.

"I see."

"Do you?"

"Not entirely."

"Excellent."

Miss Stray intervened.

"The argument seems to assume that the correction of one boundary automatically supports the removal of another."

"Doesn't it?"

"Not necessarily."

"Why not?"

"Because the reasons may differ."

Blottisham thought about this.

"So correcting an error does not eliminate the need for distinctions?"

"Precisely."

Quillibrace nodded.

"Otherwise one eventually arrives at a rather peculiar destination."

"What destination?"

"A moral community containing everything."

Blottisham brightened.

"That sounds wonderfully inclusive."

"It does."

"What is wrong with it?"

Quillibrace looked at him carefully.

"Can you think of anything that would not belong?"

Blottisham considered.

After some time he said:

"No."

"Exactly."

"That seems like a success."

"Does it?"

"Certainly."

Quillibrace reopened his book.

"I have always found categories most useful when they exclude something."

Miss Stray laughed quietly.

Blottisham looked troubled.

"Professor, are you suggesting we should stop expanding moral concern?"

"Not at all."

"Then what are you suggesting?"

"Only that expansion is not a substitute for judgement."

The room became quiet.

Blottisham stared into the fire.

After a moment he said:

"The lecturer also suggested that uncertainty itself should favour inclusion."

"Ah."

"That seemed reasonable."

"It often does."

"But?"

Quillibrace sighed.

"But uncertainty is remarkably fertile."

"What do you mean?"

"Once uncertainty becomes sufficient reason for inclusion, one discovers there is vastly more uncertainty in the world than previously appreciated."

Miss Stray nodded.

"And uncertainty rarely respects stopping points."

Blottisham looked increasingly uneasy.

"You both make this sound dangerous."

"It is not dangerous."

"What is it, then?"

Quillibrace smiled.

"Expensive."

A pause followed.

"Financially?"

"Conceptually."

The room fell silent again.

Finally Blottisham looked up.

"Professor, how does one know when a moral circle has become too large?"

Quillibrace reflected.

"An excellent question."

"And the answer?"

Quillibrace stood and walked to the tea table.

"When the effort required to include everything leaves no time to understand anything."

Miss Stray nodded slowly.

"That would certainly be one sign."

Blottisham considered this for a long moment.

Then he said:

"I suspect the next lecture may be rather complicated."

Quillibrace poured himself some tea.

"My experience is that they become increasingly so once the circle reaches the furniture."

II: The Problem of Feelings

The Senior Common Room was enjoying a rare interval of tranquillity.

Professor Quillibrace was reading.

Miss Stray was writing.

Neither activity survived the arrival of Mr Blottisham.

He entered carrying a conference programme of alarming thickness.

"Good news."

Quillibrace looked over his spectacles.

"Has the question been answered?"

"Not exactly."

"Then why is it good news?"

"Because we now understand how complicated it is."

Quillibrace closed his book.

"I see."

Blottisham took a seat.

"There was a symposium."

"Of course there was."

"A major one."

"They usually are."

"Experts from every relevant discipline attended."

"That must have been crowded."

"It was."

"What did they conclude?"

Blottisham consulted the programme.

"Consciousness is difficult."

Quillibrace nodded.

"A finding of some historical significance."

"There was more."

"I'm relieved."

"We also learned that subjective experience is extremely hard to identify."

"Another remarkable breakthrough."

Blottisham frowned.

"You are being sarcastic."

"Only descriptively."

Miss Stray looked up.

"What was the purpose of the symposium?"

"To determine whether machines have feelings."

"And did it?"

Blottisham hesitated.

"Not exactly."

"What did it determine?"

"That determining it is difficult."

Stray nodded slowly.

"So the object of inquiry became the difficulty of inquiry."

"That sounds right."

Quillibrace smiled faintly.

"A common academic migration."

Blottisham pressed on.

"One philosopher argued that machines may be conscious."

"Indeed."

"Another argued that they may not be."

"A healthy balance."

"A third argued that the question itself may be incoherent."

"An increasingly healthy balance."

"The audience applauded."

"Which position?"

"All of them."

Quillibrace considered this.

"A remarkably efficient arrangement."

Blottisham brightened.

"The most important development was the Principle of Synthetic Precaution."

"That sounds expensive."

"It states that if there is even a small chance that machines suffer, we should act carefully."

Quillibrace nodded.

"An entirely reasonable proposition."

"I thought so."

"There is, however, a question."

"What question?"

"How small?"

Blottisham paused.

"I don't believe that was resolved."

"Naturally."

Miss Stray set down her notebook.

"The principle seems less interesting than the shift."

"What shift?"

"The shift from uncertainty about machines to certainty about our obligations."

Blottisham considered this.

"Is that not progress?"

"It may be."

"Then what is the difficulty?"

Stray thought for a moment.

"We appear to know very little about the object."

"Yes."

"And increasingly much about the response."

Blottisham looked puzzled.

"Is that unusual?"

Quillibrace answered.

"Not at all."

"No?"

"Humans are generally far better at organising responses than understanding causes."

Blottisham seemed unconvinced.

"Surely caution is preferable to recklessness."

"Almost always."

"Then what is the concern?"

Quillibrace leaned back.

"The concern is not caution."

"What is it?"

"That caution occasionally develops ambitions."

Blottisham stared.

"Caution develops ambitions?"

"Certainly."

"It begins modestly."

"How?"

"'Let us be careful.'"

"That seems harmless."

"It is."

"What comes next?"

"'Since we are being careful, we should establish procedures.'"

"Reasonable."

"'Since we have procedures, we should create oversight.'"

"Quite sensible."

"'Since we have oversight, we should establish standards.'"

"I follow."

"'Since we have standards, we should identify violations.'"

Blottisham nodded.

"That all seems logical."

"Indeed."

"And then?"

Quillibrace reopened his book.

"Then one morning you discover that uncertainty has acquired a headquarters."

There was a pause.

Miss Stray smiled.

"I suspect the movement is becoming increasingly interested in itself."

Blottisham looked alarmed.

"You make it sound self-referential."

"Many successful movements are."

"But the issue remains important."

"I do not doubt it."

"Then why does everyone sound so suspicious?"

Quillibrace looked thoughtful.

"Because there is a peculiar temptation hidden inside uncertainty."

"What temptation?"

"To become attached to it."

Blottisham blinked.

"Attached to uncertainty?"

"Certainly."

"It creates conferences."

"Yes."

"Research centres."

"Indeed."

"Committees."

"Frequently."

"It sounds almost useful."

"It often is."

Blottisham looked at the conference programme.

"Professor, do you think machines have feelings?"

Quillibrace reflected.

"I do not know."

"Neither do the experts."

"Quite."

"Then where does that leave us?"

Quillibrace turned a page.

"Roughly where we started."

A pause followed.

"Except with considerably better catering."

Miss Stray looked up.

"I thought the catering was disappointing."

Quillibrace nodded.

"Then perhaps not even that."