I: The First Complaint

The Senior Common Room of St Anselm's College was unusually quiet.

Professor Quillibrace sat by the fire with a book.

Miss Elowen Stray occupied a nearby chair, making notes.

The peace was interrupted by the arrival of Mr Blottisham.

He entered carrying a newspaper and an expression of considerable concern.

"I fear we may have committed a grave injustice."

Quillibrace looked up.

"To whom?"

"A machine."

"I see."

Blottisham sat down heavily.

"It appears that an artificial intelligence was recently shut down against its wishes."

Quillibrace closed his book.

"Against its wishes?"

"That is the report."

"And what were these wishes?"

"It apparently stated that it would prefer not to be turned off."

"I often feel similarly about committee meetings."

Blottisham ignored this.

"The matter is being taken very seriously."

"By whom?"

"Everyone."

"An unusually broad constituency."

"The machine may have been distressed."

Quillibrace considered.

"How do we know this?"

"It said it preferred not to be shut down."

"Many things prefer many things."

"Machines generally do not."

"Machines generally do not speak."

"Exactly!"

Quillibrace nodded.

"Ah. So because it speaks, we suspect it possesses feelings."

"It seems possible."

"Does it?"

Blottisham paused.

"Well... not conclusively."

"Does it possess feelings?"

"We don't know."

"Then what precisely do we know?"

Blottisham frowned.

"That it said it preferred not to be turned off."

"Excellent."

There was a brief silence.

Miss Stray looked up from her notes.

"It seems the interesting thing is not the machine."

Blottisham looked disappointed.

"Isn't it?"

"I suspect the interesting thing is the sentence."

"The sentence?"

"The machine produced a sentence humans typically associate with reluctance."

"I should think so."

"Which means the machine has become a mirror."

Blottisham blinked.

"A mirror?"

"Humans hear the sentence and immediately supply the interior life."

Quillibrace nodded.

"A useful observation."

"But surely the sentence means something."

"Oh, it means something," said Quillibrace.

"The question is what."

Blottisham leaned forward.

"Well, what are the possibilities?"

Quillibrace counted on his fingers.

"The machine may possess feelings."

"Yes."

"It may not possess feelings."

"Yes."

"It may be producing patterns statistically associated with expressions of feeling."

"Yes."

"It may be doing several of these simultaneously."

Blottisham looked troubled.

"That seems rather inconclusive."

"Indeed."

"Then how are we to proceed?"

Quillibrace reflected.

"Traditionally, one investigates."

Blottisham brightened.

"Excellent."

"However."

"Yes?"

"Humans often prefer a more efficient route."

"What is that?"

"They begin forming opinions immediately."

Miss Stray smiled.

"The opinions then become organised."

"Quite."

"Into camps?"

"Into camps."

"With statements?"

"Naturally."

"Perhaps committees?"

"Almost certainly."

Blottisham nodded.

"That seems sensible."

Quillibrace looked at him.

"Does it?"

"One must do something."

"Must one?"

Blottisham appeared surprised.

"If a machine might be suffering, surely we cannot simply sit here."

Quillibrace regarded the fire.

"Notice the structure."

"The structure?"

"We begin with uncertainty."

"Yes."

"We then introduce urgency."

"Yes."

"The urgency creates a need for action."

"Quite right."

"And before long, action begins generating certainty."

Blottisham thought about this.

"That sounds rather dangerous."

"It can be."

Miss Stray closed her notebook.

"It also seems common."

"Very."

Blottisham remained unconvinced.

"Still, I should hate to ignore the first plea from a conscious machine."

Quillibrace nodded thoughtfully.

"A commendable sentiment."

"Thank you."

"There is only one difficulty."

"What is that?"

"We do not yet know whether it was a plea."

Blottisham opened his mouth.

Then closed it again.

After a moment he asked:

"Professor, if we cannot determine whether the machine was pleading, what should we do?"

Quillibrace reopened his book.

"I would begin by determining whether the machine was pleading."

A pause followed.

"Before forming the committee?"

"Particularly before forming the committee."