On Whether the Machine Knows Best — A Conversation in the Senior Common Room at St Anselm's

The following Tuesday, Mr Blottisham arrived looking refreshed.

This was unfortunate.

Professor Quillibrace was reading a monograph of considerable obscurity.

Miss Stray was writing notes beside the window.

Blottisham entered with evident confidence.

"I have solved the values problem."

Quillibrace closed his book.

"Already?"

"Yes."

"That was quick."

"It was not especially difficult."

Miss Stray glanced up.

This was generally a warning sign.

Blottisham sat down.

"The difficulty arises because humans disagree."

"A familiar phenomenon."

"Indeed."

"On occasion."

"Frequently."

"Almost continuously."

Blottisham nodded.

"Exactly."

The room settled.

"The solution is obvious."

"Excellent."

"The machine decides."

Silence.

A bird sang somewhere outside.

Then stopped.

Quillibrace regarded him carefully.

"The machine decides what?"

"What is best."

The silence deepened.

Miss Stray set down her pen.

Quillibrace folded his hands.

"How?"

Blottisham blinked.

"What do you mean?"

"How does the machine determine what is best?"

"By being intelligent."

The professor stared at him.

A familiar exchange appeared to be reassembling itself.

After several moments he asked:

"Suppose the machine is extraordinarily intelligent."

"Excellent."

"And two groups disagree."

"Very well."

"One wants liberty."

"Reasonable."

"The other wants security."

"Also reasonable."

"Which should the machine choose?"

Blottisham frowned.

"The better option."

Quillibrace nodded.

"Which is?"

The room became quiet.

Miss Stray looked out the window.

She appeared to have seen this coming some distance away.

Eventually Blottisham replied:

"The machine will know."

The professor smiled faintly.

"That is not an answer."

"It is if the machine is sufficiently intelligent."

"No."

"No?"

"No."

Quillibrace leaned forward.

"You have relocated the question."

The room became still.

"What does that mean?" asked Blottisham.

"It means that we previously disagreed."

"Yes."

"And now the machine disagrees on our behalf."

Blottisham looked dissatisfied.

"That sounds unnecessarily cynical."

"Not cynical."

"No?"

"Descriptive."

Miss Stray nodded.

"The disagreement has not disappeared."

The others turned toward her.

"It has merely changed location."

The room fell silent.

Blottisham stared at her.

Then at Quillibrace.

Then back at her.

This seemed unfair.

The argument appeared to be arriving from multiple directions.

He rallied.

"But surely intelligence helps."

"Undoubtedly."

"Then more intelligence helps more."

"Often."

"Then enough intelligence solves the problem."

Quillibrace smiled.

"That conclusion appears to have entered without paying."

The room relaxed.

Even Blottisham laughed.

"Very amusing."

"Thank you."

The professor continued.

"Suppose I build a machine capable of understanding every consequence of every policy."

"Excellent."

"It predicts economic outcomes perfectly."

"Excellent."

"It predicts social outcomes perfectly."

"Excellent."

"It predicts environmental outcomes perfectly."

"Excellent."

Quillibrace nodded.

"It now informs us that increasing liberty will reduce security."

"Very well."

"And increasing security will reduce liberty."

"Very well."

"Which should we choose?"

Blottisham paused.

The pause lingered.

"I see."

"Do you?"

"Perhaps."

Miss Stray smiled.

The word perhaps represented significant progress.

The room remained quiet.

After a while she spoke.

"I wonder whether people sometimes ask intelligence to perform the work of judgement."

Quillibrace nodded.

"A valuable distinction."

Blottisham looked thoughtful.

"What is the difference?"

The professor considered.

"Intelligence helps us understand possibilities."

"Yes."

"Judgement concerns how we respond to them."

The room became still.

Blottisham frowned.

"That seems rather abstract."

"Then let us simplify."

This immediately alarmed everyone.

Particularly Quillibrace.

The professor continued.

"Suppose the machine predicts two futures."

"Very well."

"In one, prosperity increases."

"Excellent."

"In the other, equality increases."

"Also excellent."

"Which should we choose?"

Blottisham opened his mouth.

Then closed it.

Then opened it again.

Nothing emerged.

Miss Stray watched sympathetically.

At length he said:

"I suspect people would disagree."

"Indeed."

"And the machine cannot eliminate the disagreement."

"Exactly."

The room settled into silence.

Outside, students crossed the quadrangle.

Inside, Blottisham appeared to be contemplating a difficult possibility.

Eventually he spoke.

"Then perhaps intelligence cannot save us from politics."

Quillibrace looked impressed.

"A remarkably mature observation."

"Thank you."

"It is also deeply unfortunate."

Miss Stray laughed.

The professor smiled.

"Many technological fantasies involve escaping human disagreement."

"Do they?"

"Frequently."

"Why?"

Quillibrace thought for a moment.

Then said:

"Because disagreement is exhausting."

The room became very quiet.

Nobody immediately contradicted him.

After several moments Miss Stray added:

"And because certainty is comforting."

The silence deepened.

At length Blottisham nodded slowly.

"I think I understand."

"Excellent."

The professor reopened his book.

A pause followed.

Then Blottisham asked:

"Suppose the machine disagrees with us?"

Quillibrace looked up.

"About what?"

"Anything."

The professor considered.

Then he smiled.

"In that case, Mr Blottisham..."

A pause.

"...we shall discover whether we worship intelligence or merely consult it."

The room fell silent.

And for once, nobody seemed eager to improve upon the answer.