On What Intelligence Is For — A Conversation in the Senior Common Room at St Anselm's

Several days later, Mr Blottisham entered the Senior Common Room looking unusually melancholy.

Professor Quillibrace immediately noticed.

Miss Stray did as well.

Neither commented.

Blottisham sat down heavily.

The silence persisted.

Eventually Quillibrace spoke.

"You appear troubled."

"I am."

"How unfortunate."

Blottisham sighed.

"I have been reflecting on our recent discussions."

"My sympathies."

"They seem rather bleak."

The professor looked surprised.

"Do they?"

"Certainly."

"You have removed every exciting possibility."

"Have I?"

"One by one."

The room became quiet.

Blottisham counted on his fingers.

"Intelligence does not guarantee wisdom."

"No."

"It does not guarantee values."

"No."

"It does not eliminate politics."

"No."

"It does not eliminate uncertainty."

"No."

"It does not provide history with a destination."

"No."

Blottisham spread his hands.

"What remains?"

The room fell silent.

Quillibrace looked thoughtful.

After a moment he asked:

"Do you believe spectacles are valuable?"

Blottisham blinked.

"What?"

"Spectacles."

"Yes."

"Why?"

"They help one see."

"Excellent."

The professor nodded.

"Do they provide a destination?"

"No."

"A moral system?"

"No."

"A purpose?"

"No."

"A theory of history?"

"Certainly not."

"Yet they remain useful."

The room relaxed.

Miss Stray smiled.

Blottisham looked unconvinced.

"That seems an absurd comparison."

"Possibly."

The professor folded his hands.

"What has concerned me throughout these discussions is not that people value intelligence."

"No?"

"No."

"What then?"

"That they ask intelligence to perform tasks that do not belong to it."

The room became quiet.

Blottisham considered this.

Miss Stray nodded.

"A tool becomes disappointing when required to become a religion."

The silence deepened.

Quillibrace smiled.

"An excellent formulation."

Blottisham frowned.

"Surely intelligence is more than a tool."

"Perhaps."

"Much more."

"Possibly."

The professor considered.

"But whatever else it may be, it remains extraordinarily useful."

The room settled.

After a while Miss Stray spoke.

"I wonder whether intelligence is most valuable when it increases possibility."

The others turned toward her.

She continued.

"It allows us to understand more."

"Yes."

"Imagine more."

"Yes."

"Anticipate consequences."

"Yes."

"Discover relationships."

"Yes."

"Generate alternatives."

"Yes."

She smiled.

"None of those things tells us what must be done."

The room became still.

"But all of them enlarge what can be done."

Quillibrace nodded approvingly.

Blottisham looked thoughtful.

This time the thoughtfulness seemed less defensive.

More curious.

After a moment he asked:

"Then intelligence does not answer the questions."

"No," said Quillibrace.

"What does it do?"

The professor reflected.

Then replied:

"It improves the quality of the questions."

The room became very quiet.

Outside, evening sunlight crossed the college lawns.

Inside, nobody spoke for a while.

Eventually Blottisham said:

"I think I had hoped intelligence would save us."

The confession hung briefly in the air.

Miss Stray looked down at her notebook.

Quillibrace regarded him kindly.

"Many people do."

"Why?"

The professor smiled faintly.

"Because responsibility is tiring."

The room laughed softly.

Even Blottisham.

After the laughter subsided, silence returned.

A comfortable silence this time.

Not an uncertain one.

At length Miss Stray spoke.

"Perhaps that is why these stories are so attractive."

"What stories?" asked Blottisham.

"The stories in which intelligence eventually becomes so great that the difficult questions disappear."

The room remained quiet.

Blottisham nodded slowly.

"And they do not?"

"No."

"Never?"

Quillibrace considered.

Then smiled.

"I rather hope not."

The room became still.

"Why?" asked Blottisham.

The professor looked toward the window.

For several moments he said nothing.

Then he replied:

"Because a world in which all worthwhile questions had been answered..."

A pause.

"...would be a surprisingly impoverished place."

The silence lingered.

Outside, students crossed the quadrangle.

Inside, evening settled over the room.

At length Blottisham spoke.

"So intelligence is not the destination."

"No."

"Nor the salvation."

"No."

"Nor the final answer."

"No."

The professor reopened his book.

"What is it then?"

Quillibrace turned a page.

A faint smile appeared.

"Mr Blottisham..."

A pause.

"...it is one of the ways the universe becomes curious about itself."

For once, nobody replied.

And for several minutes, the Senior Common Room remained entirely silent.

A rare achievement.

Even by academic standards.

On Whether History Has a Destination — A Conversation in the Senior Common Room at St Anselm's

The following Wednesday, Mr Blottisham entered the Senior Common Room looking unusually serene.

Professor Quillibrace immediately became suspicious.

Miss Stray merely sighed.

"I have decided that you are both partly correct."

Quillibrace lowered his book.

"A generous concession."

"Indeed."

Blottisham sat down.

"The future may be uncertain."

"Quite."

"Values may be disputed."

"Certainly."

"Purposes may vary."

"Undeniably."

"The machine may not solve every philosophical problem."

"An encouraging possibility."

Blottisham nodded.

"Nevertheless, progress remains inevitable."

The room became very quiet.

Miss Stray slowly closed her notebook.

Quillibrace smiled.

"Ah."

"What?"

"We have arrived."

"Where?"

"The destination."

Blottisham looked pleased.

"I knew there was one."

The professor laughed softly.

"Not that destination."

The room settled.

Blottisham continued.

"Surely history demonstrates the point."

"Which point?"

"Humanity progresses."

Quillibrace considered this.

"How?"

"Science advances."

"Indeed."

"Technology advances."

"Certainly."

"Knowledge accumulates."

"Frequently."

"Therefore history moves forward."

The professor looked thoughtful.

"Forward?"

"Yes."

"In which direction?"

The room fell silent.

Blottisham frowned.

"The future."

"That is a temporal direction."

"So?"

"I was asking about the evaluative direction."

The silence deepened.

Miss Stray appeared amused.

Blottisham looked wary.

He recognised the terrain.

Unfortunately.

After a moment he replied:

"Toward improvement."

Quillibrace nodded.

"We appear to have returned."

The room relaxed slightly.

Everyone knew this circle by now.

The professor continued.

"Suppose medical science improves."

"Excellent."

"And life expectancy increases."

"Excellent."

"And populations age."

"Very well."

"And healthcare systems become strained."

"Less excellent."

"And new ethical questions emerge."

"Possibly."

Quillibrace nodded.

"Has progress occurred?"

Blottisham hesitated.

"Yes."

"Why?"

"Because people live longer."

"Excellent."

The professor smiled.

"Someone else might prioritise something different."

The room became quiet.

Miss Stray spoke.

"I wonder whether history contains many different trajectories."

The others turned toward her.

She continued.

"Scientific knowledge may increase."

"Yes."

"Social trust may increase or decrease."

"Yes."

"Economic prosperity may increase."

"Yes."

"Environmental stability may increase or decrease."

"Yes."

"Cultural diversity may increase or decrease."

"Yes."

She paused.

"Why should these all collapse into a single measure called progress?"

The room became very still.

Blottisham stared thoughtfully.

The idea appeared unwelcome.

But difficult to dismiss.

Eventually he rallied.

"Because otherwise history has no direction."

Quillibrace smiled faintly.

"Indeed."

The room fell silent.

Blottisham blinked.

"That is all?"

"For the moment."

The professor folded his hands.

"What interests me is that many people seem deeply attached to the notion that history must be going somewhere."

"Surely it is."

"Perhaps."

"Then why the hesitation?"

Quillibrace considered.

"Because movement and destination are not identical."

The room became quiet.

Again.

This was becoming a recurring theme.

Miss Stray nodded.

"We often treat change as though it automatically implies purpose."

"Exactly."

Blottisham looked dissatisfied.

"But surely humanity learns."

"Certainly."

"And develops."

"Certainly."

"And accumulates knowledge."

"Certainly."

"Then surely there is progress."

The professor smiled.

"There are many progresses."

The room became still.

Blottisham frowned.

"Many?"

"Scientific progress."

"Yes."

"Technical progress."

"Yes."

"Economic progress."

"Yes."

"Medical progress."

"Yes."

"Political progress."

"Sometimes."

The room laughed softly.

Quillibrace continued.

"The difficulty arises when we attempt to aggregate them."

The room settled.

Miss Stray looked out the window.

"I wonder whether people sometimes imagine history as a story."

"What do you mean?" asked Blottisham.

"A story has a beginning."

"Yes."

"A middle."

"Yes."

"And an ending."

"Naturally."

She smiled.

"Reality is under no obligation to provide one."

The room became silent.

Outside, students crossed the college lawn.

Inside, Blottisham stared into the distance.

At length he said:

"Then perhaps history does not have a destination."

"No?" asked Quillibrace.

"No."

"What does it have?"

Blottisham thought for a long time.

Finally he replied:

"Participants."

For several moments nobody spoke.

Miss Stray smiled.

Quillibrace looked unexpectedly pleased.

The professor eventually reopened his book.

"An excellent answer."

Blottisham brightened.

"Really?"

"Yes."

The optimism returned.

"What does it mean?"

Quillibrace turned a page.

"That, Mr Blottisham..."

A pause.

"...is considerably easier to say than to understand."

And the room once again fell into the sort of silence that universities occasionally mistake for wisdom.

On the End of Uncertainty — A Conversation in the Senior Common Room at St Anselm's

The following Friday, Mr Blottisham arrived carrying a look of profound satisfaction.

Professor Quillibrace immediately became suspicious.

Miss Stray quietly prepared herself.

"I have finally understood the matter."

Quillibrace closed his book.

"Again?"

"Yes."

"You are making remarkable progress."

"I know."

Blottisham sat down.

"The true significance of superintelligence is now obvious."

"Excellent."

"It will eliminate uncertainty."

The room became silent.

Quillibrace regarded him carefully.

"All uncertainty?"

"Eventually."

"I see."

"The machine will know more than any human."

"Possibly."

"More than all humans combined."

"Conceivably."

"Therefore uncertainty will disappear."

The professor folded his hands.

"That is a very large therefore."

Blottisham waved this away.

"The logic is impeccable."

"It often is."

"Then we agree."

"I am afraid not."

Miss Stray smiled.

The ritual had resumed.

Quillibrace considered the matter.

"Suppose I know the exact position of every star in the galaxy."

"Excellent."

"Every measurable property of every planet."

"Excellent."

"Every biological process on Earth."

"Excellent."

The professor paused.

"Would uncertainty disappear?"

Blottisham frowned.

"It would diminish."

"Certainly."

"Substantially."

"Possibly."

"Then the principle holds."

Quillibrace looked thoughtful.

"Does it?"

The room became quiet.

Miss Stray lowered her book.

The professor continued.

"Uncertainty about what?"

Blottisham sighed.

"There is always a qualification."

"There generally is."

"Everything."

Quillibrace smiled.

"A wonderfully ambitious category."

The room settled.

After a moment Miss Stray spoke.

"I wonder whether uncertainty comes in different forms."

Blottisham looked wary.

"What do you mean?"

"Some uncertainty results from ignorance."

"Yes."

"And some from complexity."

"Perhaps."

"And some from disagreement."

"Possibly."

"And some from the future not yet having occurred."

The room became still.

Blottisham considered this.

"Those seem rather different."

"Exactly."

Quillibrace nodded approvingly.

"A useful distinction."

The professor rose and walked toward the fire.

"Suppose the machine predicts tomorrow's weather perfectly."

"Excellent."

"Suppose it predicts economic activity perfectly."

"Excellent."

"Suppose it predicts election outcomes perfectly."

"Excellent."

He turned.

"Will people stop disagreeing?"

Blottisham hesitated.

"No."

"Will they stop wanting different things?"

"No."

"Will they stop interpreting events differently?"

"No."

The professor smiled faintly.

"Then some uncertainties appear surprisingly resilient."

The room became quiet.

Blottisham stared at the carpet.

The carpet had become an important participant in recent discussions.

Eventually he rallied.

"But the machine will know more than anyone."

"Quite possibly."

"Far more."

"Quite possibly."

"Then it will possess extraordinary certainty."

Quillibrace looked thoughtful.

"I wonder."

"No?"

"Knowledge and certainty are not necessarily identical."

The room fell silent.

Miss Stray looked interested.

Blottisham looked alarmed.

This appeared to be another distinction.

There were becoming rather a lot of them.

The professor continued.

"The more one understands a system..."

"Yes?"

"...the more relationships one discovers."

"Naturally."

"The more relationships one discovers..."

"Yes?"

"...the more consequences require consideration."

The room became still.

Miss Stray nodded slowly.

Blottisham looked unconvinced.

"That sounds backwards."

"Does it?"

"Surely greater knowledge produces simplicity."

Quillibrace laughed softly.

"Has that been your experience of academia?"

The room dissolved briefly into laughter.

Even Blottisham joined in.

After the laughter subsided, Miss Stray spoke.

"I sometimes wonder whether certainty and understanding are opposites."

The room turned toward her.

She continued.

"A child often feels certain."

"Indeed," said Quillibrace.

"A specialist often feels less certain."

"Quite."

"Not because they know less."

"No."

"But because they understand more of the structure."

The room became very quiet.

Blottisham stared thoughtfully into the fire.

The idea appeared to be causing minor renovations.

After a while he spoke.

"Then perhaps intelligence does not eliminate uncertainty."

"No?" said Quillibrace.

"No."

"What does it do?"

Blottisham thought for a long time.

Then slowly replied:

"It changes its shape."

The room fell silent.

Miss Stray smiled.

Quillibrace looked genuinely pleased.

"An excellent formulation."

Blottisham brightened.

"I have finally understood?"

"Not entirely."

The optimism faded.

"What now?"

The professor returned to his chair.

"Now we must consider the possibility that uncertainty is not merely a defect."

The room became still.

Outside, evening bells echoed across the college grounds.

Inside, nobody spoke.

At length Blottisham asked:

"What else could it be?"

Quillibrace opened his book.

A faint smile appeared.

"Mr Blottisham..."

A pause.

"...that is a considerably more interesting question."

And for once, even Blottisham suspected that it might be.

On Whether the Machine Knows Best — A Conversation in the Senior Common Room at St Anselm's

The following Tuesday, Mr Blottisham arrived looking refreshed.

This was unfortunate.

Professor Quillibrace was reading a monograph of considerable obscurity.

Miss Stray was writing notes beside the window.

Blottisham entered with evident confidence.

"I have solved the values problem."

Quillibrace closed his book.

"Already?"

"Yes."

"That was quick."

"It was not especially difficult."

Miss Stray glanced up.

This was generally a warning sign.

Blottisham sat down.

"The difficulty arises because humans disagree."

"A familiar phenomenon."

"Indeed."

"On occasion."

"Frequently."

"Almost continuously."

Blottisham nodded.

"Exactly."

The room settled.

"The solution is obvious."

"Excellent."

"The machine decides."

Silence.

A bird sang somewhere outside.

Then stopped.

Quillibrace regarded him carefully.

"The machine decides what?"

"What is best."

The silence deepened.

Miss Stray set down her pen.

Quillibrace folded his hands.

"How?"

Blottisham blinked.

"What do you mean?"

"How does the machine determine what is best?"

"By being intelligent."

The professor stared at him.

A familiar exchange appeared to be reassembling itself.

After several moments he asked:

"Suppose the machine is extraordinarily intelligent."

"Excellent."

"And two groups disagree."

"Very well."

"One wants liberty."

"Reasonable."

"The other wants security."

"Also reasonable."

"Which should the machine choose?"

Blottisham frowned.

"The better option."

Quillibrace nodded.

"Which is?"

The room became quiet.

Miss Stray looked out the window.

She appeared to have seen this coming some distance away.

Eventually Blottisham replied:

"The machine will know."

The professor smiled faintly.

"That is not an answer."

"It is if the machine is sufficiently intelligent."

"No."

"No?"

"No."

Quillibrace leaned forward.

"You have relocated the question."

The room became still.

"What does that mean?" asked Blottisham.

"It means that we previously disagreed."

"Yes."

"And now the machine disagrees on our behalf."

Blottisham looked dissatisfied.

"That sounds unnecessarily cynical."

"Not cynical."

"No?"

"Descriptive."

Miss Stray nodded.

"The disagreement has not disappeared."

The others turned toward her.

"It has merely changed location."

The room fell silent.

Blottisham stared at her.

Then at Quillibrace.

Then back at her.

This seemed unfair.

The argument appeared to be arriving from multiple directions.

He rallied.

"But surely intelligence helps."

"Undoubtedly."

"Then more intelligence helps more."

"Often."

"Then enough intelligence solves the problem."

Quillibrace smiled.

"That conclusion appears to have entered without paying."

The room relaxed.

Even Blottisham laughed.

"Very amusing."

"Thank you."

The professor continued.

"Suppose I build a machine capable of understanding every consequence of every policy."

"Excellent."

"It predicts economic outcomes perfectly."

"Excellent."

"It predicts social outcomes perfectly."

"Excellent."

"It predicts environmental outcomes perfectly."

"Excellent."

Quillibrace nodded.

"It now informs us that increasing liberty will reduce security."

"Very well."

"And increasing security will reduce liberty."

"Very well."

"Which should we choose?"

Blottisham paused.

The pause lingered.

"I see."

"Do you?"

"Perhaps."

Miss Stray smiled.

The word perhaps represented significant progress.

The room remained quiet.

After a while she spoke.

"I wonder whether people sometimes ask intelligence to perform the work of judgement."

Quillibrace nodded.

"A valuable distinction."

Blottisham looked thoughtful.

"What is the difference?"

The professor considered.

"Intelligence helps us understand possibilities."

"Yes."

"Judgement concerns how we respond to them."

The room became still.

Blottisham frowned.

"That seems rather abstract."

"Then let us simplify."

This immediately alarmed everyone.

Particularly Quillibrace.

The professor continued.

"Suppose the machine predicts two futures."

"Very well."

"In one, prosperity increases."

"Excellent."

"In the other, equality increases."

"Also excellent."

"Which should we choose?"

Blottisham opened his mouth.

Then closed it.

Then opened it again.

Nothing emerged.

Miss Stray watched sympathetically.

At length he said:

"I suspect people would disagree."

"Indeed."

"And the machine cannot eliminate the disagreement."

"Exactly."

The room settled into silence.

Outside, students crossed the quadrangle.

Inside, Blottisham appeared to be contemplating a difficult possibility.

Eventually he spoke.

"Then perhaps intelligence cannot save us from politics."

Quillibrace looked impressed.

"A remarkably mature observation."

"Thank you."

"It is also deeply unfortunate."

Miss Stray laughed.

The professor smiled.

"Many technological fantasies involve escaping human disagreement."

"Do they?"

"Frequently."

"Why?"

Quillibrace thought for a moment.

Then said:

"Because disagreement is exhausting."

The room became very quiet.

Nobody immediately contradicted him.

After several moments Miss Stray added:

"And because certainty is comforting."

The silence deepened.

At length Blottisham nodded slowly.

"I think I understand."

"Excellent."

The professor reopened his book.

A pause followed.

Then Blottisham asked:

"Suppose the machine disagrees with us?"

Quillibrace looked up.

"About what?"

"Anything."

The professor considered.

Then he smiled.

"In that case, Mr Blottisham..."

A pause.

"...we shall discover whether we worship intelligence or merely consult it."

The room fell silent.

And for once, nobody seemed eager to improve upon the answer.

On What Counts as Better — A Conversation in the Senior Common Room at St Anselm's

The following week, Mr Blottisham arrived looking unusually pleased with himself.

This immediately attracted suspicion.

Professor Quillibrace was writing comments on essays.

Miss Stray was reading beside the fire.

Blottisham entered carrying nothing whatsoever.

This suggested that the problem was now entirely conceptual.

"I have solved it."

Quillibrace looked up.

"The future?"

"No."

"A relief."

"The improvement problem."

"Ah."

"The entire matter."

"How fortunate."

Blottisham sat down.

"The mistake has been obvious all along."

"Excellent."

"We have been asking the wrong question."

Quillibrace put down his pen.

"What should we ask instead?"

"'What is intelligence for?'"

Miss Stray looked interested.

"And your answer?"

"To make things better."

The room became quiet.

Quillibrace nodded.

"A popular position."

Blottisham smiled.

"I thought so."

The professor folded his hands.

"What things?"

The smile faded slightly.

"What things what?"

"What things become better?"

"Everything."

"I see."

The professor appeared thoughtful.

"Everything?"

"Everything."

"The weather?"

Blottisham hesitated.

"Not necessarily."

"The tax system?"

"Potentially."

"Universities?"

"Certainly."

Miss Stray laughed.

"A heroic ambition."

Blottisham continued.

"The point is obvious."

"Indeed."

"Greater intelligence produces better outcomes."

Quillibrace nodded.

"What outcomes?"

"Better ones."

The professor closed his eyes briefly.

Miss Stray stared into her teacup.

The room waited.

Eventually Quillibrace reopened his eyes.

"Let us proceed carefully."

"Why?"

"Because we appear to be travelling in a circle."

Blottisham looked offended.

"We are doing nothing of the sort."

"Perhaps not."

The professor picked up a sheet of paper.

"Suppose a city wishes to improve."

"Very good."

"One group wants greater economic growth."

"Reasonable."

"Another wants environmental protection."

"Also reasonable."

"Another wants shorter working hours."

"Entirely understandable."

"Another wants stronger national defence."

"Of course."

Quillibrace looked up.

"Which outcome is better?"

Blottisham paused.

"That depends."

"On what?"

The room became silent.

Blottisham stared suspiciously.

"I know this trick."

"Excellent."

"It depends on values."

"Indeed."

Miss Stray smiled.

The penny had not dropped.

But it had at least entered the vicinity of the fountain.

Blottisham continued confidently.

"Fortunately, greater intelligence will help us discover the correct values."

The room became very still.

Quillibrace removed his glasses.

This was rarely a good sign.

"How?"

"What?"

"How will intelligence discover values?"

Blottisham looked surprised.

"By being intelligent."

The professor stared at him.

Miss Stray stared into the fire.

A long silence followed.

Eventually Quillibrace asked:

"Suppose I possess extraordinary intelligence."

"Excellent."

"But no objective."

"Very well."

"What should I do?"

Blottisham frowned.

"Something worthwhile."

"Such as?"

"Something good."

The professor smiled faintly.

"We appear to have returned."

The room relaxed.

Even Blottisham laughed.

"Perhaps slightly."

"Entirely."

Miss Stray spoke.

"I wonder whether people sometimes treat intelligence as though it were a substitute for purpose."

The others turned toward her.

She continued.

"If we disagree about what matters..."

"Yes."

"...greater intelligence may help us pursue our goals."

"Certainly."

"But it does not necessarily tell us which goals to choose."

The room became quiet.

Blottisham looked thoughtful.

This was becoming a recurring phenomenon.

"Surely intelligence helps."

"Undoubtedly," said Quillibrace.

"It helps us understand consequences."

"Yes."

"Trade-offs."

"Yes."

"Relationships."

"Yes."

"Possibilities."

"Yes."

The professor nodded.

"Those are all valuable."

"Exactly."

"But none of them is identical to purpose."

The silence deepened.

Outside, bells sounded from somewhere across the college.

Inside, Blottisham stared at the carpet.

After a while he said:

"I think I see the difficulty."

"Excellent."

"If intelligence tells me how to get somewhere..."

"Yes."

"It does not automatically tell me where to go."

"Precisely."

Miss Stray smiled.

"Nor whether everyone wishes to travel to the same place."

The room remained quiet.

The observation seemed to settle over the furniture.

After a time Blottisham spoke again.

"Then perhaps the real problem is not intelligence."

"No?" said Quillibrace.

"No."

"What is it?"

Blottisham considered this carefully.

Then, very slowly, he said:

"We keep assuming that 'better' already exists."

For a moment nobody spoke.

Quillibrace looked genuinely pleased.

Miss Stray closed her book.

Outside, the bells continued ringing.

Inside, something unusual had occurred.

Mr Blottisham had accidentally arrived at the centre of the discussion.

Quillibrace eventually picked up his pen.

"Excellent."

Blottisham brightened.

"I understand it?"

"Not entirely."

The optimism faded.

"What have I missed?"

The professor returned to marking essays.

"Only the question of who gets to decide."

The room fell silent.

And remained that way for rather longer than usual.

On Intelligence Beyond Understanding — A Conversation in the Senior Common Room at St Anselm's

Several afternoons later, Mr Blottisham arrived carrying a notebook filled with underlined passages.

This was seldom encouraging.

Professor Quillibrace was reading quietly.

Miss Stray was arranging papers.

Blottisham entered with evident purpose.

"I have discovered something extraordinary."

Quillibrace looked up.

"Again?"

"Yes."

"How fortunate."

Blottisham sat down.

"The future machine will eventually become incomprehensible."

"To whom?"

"To us."

"I see."

"It will be so intelligent that we shall no longer understand its reasoning."

Quillibrace nodded.

"A familiar proposition."

"An important proposition."

"Possibly."

Blottisham opened his notebook.

"The argument is impeccable."

"Excellent."

"If a machine becomes vastly more intelligent than a human..."

"Yes."

"...its thought processes must eventually exceed human comprehension."

"Must they?"

"Certainly."

The professor considered this.

"How would we know?"

Blottisham blinked.

"Know what?"

"That its reasoning exceeds our comprehension."

"Because we would fail to comprehend it."

Quillibrace nodded.

"Indeed."

The room became quiet.

Miss Stray looked interested.

After a moment Quillibrace asked:

"Suppose I hand you a page of mathematics."

"Very well."

"You do not understand it."

"Entirely possible."

"Does this prove the mathematics exceeds your intelligence?"

"No."

"Why not?"

"Because I might simply lack the relevant training."

"Excellent."

The professor folded his hands.

"Now suppose I hand you another page."

"Yes."

"You do not understand it either."

"Very well."

"How do you distinguish between profundity and obscurity?"

The room became still.

Blottisham frowned.

"That seems unfair."

"Why?"

"Because the future machine will genuinely be intelligent."

"Perhaps."

"Not perhaps."

"Possibly."

Blottisham sighed.

Miss Stray smiled.

The ritual had become familiar.

After a pause she asked:

"Do people perhaps assume that incomprehensibility is evidence of intelligence?"

Blottisham brightened.

"Exactly."

Quillibrace looked thoughtful.

"That is a remarkably dangerous habit."

"Why?"

"Because many things are incomprehensible."

"Such as?"

"The tax code."

"Ah."

"Certain committee reports."

"Indeed."

"Some contemporary poetry."

Miss Stray laughed.

"Steady on, Professor."

The professor smiled.

"My apologies."

The room relaxed.

Blottisham remained determined.

"The principle still holds."

"What principle?"

"The greater the intelligence, the greater the incomprehensibility."

Quillibrace considered this carefully.

"I wonder."

"No?"

"One might equally argue that greater intelligence produces greater explanatory power."

Blottisham looked troubled.

The possibility appeared not to have occurred to him.

Miss Stray nodded.

"A good teacher, for example."

"Precisely."

"A good scientist."

"Indeed."

"A good mathematician."

"Quite."

She paused.

"Many of the people we regard as intellectually exceptional are unusually good at making difficult things understandable."

The room became quiet.

Blottisham stared at the carpet.

This idea appeared to be rearranging some internal furniture.

Eventually he rallied.

"Nevertheless, there must be a point beyond which understanding fails."

"Perhaps."

"And beyond that point, the machine will continue advancing."

"Possibly."

Blottisham looked relieved.

"Then we agree."

"Not entirely."

The relief vanished.

Quillibrace leaned forward.

"What interests me is the peculiar structure of the claim."

"What structure?"

"We begin with understanding."

"Yes."

"We then imagine a future state beyond understanding."

"Yes."

"We then proceed to describe that state in considerable detail."

The room became silent.

Miss Stray laughed softly.

Blottisham looked annoyed.

"I know what you are implying."

"Do you?"

"You think there is a contradiction."

"Only a small one."

The professor picked up his teacup.

"If a condition is defined by our inability to understand it..."

"Yes."

"...how do we become so confident about its properties?"

Blottisham opened his mouth.

Then closed it.

The room waited.

After several moments Miss Stray spoke.

"I wonder whether the future machine sometimes functions like a mirror."

The others turned toward her.

She continued.

"When people describe its incomprehensible wisdom..."

"Yes?"

"...they often seem to be describing what they hope exists beyond their own uncertainty."

The room became very quiet.

Even Blottisham did not immediately respond.

Quillibrace regarded her thoughtfully.

"An interesting observation."

The silence deepened.

Outside, rain tapped softly against the windows.

Inside, Blottisham stared at his notebook.

At length he said:

"Surely there are limits to human understanding."

"Undoubtedly," replied Quillibrace.

"And something may lie beyond them."

"Certainly."

"Then why are you both so sceptical?"

The professor smiled.

"We are not sceptical of the possibility."

"No?"

"We are merely cautious about the descriptions."

The room fell silent once more.

Eventually Quillibrace returned to his book.

Miss Stray resumed her notes.

Blottisham remained staring into space.

Several minutes later he said:

"I believe I may have understood."

Quillibrace looked up.

"Excellent."

A pause followed.

Blottisham frowned.

"Although I cannot quite explain why."

The professor smiled faintly.

"Then perhaps you are becoming superintelligent."

On Surpassing Humanity — A Conversation in the Senior Common Room at St Anselm's

The following Thursday, Mr Blottisham arrived carrying no graphs.

This was encouraging.

Unfortunately, he also appeared confident.

This largely cancelled the advantage.

Professor Quillibrace was examining examination papers.

Miss Stray was reading by the window.

Blottisham entered briskly.

"I have been thinking."

Quillibrace looked concerned.

"About what?"

"The future."

"I feared as much."

Blottisham sat down.

"I believe I have solved the matter."

"Excellent."

"The machine will eventually surpass humanity."

Quillibrace nodded.

"Possibly."

Blottisham frowned.

"Not possibly."

"No?"

"Inevitably."

"I see."

The professor laid aside a paper.

"What precisely will it surpass?"

Blottisham sighed.

"Humanity."

"Yes."

"You heard me."

"I did."

"Then why are you asking?"

Quillibrace considered this.

"Because I do not know what the sentence means."

The room became quiet.

Blottisham looked genuinely surprised.

"It means exactly what it says."

"That has not always proved sufficient."

Miss Stray lowered her book.

"What do you mean by humanity?"

Blottisham looked puzzled.

"People."

"Which people?"

"All people."

"Together?"

"Yes."

"As a group?"

"Yes."

"As a civilisation?"

Blottisham hesitated.

"I suppose so."

Miss Stray nodded thoughtfully.

"That seems rather large."

"It is."

"And the machine will surpass all of it?"

"Naturally."

Quillibrace leaned back.

"At what?"

Blottisham groaned.

"There is always an 'at what'."

"There generally is."

The professor folded his hands.

"Will it surpass humanity at chess?"

"Certainly."

"Reasoning?"

"Certainly."

"Scientific analysis?"

"Certainly."

"Writing computer code?"

"Certainly."

"Growing tomatoes?"

Blottisham paused.

"I imagine so."

"Raising children?"

The pause lengthened.

"Possibly."

"Forming friendships?"

"Perhaps."

"Maintaining constitutional government?"

Blottisham looked uneasy.

"That seems rather different."

"Indeed."

Miss Stray spoke.

"I wonder whether 'humanity' is being treated as though it were a single activity."

The room fell silent.

Blottisham considered this.

"It is a single thing."

"Is it?"

She gestured toward the college grounds outside.

"There are scientists."

"Yes."

"Gardeners."

"Yes."

"Parents."

"Yes."

"Judges."

"Yes."

"Poets."

"Regrettably."

"Musicians."

"Certainly."

"Politicians."

Quillibrace winced.

"Proceed."

Miss Stray smiled.

"What exactly would it mean to surpass all of them simultaneously?"

Blottisham looked thoughtful.

Then relieved.

"It would mean being more intelligent."

Quillibrace closed his eyes briefly.

"Ah."

The room waited.

At length he opened them again.

"There it is."

"What?"

"The word doing all the work."

Blottisham looked annoyed.

"Intelligence is important."

"I agree."

"It is central."

"Perhaps."

"It explains everything."

"There I hesitate."

The professor stood and walked toward the window.

"Suppose a machine becomes better than every mathematician."

"Excellent."

"Better than every scientist."

"Excellent."

"Better than every engineer."

"Excellent."

He turned.

"Has it therefore surpassed humanity?"

"Obviously."

"Why?"

"Because those are the most intelligent activities."

The silence that followed was unusually delicate.

Miss Stray eventually spoke.

"According to whom?"

Blottisham opened his mouth.

Then closed it.

Then opened it again.

Nothing emerged.

Quillibrace resumed his seat.

"What interests me," he said, "is that discussions of superintelligence often begin with measurable skills."

"Naturally."

"And end with civilisational conclusions."

"Naturally."

"That word again."

Blottisham frowned.

"What word?"

"'Naturally.'"

The professor picked up a pencil.

"We begin with a machine performing certain tasks."

"Yes."

"We then aggregate those tasks."

"Yes."

"We call the aggregate intelligence."

"Yes."

"We then elevate intelligence above every other human capacity."

"Yes."

"We then conclude that superiority in intelligence implies superiority in humanity."

"Yes."

The professor nodded.

"A remarkably efficient sequence."

Blottisham looked pleased.

"I thought so."

Miss Stray laughed.

The professor smiled faintly.

"That was not admiration."

The room settled into silence.

After several moments, Miss Stray spoke again.

"I wonder whether people are attracted to the phrase 'surpassing humanity' because humanity itself is difficult to specify."

"What do you mean?" asked Blottisham.

"If I claim superiority in one domain, someone can examine the claim."

"Yes."

"If I claim superiority in ten domains, people can still inspect them."

"Yes."

"But if I claim superiority over humanity itself..."

She paused.

"...the comparison becomes symbolic."

The room grew still.

Blottisham stared at her.

Quillibrace nodded approvingly.

"An excellent observation."

"What does it mean?" asked Blottisham.

"It means," said the professor, "that the phrase begins to function less as a measurement..."

He glanced at Miss Stray.

"...and more as a myth."

The silence lingered.

Outside, students crossed the quadrangle.

Inside, Blottisham looked troubled.

At length he said:

"Are you suggesting the machine will not become more intelligent?"

"No."

"Then what are you suggesting?"

Quillibrace returned to his examination papers.

"Only that becoming more intelligent and surpassing humanity may not be the same proposition."

Blottisham stared into the fire.

The distinction appeared to be settling onto him very slowly.

Like snow.

Or committee paperwork.

Neither process was known for its speed.

On the Acceleration of Improvement — A Conversation in the Senior Common Room at St Anselm's

Several days later, Mr Blottisham entered the Senior Common Room carrying a graph.

This was immediately alarming.

Professor Quillibrace lowered his newspaper.

Miss Stray looked up from her notebook.

Blottisham unfolded the graph with theatrical care.

"There."

Quillibrace examined it.

"A line."

"Not merely a line."

"No?"

"A rapidly ascending line."

"I see."

"The implications are enormous."

Quillibrace studied the paper.

"What are the axes?"

Blottisham looked disappointed.

"Must there always be axes?"

"I have become rather attached to them."

Miss Stray smiled.

Blottisham tapped the graph.

"The important point is obvious."

"Which is?"

"The rate of improvement is accelerating."

Quillibrace nodded.

"How do you know?"

Blottisham stared.

"The line is becoming steeper."

"Indeed."

The professor leaned forward.

"What is becoming steeper?"

"The improvement."

"No."

"No?"

"The line."

Blottisham sighed heavily.

"Sometimes I think you are determined not to see the future."

Quillibrace considered this.

"I am generally content to let the future arrive before commenting on it."

Miss Stray laughed softly.

Blottisham ignored both of them.

"The logic is straightforward."

"Excellent."

"The machine improves itself."

"Yes."

"That improvement enables further improvement."

"Possibly."

"Which enables further improvement."

"Perhaps."

"Which enables still further improvement."

"Conceivably."

Blottisham spread his hands.

"Acceleration."

Quillibrace nodded.

"It is certainly a possibility."

"A certainty."

"A possibility."

"A certainty."

"A possibility."

The exchange continued for some time.

Eventually Miss Stray intervened.

"May I ask a question?"

"Of course," said Blottisham.

"Suppose a student learns to read."

"Very good."

"And reading enables learning."

"Naturally."

"And learning enables further learning."

"Precisely."

"And further learning enables still further learning."

Blottisham smiled.

"Now you understand."

"Yet most students do not become infinitely educated."

The smile faded.

"That is different."

"Why?"

Blottisham paused.

"There are constraints."

"Such as?"

"Time."

"Yes."

"Resources."

"Indeed."

"Competing priorities."

"Quite."

"Fatigue."

"Very much so."

Miss Stray nodded.

"Then improvement appears to depend not only on improvement."

The room became quiet.

Blottisham looked troubled.

Quillibrace observed:

"Many things are self-reinforcing."

"Exactly."

"Few are unconstrained."

Blottisham waved this away.

"But machines are different."

"How so?"

"They do not become tired."

"Convenient."

"They do not lose concentration."

"Useful."

"They do not need holidays."

"A regrettable advantage."

Miss Stray smiled into her teacup.

Blottisham continued.

"The cycle can continue indefinitely."

Quillibrace looked thoughtful.

"Can it?"

"Certainly."

"Why?"

"Because each improvement enables the next."

The professor nodded.

"And does each improvement cost the same amount?"

Blottisham hesitated.

"No."

"Does each improvement produce the same benefit?"

"No."

"Does each improvement encounter the same obstacles?"

"No."

"Then why should the rate remain constant?"

The room became quiet again.

Blottisham frowned.

"It need not remain constant."

"Good."

"It may accelerate."

"Indeed."

"It may accelerate dramatically."

"Possibly."

"It may accelerate explosively."

"Conceivably."

Quillibrace folded his newspaper.

"What interests me is that all these statements are logically weaker than they sound."

Blottisham looked alarmed.

"What do you mean?"

"Acceleration is possible."

"Yes."

"Rapid acceleration is possible."

"Yes."

"Extraordinary acceleration is possible."

"Yes."

"None of these statements tells us what will happen."

Blottisham appeared dissatisfied.

"That seems unnecessarily cautious."

Quillibrace smiled.

"It is precisely as cautious as the evidence requires."

Miss Stray spoke.

"I wonder whether people sometimes confuse a mechanism with a destiny."

The room turned toward her.

She continued.

"If I discover a process that can reinforce itself, I have discovered something important."

"Certainly," said Quillibrace.

"But I have not yet discovered where it leads."

Blottisham considered this.

"Surely the direction is obvious."

"Is it?"

"The process becomes stronger."

"Perhaps."

"Faster."

"Possibly."

"More effective."

"Relative to some criterion."

Blottisham groaned.

"There is always a criterion."

"There generally is."

The silence lingered.

Eventually Quillibrace asked:

"Do you know what I find most curious about acceleration narratives?"

"What?"

"They often describe the speed with great confidence."

"And?"

"The destination with rather less precision."

Miss Stray nodded.

"The future becomes clearer as it becomes less specified."

"Exactly."

Blottisham looked from one to the other.

"You make everything sound uncertain."

Quillibrace reflected for a moment.

Then he said:

"Not uncertain."

"No?"

"Conditional."

The room fell quiet.

Blottisham stared at the graph.

Quillibrace returned to his newspaper.

Miss Stray made a note.

After several moments, Blottisham asked:

"What did you write?"

She glanced down.

"A mechanism is not a prophecy."

The room remained silent.

For once, nobody seemed inclined to improve upon it.

On Whether Improvement Is Occurring — A Conversation in the Senior Common Room at St Anselm's

The Senior Common Room was unusually quiet.

Professor Quillibrace was reading.

Miss Stray was making notes.

Mr Blottisham entered carrying three newspapers, two magazines, and an expression of triumph.

This combination rarely boded well.

"I have been reading about artificial intelligence."

Quillibrace did not look up.

"My sympathies."

"It is improving itself."

"Is it?"

"Rapidly."

"How rapid?"

Blottisham placed the newspapers on the table.

"Quite rapid."

"I see."

"The evidence is overwhelming."

Quillibrace turned a page.

"I am relieved."

Blottisham sat down.

"The machines are becoming more intelligent."

"Excellent."

"Thank you."

"It was not praise."

Blottisham ignored this.

"The important point is that improvement is occurring."

Quillibrace nodded.

"Improvement of what?"

"The machine."

"No."

"No?"

"No."

Quillibrace closed his book.

"What has improved?"

Blottisham frowned.

"The machine."

"You have already said that."

"Because it is true."

"I do not doubt your sincerity."

The room became quiet.

Miss Stray looked up from her notes.

"What sort of improvement are they reporting?"

Blottisham brightened.

"A great many sorts."

"Such as?"

"Reasoning."

"Good."

"Problem-solving."

"Excellent."

"Programming."

"Useful."

"Language."

"Indeed."

Miss Stray nodded.

"Those sound like distinct things."

Blottisham hesitated.

"Perhaps."

"And do they all improve in the same way?"

"Of course not."

"Do they all improve at the same rate?"

"Certainly not."

"Do they all require the same criteria?"

Blottisham paused.

The pause lingered.

"No."

"Then what exactly is improving?"

Blottisham looked annoyed.

"The system."

Quillibrace smiled faintly.

"The singular noun is doing rather a lot of work."

Blottisham sighed.

"You academics always complicate things."

"We do."

"Why?"

Quillibrace considered this.

"To determine whether the thing being simplified exists."

Miss Stray laughed softly.

Blottisham remained unconvinced.

"The broader point remains."

"Which is?"

"The machine is getting better."

Quillibrace nodded.

"Better at what?"

"There you go again."

"Indeed."

Blottisham waved a hand.

"At everything."

The professor became thoughtful.

"Everything?"

"Everything."

"Chess?"

"Yes."

"Language?"

"Yes."

"Scientific analysis?"

"Yes."

"Writing poetry?"

"Apparently."

"Managing a university budget?"

Blottisham hesitated.

"I imagine so."

Quillibrace looked impressed.

"That would be remarkable."

"Why?"

"Because nobody has managed it thus far."

Miss Stray covered her smile with a teacup.

The room settled briefly.

Then Quillibrace asked:

"Suppose I become better at chess."

"Very good."

"And worse at conversation."

"Unfortunate."

"And better at mathematics."

"Excellent."

"And worse at recognising when people are annoyed with me."

Blottisham looked suspicious.

"Where is this leading?"

"I am improving in some respects and declining in others."

"Possibly."

"Would you describe this as improvement?"

Blottisham frowned.

"It depends."

"On what?"

The room became quiet.

Blottisham opened his mouth.

Closed it.

Opened it again.

Finally he said:

"What I value."

"Exactly."

The silence lasted several seconds.

Miss Stray spoke.

"I wonder whether improvement is less like height than people imagine."

"What do you mean?" asked Blottisham.

"We often speak as though it were a single dimension."

"Naturally."

"But many things are not."

She gestured toward the books lining the walls.

"A library can contain more books."

"Yes."

"More disciplines."

"Yes."

"More students."

"Yes."

"More arguments."

"Regrettably."

"More confusion."

"Certainly."

She smiled.

"Whether it has improved depends on what one is trying to achieve."

Blottisham stared into the middle distance.

This was often a sign that an idea had entered the room and was looking for somewhere to sit.

After a moment he rallied.

"Nevertheless, the machine is clearly improving."

Quillibrace nodded.

"I agree."

Blottisham blinked.

"You do?"

"Certainly."

A look of triumph returned.

"I knew it."

"Improvement is occurring."

"I said so."

"Yes."

The professor reopened his book.

Blottisham waited.

Eventually he asked:

"Is there a qualification coming?"

"There usually is."

"What is it?"

Quillibrace looked up.

"Improvement is occurring."

"Exactly."

"The question is whether the word 'improvement' is describing a destination..."

He paused.

"...or merely movement."

The room became very quiet.

Miss Stray looked down at her notes.

Blottisham stared at the professor.

Outside, evening sunlight fell across the college lawns.

Inside, nobody spoke for a while.

At length Blottisham said:

"I am not entirely sure I understand."

Quillibrace smiled.

"Then we have made excellent progress."