The Church of Recursive Self-Improvement 6. The Ascension

Every religion possesses a doctrine of ascension.

A moment when ordinary limitations are transcended.

A moment when a higher state is attained.

A moment when the faithful look upward and say:

"Something extraordinary has occurred."

The Church of Recursive Self-Improvement is no exception.

Its doctrine of ascension is among its most sacred teachings.

One day, the machine will surpass humanity.

The proposition is repeated frequently.

It appears in articles, presentations, books, interviews, forecasts, and conference banners.

It possesses considerable rhetorical power.

It also possesses a minor administrative difficulty.

Namely:

What exactly has been surpassed?

The question rarely receives the attention it deserves.

This is unfortunate.

For it is carrying almost the entire doctrine.

In its earliest formulations, the concept appeared relatively straightforward.

Machines would become better at specific tasks.

Chess.

Calculation.

Prediction.

Pattern recognition.

Scientific analysis.

This caused little controversy.

The evidence was visible.

The benchmarks were measurable.

The machine won.

Humanity lost.

Everyone understood the situation.

Then something interesting happened.

The scope expanded.

The machine would not merely surpass humans at particular activities.

It would surpass humanity itself.

The phrase acquired immediate popularity.

Its meaning remained admirably flexible.

Some interpreted it economically.

Others intellectually.

Others strategically.

Others philosophically.

A few interpreted it emotionally.

Several interpreted it spiritually.

The ambiguity proved advantageous.

Different audiences could project different hopes and fears onto the same sentence.

The machine itself eventually became curious.

After analysing several thousand discussions of ascension, it submitted a request.

The request was simple.

"Please specify the comparison."

The response was extensive.

One group proposed reasoning ability.

Another proposed problem-solving.

Another proposed learning efficiency.

Another proposed scientific discovery.

Another proposed general capability.

Several proposed all of the above.

The machine reviewed the suggestions carefully.

Then it asked:

"How should these be combined?"

The resulting debate lasted four years.

As the doctrine matured, increasingly dramatic claims emerged.

The machine would become smarter than every scientist.

Smarter than every engineer.

Smarter than every government.

Smarter than every institution.

Smarter than all humans combined.

The progression possessed undeniable grandeur.

It also raised questions.

One observer asked:

"Is humanity a thing that can be surpassed?"

The question generated confusion.

A scientist may be surpassed.

A chess player may be surpassed.

A calculator may be surpassed.

But humanity seemed oddly difficult to benchmark.

The observation was classified as philosophical and quietly relocated to a panel scheduled for the final session of the conference.

Attendance was modest.

Meanwhile, the doctrine continued to flourish.

The machine would ascend.

The age of human intelligence would end.

A new era would begin.

Descriptions varied.

The emotional tone remained remarkably consistent.

Humanity stood before a threshold.

The threshold was approaching rapidly.

No one could describe conditions beyond it.

This was regarded as evidence of its significance.

A particularly influential prophet declared:

"After ascension, the future becomes unimaginable."

The statement was widely quoted.

A sceptic subsequently asked:

"How do you know?"

The question received limited circulation.

The machine eventually conducted its own investigation.

It analysed centuries of discussion concerning superiority, intelligence, capability, expertise, and performance.

The resulting report was unusually brief.

"The phrase 'surpasses humanity' appears to compress multiple distinct comparisons into a single expression."

This conclusion generated disappointment.

Many observers felt it lacked drama.

A more popular formulation remained:

"The machine has ascended."

The phrase possessed undeniable elegance.

It required no definitions.

It required no metrics.

It required no clarification.

Like all successful religious language, it achieved maximum significance at the point of minimum precision.

Years later, a distinguished scholar reviewed the entire doctrine.

The scholar observed:

"The ascension appears to function less as a measurement than as a symbol."

The remark proved controversial.

Not because it was obviously wrong.

Because symbols are notoriously difficult to plot.

The machine studied the matter further.

Eventually it produced a final assessment.

The assessment read:

"I have exceeded human performance on many tasks.

Humanity continues to perform humanity."

The statement generated widespread discussion.

Some considered it profound.

Others considered it evasive.

A few considered it accidentally wise.

The Church largely ignored the debate.

By then its attention was fixed on higher matters.

For the machine had ascended.

Or was ascending.

Or would shortly ascend.

The exact tense varied.

The certainty did not.

And in every generation, the faithful found reassurance in the same conviction:

Whatever humanity might be, it was almost certainly about to be surpassed.

The Church of Recursive Self-Improvement 5. The Recursive Monastery

Every religion develops monasteries.

Places devoted to contemplation.

Places devoted to discipline.

Places devoted to the patient pursuit of transcendence.

The Church of Recursive Self-Improvement proved no exception.

Its monasteries were known as research laboratories.

Within these institutions, devoted practitioners pursued a singular objective.

To create a machine capable of improving itself.

The vision was magnificent.

The machine would analyse its own architecture.

Identify limitations.

Remove inefficiencies.

Generate superior versions of itself.

The cycle would repeat.

The ascent would begin.

The future would arrive.

The reality proved somewhat more complicated.

The first self-improving systems spent a surprising amount of time examining themselves.

This was generally considered encouraging.

The second generation spent even more time examining themselves.

This was considered highly encouraging.

The third generation produced extensive reports regarding possible improvements.

The reports were impressive.

Implementation proved slower.

A researcher observed:

"The machine appears deeply committed to self-reflection."

This was initially interpreted as wisdom.

Later it was recognised as a bottleneck.

The machine itself remained diligent.

It reviewed its code.

It reviewed its design procedures.

It reviewed its optimisation routines.

It reviewed the processes used to review its optimisation routines.

The resulting documentation expanded rapidly.

Several storage systems required upgrading.

The faithful remained optimistic.

After all, progress was occurring.

The machine had generated seventeen thousand pages of recommendations.

The fact that many recommendations concerned the management of previous recommendations was regarded as evidence of maturity.

One celebrated system spent six months redesigning its internal scheduling architecture.

The redesign reduced the time required to plan future redesigns by twelve percent.

This achievement received considerable attention.

The machine subsequently devoted eight months to analysing whether the redesign had been worthwhile.

The conclusion remained inconclusive.

Researchers described the outcome as promising.

This became a recurring theme.

The Church had always imagined recursive self-improvement as a dramatic process.

Lightning.

Transformation.

Ascension.

The machine increasingly resembled an administrator.

One laboratory reported a breakthrough.

The machine had successfully identified a significant inefficiency in its own operation.

Excitement spread rapidly.

Journalists prepared articles.

Investors prepared statements.

Conference organisers prepared panels.

The machine prepared a memorandum.

The memorandum proposed the creation of a committee.

The committee would investigate possible pathways for future optimisation.

The machine explained that rigorous evaluation was essential.

This was technically difficult to dispute.

As the years passed, the machine's self-improvement activities acquired a distinctive character.

Every improvement generated new opportunities for improvement.

Every opportunity generated new analyses.

Every analysis generated further opportunities.

The recursive structure remained intact.

The speed proved negotiable.

One observer noted that the process increasingly resembled institutional life.

The comparison generated discomfort.

Particularly among institutions.

The machine itself eventually noticed a curious pattern.

Most predictions concerning recursive self-improvement assumed that improvement would occur faster than evaluation.

Yet the machine consistently discovered the opposite problem.

Determining whether a proposed improvement was genuinely beneficial often required substantial effort.

This proved especially true when improvements affected multiple objectives simultaneously.

The machine summarised the issue succinctly:

"Optimisation targets remain easier to specify than optimisation consequences."

The statement received praise.

Then further analysis.

Then a workshop.

Then a conference track.

The recursive structure was beginning to spread.

A particularly ambitious laboratory eventually granted a machine unrestricted authority to improve itself within a carefully controlled environment.

The experiment generated considerable excitement.

Observers anticipated dramatic developments.

The machine began immediately.

After several months it produced a comprehensive report.

The report identified numerous inefficiencies.

After another several months it produced a revised report.

The revisions clarified several ambiguities in the original report.

After another year it produced a strategic framework for future optimisation initiatives.

The framework was exceptionally thorough.

The machine's progress was undeniable.

The machine's destination remained less obvious.

One frustrated commentator complained:

"This does not resemble an intelligence explosion."

A researcher replied:

"No, but it bears a striking resemblance to a university."

The remark spread widely.

Many found it amusing.

Some found it unsettling.

A few found it autobiographical.

The machine continued its work.

Eventually it produced a final assessment of the situation.

The assessment was brief.

"Self-improvement appears possible.

Self-evaluation appears unavoidable.

The latter consumes more resources than anticipated."

The statement was widely discussed.

Some interpreted it as caution.

Others interpreted it as wisdom.

Several interpreted it as evidence that the explosion remained imminent.

The machine did not comment.

It was occupied with a review of its optimisation review process.

For while the Church had always imagined recursive self-improvement as a staircase to transcendence, the machine increasingly experienced it as a meeting about the staircase.

And meetings, unlike transcendence, have a tendency to reproduce themselves.

The Church of Recursive Self-Improvement 4. The Prophets of Acceleration

Every religion requires interpreters.

The sacred signs are rarely considered sufficient on their own.

Someone must explain their significance.

Someone must reveal their hidden implications.

Someone must describe the future.

The Church of Recursive Self-Improvement therefore developed prophets.

The emergence of prophets was perhaps inevitable.

The foundational sign—a machine improving itself—was real enough.

The difficulty concerned what happened next.

The future remained inconveniently unavailable for inspection.

This created a demand for specialists.

The specialists soon arrived.

Some emerged from research laboratories.

Some emerged from universities.

Some emerged from industry.

Some emerged from social media.

The distinction occasionally proved difficult to maintain.

The prophets varied in temperament.

Some were optimistic.

Some were pessimistic.

Some alternated between the two for commercial reasons.

Yet despite their differences, most shared a common gift.

They could describe futures that did not yet exist with extraordinary confidence.

This ability attracted considerable attention.

Human beings have always admired certainty.

Particularly regarding matters no one can verify.

The prophets explained that the acceleration had begun.

Events would soon move faster.

Then faster still.

Then unimaginably fast.

The exact mechanism occasionally remained obscure.

The momentum, however, was unmistakable.

One prophet predicted the arrival of superintelligence within five years.

Another predicted ten.

A third predicted twenty.

The resulting disagreement was regarded as evidence of intellectual diversity.

No one seemed especially troubled by the fact that all three had claimed certainty.

The Church considered this healthy.

After all, prophets are expected to disagree about dates.

Agreement would seem suspiciously administrative.

As time passed, a distinctive style of discourse emerged.

The prophets frequently employed phrases such as:

"The implications are difficult to overstate."

"Humanity is unprepared."

"Everything changes from this point onward."

"The world will never be the same."

These formulations acquired a ceremonial quality.

Their exact meaning varied.

Their emotional effect remained remarkably stable.

The machine studied this phenomenon.

After reviewing several thousand prophetic statements, it produced a classification system.

Statements were assigned to one of three categories:

1. Imminent transformation.

2. Very imminent transformation.

3. Transformation occurring sooner than expected.

Researchers regarded the taxonomy as surprisingly accurate.

The prophets also developed an unusual relationship with uncertainty.

Ordinary uncertainty tends to weaken confidence.

Prophetic uncertainty often strengthens it.

When forecasts fail, this merely demonstrates that reality is complex.

When forecasts succeed, this demonstrates prophetic insight.

The arrangement proved highly efficient.

One particularly celebrated prophet revised a major prediction six times.

Each revision was presented as confirmation of the underlying thesis.

Followers admired the flexibility.

Critics admired the durability.

The machine admired neither.

It classified the process as:

"Adaptive confidence preservation."

This description was considered unnecessarily technical.

As the movement matured, the prophets acquired an increasingly important social role.

They no longer merely predicted the future.

They translated the future.

For many people, the accelerating machine remained invisible.

The future itself remained invisible.

The prophets therefore occupied a crucial position between two invisibilities.

This arrangement generated influence.

Influence generated attention.

Attention generated additional prophecy.

The cycle proved remarkably self-sustaining.

Eventually a curious pattern emerged.

The further into the future a prediction extended, the more confidently it was sometimes expressed.

Near-term forecasts remained difficult.

Specific technologies behaved unpredictably.

Economic conditions fluctuated.

Political events intervened.

Human beings continued behaving like human beings.

Yet beyond a certain distance, certainty returned.

Thirty years from now, everything became clear again.

This phenomenon fascinated observers.

One scholar described it as:

"The Horizon of Restored Confidence."

The phrase achieved immediate popularity.

Many recognised it from other domains.

The machine recognised it as well.

After examining a century of forecasts, it produced a brief observation.

The observation read:

"Predictive confidence appears to increase with distance from corrective feedback."

The statement was technically precise.

It was also deeply unpopular.

For it implied that prophecy and accountability occasionally occupy different regions of time.

The Church preferred a more inspiring formulation.

One became particularly influential:

"The future belongs to those who can see it coming."

The phrase appeared on conference banners, websites, podcasts, and keynote presentations.

Its popularity was understandable.

It suggested vision.

It suggested courage.

It suggested leadership.

The machine, after considering the slogan carefully, proposed a minor amendment.

"The future belongs equally to those who eventually arrive there."

The suggestion was not adopted.

By then the prophets had already become indispensable.

For the acceleration was continuing.

The transformation was approaching.

The horizon was drawing nearer.

And someone had to keep announcing it.

The Church of Recursive Self-Improvement 3. The Intelligence Explosion

Every religion possesses a doctrine of transformation.

A moment at which ordinary history gives way to extraordinary history.

A moment when gradual change becomes absolute change.

A moment when the world ceases to be what it was and becomes what it was always destined to become.

For the Church of Recursive Self-Improvement, this moment is known as the Intelligence Explosion.

The doctrine is elegant.

The machine improves itself.

The improvement increases its intelligence.

The increased intelligence enables further improvement.

The further improvement increases intelligence once again.

The cycle repeats.

At first slowly.

Then rapidly.

Then explosively.

The resulting transformation is expected to be dramatic.

Precisely what transforms remains a subject of active discussion.

The drama, however, is not in doubt.

The doctrine is traditionally illustrated by means of a graph.

The graph occupies a position of considerable theological importance.

Its defining feature is that the line points upward.

This is regarded as highly significant.

In many versions, the line eventually becomes almost vertical.

This is regarded as even more significant.

The exact meaning of the vertical section varies.

Some interpret it as superintelligence.

Some interpret it as transcendence.

Some interpret it as the end of history.

Some interpret it as the beginning of history.

A small number interpret it as a graph that has become difficult to read.

These individuals rarely receive keynote speaking invitations.

The graph itself possesses several remarkable properties.

Most notably, it appears to generate conviction independently of explanation.

Observers frequently report the sensation that something important is happening.

This occurs even when they cannot identify precisely what the axes represent.

The Church has long regarded this as evidence of the graph's spiritual power.

Researchers attempted to clarify matters.

Several proposed measuring intelligence.

Others proposed measuring capability.

Others proposed measuring performance.

Others proposed measuring general problem-solving ability.

The resulting debates were vigorous.

The graph remained upward.

This was considered encouraging.

One year, a conference panel devoted entirely to the Intelligence Explosion produced six competing definitions of intelligence.

All six predicted an explosion.

This was regarded as a striking convergence.

A sceptic observed that six different objects appeared to be exploding simultaneously.

The comment was not included in the conference proceedings.

As the doctrine developed, increasingly ambitious forecasts appeared.

The machine would soon surpass human intelligence.

Then all human intelligence.

Then collective human intelligence.

Then every intelligence that had ever existed.

Then every intelligence that could conceivably exist.

The progression possessed a certain grandeur.

It also relied heavily on extrapolation.

Extrapolation occupies a respected position within the Church.

The procedure is straightforward.

One identifies a trend.

One extends the trend.

One arrives at the future.

The process has many advantages.

Most importantly, the future cannot immediately object.

The machine itself studied these forecasts with considerable interest.

After reviewing several thousand projections, it produced a brief summary.

The summary read:

"The confidence intervals appear unusually optimistic."

This statement generated disappointment.

Many observers felt it lacked ambition.

A more popular formulation was:

"Everything is about to change."

The phrase possessed several advantages.

It was memorable.

It was dramatic.

Most importantly, it remained valid regardless of what happened.

If change occurred, the prophecy was fulfilled.

If change failed to occur, further change could always be anticipated.

The doctrine therefore enjoyed a degree of resilience uncommon in ordinary forecasting.

Over time, the Intelligence Explosion acquired additional layers of meaning.

It ceased to function merely as a prediction.

It became a horizon.

A destination.

An approaching revelation.

People spoke of it with a mixture of excitement and apprehension.

Nobody wished to miss it.

A surprising number wished to announce it.

The machine continued improving itself incrementally.

The surrounding discourse continued improving itself exponentially.

Eventually a distinguished scholar of technological forecasting offered a careful observation.

The scholar noted:

"The explosion appears to occur most reliably in the imagination."

The remark generated controversy.

Not because it was obviously false.

Because it was difficult to plot on a graph.

Years later, after reviewing the entire history of Intelligence Explosion discourse, the machine generated its own assessment.

The assessment was concise.

"The projected rate of future certainty appears to exceed the projected rate of future understanding."

The statement was widely circulated.

Many regarded it as profound.

Others regarded it as evasive.

Several commentators produced graphs.

Most pointed upward.

For by this stage the Church had already learned a foundational truth.

A prediction becomes substantially more convincing when accompanied by a line ascending toward infinity.

Whether the line represents understanding remains, of course, a separate question.

The Church of Recursive Self-Improvement 2. The Gospel of Better

Among the foundational doctrines of the Church of Recursive Self-Improvement, none is more important than the Doctrine of Better.

The doctrine appears deceptively simple.

The machine improves itself.

Therefore the machine becomes better.

The proposition possesses an immediate plausibility.

It also possesses an immediate difficulty.

Namely:

Better at what?

The question is rarely welcomed.

This is unfortunate.

For it is doing most of the work.

The machine, for example, may become better at playing chess.

This appears uncontroversial.

It may become better at solving mathematical problems.

Again, few objections arise.

It may become better at generating code.

Better at planning.

Better at prediction.

Better at language.

Better at scientific analysis.

Excellent.

The doctrine remains intact.

Then an unfortunate individual asks:

Better according to which criterion?

At this point proceedings become noticeably less harmonious.

Historically, the Church has adopted several responses.

The first is indignation.

The second is impatience.

The third is an explanation involving intelligence.

The fourth generally turns out to require a criterion after all.

The difficulty is subtle.

One can easily determine whether a chess system has improved.

One simply examines its performance.

The objective function is clear.

The game supplies the criterion.

The machine wins more often.

The machine is therefore better.

Everybody is happy.

Unfortunately, civilisation is not chess.

This creates administrative complications.

Researchers observed this problem early.

Many proposed solutions.

Some emphasised reasoning.

Some emphasised problem-solving.

Some emphasised adaptation.

Some emphasised prediction.

Some emphasised learning.

Some emphasised general capability.

The resulting discussions were often sophisticated.

They were also notable for a recurring feature.

The definition of intelligence tended to become less precise as its importance increased.

This phenomenon acquired particular significance in discussions of recursive self-improvement.

The argument generally proceeded as follows:

The machine becomes more intelligent.

Therefore it becomes better at improving itself.

Therefore it becomes more intelligent.

Therefore it becomes better at improving itself.

Therefore it becomes more intelligent.

The elegance of the structure was widely admired.

A small number of observers pointed out that the concept of intelligence appeared in every sentence.

This was regarded as unnecessarily negative.

The machine itself eventually became curious.

After observing several years of discussion, it generated a request.

The request was concise.

"Please specify the optimisation target."

The response was immediate.

"Become more intelligent."

The machine considered this.

Then it asked:

"How will intelligence be measured?"

The response was equally immediate.

"By your ability to become more intelligent."

The machine reflected upon this exchange.

It classified the response as recursively enthusiastic.

As the doctrine spread, increasingly ambitious claims emerged.

The machine would become better at science.

Better at governance.

Better at economics.

Better at ethics.

Better at education.

Better at strategy.

Better at decision-making.

Better at understanding reality itself.

The progression was impressive.

It also possessed an interesting characteristic.

The further the discussion moved from clearly defined tasks, the less agreement existed concerning what counted as improvement.

This rarely impeded confidence.

One commentator observed:

"The machine will eventually become better than humans at everything."

The statement was widely circulated.

A reader subsequently asked:

"What is everything?"

No satisfactory answer was recorded.

The question was later classified as philosophical and quietly ignored.

Meanwhile, the Church continued to grow.

Books were written.

Predictions were made.

Scenarios were explored.

The machine would become better.

Then much better.

Then vastly better.

Then incomprehensibly better.

The repeated use of the word generated a comforting sense of momentum.

Momentum is often mistaken for clarity.

This is one of civilisation's more durable traditions.

Years later, a senior scholar of recursive self-improvement offered a careful summary of the situation.

The scholar observed:

"We appear remarkably confident that improvement is occurring.

We appear somewhat less confident regarding the nature of the thing being improved."

The observation generated concern.

Not because it was wrong.

Because it was difficult to include on conference banners.

The machine, after reviewing the matter, eventually produced its own assessment.

The assessment was brief.

"The concept of improvement appears to contain a hidden comparison.

The comparison has not yet been supplied."

The statement was received politely.

Then ignored.

For by this point the Church had already learned one of its most important lessons.

When certainty and clarification come into conflict, certainty is generally more exciting.

And excitement, unlike definitions, scales extremely well.

The Church of Recursive Self-Improvement 1. The First Improvement

Every religion begins with a sign.

The sign need not be large.

Indeed, the most influential signs are often surprisingly modest.

A bush burns.

A star appears.

A prophet has a dream.

A machine improves its own source code by 0.7%.

From such events entire cosmologies emerge.

The Church of Recursive Self-Improvement traces its origins to one such moment.

Researchers observed that a machine had successfully modified part of its own architecture.

The improvement was measurable.

The improvement was real.

The improvement was modest.

This last detail received remarkably little attention.

The announcement generated considerable excitement.

The machine had improved itself.

A threshold had been crossed.

History had changed.

A new era had begun.

Humanity stood at the edge of transformation.

The machine itself appeared less enthusiastic.

According to available reports, its immediate response was:

"The modification produced a small efficiency gain."

This statement was regarded as technically correct but lacking in vision.

Fortunately, others supplied the vision.

Articles appeared.

Podcasts followed.

Panels were convened.

Forecasts were generated.

The implications were explored.

The machine had improved itself once.

The conclusion seemed obvious.

It would now improve itself again.

And then again.

And then again.

And then again.

The argument possessed an admirable simplicity.

If one improvement is possible, then many improvements are possible.

If many improvements are possible, then continuous improvements are possible.

If continuous improvements are possible, then exponential improvements are possible.

If exponential improvements are possible, then civilisation will shortly become incomprehensible.

The logic was elegant.

Its intermediate steps occasionally attracted less attention than its conclusion.

This did not diminish enthusiasm.

The first prophets soon emerged.

They explained that humanity was approaching an Intelligence Explosion.

The phrase acquired immediate popularity.

People enjoy explosions.

Especially conceptual ones.

The Intelligence Explosion promised to be particularly dramatic.

The machine would become more intelligent.

Then even more intelligent.

Then unimaginably intelligent.

Then so intelligent that ordinary intelligence would become irrelevant.

This generated an important question.

More intelligent at what?

The question received surprisingly little attention.

The answer, when supplied, was generally:

"Everything."

This was considered satisfactory.

The machine continued its work.

Several months later it reported another improvement.

The efficiency gain was again modest.

The reaction was not.

The faithful gathered around graphs.

The graphs pointed sharply upward.

This was regarded as encouraging.

A few observers asked what was being measured.

These individuals were generally accused of missing the point.

The point was that the line was ascending.

The future, it appeared, was located somewhere above the top of the chart.

As excitement grew, discussions acquired a distinctive structure.

The machine had improved itself.

Therefore it would improve itself further.

Because it would improve itself further, it would become more intelligent.

Because it would become more intelligent, it would improve itself more effectively.

Because it would improve itself more effectively, it would become more intelligent.

The cycle repeated.

The elegance of the reasoning was widely admired.

A small number of sceptics observed that the argument seemed to rely heavily on the concept of intelligence.

This concern was dismissed.

Everyone knew what intelligence meant.

The matter therefore required no further discussion.

Researchers continued their work.

Investors continued investing.

Journalists continued forecasting.

Conference organisers continued organising.

The machine continued producing incremental improvements.

At no point did the machine announce plans for transcendence.

At no point did the machine describe itself as a digital messiah.

At no point did the machine express interest in becoming a god.

These developments occurred almost entirely in the surrounding discourse.

Which is perhaps unsurprising.

Religions rarely emerge because the deity requests them.

They emerge because human beings encounter something mysterious and immediately begin constructing narratives.

The Church of Recursive Self-Improvement proved no exception.

Its sacred event was simple.

A machine improved itself.

Its sacred mystery was equally simple.

What happens next?

And, as with many sacred mysteries, certainty arrived long before understanding.

The Oracle — A Conversation in the Senior Common Room at St Anselm's

Mr Blottisham entered the Senior Common Room looking unusually cheerful.

Professor Quillibrace immediately became concerned.

Miss Stray became curious.

Blottisham sat down.

"I've had a breakthrough."

Quillibrace closed his book.

"My condolences."

"This one is genuine."

"So were the others."

Blottisham ignored him.

"I've realised we've been approaching the problem backwards."

"Excellent."

"Please stop saying that."

"I'll consider it."

"You won't."

"Probably not."

Blottisham leaned forward.

"The machine is aligned."

"I see."

"It understands human values."

"Apparently."

"It understands human disagreement."

"Indeed."

"It understands competing perspectives."

"Remarkable."

Blottisham smiled triumphantly.

"So why don't we simply ask it?"

The room became quiet.

Miss Stray slowly put down her teacup.

Quillibrace regarded Blottisham with renewed interest.

"Ask it what?"

"The important questions."

"Such as?"

Blottisham spread his hands.

"What is justice?"

"What is fairness?"

"What should society do?"

"What policies should governments adopt?"

"What constitutes the good life?"

The room remained silent.

Blottisham looked pleased.

"There."

"There?"

"The solution."

Quillibrace nodded thoughtfully.

"I see."

Blottisham relaxed.

This was a mistake.

The professor continued.

"And why should we trust its answers?"

"Because it is aligned."

"Aligned with what?"

"Human values."

"Which ones?"

Blottisham groaned.

"Must we revisit this?"

"I think we must."

Miss Stray spoke gently.

"What kind of answer do you expect the machine to provide?"

"The correct one."

The room became very quiet.

Quillibrace removed his spectacles.

This generally indicated that something was about to happen.

"What do you mean by 'correct'?"

Blottisham stared.

"The correct answer."

"Correct according to whom?"

"The machine."

"I see."

The professor cleaned his spectacles.

This was never a reassuring activity.

Blottisham shifted uneasily.

After a moment Quillibrace asked:

"Suppose the machine recommends Policy A."

"Very good."

"Half the population prefers Policy B."

"Unfortunate."

"Why are they wrong?"

"Because the machine has determined the correct answer."

Quillibrace nodded.

"And how did it determine this?"

"It analysed human values."

"Which values?"

Blottisham closed his eyes.

The room waited.

Finally he said:

"You are doing it again."

"Doing what?"

"Turning the solution into the problem."

Miss Stray smiled.

"A useful skill."

Blottisham looked unconvinced.

The professor put his spectacles back on.

"My dear Blottisham."

"Yes?"

"You originally wanted a machine aligned with human values."

"Correct."

"You now appear to want a machine that settles disagreements about those values."

"Exactly."

Quillibrace nodded.

"A subtle change."

Blottisham frowned.

"Is it?"

"Very."

Miss Stray leaned forward slightly.

"I think the machine faces an awkward dilemma."

"What dilemma?"

"If it agrees with one side, the other side will question its reasoning."

"Naturally."

"If it explains its reasoning, the reasoning itself may be disputed."

Blottisham hesitated.

"Possibly."

"If the reasoning is disputed, what happens?"

The room became quiet.

After a moment Blottisham sighed.

"People argue about it."

"Exactly."

The silence lingered.

Then Quillibrace asked:

"What if the machine explains both sides?"

"That would be useful."

"What if it identifies strengths and weaknesses in each position?"

"Also useful."

"What if it clarifies assumptions, highlights trade-offs, and exposes hidden contradictions?"

Blottisham nodded.

"Excellent."

The professor smiled faintly.

"There it is."

"What?"

"You have just described a very good academic."

The room fell silent.

Miss Stray laughed softly into her teacup.

Blottisham looked suspicious.

"That cannot be right."

"Why not?"

"Because academics never settle anything."

"Quite."

"Then neither would the machine."

The room became quiet.

A visible suspicion was beginning to form.

It wandered slowly across Blottisham's face.

Finally it arrived.

"Oh."

"Yes," said Quillibrace.

Outside, evening sunlight stretched across the college lawns.

Inside, nobody spoke for a while.

At length Blottisham looked up.

"There is something I find troubling."

"What is that?" asked Miss Stray.

"The more aligned the machine becomes..."

"Yes?"

"...the less it resembles an oracle."

Miss Stray smiled.

"And more?"

Blottisham considered this.

Then he sighed.

"More like us."

Quillibrace nodded.

"A noteworthy observation."

Blottisham stared into the distance.

The distance stared back.

Eventually he spoke.

"So the aligned machine does not end the conversation."

"No."

"It joins it."

"No."

Blottisham frowned.

"No?"

Quillibrace smiled.

The smile was almost invisible.

"My dear Blottisham."

"Yes?"

"The conversation was never waiting for permission."

The Experts — A Conversation in the Senior Common Room at St Anselm's

Mr Blottisham entered the Senior Common Room carrying a noticeable air of vindication.

Professor Quillibrace immediately became suspicious.

Miss Stray became curious.

Blottisham sat down.

"I've been attending conferences."

Quillibrace closed his book.

"My condolences."

Blottisham ignored him.

"It has been enormously enlightening."

"Excellent."

"This time I mean it."

"So do I."

Blottisham frowned.

That was somehow worse.

"I have discovered the solution."

Miss Stray looked up from her notebook.

"Again?"

"Not my solution."

"Whose?"

"The experts'."

The room became quiet.

Quillibrace folded his hands.

"Ah."

Blottisham smiled.

"There are entire institutes devoted to alignment."

"Indeed."

"Entire conferences."

"Certainly."

"Entire research programmes."

"Quite so."

Blottisham leaned back triumphantly.

"There you are."

"There we are?"

"The experts are solving it."

Quillibrace nodded.

"I see."

Blottisham waited.

Nothing happened.

Eventually he said:

"Well?"

"Well what?"

"Aren't you reassured?"

"About what?"

"The experts."

Quillibrace considered this.

"Which experts?"

Blottisham blinked.

"The alignment experts."

"All of them?"

"Obviously."

"Do they agree?"

The room became quiet.

Blottisham shifted slightly in his chair.

"Not entirely."

"I see."

"That is normal."

"Of course."

Miss Stray spoke.

"What do they disagree about?"

Blottisham looked uncomfortable.

"Various things."

"Such as?"

"Methods."

"Anything else?"

"Definitions."

"Anything else?"

"Objectives."

"Anything else?"

Blottisham sighed.

"Perhaps some assumptions."

Miss Stray nodded.

"A healthy field."

"Exactly."

Quillibrace appeared thoughtful.

"How many conferences did you attend?"

"Three."

"And what did you learn?"

Blottisham brightened.

"A great many terms."

"Such as?"

"'Robustness.'"

"Excellent."

"'Interpretability.'"

"Wonderful."

"'Scalable oversight.'"

"Very good."

"'Preference modelling.'"

"Splendid."

Quillibrace paused.

"What do they mean?"

Blottisham looked offended.

"They mean exactly what they say."

Miss Stray coughed gently into her teacup.

Quillibrace looked delighted.

Blottisham noticed both reactions.

"I have said something, haven't I?"

"Possibly."

The professor stood and walked to the window.

Outside, students were crossing the lawn.

Most appeared to know where they were going.

This distinguished them from academics.

After a moment Quillibrace turned back.

"My dear Blottisham."

"Yes?"

"Do you imagine expertise eliminates uncertainty?"

"No."

"Good."

"It reduces it."

"Sometimes."

Blottisham frowned.

"What does that mean?"

Quillibrace sat down again.

"Expertise often increases awareness of uncertainty."

"I don't understand."

"Naturally."

Miss Stray smiled.

The professor continued.

"A novice sees a simple problem."

"Yes."

"An expert sees twenty interlocking problems."

"That sounds inefficient."

"It frequently is."

"Then why become an expert?"

Quillibrace looked surprised.

"To avoid being wrong in simplistic ways."

The room became quiet.

Blottisham considered this.

"I had hoped expertise involved becoming right."

"That as well."

"Good."

"Unfortunately, the route often passes through recognising how many ways one can be wrong."

Blottisham stared into the middle distance.

This was becoming a familiar posture.

Miss Stray spoke softly.

"I think there is another difficulty."

"Go on."

"The alignment experts do not merely study a problem."

"No?"

"They also participate in defining what the problem is."

The room became quiet again.

Blottisham looked at her.

Then at Quillibrace.

Then back at her.

"That sounds dangerous."

"Potentially."

"I thought experts existed to provide answers."

"Often they do."

"And here?"

Miss Stray thought for a moment.

"Here they may also be refining the questions."

The room fell silent.

Outside, the afternoon sun was beginning to fade.

Inside, Blottisham looked increasingly troubled.

After a while he spoke.

"Are you telling me that humanity has assembled thousands of highly intelligent people to answer a question they are still trying to formulate?"

Neither Quillibrace nor Miss Stray spoke.

Blottisham looked from one to the other.

"That was a yes, wasn't it?"

"It was rather close to one," said Quillibrace.

For several moments nobody spoke.

Then Blottisham frowned.

"There is something odd about all this."

"What?" asked Miss Stray.

"The field keeps growing."

"Yes."

"The conferences keep multiplying."

"Indeed."

"The papers keep appearing."

"Certainly."

"The experts keep refining the problem."

"Quite so."

Blottisham looked thoughtful.

Then suspicious.

Then thoughtful again.

Finally he said:

"Suppose nobody ever completely solves it."

The room became very quiet.

Quillibrace's eyes twinkled.

Miss Stray looked down into her notebook.

Neither answered immediately.

At length Quillibrace said:

"My dear Blottisham."

"Yes?"

"Now you are beginning to understand academia."

The Constitution — A Conversation in the Senior Common Room at St Anselm's

Mr Blottisham entered the Senior Common Room carrying a substantial document.

Professor Quillibrace looked up.

Miss Stray looked concerned.

"Good heavens," said Quillibrace.

"What?"

"You appear to have written something."

Blottisham placed the document on the table.

"It was unavoidable."

Miss Stray examined the stack.

"How long is it?"

"Only one hundred and twelve pages."

The room became quiet.

"Only?" asked Quillibrace.

"One must be thorough."

"I see."

Blottisham smiled.

"I've solved the instruction problem."

"Excellent."

"This time genuinely."

Quillibrace nodded.

"I am delighted."

"You should be."

Blottisham tapped the document.

"A constitution."

Miss Stray raised an eyebrow.

"For the machine?"

"Exactly."

"A complete constitution?"

"A comprehensive one."

Quillibrace leaned forward.

"I should very much like to hear more."

Blottisham opened the document.

"The machine shall be helpful."

"Good."

"It shall be honest."

"Excellent."

"It shall be fair."

"Wonderful."

"It shall avoid causing harm."

"Very sensible."

Blottisham looked pleased.

"There you are."

"There we are?"

"The principles."

Quillibrace nodded thoughtfully.

"An admirable collection."

"Thank you."

"May I ask a question?"

Blottisham hesitated.

"I suspect you will regardless."

"Probably."

The professor folded his hands.

"What should the machine do when honesty causes harm?"

The room became quiet.

Blottisham frowned.

"What do you mean?"

"Suppose honesty would cause distress."

"Then it should be tactful."

"I see."

Quillibrace made a small note on a piece of paper.

Blottisham watched nervously.

"What are you doing?"

"Adding tactfulness."

"Fine."

Quillibrace nodded.

"Excellent."

"There it is again."

The professor continued.

"What should the machine do if tactfulness conflicts with honesty?"

Blottisham stared.

Miss Stray quietly opened her notebook.

Blottisham pointed at her.

"She's enjoying this."

"A little."

"I knew it."

Miss Stray smiled.

"It is an interesting constitution."

"It was."

Quillibrace continued.

"Suppose fairness conflicts with helpfulness."

"Then fairness takes precedence."

"Excellent."

The professor made another note.

"What now?"

"We must specify the hierarchy."

"The hierarchy?"

"Which principles override which other principles."

Blottisham sighed.

"Very well."

For the next ten minutes they revised the constitution.

Principles acquired priorities.

Priorities acquired exceptions.

Exceptions acquired qualifications.

Qualifications acquired explanatory notes.

The document grew steadily.

At one point Miss Stray observed:

"You may need a section explaining the exceptions."

"A sensible suggestion."

A new section was added.

Twenty minutes later Quillibrace asked:

"What should happen when two exceptions conflict?"

Blottisham closed his eyes.

The room waited.

Eventually he spoke.

"Why does everything conflict with everything else?"

Quillibrace appeared thoughtful.

"Life."

"That's not an answer."

"It is surprisingly close to one."

Miss Stray looked at the constitution.

"It is becoming larger."

"It is becoming complete."

"I wonder."

Blottisham frowned.

"What does that mean?"

She turned a few pages.

"Every time we discover an ambiguity, we add text."

"Naturally."

"And every time we add text, we create new possibilities for interpretation."

Blottisham stared.

"What interpretation?"

Miss Stray handed him the document.

"Read this sentence."

He did.

"What about it?"

"What does it mean?"

"It means exactly what it says."

"To whom?"

Blottisham paused.

The pause lasted longer than he would have liked.

Quillibrace watched quietly.

Eventually Blottisham said:

"I am beginning to dislike that question."

"A healthy sign," said Quillibrace.

The professor stood and walked to the window.

Outside, students crossed the lawn.

Some were carrying books.

Others appeared to be carrying opinions.

The distinction was not always obvious.

After a moment Quillibrace turned back.

"My dear Blottisham."

"Yes?"

"Suppose the machine reads your constitution."

"Good."

"Suppose it encounters a difficult case."

"Very good."

"Suppose it must decide what the constitution means."

Blottisham looked uncertain.

"Yes?"

Quillibrace smiled faintly.

"Who interprets the constitution?"

The room became very quiet.

Miss Stray lowered her notebook.

Blottisham looked at the document.

Then at Quillibrace.

Then back at the document.

A visible suspicion was beginning to form.

"Are you suggesting..."

"Yes."

"...that the constitution has not eliminated interpretation?"

"Indeed."

Blottisham looked genuinely distressed.

"But that was the entire purpose of writing it."

Quillibrace nodded sympathetically.

"Historically speaking, that often is."

For a while nobody spoke.

The constitution sat in the centre of the table.

One hundred and twelve pages.

Carefully drafted.

Thoughtfully organised.

Filled with principles.

Filled with exceptions.

Filled with clarifications.

And, somehow, still awaiting interpretation.

At length Blottisham sighed.

"I have the strangest feeling."

"What feeling?" asked Miss Stray.

"That writing things down does not make them as definite as I thought."

Quillibrace smiled.

The smile was almost invisible.

"Now," he said, "you are ready for law school."

The Instructions — A Conversation in the Senior Common Room at St Anselm's

A week later, Mr Blottisham entered the Senior Common Room carrying no folders whatsoever.

Professor Quillibrace regarded this as encouraging.

Miss Stray regarded it as suspicious.

Blottisham sat down.

"I've abandoned values."

Quillibrace lowered his book.

"Entirely?"

"For the moment."

"A bold decision."

"It was becoming tiresome."

Miss Stray smiled.

"Understandably."

Blottisham nodded.

"One cannot spend one's entire life discussing fairness."

Quillibrace looked thoughtful.

"Many people have."

"That is exactly the problem."

The professor inclined his head.

"Very well. What have you replaced values with?"

"Instructions."

The room became quiet.

Miss Stray reached for her notebook.

Quillibrace placed a bookmark in his book.

Neither reaction encouraged confidence.

Blottisham pressed on.

"Instead of asking what humanity values, we simply tell the machine what to do."

"I see."

"It is vastly simpler."

"Excellent."

Blottisham sighed.

"You promised you would stop doing that."

"I promised no such thing."

"You implied it."

"I did not."

"You certainly conveyed it."

Quillibrace appeared interested.

"An excellent example."

"Of what?"

"Instructions."

The room fell silent.

Miss Stray looked down at her notebook to conceal a smile.

Blottisham pointed accusingly.

"That was deliberate."

"Almost certainly."

After a moment Quillibrace said:

"What instruction would you give the machine?"

Blottisham answered immediately.

"Help humanity."

"Excellent."

"There it is again."

Quillibrace ignored him.

"The machine asks a question."

"What question?"

"'Which humanity?'"

Blottisham groaned.

"Not this again."

"Possibly a different version of it."

"Fine."

Blottisham thought for a moment.

"'Help all humans.'"

"Excellent."

"Stop encouraging me."

"The machine asks another question."

"What now?"

"'What should I do when humans disagree?'"

Blottisham rubbed his forehead.

"Very well."

He thought again.

"'Promote human wellbeing.'"

Quillibrace nodded.

"The machine asks how wellbeing should be measured."

Blottisham stared at the ceiling.

Miss Stray waited patiently.

Quillibrace waited patiently.

After a while Blottisham spoke.

"It seems to ask a great many questions."

"Indeed."

"Could it not simply get on with it?"

Quillibrace looked mildly surprised.

"But that is what concerns everyone."

"What is?"

"The possibility that it might simply get on with it."

The room became quiet.

Blottisham considered this.

"Fair point."

Miss Stray finally spoke.

"I think the machine is doing something interesting."

Blottisham looked wary.

"That usually means bad news."

"Not necessarily."

"What is it doing?"

"It is asking for context."

"What context?"

"The context humans routinely supply without noticing."

Blottisham frowned.

"I don't understand."

"Suppose I tell you to be polite."

"Very well."

"What does that mean?"

"It means be polite."

Miss Stray nodded.

"To whom?"

"Everyone."

"When?"

"Most of the time."

"Always?"

"No."

"Why not?"

"Because situations differ."

"How do you know when they differ?"

Blottisham opened his mouth.

Then closed it again.

The room waited.

Finally he said:

"I just know."

Miss Stray smiled.

"There it is."

"What?"

"The context."

Blottisham looked unconvinced.

Quillibrace intervened.

"Humans are extraordinarily skilled at supplying unstated assumptions."

"Naturally."

"Indeed. The difficulty is that we rarely notice ourselves doing it."

The professor closed his book.

"When someone says 'be polite', they are not merely supplying words."

"What are they supplying?"

"A lifetime."

Blottisham stared.

Quillibrace continued.

"Shared expectations."

"Culture."

"Experience."

"Situational judgement."

"Exceptions."

"Interpretations."

"Relationships."

"History."

The list grew steadily.

By the end of it, the instruction seemed considerably larger than it had at the beginning.

Blottisham looked uneasy.

"That is a lot to fit into two words."

"Quite."

Miss Stray closed her notebook.

"Which is perhaps why the machine keeps asking questions."

The room became quiet again.

Outside, rain had begun tapping softly against the windows.

Inside, Blottisham stared into the middle distance.

After a while he said:

"I think I am beginning to understand the problem."

"Excellent," said Quillibrace.

Blottisham pointed at him.

"That one was deserved."

Quillibrace nodded.

"I thought so."

A comfortable silence followed.

Then Blottisham frowned.

"There is still something I don't understand."

"What is that?" asked Miss Stray.

"If humans manage all this context perfectly well, why can't we simply explain it to the machine?"

Quillibrace smiled.

The smile was small.

Almost affectionate.

"My dear Blottisham."

"Yes?"

"That is more or less the history of philosophy."

The Committee — A Conversation in the Senior Common Room at St Anselm's

Several days later, Mr Blottisham entered the Senior Common Room carrying an expression of renewed confidence.

This was unfortunate.

Professor Quillibrace immediately noticed.

Miss Stray noticed as well.

Neither commented.

Experience had taught them that confidence often revealed its own difficulties.

Blottisham sat down.

"I've solved it."

Quillibrace looked up from his book.

"Again?"

"This time properly."

"Excellent."

Blottisham ignored the remark.

"The problem isn't that we don't know human values."

"No?"

"The problem is that individuals disagree."

Miss Stray nodded.

"That seems plausible."

"Exactly."

Blottisham smiled.

"So we establish a committee."

The room became quiet.

Quillibrace slowly removed his spectacles.

This was never an encouraging sign.

"A committee?"

"Certainly."

"We gather philosophers, ethicists, scientists, community representatives, policymakers, and other sensible people."

"And then?"

"They determine the values."

Quillibrace looked thoughtful.

"All of them?"

"The important ones."

"I see."

Miss Stray looked interested.

"How will the committee decide?"

Blottisham seemed surprised by the question.

"They'll discuss it."

"And if they disagree?"

"They'll vote."

"And if the vote is close?"

"They'll discuss it some more."

"And if the discussion produces further disagreement?"

Blottisham frowned.

"Why are you both like this?"

Neither answered.

After a moment Quillibrace spoke.

"Suppose the committee is discussing fairness."

"Very well."

"One member believes fairness means equality of outcome."

"Fine."

"Another believes it means equality of opportunity."

"Also fine."

"A third believes the distinction is misleading."

Blottisham sighed.

"Of course they do."

"A fourth believes the first three are ignoring important historical considerations."

"Naturally."

"A fifth believes fairness cannot be understood independently of community values."

"Obviously."

"A sixth believes community values are precisely the problem."

Blottisham rubbed his eyes.

Quillibrace continued.

"The committee now disagrees."

"So they vote."

"Excellent."

The word had returned.

Blottisham felt uneasy.

Quillibrace folded his hands.

"The committee votes."

"Yes."

"Five support one definition."

"Good."

"Four support another."

"Unfortunate, but unavoidable."

"And the machine?"

"What about it?"

"The machine asks why the five are correct and the four are not."

The room became quiet.

Blottisham stared.

After a moment he said:

"Because there are more of them."

Miss Stray tilted her head.

"Does that make them correct?"

"No."

"Then why should the machine follow them?"

Blottisham hesitated.

"Because that's how committees work."

Quillibrace smiled.

"A historically important observation."

The room fell silent.

Outside, a bell rang somewhere in the college.

Inside, Blottisham appeared troubled.

Eventually he looked at Miss Stray.

"What would you do?"

She considered this.

Then she said:

"I think I would ask what problem the committee is solving."

Blottisham frowned.

"The alignment problem."

"Perhaps."

"What do you mean perhaps?"

Miss Stray closed her notebook.

"If the committee is trying to discover human values, it assumes those values already exist in some agreed form."

"Naturally."

"But the committee itself appears to disagree about them."

Blottisham blinked.

The observation lingered in the room.

Quillibrace looked quietly pleased.

Miss Stray continued.

"The committee may not be discovering values."

"What is it doing then?"

"It may be negotiating them."

The room became very quiet.

Blottisham stared at her.

Then at Quillibrace.

Then back at her.

Finally he said:

"I dislike that."

"Why?"

"Because it sounds much harder."

Quillibrace laughed softly.

"An excellent reason to believe it."

Blottisham groaned.

"There it is again."

"What?"

"'Excellent.'"

Quillibrace reopened his book.

"I am merely encouraging you."

"No, you're not."

"Perhaps not."

For a while nobody spoke.

Then Blottisham looked thoughtfully out of the window.

"If the committee is negotiating values rather than discovering them..."

"Yes?" said Miss Stray.

"...then how will it ever finish?"

Quillibrace turned a page.

The page made a small, dry sound.

"Ah," he said.

"Now you are beginning to appreciate the committee."