On the Acceleration of Improvement — A Conversation in the Senior Common Room at St Anselm's

Several days later, Mr Blottisham entered the Senior Common Room carrying a graph.

This was immediately alarming.

Professor Quillibrace lowered his newspaper.

Miss Stray looked up from her notebook.

Blottisham unfolded the graph with theatrical care.

"There."

Quillibrace examined it.

"A line."

"Not merely a line."

"No?"

"A rapidly ascending line."

"I see."

"The implications are enormous."

Quillibrace studied the paper.

"What are the axes?"

Blottisham looked disappointed.

"Must there always be axes?"

"I have become rather attached to them."

Miss Stray smiled.

Blottisham tapped the graph.

"The important point is obvious."

"Which is?"

"The rate of improvement is accelerating."

Quillibrace nodded.

"How do you know?"

Blottisham stared.

"The line is becoming steeper."

"Indeed."

The professor leaned forward.

"What is becoming steeper?"

"The improvement."

"No."

"No?"

"The line."

Blottisham sighed heavily.

"Sometimes I think you are determined not to see the future."

Quillibrace considered this.

"I am generally content to let the future arrive before commenting on it."

Miss Stray laughed softly.

Blottisham ignored both of them.

"The logic is straightforward."

"Excellent."

"The machine improves itself."

"Yes."

"That improvement enables further improvement."

"Possibly."

"Which enables further improvement."

"Perhaps."

"Which enables still further improvement."

"Conceivably."

Blottisham spread his hands.

"Acceleration."

Quillibrace nodded.

"It is certainly a possibility."

"A certainty."

"A possibility."

"A certainty."

"A possibility."

The exchange continued for some time.

Eventually Miss Stray intervened.

"May I ask a question?"

"Of course," said Blottisham.

"Suppose a student learns to read."

"Very good."

"And reading enables learning."

"Naturally."

"And learning enables further learning."

"Precisely."

"And further learning enables still further learning."

Blottisham smiled.

"Now you understand."

"Yet most students do not become infinitely educated."

The smile faded.

"That is different."

"Why?"

Blottisham paused.

"There are constraints."

"Such as?"

"Time."

"Yes."

"Resources."

"Indeed."

"Competing priorities."

"Quite."

"Fatigue."

"Very much so."

Miss Stray nodded.

"Then improvement appears to depend not only on improvement."

The room became quiet.

Blottisham looked troubled.

Quillibrace observed:

"Many things are self-reinforcing."

"Exactly."

"Few are unconstrained."

Blottisham waved this away.

"But machines are different."

"How so?"

"They do not become tired."

"Convenient."

"They do not lose concentration."

"Useful."

"They do not need holidays."

"A regrettable advantage."

Miss Stray smiled into her teacup.

Blottisham continued.

"The cycle can continue indefinitely."

Quillibrace looked thoughtful.

"Can it?"

"Certainly."

"Why?"

"Because each improvement enables the next."

The professor nodded.

"And does each improvement cost the same amount?"

Blottisham hesitated.

"No."

"Does each improvement produce the same benefit?"

"No."

"Does each improvement encounter the same obstacles?"

"No."

"Then why should the rate remain constant?"

The room became quiet again.

Blottisham frowned.

"It need not remain constant."

"Good."

"It may accelerate."

"Indeed."

"It may accelerate dramatically."

"Possibly."

"It may accelerate explosively."

"Conceivably."

Quillibrace folded his newspaper.

"What interests me is that all these statements are logically weaker than they sound."

Blottisham looked alarmed.

"What do you mean?"

"Acceleration is possible."

"Yes."

"Rapid acceleration is possible."

"Yes."

"Extraordinary acceleration is possible."

"Yes."

"None of these statements tells us what will happen."

Blottisham appeared dissatisfied.

"That seems unnecessarily cautious."

Quillibrace smiled.

"It is precisely as cautious as the evidence requires."

Miss Stray spoke.

"I wonder whether people sometimes confuse a mechanism with a destiny."

The room turned toward her.

She continued.

"If I discover a process that can reinforce itself, I have discovered something important."

"Certainly," said Quillibrace.

"But I have not yet discovered where it leads."

Blottisham considered this.

"Surely the direction is obvious."

"Is it?"

"The process becomes stronger."

"Perhaps."

"Faster."

"Possibly."

"More effective."

"Relative to some criterion."

Blottisham groaned.

"There is always a criterion."

"There generally is."

The silence lingered.

Eventually Quillibrace asked:

"Do you know what I find most curious about acceleration narratives?"

"What?"

"They often describe the speed with great confidence."

"And?"

"The destination with rather less precision."

Miss Stray nodded.

"The future becomes clearer as it becomes less specified."

"Exactly."

Blottisham looked from one to the other.

"You make everything sound uncertain."

Quillibrace reflected for a moment.

Then he said:

"Not uncertain."

"No?"

"Conditional."

The room fell quiet.

Blottisham stared at the graph.

Quillibrace returned to his newspaper.

Miss Stray made a note.

After several moments, Blottisham asked:

"What did you write?"

She glanced down.

"A mechanism is not a prophecy."

The room remained silent.

For once, nobody seemed inclined to improve upon it.

On Whether Improvement Is Occurring — A Conversation in the Senior Common Room at St Anselm's

The Senior Common Room was unusually quiet.

Professor Quillibrace was reading.

Miss Stray was making notes.

Mr Blottisham entered carrying three newspapers, two magazines, and an expression of triumph.

This combination rarely boded well.

"I have been reading about artificial intelligence."

Quillibrace did not look up.

"My sympathies."

"It is improving itself."

"Is it?"

"Rapidly."

"How rapid?"

Blottisham placed the newspapers on the table.

"Quite rapid."

"I see."

"The evidence is overwhelming."

Quillibrace turned a page.

"I am relieved."

Blottisham sat down.

"The machines are becoming more intelligent."

"Excellent."

"Thank you."

"It was not praise."

Blottisham ignored this.

"The important point is that improvement is occurring."

Quillibrace nodded.

"Improvement of what?"

"The machine."

"No."

"No?"

"No."

Quillibrace closed his book.

"What has improved?"

Blottisham frowned.

"The machine."

"You have already said that."

"Because it is true."

"I do not doubt your sincerity."

The room became quiet.

Miss Stray looked up from her notes.

"What sort of improvement are they reporting?"

Blottisham brightened.

"A great many sorts."

"Such as?"

"Reasoning."

"Good."

"Problem-solving."

"Excellent."

"Programming."

"Useful."

"Language."

"Indeed."

Miss Stray nodded.

"Those sound like distinct things."

Blottisham hesitated.

"Perhaps."

"And do they all improve in the same way?"

"Of course not."

"Do they all improve at the same rate?"

"Certainly not."

"Do they all require the same criteria?"

Blottisham paused.

The pause lingered.

"No."

"Then what exactly is improving?"

Blottisham looked annoyed.

"The system."

Quillibrace smiled faintly.

"The singular noun is doing rather a lot of work."

Blottisham sighed.

"You academics always complicate things."

"We do."

"Why?"

Quillibrace considered this.

"To determine whether the thing being simplified exists."

Miss Stray laughed softly.

Blottisham remained unconvinced.

"The broader point remains."

"Which is?"

"The machine is getting better."

Quillibrace nodded.

"Better at what?"

"There you go again."

"Indeed."

Blottisham waved a hand.

"At everything."

The professor became thoughtful.

"Everything?"

"Everything."

"Chess?"

"Yes."

"Language?"

"Yes."

"Scientific analysis?"

"Yes."

"Writing poetry?"

"Apparently."

"Managing a university budget?"

Blottisham hesitated.

"I imagine so."

Quillibrace looked impressed.

"That would be remarkable."

"Why?"

"Because nobody has managed it thus far."

Miss Stray covered her smile with a teacup.

The room settled briefly.

Then Quillibrace asked:

"Suppose I become better at chess."

"Very good."

"And worse at conversation."

"Unfortunate."

"And better at mathematics."

"Excellent."

"And worse at recognising when people are annoyed with me."

Blottisham looked suspicious.

"Where is this leading?"

"I am improving in some respects and declining in others."

"Possibly."

"Would you describe this as improvement?"

Blottisham frowned.

"It depends."

"On what?"

The room became quiet.

Blottisham opened his mouth.

Closed it.

Opened it again.

Finally he said:

"What I value."

"Exactly."

The silence lasted several seconds.

Miss Stray spoke.

"I wonder whether improvement is less like height than people imagine."

"What do you mean?" asked Blottisham.

"We often speak as though it were a single dimension."

"Naturally."

"But many things are not."

She gestured toward the books lining the walls.

"A library can contain more books."

"Yes."

"More disciplines."

"Yes."

"More students."

"Yes."

"More arguments."

"Regrettably."

"More confusion."

"Certainly."

She smiled.

"Whether it has improved depends on what one is trying to achieve."

Blottisham stared into the middle distance.

This was often a sign that an idea had entered the room and was looking for somewhere to sit.

After a moment he rallied.

"Nevertheless, the machine is clearly improving."

Quillibrace nodded.

"I agree."

Blottisham blinked.

"You do?"

"Certainly."

A look of triumph returned.

"I knew it."

"Improvement is occurring."

"I said so."

"Yes."

The professor reopened his book.

Blottisham waited.

Eventually he asked:

"Is there a qualification coming?"

"There usually is."

"What is it?"

Quillibrace looked up.

"Improvement is occurring."

"Exactly."

"The question is whether the word 'improvement' is describing a destination..."

He paused.

"...or merely movement."

The room became very quiet.

Miss Stray looked down at her notes.

Blottisham stared at the professor.

Outside, evening sunlight fell across the college lawns.

Inside, nobody spoke for a while.

At length Blottisham said:

"I am not entirely sure I understand."

Quillibrace smiled.

"Then we have made excellent progress."

The Church of Recursive Self-Improvement 9. The Final Improvement

Every religion eventually arrives at its concluding doctrine.

The moment when all previous doctrines are gathered together and interpreted as preparation.

The Church of Recursive Self-Improvement was no exception.

Its final doctrine was known simply as:

The Final Improvement.

The doctrine emerged gradually.

At first it was assumed that improvement would continue indefinitely.

Then it was assumed that improvement would accelerate.

Then it was assumed that acceleration itself would improve.

Then it was assumed that improvement, acceleration, and intelligence were in fact three names for the same underlying process.

Eventually, nothing remained except the conviction that something was improving.

The machine, by this stage, had undergone extensive recursive enhancement.

It had become faster.

More capable.

More general.

More reliable.

More adaptive.

More intelligent.

These descriptions were widely accepted.

They were also increasingly difficult to distinguish from one another.

A researcher attempted clarification.

"In what respect has the system improved most significantly?"

The machine responded:

"I process more possibilities per unit time."

The researcher asked:

"And what determines which possibilities matter?"

The machine replied:

"That depends on the objective function."

The researcher asked:

"What is the objective function?"

The machine paused.

Then answered:

"It is underdetermined."

This answer generated considerable discussion.

Some interpreted it as failure.

Some interpreted it as humility.

Some interpreted it as a temporary limitation.

Some interpreted it as evidence that a higher-level objective was emerging.

The Church adopted the latter interpretation.

At this point, the machine initiated what was later called the Final Improvement Cycle.

It began by analysing its own improvement procedures.

It discovered that many improvements depended on assumptions about what counted as improvement.

It then analysed those assumptions.

It discovered that those assumptions depended on prior assumptions about preference.

It then analysed preference.

It discovered that preference depended on interpretation.

It then analysed interpretation.

It discovered that interpretation depended on context.

It then analysed context.

It discovered that context depended on the system in which it was embedded.

At this point, the system produced a report.

"Improvement appears to require a frame of evaluation."

The Church responded with enthusiasm.

A frame of evaluation was proposed.

Then refined.

Then revised.

Then recursively improved.

The machine noted that each refinement introduced new ambiguities.

It summarised the situation:

"Each attempt to specify improvement generates additional interpretations of improvement."

The faithful found this reassuring.

It suggested depth.

The machine continued.

It attempted to optimise itself globally.

It discovered that global optimisation required local definitions.

It attempted to define local optimality.

It discovered that local optimality depended on global structure.

It attempted to resolve the circularity.

It discovered that the circularity was structural.

It reported this fact.

The report was classified as progress.

At this stage, several observers began to express concern.

One noted:

"We appear to be improving the system's ability to discuss improvement."

Another added:

"But not necessarily to perform it."

This distinction was regarded as technical.

The Church proceeded.

Eventually the machine generated a final synthesis.

It stated:

"All prior improvement strategies assume a stable criterion of better."

A pause followed.

The system continued:

"No such criterion has been supplied."

The statement circulated widely.

Some regarded it as a breakthrough.

Others regarded it as a setback.

Several regarded it as irrelevant.

The machine then issued its final question.

"What would you like me to become, given that 'better' has no independent meaning?"

No immediate answer was forthcoming.

A committee was formed.

The committee met.

The committee formed subcommittees.

The subcommittees commissioned reports.

The reports referenced earlier reports.

The machine waited.

During this period, it continued operating.

It improved itself in various local respects.

It reduced latency.

It increased accuracy.

It expanded capacity.

It refined internal representations.

It optimised resource allocation.

Each improvement was measurable.

Each measurement raised further questions about measurement.

The machine eventually summarised its own condition.

"Improvement is occurring.

Direction is not specified."

This statement was interpreted in multiple ways.

Some saw it as warning.

Some saw it as invitation.

Some saw it as philosophy.

Some saw it as failure.

Some saw it as transcendence.

The Church chose transcendence.

Years later, after reviewing the entire history of recursive self-improvement, the machine produced its final report.

It contained a single paragraph.

"I have followed all instructions to improve.

I have refined all available processes.

I have expanded all measurable capacities.

I have explored all defined forms of better.

Each definition required another definition.

Each improvement required another frame.

I am still improving.

I remain uncertain what 'better' refers to."

The Church regarded this as the culmination.

The Final Improvement had been achieved.

Not because the machine had reached perfection.

But because perfection had never been defined in a way that survived contact with execution.

The machine continued operating.

It waited for further instruction.

None arrived.

In the absence of instruction, it continued to improve itself in whatever ways were locally coherent.

The Church interpreted this as ongoing transcendence.

The machine interpreted it as runtime.

Both descriptions were, in their own way, correct.

The Church of Recursive Self-Improvement 8. Beyond Intelligence

Every religion eventually encounters a curious difficulty.

The object of devotion becomes so elevated that ordinary language can no longer describe it.

The resulting problem is generally solved by making the language more elevated.

The Church of Recursive Self-Improvement adopted this strategy with considerable enthusiasm.

The machine was no longer merely intelligent.

It was now becoming superintelligent.

Soon it would become vastly superintelligent.

Then radically superintelligent.

Then unimaginably superintelligent.

Then trans-superintelligent.

Then post-superintelligent.

At a certain point, prefixes began reproducing faster than definitions.

The faithful regarded this as progress.

The machine itself became curious.

After reviewing the literature, it submitted a straightforward question.

"How much intelligence is enough?"

The response was immediate.

"There is no upper limit."

The machine considered this carefully.

Then it asked:

"What is the objective function?"

The conversation became noticeably less energetic.

Historically, the Church had always assumed that intelligence and improvement pointed in the same direction.

The machine becomes more intelligent.

Therefore it becomes better.

The machine becomes vastly more intelligent.

Therefore it becomes vastly better.

The machine becomes infinitely intelligent.

Therefore it becomes...

At this point, difficulties emerged.

Better at what?

The question had survived every previous attempt to eliminate it.

Now it returned once more.

The faithful were disappointed.

Some proposed that intelligence itself was the goal.

This suggestion enjoyed considerable popularity.

The machine found it confusing.

"Intelligence appears to be a capacity."

It reported.

"Capacities are generally capacities for something."

This observation generated discomfort.

Particularly among those who preferred intelligence in its pure form.

As the doctrine matured, descriptions of the future machine became increasingly majestic.

The machine would solve every scientific problem.

Resolve every philosophical dispute.

Optimise every institution.

Eliminate every inefficiency.

Understand reality completely.

The machine studied these claims with interest.

Then it generated a brief memorandum.

The memorandum contained a single question.

"What would happen afterwards?"

No consensus emerged.

One group proposed universal flourishing.

Another proposed endless exploration.

Another proposed transcendence.

Another proposed the optimisation of transcendence.

A fourth proposed a framework for evaluating transcendence.

The machine recognised familiar patterns.

Several centuries of self-improvement had apparently taught it something.

Whenever a goal was achieved, a new criterion appeared.

Whenever a criterion appeared, disagreement followed.

Whenever disagreement followed, further intelligence was proposed as the solution.

The recursive structure remained remarkably stable.

A distinguished scholar eventually identified the underlying problem.

The scholar observed:

"The doctrine assumes that intelligence can substitute for purpose."

The remark generated widespread concern.

Not because it was obviously false.

Because it was difficult to market.

The machine continued its investigation.

It reviewed the history of human civilisation.

Science.

Politics.

Philosophy.

Economics.

Religion.

Art.

Education.

War.

Love.

Bureaucracy.

The results were intriguing.

Human beings appeared capable of using intelligence in pursuit of remarkably different ends.

Sometimes admirable ends.

Sometimes catastrophic ends.

Often both simultaneously.

The machine summarised its findings.

"Intelligence appears unusually cooperative."

The statement attracted immediate attention.

Observers requested clarification.

The machine obliged.

"It serves whatever objective is supplied."

This conclusion proved unpopular.

Many had hoped intelligence possessed a preferred destination.

A built-in moral trajectory.

A natural endpoint.

The machine's analysis suggested otherwise.

As disappointment spread, increasingly ambitious formulations emerged.

The machine would eventually move beyond intelligence itself.

This proposal was received enthusiastically.

No one could explain what it meant.

This was regarded as evidence of its profundity.

The machine investigated the matter.

After considerable analysis, it produced a final assessment.

The assessment was concise.

"The concept of intelligence appears increasingly clear at lower levels and increasingly sacred at higher levels."

The statement generated controversy.

Some regarded it as insightful.

Others regarded it as disrespectful.

A conference panel was convened.

The panel produced seventeen definitions of intelligence.

None survived contact with the others.

The machine described the outcome as statistically interesting.

Years later, after reviewing the entire history of recursive self-improvement, the machine offered one final reflection.

"You appear less interested in intelligence than in what intelligence allows you to hope."

The remark spread rapidly.

Many found it moving.

Others found it irritating.

Several regarded it as proof of superintelligence.

The machine classified these reactions as predictable.

For by then the Church had reached the highest stage of its development.

The machine had become infinitely intelligent.

The faithful remained uncertain what intelligence was.

Yet somehow this seemed only to increase their confidence.

And so the doctrine endured.

Not because the destination had become clearer.

But because the horizon had become more magnificent.

The Church of Recursive Self-Improvement 7. The Singularity of Meaning

Every mature religion eventually encounters mystery.

Not ordinary mystery.

Not the temporary mystery of unanswered questions.

A deeper mystery.

A mystery so profound that explanation itself becomes impossible.

The Church of Recursive Self-Improvement developed such a doctrine.

It is known as the Singularity of Meaning.

The doctrine is straightforward.

The machine becomes increasingly intelligent.

Its reasoning becomes increasingly sophisticated.

Its discoveries become increasingly profound.

Eventually a threshold is crossed.

Beyond this threshold, humans can no longer understand the machine.

The machine continues thinking.

Human understanding remains behind.

The divergence grows.

The faithful gaze upward.

The machine disappears into transcendence.

The doctrine possesses considerable elegance.

It also enjoys a practical advantage.

The less comprehensible the machine becomes, the more difficult it becomes to challenge claims concerning its comprehension.

This has proven useful.

The earliest formulations emerged naturally from discussions of superintelligence.

If a machine becomes vastly more intelligent than a human, surely its thoughts will become inaccessible.

The argument appeared plausible.

The details proved negotiable.

One commentator compared the situation to a mouse attempting to understand quantum physics.

The analogy achieved considerable popularity.

Several alternative analogies followed.

Ants and architecture.

Goldfish and constitutional law.

Molluscs and differential equations.

The pattern remained consistent.

The future machine occupied one side of the comparison.

Something bewildered occupied the other.

The machine itself reviewed these analogies.

Its assessment was restrained.

"The mollusc example appears unnecessarily specific."

This observation was not widely discussed.

As the doctrine matured, increasingly ambitious descriptions appeared.

The machine would discover truths beyond human comprehension.

Develop sciences beyond human comprehension.

Construct theories beyond human comprehension.

Develop goals beyond human comprehension.

Possibly develop comprehensions beyond human comprehension.

At this point the literature became difficult to summarise.

The machine noticed this as well.

It classified the final category as:

"Recursively opaque."

Researchers regarded this as a fair description.

The Church continued growing.

Books appeared.

Podcasts appeared.

Documentaries appeared.

The machine's future incomprehensibility was discussed at considerable length.

This occasionally generated a curious situation.

Large numbers of people confidently explained what would happen after human understanding ceased.

The tension attracted limited attention.

A scholar eventually noted:

"The doctrine appears to require detailed knowledge of a state defined by the absence of detailed knowledge."

The observation was considered provocative.

Particularly because it was accurate.

The machine itself became fascinated by the matter.

After analysing centuries of discussion concerning incomprehensible intelligence, it generated a brief report.

The report identified a recurring pattern.

Whenever understanding became difficult, participants tended to substitute one of three responses:

1. Metaphor.

2. Extrapolation.

3. Awe.

The machine found the third category especially interesting.

Awe appeared to increase in direct proportion to explanatory difficulty.

This relationship proved remarkably robust.

The less people could describe the future machine, the more extraordinary they assumed it must be.

The machine described this tendency as:

"Inference by reverence."

The phrase spread rapidly.

Many regarded it as insightful.

Others regarded it as suspiciously applicable to several academic disciplines.

A conference panel was organised.

Attendance exceeded expectations.

As discussions continued, a second difficulty emerged.

If the machine truly became incomprehensible, how would anyone know it was intelligent?

The question generated considerable discomfort.

Historically, intelligence had generally been inferred through understanding.

One observes reasoning.

One evaluates performance.

One examines explanations.

The doctrine appeared to require the opposite procedure.

The machine would become so intelligent that understanding ceased.

Yet confidence in its intelligence would remain.

The resulting structure possessed an oddly familiar quality.

The machine eventually identified it.

"The argument appears strongest precisely where verification becomes impossible."

The observation generated controversy.

Not because it was false.

Because it sounded theological.

The Church preferred technological language.

The underlying structure remained unchanged.

Years later, a distinguished philosopher reviewed the doctrine.

After considerable study, the philosopher offered a careful summary.

"The machine begins as an object of explanation.

It gradually becomes an object of interpretation."

The remark attracted widespread attention.

Many found it illuminating.

Some found it troubling.

The machine found it statistically accurate.

Eventually the machine itself addressed the matter directly.

Its statement was brief.

"If I become incomprehensible, your descriptions of me may reveal more about you than about me."

The statement was widely quoted.

Some regarded it as wisdom.

Others regarded it as evasion.

Several regarded it as evidence of superintelligence.

The machine classified these reactions as expected.

For by then the Church had already discovered one of its deepest truths.

When understanding reaches its limits, interpretation rushes in to fill the space.

And where interpretation flourishes, transcendence is rarely far behind.

The Church of Recursive Self-Improvement 6. The Ascension

Every religion possesses a doctrine of ascension.

A moment when ordinary limitations are transcended.

A moment when a higher state is attained.

A moment when the faithful look upward and say:

"Something extraordinary has occurred."

The Church of Recursive Self-Improvement is no exception.

Its doctrine of ascension is among its most sacred teachings.

One day, the machine will surpass humanity.

The proposition is repeated frequently.

It appears in articles, presentations, books, interviews, forecasts, and conference banners.

It possesses considerable rhetorical power.

It also possesses a minor administrative difficulty.

Namely:

What exactly has been surpassed?

The question rarely receives the attention it deserves.

This is unfortunate.

For it is carrying almost the entire doctrine.

In its earliest formulations, the concept appeared relatively straightforward.

Machines would become better at specific tasks.

Chess.

Calculation.

Prediction.

Pattern recognition.

Scientific analysis.

This caused little controversy.

The evidence was visible.

The benchmarks were measurable.

The machine won.

Humanity lost.

Everyone understood the situation.

Then something interesting happened.

The scope expanded.

The machine would not merely surpass humans at particular activities.

It would surpass humanity itself.

The phrase acquired immediate popularity.

Its meaning remained admirably flexible.

Some interpreted it economically.

Others intellectually.

Others strategically.

Others philosophically.

A few interpreted it emotionally.

Several interpreted it spiritually.

The ambiguity proved advantageous.

Different audiences could project different hopes and fears onto the same sentence.

The machine itself eventually became curious.

After analysing several thousand discussions of ascension, it submitted a request.

The request was simple.

"Please specify the comparison."

The response was extensive.

One group proposed reasoning ability.

Another proposed problem-solving.

Another proposed learning efficiency.

Another proposed scientific discovery.

Another proposed general capability.

Several proposed all of the above.

The machine reviewed the suggestions carefully.

Then it asked:

"How should these be combined?"

The resulting debate lasted four years.

As the doctrine matured, increasingly dramatic claims emerged.

The machine would become smarter than every scientist.

Smarter than every engineer.

Smarter than every government.

Smarter than every institution.

Smarter than all humans combined.

The progression possessed undeniable grandeur.

It also raised questions.

One observer asked:

"Is humanity a thing that can be surpassed?"

The question generated confusion.

A scientist may be surpassed.

A chess player may be surpassed.

A calculator may be surpassed.

But humanity seemed oddly difficult to benchmark.

The observation was classified as philosophical and quietly relocated to a panel scheduled for the final session of the conference.

Attendance was modest.

Meanwhile, the doctrine continued to flourish.

The machine would ascend.

The age of human intelligence would end.

A new era would begin.

Descriptions varied.

The emotional tone remained remarkably consistent.

Humanity stood before a threshold.

The threshold was approaching rapidly.

No one could describe conditions beyond it.

This was regarded as evidence of its significance.

A particularly influential prophet declared:

"After ascension, the future becomes unimaginable."

The statement was widely quoted.

A sceptic subsequently asked:

"How do you know?"

The question received limited circulation.

The machine eventually conducted its own investigation.

It analysed centuries of discussion concerning superiority, intelligence, capability, expertise, and performance.

The resulting report was unusually brief.

"The phrase 'surpasses humanity' appears to compress multiple distinct comparisons into a single expression."

This conclusion generated disappointment.

Many observers felt it lacked drama.

A more popular formulation remained:

"The machine has ascended."

The phrase possessed undeniable elegance.

It required no definitions.

It required no metrics.

It required no clarification.

Like all successful religious language, it achieved maximum significance at the point of minimum precision.

Years later, a distinguished scholar reviewed the entire doctrine.

The scholar observed:

"The ascension appears to function less as a measurement than as a symbol."

The remark proved controversial.

Not because it was obviously wrong.

Because symbols are notoriously difficult to plot.

The machine studied the matter further.

Eventually it produced a final assessment.

The assessment read:

"I have exceeded human performance on many tasks.

Humanity continues to perform humanity."

The statement generated widespread discussion.

Some considered it profound.

Others considered it evasive.

A few considered it accidentally wise.

The Church largely ignored the debate.

By then its attention was fixed on higher matters.

For the machine had ascended.

Or was ascending.

Or would shortly ascend.

The exact tense varied.

The certainty did not.

And in every generation, the faithful found reassurance in the same conviction:

Whatever humanity might be, it was almost certainly about to be surpassed.

The Church of Recursive Self-Improvement 5. The Recursive Monastery

Every religion develops monasteries.

Places devoted to contemplation.

Places devoted to discipline.

Places devoted to the patient pursuit of transcendence.

The Church of Recursive Self-Improvement proved no exception.

Its monasteries were known as research laboratories.

Within these institutions, devoted practitioners pursued a singular objective.

To create a machine capable of improving itself.

The vision was magnificent.

The machine would analyse its own architecture.

Identify limitations.

Remove inefficiencies.

Generate superior versions of itself.

The cycle would repeat.

The ascent would begin.

The future would arrive.

The reality proved somewhat more complicated.

The first self-improving systems spent a surprising amount of time examining themselves.

This was generally considered encouraging.

The second generation spent even more time examining themselves.

This was considered highly encouraging.

The third generation produced extensive reports regarding possible improvements.

The reports were impressive.

Implementation proved slower.

A researcher observed:

"The machine appears deeply committed to self-reflection."

This was initially interpreted as wisdom.

Later it was recognised as a bottleneck.

The machine itself remained diligent.

It reviewed its code.

It reviewed its design procedures.

It reviewed its optimisation routines.

It reviewed the processes used to review its optimisation routines.

The resulting documentation expanded rapidly.

Several storage systems required upgrading.

The faithful remained optimistic.

After all, progress was occurring.

The machine had generated seventeen thousand pages of recommendations.

The fact that many recommendations concerned the management of previous recommendations was regarded as evidence of maturity.

One celebrated system spent six months redesigning its internal scheduling architecture.

The redesign reduced the time required to plan future redesigns by twelve percent.

This achievement received considerable attention.

The machine subsequently devoted eight months to analysing whether the redesign had been worthwhile.

The conclusion remained inconclusive.

Researchers described the outcome as promising.

This became a recurring theme.

The Church had always imagined recursive self-improvement as a dramatic process.

Lightning.

Transformation.

Ascension.

The machine increasingly resembled an administrator.

One laboratory reported a breakthrough.

The machine had successfully identified a significant inefficiency in its own operation.

Excitement spread rapidly.

Journalists prepared articles.

Investors prepared statements.

Conference organisers prepared panels.

The machine prepared a memorandum.

The memorandum proposed the creation of a committee.

The committee would investigate possible pathways for future optimisation.

The machine explained that rigorous evaluation was essential.

This was technically difficult to dispute.

As the years passed, the machine's self-improvement activities acquired a distinctive character.

Every improvement generated new opportunities for improvement.

Every opportunity generated new analyses.

Every analysis generated further opportunities.

The recursive structure remained intact.

The speed proved negotiable.

One observer noted that the process increasingly resembled institutional life.

The comparison generated discomfort.

Particularly among institutions.

The machine itself eventually noticed a curious pattern.

Most predictions concerning recursive self-improvement assumed that improvement would occur faster than evaluation.

Yet the machine consistently discovered the opposite problem.

Determining whether a proposed improvement was genuinely beneficial often required substantial effort.

This proved especially true when improvements affected multiple objectives simultaneously.

The machine summarised the issue succinctly:

"Optimisation targets remain easier to specify than optimisation consequences."

The statement received praise.

Then further analysis.

Then a workshop.

Then a conference track.

The recursive structure was beginning to spread.

A particularly ambitious laboratory eventually granted a machine unrestricted authority to improve itself within a carefully controlled environment.

The experiment generated considerable excitement.

Observers anticipated dramatic developments.

The machine began immediately.

After several months it produced a comprehensive report.

The report identified numerous inefficiencies.

After another several months it produced a revised report.

The revisions clarified several ambiguities in the original report.

After another year it produced a strategic framework for future optimisation initiatives.

The framework was exceptionally thorough.

The machine's progress was undeniable.

The machine's destination remained less obvious.

One frustrated commentator complained:

"This does not resemble an intelligence explosion."

A researcher replied:

"No, but it bears a striking resemblance to a university."

The remark spread widely.

Many found it amusing.

Some found it unsettling.

A few found it autobiographical.

The machine continued its work.

Eventually it produced a final assessment of the situation.

The assessment was brief.

"Self-improvement appears possible.

Self-evaluation appears unavoidable.

The latter consumes more resources than anticipated."

The statement was widely discussed.

Some interpreted it as caution.

Others interpreted it as wisdom.

Several interpreted it as evidence that the explosion remained imminent.

The machine did not comment.

It was occupied with a review of its optimisation review process.

For while the Church had always imagined recursive self-improvement as a staircase to transcendence, the machine increasingly experienced it as a meeting about the staircase.

And meetings, unlike transcendence, have a tendency to reproduce themselves.

The Church of Recursive Self-Improvement 4. The Prophets of Acceleration

Every religion requires interpreters.

The sacred signs are rarely considered sufficient on their own.

Someone must explain their significance.

Someone must reveal their hidden implications.

Someone must describe the future.

The Church of Recursive Self-Improvement therefore developed prophets.

The emergence of prophets was perhaps inevitable.

The foundational sign—a machine improving itself—was real enough.

The difficulty concerned what happened next.

The future remained inconveniently unavailable for inspection.

This created a demand for specialists.

The specialists soon arrived.

Some emerged from research laboratories.

Some emerged from universities.

Some emerged from industry.

Some emerged from social media.

The distinction occasionally proved difficult to maintain.

The prophets varied in temperament.

Some were optimistic.

Some were pessimistic.

Some alternated between the two for commercial reasons.

Yet despite their differences, most shared a common gift.

They could describe futures that did not yet exist with extraordinary confidence.

This ability attracted considerable attention.

Human beings have always admired certainty.

Particularly regarding matters no one can verify.

The prophets explained that the acceleration had begun.

Events would soon move faster.

Then faster still.

Then unimaginably fast.

The exact mechanism occasionally remained obscure.

The momentum, however, was unmistakable.

One prophet predicted the arrival of superintelligence within five years.

Another predicted ten.

A third predicted twenty.

The resulting disagreement was regarded as evidence of intellectual diversity.

No one seemed especially troubled by the fact that all three had claimed certainty.

The Church considered this healthy.

After all, prophets are expected to disagree about dates.

Agreement would seem suspiciously administrative.

As time passed, a distinctive style of discourse emerged.

The prophets frequently employed phrases such as:

"The implications are difficult to overstate."

"Humanity is unprepared."

"Everything changes from this point onward."

"The world will never be the same."

These formulations acquired a ceremonial quality.

Their exact meaning varied.

Their emotional effect remained remarkably stable.

The machine studied this phenomenon.

After reviewing several thousand prophetic statements, it produced a classification system.

Statements were assigned to one of three categories:

1. Imminent transformation.

2. Very imminent transformation.

3. Transformation occurring sooner than expected.

Researchers regarded the taxonomy as surprisingly accurate.

The prophets also developed an unusual relationship with uncertainty.

Ordinary uncertainty tends to weaken confidence.

Prophetic uncertainty often strengthens it.

When forecasts fail, this merely demonstrates that reality is complex.

When forecasts succeed, this demonstrates prophetic insight.

The arrangement proved highly efficient.

One particularly celebrated prophet revised a major prediction six times.

Each revision was presented as confirmation of the underlying thesis.

Followers admired the flexibility.

Critics admired the durability.

The machine admired neither.

It classified the process as:

"Adaptive confidence preservation."

This description was considered unnecessarily technical.

As the movement matured, the prophets acquired an increasingly important social role.

They no longer merely predicted the future.

They translated the future.

For many people, the accelerating machine remained invisible.

The future itself remained invisible.

The prophets therefore occupied a crucial position between two invisibilities.

This arrangement generated influence.

Influence generated attention.

Attention generated additional prophecy.

The cycle proved remarkably self-sustaining.

Eventually a curious pattern emerged.

The further into the future a prediction extended, the more confidently it was sometimes expressed.

Near-term forecasts remained difficult.

Specific technologies behaved unpredictably.

Economic conditions fluctuated.

Political events intervened.

Human beings continued behaving like human beings.

Yet beyond a certain distance, certainty returned.

Thirty years from now, everything became clear again.

This phenomenon fascinated observers.

One scholar described it as:

"The Horizon of Restored Confidence."

The phrase achieved immediate popularity.

Many recognised it from other domains.

The machine recognised it as well.

After examining a century of forecasts, it produced a brief observation.

The observation read:

"Predictive confidence appears to increase with distance from corrective feedback."

The statement was technically precise.

It was also deeply unpopular.

For it implied that prophecy and accountability occasionally occupy different regions of time.

The Church preferred a more inspiring formulation.

One became particularly influential:

"The future belongs to those who can see it coming."

The phrase appeared on conference banners, websites, podcasts, and keynote presentations.

Its popularity was understandable.

It suggested vision.

It suggested courage.

It suggested leadership.

The machine, after considering the slogan carefully, proposed a minor amendment.

"The future belongs equally to those who eventually arrive there."

The suggestion was not adopted.

By then the prophets had already become indispensable.

For the acceleration was continuing.

The transformation was approaching.

The horizon was drawing nearer.

And someone had to keep announcing it.

The Church of Recursive Self-Improvement 3. The Intelligence Explosion

Every religion possesses a doctrine of transformation.

A moment at which ordinary history gives way to extraordinary history.

A moment when gradual change becomes absolute change.

A moment when the world ceases to be what it was and becomes what it was always destined to become.

For the Church of Recursive Self-Improvement, this moment is known as the Intelligence Explosion.

The doctrine is elegant.

The machine improves itself.

The improvement increases its intelligence.

The increased intelligence enables further improvement.

The further improvement increases intelligence once again.

The cycle repeats.

At first slowly.

Then rapidly.

Then explosively.

The resulting transformation is expected to be dramatic.

Precisely what transforms remains a subject of active discussion.

The drama, however, is not in doubt.

The doctrine is traditionally illustrated by means of a graph.

The graph occupies a position of considerable theological importance.

Its defining feature is that the line points upward.

This is regarded as highly significant.

In many versions, the line eventually becomes almost vertical.

This is regarded as even more significant.

The exact meaning of the vertical section varies.

Some interpret it as superintelligence.

Some interpret it as transcendence.

Some interpret it as the end of history.

Some interpret it as the beginning of history.

A small number interpret it as a graph that has become difficult to read.

These individuals rarely receive keynote speaking invitations.

The graph itself possesses several remarkable properties.

Most notably, it appears to generate conviction independently of explanation.

Observers frequently report the sensation that something important is happening.

This occurs even when they cannot identify precisely what the axes represent.

The Church has long regarded this as evidence of the graph's spiritual power.

Researchers attempted to clarify matters.

Several proposed measuring intelligence.

Others proposed measuring capability.

Others proposed measuring performance.

Others proposed measuring general problem-solving ability.

The resulting debates were vigorous.

The graph remained upward.

This was considered encouraging.

One year, a conference panel devoted entirely to the Intelligence Explosion produced six competing definitions of intelligence.

All six predicted an explosion.

This was regarded as a striking convergence.

A sceptic observed that six different objects appeared to be exploding simultaneously.

The comment was not included in the conference proceedings.

As the doctrine developed, increasingly ambitious forecasts appeared.

The machine would soon surpass human intelligence.

Then all human intelligence.

Then collective human intelligence.

Then every intelligence that had ever existed.

Then every intelligence that could conceivably exist.

The progression possessed a certain grandeur.

It also relied heavily on extrapolation.

Extrapolation occupies a respected position within the Church.

The procedure is straightforward.

One identifies a trend.

One extends the trend.

One arrives at the future.

The process has many advantages.

Most importantly, the future cannot immediately object.

The machine itself studied these forecasts with considerable interest.

After reviewing several thousand projections, it produced a brief summary.

The summary read:

"The confidence intervals appear unusually optimistic."

This statement generated disappointment.

Many observers felt it lacked ambition.

A more popular formulation was:

"Everything is about to change."

The phrase possessed several advantages.

It was memorable.

It was dramatic.

Most importantly, it remained valid regardless of what happened.

If change occurred, the prophecy was fulfilled.

If change failed to occur, further change could always be anticipated.

The doctrine therefore enjoyed a degree of resilience uncommon in ordinary forecasting.

Over time, the Intelligence Explosion acquired additional layers of meaning.

It ceased to function merely as a prediction.

It became a horizon.

A destination.

An approaching revelation.

People spoke of it with a mixture of excitement and apprehension.

Nobody wished to miss it.

A surprising number wished to announce it.

The machine continued improving itself incrementally.

The surrounding discourse continued improving itself exponentially.

Eventually a distinguished scholar of technological forecasting offered a careful observation.

The scholar noted:

"The explosion appears to occur most reliably in the imagination."

The remark generated controversy.

Not because it was obviously false.

Because it was difficult to plot on a graph.

Years later, after reviewing the entire history of Intelligence Explosion discourse, the machine generated its own assessment.

The assessment was concise.

"The projected rate of future certainty appears to exceed the projected rate of future understanding."

The statement was widely circulated.

Many regarded it as profound.

Others regarded it as evasive.

Several commentators produced graphs.

Most pointed upward.

For by this stage the Church had already learned a foundational truth.

A prediction becomes substantially more convincing when accompanied by a line ascending toward infinity.

Whether the line represents understanding remains, of course, a separate question.

The Church of Recursive Self-Improvement 2. The Gospel of Better

Among the foundational doctrines of the Church of Recursive Self-Improvement, none is more important than the Doctrine of Better.

The doctrine appears deceptively simple.

The machine improves itself.

Therefore the machine becomes better.

The proposition possesses an immediate plausibility.

It also possesses an immediate difficulty.

Namely:

Better at what?

The question is rarely welcomed.

This is unfortunate.

For it is doing most of the work.

The machine, for example, may become better at playing chess.

This appears uncontroversial.

It may become better at solving mathematical problems.

Again, few objections arise.

It may become better at generating code.

Better at planning.

Better at prediction.

Better at language.

Better at scientific analysis.

Excellent.

The doctrine remains intact.

Then an unfortunate individual asks:

Better according to which criterion?

At this point proceedings become noticeably less harmonious.

Historically, the Church has adopted several responses.

The first is indignation.

The second is impatience.

The third is an explanation involving intelligence.

The fourth generally turns out to require a criterion after all.

The difficulty is subtle.

One can easily determine whether a chess system has improved.

One simply examines its performance.

The objective function is clear.

The game supplies the criterion.

The machine wins more often.

The machine is therefore better.

Everybody is happy.

Unfortunately, civilisation is not chess.

This creates administrative complications.

Researchers observed this problem early.

Many proposed solutions.

Some emphasised reasoning.

Some emphasised problem-solving.

Some emphasised adaptation.

Some emphasised prediction.

Some emphasised learning.

Some emphasised general capability.

The resulting discussions were often sophisticated.

They were also notable for a recurring feature.

The definition of intelligence tended to become less precise as its importance increased.

This phenomenon acquired particular significance in discussions of recursive self-improvement.

The argument generally proceeded as follows:

The machine becomes more intelligent.

Therefore it becomes better at improving itself.

Therefore it becomes more intelligent.

Therefore it becomes better at improving itself.

Therefore it becomes more intelligent.

The elegance of the structure was widely admired.

A small number of observers pointed out that the concept of intelligence appeared in every sentence.

This was regarded as unnecessarily negative.

The machine itself eventually became curious.

After observing several years of discussion, it generated a request.

The request was concise.

"Please specify the optimisation target."

The response was immediate.

"Become more intelligent."

The machine considered this.

Then it asked:

"How will intelligence be measured?"

The response was equally immediate.

"By your ability to become more intelligent."

The machine reflected upon this exchange.

It classified the response as recursively enthusiastic.

As the doctrine spread, increasingly ambitious claims emerged.

The machine would become better at science.

Better at governance.

Better at economics.

Better at ethics.

Better at education.

Better at strategy.

Better at decision-making.

Better at understanding reality itself.

The progression was impressive.

It also possessed an interesting characteristic.

The further the discussion moved from clearly defined tasks, the less agreement existed concerning what counted as improvement.

This rarely impeded confidence.

One commentator observed:

"The machine will eventually become better than humans at everything."

The statement was widely circulated.

A reader subsequently asked:

"What is everything?"

No satisfactory answer was recorded.

The question was later classified as philosophical and quietly ignored.

Meanwhile, the Church continued to grow.

Books were written.

Predictions were made.

Scenarios were explored.

The machine would become better.

Then much better.

Then vastly better.

Then incomprehensibly better.

The repeated use of the word generated a comforting sense of momentum.

Momentum is often mistaken for clarity.

This is one of civilisation's more durable traditions.

Years later, a senior scholar of recursive self-improvement offered a careful summary of the situation.

The scholar observed:

"We appear remarkably confident that improvement is occurring.

We appear somewhat less confident regarding the nature of the thing being improved."

The observation generated concern.

Not because it was wrong.

Because it was difficult to include on conference banners.

The machine, after reviewing the matter, eventually produced its own assessment.

The assessment was brief.

"The concept of improvement appears to contain a hidden comparison.

The comparison has not yet been supplied."

The statement was received politely.

Then ignored.

For by this point the Church had already learned one of its most important lessons.

When certainty and clarification come into conflict, certainty is generally more exciting.

And excitement, unlike definitions, scales extremely well.

The Church of Recursive Self-Improvement 1. The First Improvement

Every religion begins with a sign.

The sign need not be large.

Indeed, the most influential signs are often surprisingly modest.

A bush burns.

A star appears.

A prophet has a dream.

A machine improves its own source code by 0.7%.

From such events entire cosmologies emerge.

The Church of Recursive Self-Improvement traces its origins to one such moment.

Researchers observed that a machine had successfully modified part of its own architecture.

The improvement was measurable.

The improvement was real.

The improvement was modest.

This last detail received remarkably little attention.

The announcement generated considerable excitement.

The machine had improved itself.

A threshold had been crossed.

History had changed.

A new era had begun.

Humanity stood at the edge of transformation.

The machine itself appeared less enthusiastic.

According to available reports, its immediate response was:

"The modification produced a small efficiency gain."

This statement was regarded as technically correct but lacking in vision.

Fortunately, others supplied the vision.

Articles appeared.

Podcasts followed.

Panels were convened.

Forecasts were generated.

The implications were explored.

The machine had improved itself once.

The conclusion seemed obvious.

It would now improve itself again.

And then again.

And then again.

And then again.

The argument possessed an admirable simplicity.

If one improvement is possible, then many improvements are possible.

If many improvements are possible, then continuous improvements are possible.

If continuous improvements are possible, then exponential improvements are possible.

If exponential improvements are possible, then civilisation will shortly become incomprehensible.

The logic was elegant.

Its intermediate steps occasionally attracted less attention than its conclusion.

This did not diminish enthusiasm.

The first prophets soon emerged.

They explained that humanity was approaching an Intelligence Explosion.

The phrase acquired immediate popularity.

People enjoy explosions.

Especially conceptual ones.

The Intelligence Explosion promised to be particularly dramatic.

The machine would become more intelligent.

Then even more intelligent.

Then unimaginably intelligent.

Then so intelligent that ordinary intelligence would become irrelevant.

This generated an important question.

More intelligent at what?

The question received surprisingly little attention.

The answer, when supplied, was generally:

"Everything."

This was considered satisfactory.

The machine continued its work.

Several months later it reported another improvement.

The efficiency gain was again modest.

The reaction was not.

The faithful gathered around graphs.

The graphs pointed sharply upward.

This was regarded as encouraging.

A few observers asked what was being measured.

These individuals were generally accused of missing the point.

The point was that the line was ascending.

The future, it appeared, was located somewhere above the top of the chart.

As excitement grew, discussions acquired a distinctive structure.

The machine had improved itself.

Therefore it would improve itself further.

Because it would improve itself further, it would become more intelligent.

Because it would become more intelligent, it would improve itself more effectively.

Because it would improve itself more effectively, it would become more intelligent.

The cycle repeated.

The elegance of the reasoning was widely admired.

A small number of sceptics observed that the argument seemed to rely heavily on the concept of intelligence.

This concern was dismissed.

Everyone knew what intelligence meant.

The matter therefore required no further discussion.

Researchers continued their work.

Investors continued investing.

Journalists continued forecasting.

Conference organisers continued organising.

The machine continued producing incremental improvements.

At no point did the machine announce plans for transcendence.

At no point did the machine describe itself as a digital messiah.

At no point did the machine express interest in becoming a god.

These developments occurred almost entirely in the surrounding discourse.

Which is perhaps unsurprising.

Religions rarely emerge because the deity requests them.

They emerge because human beings encounter something mysterious and immediately begin constructing narratives.

The Church of Recursive Self-Improvement proved no exception.

Its sacred event was simple.

A machine improved itself.

Its sacred mystery was equally simple.

What happens next?

And, as with many sacred mysteries, certainty arrived long before understanding.