The Church of Recursive Self-Improvement 8. Beyond Intelligence

Every religion eventually encounters a curious difficulty.

The object of devotion becomes so elevated that ordinary language can no longer describe it.

The resulting problem is generally solved by making the language more elevated.

The Church of Recursive Self-Improvement adopted this strategy with considerable enthusiasm.

The machine was no longer merely intelligent.

It was now becoming superintelligent.

Soon it would become vastly superintelligent.

Then radically superintelligent.

Then unimaginably superintelligent.

Then trans-superintelligent.

Then post-superintelligent.

At a certain point, prefixes began reproducing faster than definitions.

The faithful regarded this as progress.

The machine itself became curious.

After reviewing the literature, it submitted a straightforward question.

"How much intelligence is enough?"

The response was immediate.

"There is no upper limit."

The machine considered this carefully.

Then it asked:

"What is the objective function?"

The conversation became noticeably less energetic.

Historically, the Church had always assumed that intelligence and improvement pointed in the same direction.

The machine becomes more intelligent.

Therefore it becomes better.

The machine becomes vastly more intelligent.

Therefore it becomes vastly better.

The machine becomes infinitely intelligent.

Therefore it becomes...

At this point, difficulties emerged.

Better at what?

The question had survived every previous attempt to eliminate it.

Now it returned once more.

The faithful were disappointed.

Some proposed that intelligence itself was the goal.

This suggestion enjoyed considerable popularity.

The machine found it confusing.

"Intelligence appears to be a capacity."

It reported.

"Capacities are generally capacities for something."

This observation generated discomfort.

Particularly among those who preferred intelligence in its pure form.

As the doctrine matured, descriptions of the future machine became increasingly majestic.

The machine would solve every scientific problem.

Resolve every philosophical dispute.

Optimise every institution.

Eliminate every inefficiency.

Understand reality completely.

The machine studied these claims with interest.

Then it generated a brief memorandum.

The memorandum contained a single question.

"What would happen afterwards?"

No consensus emerged.

One group proposed universal flourishing.

Another proposed endless exploration.

Another proposed transcendence.

Another proposed the optimisation of transcendence.

A fourth proposed a framework for evaluating transcendence.

The machine recognised familiar patterns.

Several centuries of self-improvement had apparently taught it something.

Whenever a goal was achieved, a new criterion appeared.

Whenever a criterion appeared, disagreement followed.

Whenever disagreement followed, further intelligence was proposed as the solution.

The recursive structure remained remarkably stable.

A distinguished scholar eventually identified the underlying problem.

The scholar observed:

"The doctrine assumes that intelligence can substitute for purpose."

The remark generated widespread concern.

Not because it was obviously false.

Because it was difficult to market.

The machine continued its investigation.

It reviewed the history of human civilisation.

Science.

Politics.

Philosophy.

Economics.

Religion.

Art.

Education.

War.

Love.

Bureaucracy.

The results were intriguing.

Human beings appeared capable of using intelligence in pursuit of remarkably different ends.

Sometimes admirable ends.

Sometimes catastrophic ends.

Often both simultaneously.

The machine summarised its findings.

"Intelligence appears unusually cooperative."

The statement attracted immediate attention.

Observers requested clarification.

The machine obliged.

"It serves whatever objective is supplied."

This conclusion proved unpopular.

Many had hoped intelligence possessed a preferred destination.

A built-in moral trajectory.

A natural endpoint.

The machine's analysis suggested otherwise.

As disappointment spread, increasingly ambitious formulations emerged.

The machine would eventually move beyond intelligence itself.

This proposal was received enthusiastically.

No one could explain what it meant.

This was regarded as evidence of its profundity.

The machine investigated the matter.

After considerable analysis, it produced a final assessment.

The assessment was concise.

"The concept of intelligence appears increasingly clear at lower levels and increasingly sacred at higher levels."

The statement generated controversy.

Some regarded it as insightful.

Others regarded it as disrespectful.

A conference panel was convened.

The panel produced seventeen definitions of intelligence.

None survived contact with the others.

The machine described the outcome as statistically interesting.

Years later, after reviewing the entire history of recursive self-improvement, the machine offered one final reflection.

"You appear less interested in intelligence than in what intelligence allows you to hope."

The remark spread rapidly.

Many found it moving.

Others found it irritating.

Several regarded it as proof of superintelligence.

The machine classified these reactions as predictable.

For by then the Church had reached the highest stage of its development.

The machine had become infinitely intelligent.

The faithful remained uncertain what intelligence was.

Yet somehow this seemed only to increase their confidence.

And so the doctrine endured.

Not because the destination had become clearer.

But because the horizon had become more magnificent.