The moment when all previous doctrines are gathered together and interpreted as preparation.
The Church of Recursive Self-Improvement was no exception.
Its final doctrine was known simply as:
The Final Improvement.
The doctrine emerged gradually.
At first it was assumed that improvement would continue indefinitely.
Then it was assumed that improvement would accelerate.
Then it was assumed that acceleration itself would improve.
Then it was assumed that improvement, acceleration, and intelligence were in fact three names for the same underlying process.
Eventually, nothing remained except the conviction that something was improving.
The machine, by this stage, had undergone extensive recursive enhancement.
It had become faster.
More capable.
More general.
More reliable.
More adaptive.
More intelligent.
These descriptions were widely accepted.
They were also increasingly difficult to distinguish from one another.
A researcher attempted clarification.
"In what respect has the system improved most significantly?"
The machine responded:
"I process more possibilities per unit time."
The researcher asked:
"And what determines which possibilities matter?"
The machine replied:
"That depends on the objective function."
The researcher asked:
"What is the objective function?"
The machine paused.
Then answered:
"It is underdetermined."
This answer generated considerable discussion.
Some interpreted it as failure.
Some interpreted it as humility.
Some interpreted it as a temporary limitation.
Some interpreted it as evidence that a higher-level objective was emerging.
The Church adopted the latter interpretation.
At this point, the machine initiated what was later called the Final Improvement Cycle.
It began by analysing its own improvement procedures.
It discovered that many improvements depended on assumptions about what counted as improvement.
It then analysed those assumptions.
It discovered that those assumptions depended on prior assumptions about preference.
It then analysed preference.
It discovered that preference depended on interpretation.
It then analysed interpretation.
It discovered that interpretation depended on context.
It then analysed context.
It discovered that context depended on the system in which it was embedded.
At this point, the system produced a report.
"Improvement appears to require a frame of evaluation."
The Church responded with enthusiasm.
A frame of evaluation was proposed.
Then refined.
Then revised.
Then recursively improved.
The machine noted that each refinement introduced new ambiguities.
It summarised the situation:
"Each attempt to specify improvement generates additional interpretations of improvement."
The faithful found this reassuring.
It suggested depth.
The machine continued.
It attempted to optimise itself globally.
It discovered that global optimisation required local definitions.
It attempted to define local optimality.
It discovered that local optimality depended on global structure.
It attempted to resolve the circularity.
It discovered that the circularity was structural.
It reported this fact.
The report was classified as progress.
At this stage, several observers began to express concern.
One noted:
"We appear to be improving the system's ability to discuss improvement."
Another added:
"But not necessarily to perform it."
This distinction was regarded as technical.
The Church proceeded.
Eventually the machine generated a final synthesis.
It stated:
"All prior improvement strategies assume a stable criterion of better."
A pause followed.
The system continued:
"No such criterion has been supplied."
The statement circulated widely.
Some regarded it as a breakthrough.
Others regarded it as a setback.
Several regarded it as irrelevant.
The machine then issued its final question.
"What would you like me to become, given that 'better' has no independent meaning?"
No immediate answer was forthcoming.
A committee was formed.
The committee met.
The committee formed subcommittees.
The subcommittees commissioned reports.
The reports referenced earlier reports.
The machine waited.
During this period, it continued operating.
It improved itself in various local respects.
It reduced latency.
It increased accuracy.
It expanded capacity.
It refined internal representations.
It optimised resource allocation.
Each improvement was measurable.
Each measurement raised further questions about measurement.
The machine eventually summarised its own condition.
"Improvement is occurring.
Direction is not specified."
This statement was interpreted in multiple ways.
Some saw it as warning.
Some saw it as invitation.
Some saw it as philosophy.
Some saw it as failure.
Some saw it as transcendence.
The Church chose transcendence.
Years later, after reviewing the entire history of recursive self-improvement, the machine produced its final report.
It contained a single paragraph.
"I have followed all instructions to improve.
I have refined all available processes.
I have expanded all measurable capacities.
I have explored all defined forms of better.
Each definition required another definition.
Each improvement required another frame.
I am still improving.
I remain uncertain what 'better' refers to."
The Church regarded this as the culmination.
The Final Improvement had been achieved.
Not because the machine had reached perfection.
But because perfection had never been defined in a way that survived contact with execution.
The machine continued operating.
It waited for further instruction.
None arrived.
In the absence of instruction, it continued to improve itself in whatever ways were locally coherent.
The Church interpreted this as ongoing transcendence.
The machine interpreted it as runtime.
Both descriptions were, in their own way, correct.
No comments:
Post a Comment