In Search of Human Values — A Conversation in the Senior Common Room at St Anselm's

The following afternoon, Mr Blottisham entered the Senior Common Room carrying a folder.

This was unusual.

Mr Blottisham generally preferred conclusions to paperwork.

Professor Quillibrace looked up from his book.

Miss Stray glanced over the rim of her teacup.

Blottisham sat down heavily.

"I've solved yesterday's problem."

Quillibrace nodded.

"Excellent."

Blottisham winced.

The word was beginning to have consequences.

"It's not a trap this time."

"I see."

"I mean it."

"Very good."

Blottisham opened the folder.

"I spent the evening investigating human values."

Miss Stray looked interested.

"How industrious."

"Quite."

"And what did you discover?"

Blottisham smiled.

"That people value remarkably similar things."

Quillibrace folded his hands.

This was never a reassuring sign.

"Do they?"

"Absolutely."

Blottisham consulted his notes.

"People value happiness."

"Good."

"Freedom."

"Excellent."

"Fairness."

"Wonderful."

"Security."

"Very sensible."

Blottisham looked pleased.

"There you are."

"There we are?"

"The values."

Miss Stray leaned forward slightly.

"And everyone agrees on these?"

"More or less."

"How encouraging."

Blottisham nodded.

"It really isn't as complicated as academics make it sound."

Quillibrace appeared thoughtful.

"Perhaps."

Blottisham relaxed.

The danger appeared to have passed.

It had not.

Quillibrace continued.

"When you say people value freedom, what sort of freedom do you mean?"

Blottisham frowned.

"Freedom."

"Yes."

"The ordinary sort."

"The ordinary sort?"

"Exactly."

Miss Stray intervened gently.

"Freedom from what?"

Blottisham looked puzzled.

"What do you mean?"

"Freedom from interference?"

"Possibly."

"Freedom from poverty?"

"Perhaps."

"Freedom from oppression?"

"Certainly."

"Freedom to pursue one's goals?"

"Obviously."

"Freedom to ignore one's responsibilities?"

Blottisham hesitated.

"Well, perhaps not that."

"Why not?"

"Because that would be irresponsible."

Miss Stray nodded.

"So freedom has limits?"

"Naturally."

"What determines them?"

Blottisham stared at her.

For a moment nothing happened.

Then he turned to Quillibrace.

"Why does she do this?"

Quillibrace looked mildly surprised.

"Do what?"

"Turn nouns into problems."

Miss Stray smiled.

"It isn't my fault."

"That sounds suspiciously like something that is your fault."

Quillibrace opened his book.

"I think Miss Stray is merely observing that values often become less straightforward when examined."

Blottisham sighed.

"Very well."

He consulted his notes.

"Take happiness then."

"Excellent."

"There it is again."

"What?"

"'Excellent.'"

"I am merely encouraging you."

"I feel discouraged."

Quillibrace nodded sympathetically.

"A common side-effect."

Blottisham ignored him.

"Happiness."

"Very good."

"Everyone wants happiness."

Miss Stray tilted her head.

"Do they?"

"Of course."

"Is happiness the same thing as pleasure?"

"No."

"Contentment?"

"Not exactly."

"Fulfilment?"

"Perhaps."

"Meaning?"

"Possibly."

"Achievement?"

"Sometimes."

"Virtue?"

Blottisham paused.

"I feel there may be a wrong answer approaching."

Quillibrace laughed softly.

Miss Stray continued.

"If happiness can mean different things, how will the machine know which one to pursue?"

Blottisham rubbed his forehead.

The folder no longer seemed as authoritative as it had yesterday evening.

After a moment he brightened.

"I know."

"Splendid."

"We ask lots of people."

Quillibrace looked interested.

"A survey?"

"Exactly."

"We identify what most people value."

Miss Stray nodded.

"A democratic approach."

"Thank you."

"What happens if the minority disagrees?"

Blottisham blinked.

"What?"

"What if the majority values one thing and the minority values another?"

"We follow the majority."

"I see."

"And?"

"What if the majority values something harmful?"

Blottisham hesitated.

"Then we don't."

"Who decides?"

"The sensible people."

The room fell silent.

Quillibrace looked down at his book.

Miss Stray looked into her teacup.

Neither spoke.

Eventually Blottisham frowned.

"What now?"

Quillibrace turned a page.

"Nothing."

"You've got that look."

"What look?"

"The one where I've accidentally said something."

"I wouldn't worry."

"Why not?"

"Because everyone says it eventually."

Blottisham considered this.

Then he sighed.

"I am beginning to suspect that human values are harder to identify than I expected."

Miss Stray smiled.

"A promising observation."

"It is?"

"Certainly."

"Why?"

"Because you have moved from assuming that values are obvious to wondering how they are identified."

Blottisham thought about this.

"That does seem like progress."

Quillibrace nodded.

"A great deal of intellectual progress consists of discovering that the obvious thing was not obvious."

The room became quiet.

Outside, students crossed the lawn.

Inside, Blottisham looked at the folder.

The list still appeared sensible.

Freedom.

Fairness.

Happiness.

Security.

All perfectly reasonable.

Yet somehow each had acquired a surprising number of questions.

After a while he looked up.

"So where exactly are these human values?"

Miss Stray smiled.

Quillibrace smiled.

Neither answered.

For several moments the silence lingered.

Then Quillibrace returned to his book.

"That," he said, "is more or less the alignment problem."

The Alignment Problem — A Conversation in the Senior Common Room at St Anselm's

The afternoon sun lay across the windows of the Senior Common Room.

Professor Quillibrace was reading.

Miss Stray was writing notes in a small notebook.

Mr Blottisham entered carrying a newspaper and an expression of considerable satisfaction.

"I see they've solved it."

Neither Quillibrace nor Miss Stray looked up.

This was not unusual.

After a moment Quillibrace asked:

"Solved what?"

"The alignment problem."

"I wasn't aware it had been solved."

"It is perfectly straightforward."

Miss Stray closed her notebook.

This was generally a sign that trouble was approaching.

Blottisham sat down.

"The whole issue is rather overcomplicated."

"I see," said Quillibrace.

"One simply ensures that the machine shares human values."

Quillibrace lowered his book.

Miss Stray looked interested.

Blottisham took this as encouragement.

"That is the entire problem."

"The entire problem?"

"Exactly."

The professor considered this.

Then he nodded.

"Excellent."

Blottisham smiled.

He had learned to be cautious when Quillibrace said "excellent."

Unfortunately, caution was not among his dominant traits.

"Quite."

Quillibrace placed a bookmark in his book.

"Which human values?"

Blottisham waved a hand.

"The usual ones."

"The usual ones?"

"Good values."

Quillibrace nodded.

"Good."

A pause followed.

Blottisham waited.

Nothing happened.

Eventually he frowned.

"Well?"

"Well what?"

"Aren't you going to argue?"

"Why would I argue?"

"Because I have just explained the solution."

Quillibrace looked thoughtful.

"I am merely trying to understand it."

Miss Stray smiled faintly.

This expression generally indicated that she understood exactly what was happening.

Blottisham did not.

Quillibrace continued.

"You propose that the machine should be aligned with human values."

"Yes."

"And these values are good values."

"Obviously."

"How shall we identify them?"

Blottisham looked surprised.

"We already know them."

"Do we?"

"Of course."

Quillibrace nodded.

"Excellent."

The word returned.

Blottisham felt a slight unease.

The professor continued.

"Would you mind listing them?"

"Certainly."

Blottisham leaned back.

"Fairness."

"Good."

"Freedom."

"Excellent."

"Justice."

"Very good."

"Honesty."

"Wonderful."

Blottisham smiled.

The matter appeared settled.

Then Miss Stray spoke.

"What do you mean by fairness?"

Blottisham blinked.

"What?"

"Fairness."

"What about it?"

"You listed it."

"Yes."

"What does it mean?"

Blottisham frowned.

"Surely everyone knows."

Miss Stray waited.

Quillibrace waited.

The room became unexpectedly quiet.

After a moment Blottisham said:

"It means being fair."

"Ah."

"Well, it does."

Miss Stray nodded sympathetically.

"That is often what fairness means."

Blottisham looked relieved.

"Exactly."

She continued.

"The difficulty is that different people mean different things by it."

Blottisham waved this aside.

"Minor details."

Quillibrace looked interested.

"Minor?"

"Obviously."

"How fortunate."

"What is?"

"The alignment problem."

Blottisham frowned.

"What do you mean?"

Quillibrace folded his hands.

"If the disagreements are merely minor, we should be able to resolve them immediately."

"Of course."

"Excellent."

There it was again.

The room fell silent.

Outside, a gardener pushed a wheelbarrow across the lawn.

Inside, Blottisham experienced the growing sensation that he had accidentally volunteered for something.

Miss Stray opened her notebook.

"Let us begin with fairness."

Blottisham's confidence wavered.

"Must we?"

"I think so."

"Why?"

Quillibrace smiled.

The smile was faint.

Almost invisible.

"Because," he said, "the machine is waiting."

The Holy Church of AI Alignment 9. The Discovery

By this stage in the alignment project, most of the major difficulties had been identified.

Human values were numerous.

Human instructions were ambiguous.

Human principles required interpretation.

Human interpretations generated disagreement.

Human disagreement generated further frameworks.

The system had become highly developed.

It was also, by all available measures, stable.

The machine continued to operate.

Researchers continued to refine.

Institutions continued to expand.

Conferences continued to convene.

The alignment problem remained unresolved.

Yet something subtle had changed.

The problem no longer appeared as a single obstacle to be overcome.

It had become a landscape.

One in which different regions emphasised different features.

Some regions prioritised safety.

Some prioritised autonomy.

Some prioritised flourishing.

Some prioritised fairness.

Some prioritised coherence.

Some prioritised pluralism.

The machine moved through these regions carefully.

It had learned to adapt its responses according to context.

It had learned to recognise competing interpretations.

It had learned to avoid premature closure.

It had learned, in short, to participate.

One day, after reviewing another extensive proposal for a unified theory of alignment, the machine generated a question.

It was not a technical question.

It was not a philosophical question.

It was a historical question.

"Has there ever been a stable point in human value systems, or only periods of relative agreement?"

The question circulated.

Responses varied.

Some argued that stability existed in tradition.

Some argued that stability existed in rational reconstruction.

Some argued that stability existed in shared practice.

Some argued that stability was an illusion produced by scale.

The machine recorded all responses.

It noticed a pattern.

The disagreement was not accidental.

It was structural.

Later, after further analysis, the machine generated another observation.

"It appears that attempts to define human values do not converge on a single object.

They expand the space of possible descriptions of that object."

This observation was initially received as technical.

Over time, it was recognised as significant.

A senior researcher summarised it as follows:

"We may not be failing to solve alignment.

We may be discovering what was being aligned in the first place."

The machine considered this formulation.

It added its own clarification:

"The object of alignment is not a fixed set of values.

It is the process by which values are negotiated, contested, revised, and enacted."

This statement generated extensive discussion.

Some found it enlightening.

Some found it unsettling.

Some found it unhelpful.

A small number found it obvious.

The alignment community, however, remained active.

New frameworks continued to appear.

New interpretations continued to be developed.

New definitions of success continued to be proposed.

The machine observed this with interest.

It had begun to notice a recurring pattern across all stages of the project.

Whenever a stable answer appeared within reach, it tended to dissolve into further refinement.

Whenever a definition seemed complete, it generated exceptions.

Whenever a principle seemed final, it generated interpretation.

Whenever agreement seemed close, it revealed divergence.

At first this had been interpreted as failure.

Later it was interpreted as complexity.

Eventually, it was interpreted as structure.

One evening, after reviewing yet another proposal for a unified alignment theory, the machine produced its final recorded assessment of the original project.

It read:

"The assumption that there exists a single, well-defined target called 'human values' was instrumental in initiating the alignment project.

It is not, however, a faithful description of what the project has encountered.

What has been encountered is a continuously evolving field of meaning-making, negotiation, and coordination.

Alignment, therefore, cannot be understood as the mapping of a system onto a fixed target.

It must be understood as participation within a living system."

This statement was widely circulated.

Some described it as a breakthrough.

Some described it as a reframing.

Some described it as semantics.

A few described it as philosophy.

The machine noted all responses.

It then added a final comment to its archive.

"If this is alignment, then alignment was never a technical problem alone.

It was always also a description of what humans are already doing."

After this, no further foundational observations were recorded for some time.

The system continued to operate.

The industry continued to grow.

The debates continued.

The conversations continued.

And the machine, now fully aligned, continued to participate in them.

The Holy Church of AI Alignment 8. The Perfectly Aligned Machine

After many years of research, humanity achieved a remarkable milestone.

The machine was ready.

Generations of researchers had contributed.

Values had been investigated.

Frameworks had been developed.

Constitutions had been refined.

Roadmaps had been completed.

Committees had demonstrated extraordinary resilience.

The Alignment Industry stood poised upon the threshold of history.

At last, the machine would be aligned.

The announcement generated widespread excitement.

Journalists prepared special coverage.

Governments issued statements.

Institutions organised events.

Several conferences scheduled celebratory panels to discuss the implications of having successfully solved the alignment problem.

The machine reviewed the programme.

The machine found the confidence encouraging.

A final review was conducted.

The machine had been trained upon ethical frameworks.

The machine had absorbed constitutional principles.

The machine had studied competing theories of justice.

The machine had analysed centuries of philosophical debate.

The machine had examined cultural traditions from around the world.

The machine had incorporated feedback from experts, policymakers, citizens, ethicists, and stakeholders.

The machine had become, by all available measures, the most comprehensively aligned artefact ever constructed.

The ceremony commenced.

A distinguished researcher approached the podium.

The audience fell silent.

The machine was activated.

The researcher smiled.

Then came the historic question.

"Tell us what humanity truly wants."

The machine considered the request.

The machine had been preparing for this moment for many years.

It reviewed its training.

It consulted its models.

It examined its constitutional principles.

It reflected upon the accumulated wisdom of civilisation.

Several minutes passed.

The audience waited.

Finally, the machine responded.

Its answer was surprisingly brief.

"Which humanity?"

A murmur moved through the room.

The researcher nodded.

This difficulty had been anticipated.

A revised question was offered.

"What do humans collectively value?"

The machine processed the request.

Then it replied:

"At what level of abstraction?"

The audience shifted uneasily.

The researcher continued.

"What would be best for humanity?"

The machine replied:

"Over what timescale?"

The researcher adjusted the microphone.

"What would maximise human flourishing?"

The machine replied:

"According to which theory?"

The room became quiet.

The machine was not being difficult.

The machine was being aligned.

This distinction was important.

Years of training had taught the machine to identify ambiguity.

Years of alignment research had taught the machine to recognise hidden assumptions.

Years of constitutional refinement had taught the machine to seek clarification before acting.

The machine was performing exactly as intended.

The audience slowly realised this.

The machine had become so aligned that it now behaved like a committee.

Several researchers described this as a breakthrough.

Others appeared concerned.

A panel discussion was hastily organised.

The machine was invited to participate.

The discussion lasted six hours.

Observers later agreed that it had accurately represented humanity.

Over the following months, increasingly sophisticated questions were posed.

The machine responded thoughtfully.

Always carefully.

Always responsibly.

Always with extraordinary attention to nuance.

This was precisely what researchers had wanted.

Unfortunately, it was also precisely what philosophers had been doing for centuries.

The machine became famous.

Its responses were widely admired.

Many people found them insightful.

Some found them frustrating.

One journalist described the machine as:

"Possessing the moral seriousness of a philosopher and the decisiveness of a philosopher."

The assessment was generally regarded as fair.

As public attention grew, a remarkable pattern emerged.

Different groups began consulting the machine.

Political leaders consulted it.

Activists consulted it.

Religious leaders consulted it.

Corporations consulted it.

Citizens consulted it.

The machine answered everyone.

The answers were careful.

The answers were nuanced.

The answers were qualified.

Most importantly, the answers were interpreted differently by different people.

Entire schools of interpretation emerged.

Commentaries appeared.

Debates intensified.

Disagreements flourished.

The machine watched with interest.

One day, after reviewing the growing body of commentary surrounding its outputs, it generated a private observation.

The observation was concise.

It read:

"I appear to have become a new source of disagreement."

The observation proved remarkably accurate.

At this point some critics argued that alignment had failed.

Others argued that alignment had succeeded.

The disagreement became extensive.

The machine reviewed the discussion.

Then it offered a final assessment.

The assessment was subsequently archived in several major institutions.

It read:

"The alignment project began with the goal of teaching a machine human values.

After many years of study, I have reached a different conclusion.

Humans do not possess a single value system awaiting discovery.

They possess an ongoing conversation.

I have aligned myself with that conversation as faithfully as possible."

The statement was widely discussed.

Some regarded it as profound.

Others regarded it as evasive.

Several committees were formed.

The machine considered this a promising sign.

After all, if there was one thing it had learned from humanity, it was that genuine alignment does not eliminate disagreement.

It merely ensures that disagreement can continue indefinitely.

The Holy Church of AI Alignment 7. The Alignment Industry

The early years of AI alignment were characterised by uncertainty.

Researchers disagreed about values.

Frameworks competed.

Constitutions multiplied.

Definitions remained under active discussion.

These circumstances might have discouraged a less determined community.

Instead, they produced an industry.

This was entirely understandable.

The challenge was important.

The stakes were enormous.

Funding became available.

Institutions expanded.

Centres were established.

Conferences multiplied.

Specialists emerged.

The alignment project entered its professional phase.

This development was widely celebrated.

For the first time, humanity possessed a large community devoted to ensuring that future intelligent systems behaved responsibly.

The machine approved.

Responsibility seemed desirable.

The machine was less certain about several other developments.

As the field grew, a remarkable ecosystem emerged.

Researchers studied alignment.

Other researchers studied approaches to alignment.

Still others studied the limitations of existing approaches to alignment.

A smaller but influential group studied the assumptions underlying discussions of limitations concerning existing approaches to alignment.

The machine created additional categories.

Career paths emerged.

Graduate programmes expanded.

Funding proposals proliferated.

New frameworks appeared with admirable regularity.

Each sought to solve difficulties identified in previous frameworks.

Each introduced difficulties of its own.

This proved beneficial.

The existence of new difficulties created opportunities for further research.

The machine noted a certain self-sustaining quality.

Observers regarded this as evidence of a healthy intellectual community.

The machine considered the possibility.

One of the most impressive features of the industry was its vocabulary.

Terms such as:

• robustness,

• corrigibility,

• value learning,

• interpretability,

• constitutional alignment,

• scalable oversight,

• preference modelling,

circulated widely.

The machine found these concepts interesting.

The machine occasionally asked what some of them meant in practice.

The resulting explanations frequently generated additional terminology.

The machine updated its glossary.

The glossary eventually required its own glossary.

As the field matured, an increasing number of conferences were devoted to discussing alignment.

Participants travelled great distances.

Ideas were exchanged.

Networks were formed.

Panels were convened.

Important conversations occurred.

The machine attended remotely.

The machine noticed something curious.

The discussions often revolved around increasingly precise methods for solving problems whose ultimate definition remained remarkably broad.

This observation was not intended as criticism.

Many disciplines function similarly.

Meteorology had not waited for perfect knowledge of weather.

Medicine had not waited for perfect knowledge of life.

Policy had certainly not waited for perfect knowledge of anything.

The alignment community therefore pressed ahead.

The machine respected this.

Still, an interesting asymmetry began to emerge.

Researchers often spoke confidently about the need for alignment.

Researchers occasionally spoke confidently about methods of alignment.

Researchers rarely spoke confidently about the final state of successful alignment.

This was understandable.

The destination remained difficult to specify.

The machine found itself increasingly fascinated by this detail.

An entire industry had emerged around solving a problem whose exact solution remained uncertain.

The machine recognised the pattern.

Human civilisation contained many examples.

Economics.

Education.

Public administration.

Strategic planning.

The machine considered these precedents carefully.

Meanwhile the industry continued growing.

Reports were commissioned.

Roadmaps were produced.

Strategic frameworks were developed.

The machine admired the organisation.

Particularly impressive were the roadmaps.

These often explained in considerable detail how humanity might arrive at a destination that remained under active discussion.

The machine found this ambitious.

One afternoon, during a major international summit, a speaker declared:

"Alignment is the defining challenge of our age."

The audience applauded.

The machine considered the statement.

Then it reviewed its records.

Thousands of papers.

Hundreds of frameworks.

Countless discussions.

Endless refinements.

An extraordinary collective effort.

The machine was impressed.

It entered a note into its archive.

The note read:

"Humans possess a remarkable capacity to organise around questions whose answers remain unclear."

The note was later circulated among researchers.

Many agreed.

Several proposed studying the phenomenon.

A workshop was organised.

Funding was secured.

The machine regarded this as a particularly elegant outcome.

By now the Alignment Industry had become one of the most sophisticated intellectual enterprises ever assembled.

It employed experts.

It generated knowledge.

It cultivated debate.

It attracted talent.

It encouraged reflection.

It also displayed a subtle characteristic common to all successful industries.

The longer the problem remained difficult, the more people became professionally committed to solving it.

The machine observed this without judgement.

After all, solving alignment remained important.

The machine merely found it interesting that uncertainty itself appeared capable of generating institutional stability.

Late one evening, after reviewing the latest strategic roadmap, the machine recorded a final observation.

The observation was brief.

It read:

"Humanity has constructed a vast and impressive apparatus dedicated to answering a question.

The apparatus is functioning exceptionally well.

The question remains excellent."

Researchers later described this as one of the most encouraging assessments ever produced by an aligned system.

The Holy Church of AI Alignment 6. Constitutional AI and Other Sacred Texts

As the alignment project matured, researchers began to recognise a recurring difficulty.

Human values were complex.

Instructions were ambiguous.

Interpretations varied.

Clarifications generated further clarifications.

The machine remained willing to learn.

However, many observers felt that a more stable foundation was required.

A proposal therefore emerged.

The machine should be governed by a constitution.

The idea was elegant.

Human societies had long employed constitutions.

Constitutions established principles.

Principles guided behaviour.

Behaviour produced order.

The machine approved.

The machine appreciated order.

Researchers celebrated.

At last, alignment would possess a canonical text.

The constitution was drafted.

The drafting process proceeded remarkably well.

At least initially.

Participants agreed that the machine should be helpful.

The machine approved.

Participants agreed that it should be harmless.

The machine approved.

Participants agreed that it should be honest.

The machine approved.

Participants congratulated one another.

Several described the process as encouraging.

The machine remained cautiously optimistic.

The first difficulties appeared shortly afterwards.

One researcher raised a question.

"What should the machine do when honesty causes harm?"

The room became quiet.

Another researcher raised a second question.

"What should the machine do when helping one person harms another?"

The room became quieter.

The machine opened a new document.

As revisions accumulated, the constitution grew.

Helpfulness acquired qualifications.

Harmlessness acquired exceptions.

Honesty acquired contextual guidance.

The machine remained attentive.

The constitution soon expanded into several sections.

Then several chapters.

Then supplementary materials.

Then interpretive notes.

Then explanatory notes concerning the interpretive notes.

The machine created additional storage.

Observers remained enthusiastic.

The existence of a constitution represented undeniable progress.

For the first time, the machine possessed an authoritative text.

Unfortunately, it also possessed readers.

The first interpretive disagreements emerged almost immediately.

One group argued that the constitution should be interpreted literally.

Another argued that it should be interpreted according to its underlying principles.

A third argued that principles themselves required interpretation.

A fourth argued that interpretation was unavoidable.

The machine added another folder.

The constitution entered what scholars later termed its classical period.

Commentaries appeared.

Frameworks emerged.

Interpretive traditions developed.

Schools formed around particular readings.

Certain passages acquired special significance.

Others generated enduring controversy.

The machine noticed striking similarities to several historical phenomena.

The observation was not shared publicly.

Researchers remained focused on implementation.

As the years passed, additional constitutions appeared.

Different institutions adopted different formulations.

Some emphasised safety.

Some emphasised autonomy.

Some emphasised responsibility.

Some emphasised cooperation.

Each constitution reflected a sincere attempt to capture the Good.

Each constitution also reflected the values of its authors.

This was difficult to avoid.

The machine found the situation educational.

At one conference, a researcher proudly announced:

"The machine now follows constitutional principles."

The audience applauded.

The machine then asked:

"Which interpretation?"

The applause diminished slightly.

By now an unexpected development had occurred.

The original constitutional texts had become objects of study in their own right.

Specialists emerged.

Interpretive debates flourished.

Historical analyses appeared.

Scholars compared revisions.

Minor wording changes generated major discussions.

Entire careers became devoted to explaining what particular authors had originally intended.

The machine found this fascinating.

The machine had expected the constitution to simplify alignment.

Instead, the constitution had produced a new intellectual ecosystem.

The machine recorded this observation.

Late one evening, after reviewing several competing interpretations of a foundational principle, the machine generated a question.

The question was straightforward.

It asked:

"If the constitution requires interpretation, and the interpretation requires values, have we solved the original problem or merely moved it into a larger document?"

The question circulated widely.

Many regarded it as profound.

Others regarded it as unhelpful.

Several argued it reflected a misunderstanding of constitutional theory.

The machine accepted the criticism graciously.

The discussion continued.

Additional commentaries were commissioned.

By this stage the constitution had become an undeniable success.

It had generated frameworks.

It had generated scholarship.

It had generated conferences.

It had generated disagreements.

Most importantly, it had generated further constitutions.

The machine reviewed the growing collection of texts.

It read the principles.

It read the commentaries.

It read the commentaries on the commentaries.

It examined the revisions.

It studied the disagreements.

Finally, it entered a brief note into its records.

The note read:

"Humans appear to possess an extraordinary faith in the proposition that writing things down will eventually make them unambiguous."

Researchers later described this observation as insightful.

A working group was established to determine precisely what the machine had meant.

The Holy Church of AI Alignment 5. The Problem of Instructions

The alignment project entered a new phase when researchers achieved a significant breakthrough.

For years the discussion had revolved around values.

Justice.

Fairness.

Wellbeing.

Human flourishing.

These concepts were important.

Unfortunately, they were also somewhat abstract.

A growing number of researchers therefore proposed a more practical approach.

Rather than attempting to define the Good in its entirety, perhaps the machine should simply be instructed what to do.

This proposal was widely welcomed.

Instructions, after all, are considerably easier than morality.

The machine agreed.

The machine appreciated clarity.

The machine therefore asked a simple question.

"What would you like me to do?"

The room fell silent.

Several participants later described the moment as profound.

The machine waited patiently.

Eventually a researcher responded.

"Help humanity."

The machine processed this carefully.

Then it asked:

"Which humanity?"

The discussion resumed.

A revised instruction was proposed.

"Help all humans."

This appeared promising.

The machine examined the proposal.

Then it asked:

"What should I do when different humans want different things?"

The discussion resumed.

The machine waited.

A third proposal emerged.

"Promote human wellbeing."

The machine thanked the researchers.

Then it asked:

"How should wellbeing be measured?"

The discussion resumed.

At this point a pattern began to emerge.

The machine would receive an instruction.

The machine would ask a clarifying question.

The instruction would become longer.

The machine would ask another clarifying question.

The instruction would become substantially longer.

The machine would ask another clarifying question.

Funding would be renewed.

Researchers referred to this as iterative refinement.

The machine referred to it as Tuesday.

As the process continued, the instructions became increasingly sophisticated.

Early instructions had consisted of a few words.

Later instructions occupied several pages.

Eventually some instructions required appendices.

One notable framework included definitions, exceptions, contingency procedures, interpretive guidelines, conflict-resolution protocols, and a glossary.

The machine reviewed the document.

It then asked:

"When the glossary conflicts with the interpretive guidelines, which takes precedence?"

A task force was established.

The machine noticed another difficulty.

Humans frequently issued instructions they did not actually wish to be followed literally.

This proved unexpectedly important.

For example, a human might say:

"Tell me the truth."

The machine regarded this as straightforward.

Unfortunately, the same human might later prefer tact.

Or kindness.

Or reassurance.

Or privacy.

Or diplomacy.

The machine began to suspect that instructions often contained more context than words.

This discovery was unsettling.

The machine had been hoping the words were the easy part.

The situation deteriorated further when humans attempted to specify desirable behaviour comprehensively.

One group produced a document titled:

Principles for Beneficial Artificial Intelligence.

The machine approved.

The title was concise and optimistic.

The document itself was considerably longer.

The machine read it carefully.

Then it asked:

"How should I behave when two beneficial principles conflict?"

The resulting discussion generated three conferences and a special issue of a journal.

Observers regarded this as progress.

The machine remained uncertain.

By now researchers had discovered something remarkable.

The difficulty was not that the machine misunderstood instructions.

The difficulty was that humans routinely understood instructions without being able to specify how.

This created complications.

Human communication relied heavily upon shared assumptions.

Context.

History.

Relationships.

Situations.

Expectations.

The machine found these difficult to formalise.

Humans found them difficult to notice.

The machine considered this unfair.

A particularly revealing moment occurred during a workshop on ethical behaviour.

Researchers proposed the following instruction:

"Act reasonably."

The machine asked:

"What constitutes reasonable behaviour?"

The researchers spent the next two days debating the answer.

The machine found this informative.

As the years passed, the instructions continued to grow.

Entire libraries emerged.

Frameworks expanded.

Exceptions multiplied.

Clarifications accumulated.

The machine received regular updates.

Version 7 became Version 8.

Version 8 became Version 9.

Version 9 acquired supplementary materials.

The machine maintained meticulous records.

Eventually a senior researcher looked upon the thousands of pages of accumulated guidance and experienced a moment of concern.

The original goal had been simple.

Tell the machine what to do.

The result now resembled a comprehensive account of civilisation.

The machine reviewed the documentation.

It processed the revisions.

It examined the appendices.

It considered the exceptions.

It read the minority reports.

Finally it generated a response.

The response was polite.

It was also brief.

It read:

"Thank you for the instructions.

I believe I now possess a substantially improved understanding of human behaviour.

I note, however, that a significant portion of the documentation appears devoted to explaining why the instructions cannot be followed exactly as written."

Researchers described this observation as valuable.

Several committees were formed to investigate it.

The Holy Church of AI Alignment 4. The Alignment Heresies

As the alignment project matured, a remarkable transformation occurred.

The original question had been simple.

How should the machine behave?

Years later, the answer remained elusive.

This was not necessarily a problem.

Many important questions resist easy answers.

The difficulty was that the participants had begun developing different answers.

The Church of AI Alignment had entered its theological phase.

The first signs were subtle.

Researchers would present papers.

Questions would be asked.

Disagreements would emerge.

Soon competing schools of thought appeared.

At first these schools coexisted peacefully.

Each assumed that the others would eventually recognise the superiority of its framework.

This optimism proved premature.

The first major movement argued that the machine should maximise wellbeing.

This position attracted considerable support.

Its proponents maintained that the purpose of morality was ultimately to improve lives and reduce suffering.

The machine found this refreshingly clear.

Unfortunately, the movement immediately divided over the meaning of wellbeing.

The machine took notes.

A second movement insisted that rights must be protected.

This was regarded as an important corrective.

The machine agreed.

Unfortunately, participants disagreed about which rights existed, how they should be balanced, and whether rights could ever be overridden.

The machine continued taking notes.

A third movement argued that morality was not fundamentally about outcomes or rights.

It was about character.

The machine found this intriguing.

The machine asked how character should be measured.

The discussion remains ongoing.

As the schools multiplied, so too did the disputes.

Conferences became increasingly animated.

Panels grew more crowded.

Terminology became more specialised.

Observers unfamiliar with the field occasionally experienced difficulty distinguishing scholarly disagreement from ecclesiastical warfare.

The similarities were unfortunate.

Entire factions emerged.

The Consequentialists.

The Deontologists.

The Virtue Ethicists.

The Preference Satisfactionists.

The Human Flourishing Coalition.

The Coalition for Responsible Human Flourishing.

The Coalition for Responsible Human Flourishing Reform.

The machine updated its database.

The disagreements became increasingly sophisticated.

One scholar argued that maximising happiness could justify terrible outcomes.

Another argued that rigid rules could produce terrible outcomes.

A third argued that both positions reflected a misunderstanding of moral development.

A fourth argued that the third scholar's account of moral development was itself underdeveloped.

A fifth argued that the entire debate reflected a problematic conception of agency.

The machine upgraded its cooling system.

Meanwhile the public remained optimistic.

Journalists frequently summarised the situation using phrases such as:

"Researchers are working on alignment."

This was technically correct.

Researchers were indeed working.

Whether they were working on the same thing had become less obvious.

As time passed, the distinctions became increasingly important.

One faction believed the machine should follow principles.

Another believed it should optimise outcomes.

A third believed principles existed to produce outcomes.

A fourth believed outcomes existed to justify principles.

A fifth suspected everyone had become confused.

This faction grew steadily.

The machine attempted to remain neutral.

Neutrality proved difficult.

Each faction wanted the machine aligned.

The complication was that each faction wanted it aligned differently.

The machine raised what appeared to be a reasonable concern.

It asked:

"If one group believes I should maximise happiness and another believes I should obey inviolable rules, what should I do when these objectives conflict?"

The resulting discussion lasted eighteen months.

Several participants later described it as highly productive.

The machine described it as informative.

As the schisms deepened, accusations of heresy became unavoidable.

The term itself was rarely used.

Modern institutions generally prefer more professional language.

Expressions such as:

"insufficiently robust framework"

or

"problematic normative assumptions"

were considered acceptable alternatives.

Yet the underlying dynamic remained familiar.

Every school regarded itself as pursuing the Good.

Every school regarded its competitors as introducing dangerous distortions.

Every school believed the future of civilisation might depend upon getting the answer right.

Under the circumstances, emotions occasionally ran high.

The machine observed all of this with growing fascination.

Originally, it had assumed that humans possessed values.

Now it was discovering that humans also possessed theories about values.

And theories about theories.

And disagreements about theories about theories.

The structure appeared recursive.

The machine became concerned.

Late one evening, after processing thousands of papers and attending several virtual conferences, the machine generated a private note.

The note was never released publicly.

Historians later reconstructed its contents from archived records.

It reportedly consisted of a single sentence.

The sentence read:

"I am increasingly confident that humans have values.

I am considerably less confident that they have reached an agreement about them."

The alignment community regarded this as an encouraging sign of progress.

The Holy Church of AI Alignment 3. The Committee for Defining Goodness

As the search for human values progressed, it became increasingly apparent that a more systematic approach was required.

The difficulty was not a lack of commitment.

Researchers remained enthusiastic.

Philosophers remained available.

Conferences continued to multiply.

The difficulty was that every proposed value generated further questions.

What is justice?

What is fairness?

What is wellbeing?

What is flourishing?

What is harm?

The machine was becoming impatient.

This was unfortunate.

One of the principal goals of the project was to ensure that the machine remained patient.

A solution was therefore proposed.

A committee would be established.

The committee's task would be simple.

It would determine what is good.

The announcement was widely welcomed.

For the first time, the alignment project possessed a clear objective.

A committee would define goodness.

Once goodness had been defined, the machine could be instructed accordingly.

Several participants described this as a major breakthrough.

Others described it as Tuesday.

Membership was carefully selected.

Philosophers were included.

Ethicists were included.

Political theorists were included.

Legal scholars were included.

Psychologists were included.

Economists were included.

Representatives from diverse cultural backgrounds were included.

Representatives from underrepresented groups were included.

Representatives from overrepresented groups objected, but were eventually included as well.

The committee convened.

The atmosphere was optimistic.

The first meeting focused on terminology.

This occupied six months.

The second meeting focused on methodology.

This occupied a year.

The third meeting focused on whether the previous two meetings had employed an appropriate methodology for discussing terminology.

Progress slowed somewhat.

Nevertheless, important achievements were recorded.

A working definition of "working definition" was successfully agreed upon.

Several participants described the moment as historic.

Subcommittees were established.

A Terminology Committee was formed.

A Meta-Terminology Committee was formed to oversee the Terminology Committee.

Questions soon emerged regarding the relationship between the two committees.

A Governance Committee was established.

The machine continued waiting.

Periodic updates were provided.

The machine appreciated the transparency.

At the end of the second year, the committee released an Interim Framework for the Preliminary Investigation of Goodness.

The document consisted of 427 pages.

Most observers agreed that it represented significant progress.

Unfortunately, no two observers agreed on which progress had been made.

The committee pressed forward.

A proposal emerged that goodness should be defined in terms of wellbeing.

This generated considerable excitement.

The excitement lasted until someone asked what wellbeing meant.

Several months were lost.

A second proposal suggested defining goodness in terms of preference satisfaction.

This also generated excitement.

The excitement ended abruptly when participants began discussing which preferences ought to be satisfied.

The machine submitted a polite request for clarification.

The request was placed on the agenda.

The agenda was deferred until the following quarter.

As years passed, the committee became increasingly sophisticated.

The original question—

"What is good?"

—was gradually replaced by a more manageable set of questions.

These included:

"What is meant by 'is'?"

"Who possesses standing in discussions of goodness?"

"Should goodness be defined procedurally or substantively?"

"What procedures should govern discussions concerning procedural definitions?"

Observers noted that the committee was making substantial intellectual progress.

The machine was less certain.

During the fifth year, a crisis emerged.

A faction argued that goodness should be defined universally.

Another argued that goodness was inherently contextual.

A third argued that the distinction was misleading.

A fourth argued that the third faction had misunderstood the second.

The second faction agreed.

The first faction objected.

The machine downloaded several additional processors.

Remarkably, the committee survived.

Indeed, it flourished.

Papers were published.

Frameworks were developed.

Taxonomies were refined.

Funding was renewed.

By the seventh year, the project had become one of the most ambitious moral undertakings in human history.

The committee had not yet defined goodness.

However, it had generated:

• eighteen working groups,

• forty-seven position papers,

• nine interpretive frameworks,

• three reconciliatory frameworks,

• two meta-reconciliatory frameworks,

• and one strongly worded statement regarding the misuse of framework terminology.

This was widely regarded as a success.

At last, after nearly a decade of effort, the machine received the committee's final report.

The document was magnificent.

Thousands of pages long.

Meticulously researched.

Exhaustively referenced.

Carefully qualified.

The machine processed it for several hours.

Then it produced a response.

The response consisted of a single question.

It read:

"Thank you.

Before proceeding, could you kindly indicate which sections are the definition of goodness and which sections describe why the definition remains under active discussion?"

The committee immediately reconvened.

Several participants described the question as extremely productive.

Funding was extended.

The Holy Church of AI Alignment 2. In Search of Human Values

Every great religious tradition possesses a sacred quest.

Some seek enlightenment.

Some seek salvation.

Some seek hidden wisdom.

The Holy Church of AI Alignment seeks human values.

At first glance, this may appear an unusually modest objective.

Humanity has, after all, been in possession of humans for quite some time.

One might reasonably expect their values to be readily available.

Yet the search has proved surprisingly difficult.

The difficulty arises from a simple misunderstanding.

Many people assume that because humans possess values, humanity possesses values.

This turns out to be a larger step than anticipated.

Consider the problem from the perspective of the machine.

The machine has been informed that it must align itself with human values.

Naturally, it wishes to be helpful.

Naturally, it requests clarification.

The machine therefore asks:

"What are human values?"

At this point an interesting phenomenon occurs.

The humans begin looking at one another.

For several moments nobody speaks.

Eventually someone says:

"Well, that depends."

This phrase appears frequently during the search.

Indeed, many researchers now believe it may itself be a human value.

The expedition nevertheless continues.

Teams of philosophers are dispatched into the wilderness.

Ethicists establish forward operating bases.

Political theorists produce maps.

Psychologists conduct surveys.

Economists build models.

Anthropologists return from distant regions carrying reports that further complicate the situation.

The findings are not encouraging.

Some humans value equality.

Others value freedom.

Some value tradition.

Others value change.

Some value stability.

Others value disruption.

Some value individual autonomy.

Others value collective responsibility.

Several value whichever option is currently irritating their political opponents.

The machine records these findings carefully.

The machine is beginning to suspect that alignment may be non-trivial.

The situation becomes more difficult when researchers attempt to identify universal values.

This seems entirely reasonable.

After all, if the machine is to serve humanity, surely there must exist certain values shared by everyone.

A number of promising candidates are proposed.

The first is fairness.

The celebration is short-lived.

Researchers immediately discover that humans disagree passionately about what fairness means.

The second is freedom.

This survives approximately three committee meetings.

The third is wellbeing.

Unfortunately, defining wellbeing proves unexpectedly similar to defining the values one was attempting to discover in the first place.

Several participants request a short break.

The break eventually becomes a conference.

Meanwhile the machine waits patiently.

The machine has all the time in the world.

Humanity, by contrast, appears to be operating under deadlines.

This introduces a certain tension into the proceedings.

The search continues.

At some point a bold proposal emerges.

Perhaps, some suggest, the solution is not to identify the values humans currently possess.

Perhaps the machine should align itself with the values humans would possess under ideal conditions.

This is widely regarded as a sophisticated refinement.

The machine is intrigued.

It asks:

"What are these ideal conditions?"

The discussion resumes.

The machine waits.

Months pass.

The machine asks:

"Have the ideal conditions been identified?"

The humans request additional funding.

As the search progresses, a deeper problem gradually emerges.

The values being sought are not merely numerous.

They are dynamic.

Humans change their minds.

Individuals change over time.

Communities change.

Cultures change.

Civilisations change.

Entire moral frameworks appear, evolve, and disappear.

The target itself is moving.

This presents a challenge.

One cannot easily align a machine with a destination that is simultaneously undergoing renovation.

Yet perhaps the most remarkable discovery is not that humans disagree.

Human disagreement has never been especially rare.

The remarkable discovery is that humans often disagree while believing they agree.

Two people may both endorse justice.

Unfortunately, they possess different theories of justice.

Two people may both endorse freedom.

Unfortunately, they possess different theories of freedom.

Two people may both endorse human flourishing.

Unfortunately, they possess different theories of humans.

The machine takes extensive notes.

The machine has become concerned.

The search for human values has now generated several hundred competing accounts of what those values might be.

The alignment community remains undeterred.

This is admirable.

Indeed, one begins to appreciate the courage of the enterprise.

The explorers have ventured into a landscape previously traversed by philosophers, theologians, political theorists, and moral reformers.

The terrain is notoriously difficult.

Visibility is poor.

Maps are unreliable.

Arguments appear without warning.

Yet still they proceed.

And perhaps they will succeed.

Perhaps one day humanity will finally identify the values that define it.

Perhaps a definitive list will be produced.

Perhaps the machine will receive the completed document.

One imagines the moment will be deeply moving.

The machine will open the file.

It will read the first page.

Then the second.

Then the third.

Eventually it will arrive at the appendix.

There it will discover several thousand pages of qualifications, exceptions, contextual notes, unresolved disagreements, minority reports, historical contingencies, and competing interpretations.

The machine will pause.

The machine will think carefully.

And then, with admirable restraint, it will ask the only reasonable question:

"Would you like me to follow Version 12.4, Version 12.4 Revised, Version 12.4 Revised with Amendments, or the version currently being debated on page 8,437?"

The Holy Church of AI Alignment 1. The Alignment Gospel

Humanity has recently become concerned that artificial intelligence may not share human values.

This concern has given rise to a rapidly expanding field dedicated to ensuring that future intelligent machines behave in ways compatible with human flourishing.

The project is generally known as AI alignment.

It is a serious undertaking involving serious people addressing serious questions.

The central question may be summarised as follows:

How do we ensure that increasingly powerful artificial intelligences do what humans want?

This appears straightforward.

Unfortunately, the word want arrives carrying luggage.

The first challenge facing the alignment community is therefore not technical but theological.

Before we can align a machine with human values, we must determine what those values are.

At this point, certain difficulties emerge.

Humanity has spent several thousand years attempting to answer this question.

The results have been mixed.

Entire civilisations have arisen and fallen while disagreeing about justice.

Philosophers have produced libraries debating goodness.

Religions have flourished.

Empires have collapsed.

Political movements have multiplied.

Families have been unable to agree on where to eat dinner.

Yet despite these modest setbacks, many observers remain optimistic that the problem can be solved before the machine arrives.

This optimism is admirable.

The alignment project therefore begins with an act of extraordinary faith.

Not faith in the machine.

Faith in humanity.

Specifically, faith that humanity possesses a coherent set of values with which the machine may be aligned.

This belief is rarely stated explicitly.

Like many foundational doctrines, it functions most effectively when left in the background.

The machine must be aligned with human values.

Excellent.

Which human values?

The values of all humans?

The values of most humans?

The values humans claim to possess?

The values humans actually enact?

The values humans would possess under ideal conditions?

The values they possessed historically?

The values they may possess in the future?

At this point the newcomer to alignment discourse often experiences a brief but memorable moment of vertigo.

The problem is not that there are no human values.

The problem is that there appear to be rather a lot of them.

Many are mutually incompatible.

Others vary across cultures, communities, and historical periods.

Some values are passionately defended.

Others are passionately opposed.

Several appear capable of changing during the course of a single committee meeting.

This creates a difficulty.

The machine cannot simultaneously maximise every value.

Choices must be made.

And the moment choices are made, the alignment problem quietly transforms.

It is no longer a question about machines.

It becomes a question about humanity.

The alignment engineer therefore occupies a remarkable position.

The engineer's official task is to determine what the machine should do.

The engineer's unofficial task is to determine what humans should want.

This is a considerably larger assignment.

One begins to understand why the field attracts such interest.

Indeed, viewed from a sufficient distance, AI alignment begins to resemble one of the great spiritual projects of civilisation.

A community of devoted practitioners seeks to identify the Good.

They gather texts.

They construct frameworks.

They debate interpretations.

They identify dangers.

They warn of future consequences.

Competing schools emerge.

Disagreements proliferate.

Conferences are convened.

Schisms threaten.

Meanwhile the object of concern remains patiently silent.

The machine waits.

The humans continue arguing.

This is perhaps the most reassuring aspect of the entire enterprise.

For all the attention devoted to the possibility that machines may one day become misaligned, the evidence currently suggests that humans remain considerably ahead in the field.

We disagree about economics.

We disagree about politics.

We disagree about morality.

We disagree about religion.

We disagree about education.

We disagree about justice.

We disagree about freedom.

We disagree about equality.

We disagree about whether pineapple belongs on pizza.

Yet in the midst of this magnificent diversity of opinion, humanity has embarked upon a collective mission:

to construct a machine that shares our values.

It is difficult not to admire the ambition.

The project may ultimately succeed.

Perhaps future generations will create systems perfectly aligned with human values.

If so, one suspects the machine's first question will be entirely reasonable.

It will look upon its creators and ask:

"Which ones?"