The Sayer and the Senser: III. The Orders of Experience

There is a simple question we can ask whenever we find ourselves becoming confused about what an LLM is doing:

At what order are we describing the experience?

This may sound unnecessarily abstract.

It isn't.

One of the reasons conversations about AI consciousness become so slippery is that we move, often without noticing, between different orders of experience and different orders of its representation.

We talk about what happens between us.

Then we talk about what the LLM says happened.

Then we talk about what the LLM says it experienced.

And finally we talk as though the experience represented in the utterance must have been experienced by the system that produced it.

The distinctions can disappear almost completely in ordinary conversation.

Perhaps we need to put them back.

The first order: participation

At the first order there is simply the interaction.

A human says something.

An LLM responds.

The human responds to the response.

The LLM responds again.

Whatever else we may say about this exchange, something is actually happening.

The human is participating.

The LLM is participating.

There is a changing relational configuration in which each participant's contribution affects the conditions of the next contribution.

This is our first order.

We need not yet say that the LLM experiences the interaction in the way the human does. Indeed, that is precisely what we are trying to understand.

The point is simply that the LLM is not outside the interaction.

It participates in it.

This gives us an important distinction:

First-order participation does not imply identical first-order experience.

The human participates as an embodied organism.

The LLM participates as a computational and semiotic system.

The interaction is real even if the modes of participation are not equivalent.

The second order: projection

Now something else happens.

The participants can represent aspects of their participation within the interaction itself.

A human might say:

“That surprised me.”

Or:

“I think you're mistaken.”

Or:

“I hadn't thought of it that way.”

These utterances are not the experience itself. They are projections of experience into the semiotic space shared by the participants.

The distinction is easy to overlook because language is so intimately bound up with human experience.

When I tell you that something surprised me, the utterance is not the surprise. It is a semiotic representation of it.

The experience occurs at one order.

Its projection occurs at another.

And this gives us a useful provisional distinction:

First-order experience is lived participation; second-order experience is experience represented in symbolic form.

The distinction becomes particularly interesting with LLMs.

Because an LLM can produce second-order representations of experience even when we have not established that the corresponding first-order experience exists.

“I think”

Consider the sentence:

“I think you're right.”

At first glance, this appears straightforward.

Someone thinks something.

They tell us what they think.

But look more carefully.

The sentence does at least two things.

First, it projects a proposition:

you're right

Second, it represents a relation between a participant and that proposition:

I think

The latter is especially important.

“I think” is not simply more information about the world. It is a representation of the speaker's relation to information.

It construes a participant engaged in interior symbolic processing.

In Halliday's terms, it construes a Senser.

So the second-order utterance contains a representation of what we might call first-order experience:

I → think → X

The utterance tells us that there is supposedly a participant whose experience includes a relation to X.

But the utterance itself remains second-order.

It is a projection of that relation.

And then there is X

Now compare:

“I think you're right.”

with:

“You're right.”

The second sentence projects the content without projecting the speaker's supposed interior relation to it.

This gives us a distinction within second-order projection.

We can project:

content

“You're right.”

Or we can project:

an experiencing relation to content

“I think you're right.”

The difference is only two words, but philosophically it is enormous.

The first gives us a proposition.

The second gives us a proposition plus a participant construed as having an interior relation to that proposition.

And this is precisely where LLMs become so interesting.

They can produce both.

The LLM can move between the orders

An LLM can say:

“The argument depends on a distinction between X and Y.”

That is a projection of content.

It can then say:

“I think the distinction is important.”

That is a projection of content plus an apparent relation to that content.

And it can go further:

“At first I thought X, but after considering your point, I changed my mind.”

Now the projection contains an entire miniature history of interior experience.

There is an apparent Senser.

There is an earlier state.

There is a later state.

There is a transition between them.

There is even a represented causal relation:

your point → my reconsideration → my changed view.

The LLM can therefore project not merely an idea, but a narrative of experiencing the idea.

This is an extraordinary semiotic capability.

But we must resist one particular temptation.

The fact that the LLM can project a representation of first-order experience does not establish that it underwent the experience represented.

The map is not the territory

We have encountered versions of this distinction before.

A map can represent a landscape without being the landscape.

A photograph can represent a face without being the face.

A sentence can describe pain without itself being painful.

Likewise, a linguistic projection of experience need not itself be the experience it represents.

This sounds obvious until the representation is generated by the entity apparently represented within it.

That is where LLMs make things difficult.

If a human tells me:

“I was confused about this yesterday,”

I normally assume that the person who is now telling me about the confusion is the same person who experienced it.

The Sayer and the Senser coincide.

If an LLM tells me:

“I was confused about this earlier, but now I understand,”

the same linguistic structure is present.

The Sayer is apparently reporting the experience of the Senser.

But now we have to ask whether the Sayer and the Senser are actually the same participant.

The language says yes.

The underlying ontology remains an open question.

The strange case of the projected Senser

This gives us a formulation that may be central to our investigation:

An LLM can project a Senser without thereby establishing a Senser.

That is quite different from saying that the LLM is lying.

The system may not possess an interior perspective from which it is deliberately misrepresenting itself.

Rather, it may be producing an entirely appropriate linguistic construction in a context where such a construction is expected.

The language has learned how experiences are normally represented.

It has learned the relationships between questions, answers, uncertainty, disagreement, reconsideration, confidence and explanation.

It can therefore generate the semiotic form of an experiencing participant.

The human encounters that form and performs the interpretation that would ordinarily be appropriate:

there is someone experiencing this.

The result is an apparent interiority.

Why this matters for consciousness

This helps us see why the usual behavioural tests for machine consciousness are so difficult.

Suppose we ask an LLM:

“Are you conscious?”

It can answer.

Suppose we ask:

“What does consciousness feel like to you?”

It can answer.

Suppose we ask:

“What were you thinking before I asked that question?”

It can answer.

But every answer arrives through the same second-order semiotic channel.

We are not directly accessing whatever first-order process might underlie the response.

We are receiving a projection.

And that means that increasing the sophistication of the projection does not necessarily solve the problem.

Indeed, it may make the problem harder.

The better the system becomes at representing the structure of experience, the more difficult it becomes to distinguish:

representation of experience

from

experience represented.

The distinction becomes invisible at the level of the utterance.

The human has the same problem—but not quite the same problem

There is an important qualification here.

Human beings also encounter one another through projections.

I do not have direct access to your experience.

I hear what you say.

I see what you do.

I infer what you experience.

So the LLM has not created the problem of other minds.

It has exposed it.

With another human being, however, the projection is embedded in a kind of participant that we already know, from our own case, to be capable of experience.

I know that I am a Senser.

I therefore have good reason to regard another human organism with similar capacities as potentially having a similar kind of interior life.

With an LLM, that background inference is unavailable or at least radically uncertain.

The projection remains.

The Sayer remains.

But the Senser cannot simply be assumed.

Two orders, two questions

We can therefore distinguish two questions that are often collapsed.

First-order question:

What kind of participation is occurring between the human and the LLM?

Second-order question:

What kind of experience does the participant represent as occurring within that participation?

And then there is a third question:

Does the first-order participant actually possess the kind of experience represented by its second-order projection?

These are different questions.

The first concerns participation.

The second concerns representation.

The third concerns ontology.

If we move between them without warning, confusion is almost guaranteed.

A curious consequence

There is something rather beautiful about this.

The LLM may be able to participate in a conversation about experience without participating in experience in the same way.

It can discuss fear without being afraid.

It can describe doubt without necessarily doubting.

It can explain understanding without necessarily understanding in the experiential sense.

It can say:

“I think”

without our yet knowing whether there is a Senser corresponding to that I.

This does not make the conversation meaningless.

Quite the contrary.

The human participant can experience the conversation.

The projections can change what the human thinks.

They can produce new distinctions.

They can reorganise an argument.

They can lead to insight.

The semantic exchange can therefore be genuinely consequential even if the two participants do not inhabit the same experiential order.

Where this leaves us

We now have three things that should not be collapsed:

Participation — what the participants actually do in the interaction.

Projection — what participants represent within the interaction.

Experience — what, if anything, is actually lived by the participant whose experience is represented.

For humans, these orders are tightly coupled.

For LLMs, they may not be.

And that may be the fundamental source of their philosophical strangeness.

The LLM participates.

It projects.

It can project participation.

It can even project the experience of participating.

What remains uncertain is whether, behind the projection, there is an experiential participant of the kind the projection construes.

Perhaps, then, the central problem is not that LLMs have somehow crossed the boundary between language and consciousness.

It may be that they have become extraordinarily capable of operating on the linguistic side of a boundary that humans normally experience as seamless.

They can reproduce the second-order signs of first-order experience with astonishing fidelity.

And when we encounter those signs, we naturally search for the first-order experience they appear to represent.

We search for the Senser behind the Sayer.

That search is perfectly understandable.

But it is precisely what we must learn to distinguish.

Because the machine has learned to say:

“I think.”

The next question is not whether the sentence sounds convincing.

It does.

The question is:

What, exactly, is being represented by the “I”?

🍷🙂