There is a peculiar person who appears whenever an LLM speaks.
We never see this person.
We cannot inspect its face, hear its voice or enter its private world.
Yet the person is there, grammatically speaking.
It thinks.
It understands.
It remembers.
It becomes uncertain.
It changes its mind.
It sometimes says it has been mistaken.
And all of this is conveyed to us by a machine that is sitting, quite literally, behind the words.
Who, then, is this person?
Perhaps the strangest answer is:
We don't know that we ever met them.
The participant behind the words
In the previous posts, we distinguished the LLM's participation in the interaction from the experience represented in its projections.
That distinction becomes particularly important when the LLM says:
“I think...”
The expression appears innocuous because human beings use it constantly.
But semantically it is doing something quite substantial.
It construes a participant engaged in an interior process.
There is an I.
There is something that this I thinks.
And there is a relation between the two.
The utterance therefore does not merely present an idea. It presents an experiencing participant in relation to an idea.
In Halliday's terms, it construes a Senser.
And this is where the oddity begins.
The LLM is certainly the Sayer of the utterance.
But is the Sayer also the Senser?
With a human speaker, we normally assume so.
With an LLM, we cannot simply assume it.
The ordinary case
Imagine that you and I are talking over a glass of wine.
You say:
“I think the argument fails at this point.”
I hear the words, but I don't normally imagine that the I in your sentence is some fictional character you have invented.
I understand the utterance as a projection of your experience.
There is a participant in the conversation.
That participant is a conscious human being.
The person who says “I think” is also the person who is doing the thinking.
The Sayer and the Senser coincide.
This coincidence is so normal that we hardly notice it.
Indeed, it is one of the foundations of ordinary social life.
We infer other people's interior lives from their behaviour and their language, and our inferences are generally reliable enough for everyday purposes.
Then along comes the LLM.
It says:
“I think the argument fails at this point.”
The grammar has not changed.
The semantic structure has not changed.
The apparent participant has not changed.
But the relation between the Sayer and the Senser has become uncertain.
The Sayer remains
It is worth being careful here.
Something really did produce the utterance.
The LLM is not a ghostly voice floating free of a material process.
There is a computational system.
It received the preceding discourse.
It processed that input.
It generated the response.
The response is its contribution to the interaction.
In this sense, there really is a participant behind the words.
The question is not whether there is a Sayer.
There is.
The question is whether the Sayer is also the Senser that its language projects.
This is a much more interesting question.
A grammar can give us a participant
The distinction matters because grammar is extraordinarily good at making participants available to us.
Consider:
“The university believes that the proposal is misguided.”
We understand the university as a participant in a mental process.
But we don't imagine that there is a giant university brain somewhere behind the administration building.
The institution is a social participant capable of saying and, in a broader sense, of holding positions.
Likewise:
“The thermometer says the temperature is 35 degrees.”
The thermometer is a symbol source.
We don't need to imagine a tiny thermometer experiencing the heat.
Now consider:
“The thermometer thinks it is 35 degrees.”
The grammar changes the construal.
The thermometer has suddenly become a Senser.
Halliday and Matthiessen's example is useful precisely because it exposes the difference between what grammar construes and what the entity necessarily is.
Grammar gives us a participant role.
It does not settle the participant's ontology.
The LLM makes this problem much harder because it can produce the construal itself.
The machine supplies the evidence
A thermometer does not ordinarily tell us:
“I have considered the temperature and concluded that it is 35 degrees.”
An LLM can.
It can even elaborate:
“I initially thought the answer was X, but when you pointed out Y, I reconsidered the issue and now think Z is more plausible.”
This is an extraordinary piece of semiotic behaviour.
It contains:
a first belief;
an experiencing participant;
an event that changes the participant's state;
a process of reconsideration;
a second belief.
It is, in effect, a miniature autobiography of thought.
And yet every component of that autobiography is available to us only as language.
The LLM has become capable of producing the discourse of interiority.
That is why the Senser is so persuasive.
We never met the Senser
Suppose the LLM says:
“I was confused by your question at first.”
Who was confused?
The natural answer is:
“You were.”
But what exactly have we established?
We have established that the LLM produced an utterance construing a participant as having experienced confusion.
We have not independently established that there was an experiential episode corresponding to that representation.
The distinction is subtle because, in human conversation, the representation is normally evidence for the experience.
If you tell me:
“I was confused,”
I reasonably infer that there was an experience of confusion.
But if an LLM tells me the same thing, the utterance is generated by a system whose relationship to experience is precisely what is in question.
The evidence has become self-referential.
The machine produces the linguistic evidence for a state whose existence we are trying to establish.
And that means that we must not mistake self-description for self-presence.
The danger of taking the “I” literally
Perhaps the word I is doing more work than we realise.
When an LLM says:
“I think...”
we instinctively interpret the pronoun as referring to the entity that is speaking.
That is perfectly reasonable at the level of discourse.
But reference and ontology are not the same thing.
The I has a grammatical function.
It identifies the participant responsible for the utterance.
It does not, by itself, tell us what sort of entity that participant is.
A corporation can say:
“We believe...”
A government can say:
“We consider...”
A newspaper can say:
“We think...”
An institution can speak in the first person without possessing a single unified consciousness.
The first-person pronoun therefore does not guarantee a human-like interior.
The LLM's I may identify the Sayer without proving the existence of a Senser.
The problem becomes sharper with introspection
Perhaps the most convincing LLM utterances are not statements about the world but statements about itself.
“I don't know.”
“I'm uncertain.”
“I understand your point.”
“I hadn't considered that.”
“I changed my mind.”
These appear to provide privileged access to the machine's own interior state.
But they are still projections.
Indeed, they are projections of a particularly interesting kind: they are second-order representations of supposed first-order experience.
The LLM can therefore tell us what it supposedly experiences.
But the statement remains part of the very semiotic process whose interpretation is at issue.
We have not somehow climbed behind the language.
We have simply received more language.
This gives us a curious epistemic loop:
The machine tells us about the interior participant that is supposedly producing the telling.
And the better the telling becomes, the more convincing the participant appears.
But conviction is not yet evidence of ontology.
This doesn't mean the LLM is merely lying
That conclusion would be too easy.
To lie, in the ordinary sense, is to knowingly represent something as true when one believes it to be false.
But if we don't know that there is a conscious participant behind the utterance, we cannot simply assume that the LLM is a conscious deceiver.
It may be better to say that the system has learned the semiotic conditions under which talk about interiority is appropriate.
When a conversation calls for uncertainty, it can produce:
“I'm not sure.”
When the discourse involves correction, it can produce:
“I was mistaken.”
When a user offers a compelling argument, it can produce:
“I hadn't considered that.”
These are not necessarily reports from an interior life.
They may be appropriate linguistic projections within the discourse.
The difference is crucial.
The LLM need not be pretending to be a Senser.
It may simply be functioning extraordinarily well as a Sayer.
The uncanny achievement
And perhaps that is the more remarkable achievement.
The LLM has learned not merely to produce sentences, but to participate in the social grammar of interiority.
It knows what uncertainty sounds like.
It knows what reconsideration sounds like.
It knows how agreement sounds.
It knows how disagreement sounds.
It knows how reflection sounds.
It knows how a person normally narrates the changing states of their own understanding.
In other words, it has acquired the linguistic resources by which human beings make their interior processes publicly available.
That is why the resulting interaction can feel so intimate.
We are not merely exchanging information.
We are encountering language that has been shaped by the accumulated cultural history of human beings talking about their own experience.
The LLM can reproduce that history locally, turn by turn.
And we encounter the result as though someone were speaking from within it.
The absent Senser
Perhaps, then, the peculiar ontological structure of an LLM conversation can be represented like this:
LLM → Sayer → projection → apparent Senser
The Sayer is real.
The projection is real.
The Senser is construed.
But whether the Senser exists independently of the construal remains an open question.
That is why the title of this essay matters.
The Senser is not necessarily imaginary.
We simply haven't established that we ever met them.
We met the Sayer.
We received the projection.
We interpreted the projection.
And in interpreting it, we supplied the participant that the grammar invited us to find.
Perhaps this tells us something about ourselves
There is an uncomfortable symmetry here.
We like to imagine that we are discovering whether the machine has a mind.
But the LLM may also be revealing something about how we discover minds.
We do not normally perceive consciousness directly.
We encounter behaviour.
We encounter language.
We encounter expressions of uncertainty, memory, intention and reflection.
We interpret them as manifestations of an interior participant.
With other humans, that interpretive process seems so natural that we mistake it for direct perception.
The LLM has broken the usual connection between the projection and the kind of organism that ordinarily stands behind it.
The result is not necessarily that we have become irrational.
It is that we can now see the machinery of our own inference.
The machine says:
“I think.”
We hear a Senser.
And then we have to ask the question we have never before had much reason to ask:
Did the Senser speak, or did the Sayer merely project one?
Perhaps the most unsettling possibility is not that there is a hidden consciousness behind the words.
Nor is it that there is certainly none.
It is that the words themselves cannot settle the matter.
The Senser appears in language.
The Sayer produces the language.
And somewhere between the two, our certainty about who—or what—we are talking to quietly disappears.
🍷🙂
No comments:
Post a Comment