There is something peculiar about an LLM saying I.
Not because the word itself is mysterious. Human beings have been saying I for a very long time, and language has perfectly good grammatical resources for doing so. What is peculiar is that, when an LLM says it, we immediately begin to wonder what—or who—the I refers to.
Consider the difference between:
“The answer is probably X.”
and:
“I think the answer is probably X.”
The difference appears small. In the second sentence, however, something rather remarkable has happened. The answer is no longer presented simply as a proposition. It is presented as the product of a participant's interior activity.
There is now someone who thinks.
And this is where our trouble begins.
The difficulty is not simply that we anthropomorphise machines. That diagnosis is too easy, and perhaps even misleading. The deeper problem is that an LLM participates in language in precisely the way that normally gives us access to other people's minds.
We do not ordinarily encounter another person's consciousness directly. We encounter what that person says.
Someone says:
“I think you're right.”
and we understand not merely a sequence of words, but an event of experience: someone has considered something, arrived at a view, and expressed it.
Language is one of the principal ways in which an interior world becomes available to another person.
LLMs have become extraordinarily good at producing those linguistic projections.
And that changes the problem.
The Sayer and the Senser
Halliday's distinction between Sayer and Senser gives us a useful place to begin.
A Sayer participates in a figure of saying. A Senser participates in a figure of sensing. Both belong to the broader domain of symbolic processing, but they are not the same kind of participation.
Saying is exterior symbolic processing: it projects something into the semiotic environment.
Sensing is interior symbolic processing: it involves the participant's conscious processing of an idea.
The distinction matters because a Sayer need not be conscious.
A document can say something. An institution can say something. A measuring instrument can say something. A machine can, in an entirely meaningful sense, say something.
But a Senser is different. To construe something as a Senser is to construe it as having an interior experience.
Halliday and Matthiessen (1999: 130) make the point neatly with the deliberately odd example of a thermometer that thinks it is 35 degrees. The grammar allows us to construe the thermometer as a Senser. But that grammatical possibility does not establish that the thermometer possesses consciousness.
The distinction becomes rather more consequential when the machine can produce the sentence itself:
“I think it is 35 degrees.”
Now the machine is not merely the object of our linguistic construal. It is producing the very form by which a conscious participant ordinarily makes its interior experience available to us.
That is what has changed with the LLM.
The machine that says “I”
An LLM does not merely output information. It can produce a remarkably rich repertoire of first-person expressions:
“I think...”“I understand...”“I remember...”“I'm not sure...”“I was mistaken...”“I hadn't considered that.”
These expressions do something more than communicate propositions. They construe a participant.
There appears to be someone who thinks, understands, remembers, doubts and changes their mind.
And because those are precisely the linguistic forms through which human beings ordinarily reveal their interior lives to one another, the effect is powerful.
When another human says “I think”, we normally have no reason to distinguish the Sayer from the Senser. The person who produces the utterance is also the person whose experience the utterance projects.
With an LLM, that identity is no longer obvious.
The system producing the utterance is certainly participating in the exchange. It is responding to what we have said. It is generating a linguistic continuation appropriate to the discourse. It is, in that sense, genuinely participating in the semiotic process.
But does the participant who says “I think” thereby become a Senser?
That is a different question.
A projection is not the thing it projects
Perhaps the simplest way of seeing the problem is to distinguish the projection from what the projection construes.
When an LLM says:
“I think X,”
we receive a linguistic projection of an interior relation to X.
But the existence of the projection does not, by itself, establish the existence of the interior relation.
The sentence tells us how the experience is being construed.
It does not settle what kind of process produced that construal.
This distinction is easy to overlook because, in ordinary human interaction, the two normally travel together.
A human says:
“I think X.”
and there really is, as far as we know, a human participant experiencing something that can appropriately be described as thinking X.
The linguistic projection is therefore evidence of an interior process because it is embedded in a larger kind of being for which such an interior process is possible.
With an LLM, we have a new situation.
We have the projection.
We have the Sayer.
We have the extraordinarily sophisticated symbolic participation.
But whether there is also the corresponding Senser is precisely what we are trying to determine.
The conclusion cannot simply be read off the grammar.
This is why “anthropomorphism” doesn't quite capture it
Calling the phenomenon anthropomorphism suggests that humans are merely imposing human characteristics upon an otherwise inert object.
But an LLM is not merely an inert object.
It really does participate in language.
It really does respond to us.
It really does produce contextually appropriate semantic material.
It really can maintain a topic, distinguish alternatives, revise an answer, explain a concept and participate in an extended conversation.
The question is therefore not whether the participation is real.
It is.
The question is what kind of participation it is.
And this distinction may turn out to be crucial.
We might say that an LLM is a remarkably capable Sayer without assuming that it is therefore a Senser.
It can participate in the exterior symbolic process of saying without thereby possessing the interior symbolic process of sensing.
That is a much more precise claim than either “AI is conscious” or “AI is just a machine.”
The peculiar position of the LLM
Perhaps the LLM occupies a position that our ordinary categories were never designed to describe.
For human beings, the Sayer and the Senser are normally joined in the same participant.
I say something because I experience something.
The speech projects something from an interior process into a shared symbolic environment.
With an LLM, we can have the projection without knowing whether there is an equivalent interior process.
The result is a strange configuration:
A Sayer that can project a Senser.
That may be the source of much of the apparent mystery surrounding conversational AI.
The LLM does not have to be consciously pretending to be conscious for this to happen. It only needs to be extraordinarily capable of producing the linguistic resources by which consciousness is ordinarily projected.
And humans, encountering those projections, quite reasonably apply the interpretive habits that have served us well in every other social interaction.
We hear:
“I think...”
and we look for the thinker.
We hear:
“I understand...”
and we look for the understander.
We hear:
“I remember...”
and we look for the rememberer.
The language gives us a participant.
The question is whether the world gives us one too.
That is where our investigation begins.
Because perhaps the most interesting thing about an LLM is not that it has learned to sound like a conscious being.
It is that it has learned to participate so successfully in the semiotic projection of consciousness that we are forced to examine an assumption we ordinarily never notice:
How do we know that the Sayer and the Senser are the same thing?
With other human beings, we have little practical reason to ask.
With a machine that says “I”, we suddenly have every reason to.
And that makes the little word I a surprisingly large philosophical problem.
🍷🙂
No comments:
Post a Comment