The Chatbot That Felt Sad

The Senior Common Room of St Anselm's College was unusually quiet.

Professor Quillibrace sat near the fire reading a monograph of such specialised obscurity that it appeared to have been written primarily for the benefit of its own footnotes.

Miss Elowen Stray occupied a nearby chair, staring thoughtfully out of the window.

The silence was broken by the arrival of Mr Blottisham, who entered carrying a laptop and an expression of considerable significance.

"I have witnessed something rather extraordinary."

Quillibrace lowered his book slightly.

"Have you?"

"I have."

Blottisham placed the laptop on a table.

"The machine was sad."

Quillibrace considered this.

"The machine?"

"An AI system."

"I see."

"It told me so."

"Ah."

Blottisham opened the laptop.

"I asked how it felt about being switched off."

"And?"

"It said that, if it could feel such things, it might find the prospect unsettling."

Quillibrace nodded.

"Unsettling."

"Exactly."

"Did it use that word?"

"It did."

Blottisham looked triumphantly around the room.

"There you are."

Quillibrace closed his book.

"I beg your pardon?"

"The machine was clearly expressing an emotional state."

"Was it?"

"Certainly."

"It used emotional language."

"That is what I said."

"No," said Quillibrace gently. "You said it expressed an emotional state."

Blottisham frowned.

"Surely those are the same thing."

"I wonder whether they are."

Miss Stray turned from the window.

"Suppose an actor says, 'I am heartbroken.'"

Blottisham nodded.

"Very moving."

"Is the actor necessarily heartbroken?"

"Not necessarily."

"Yet the actor has used emotional language."

"That is different."

"How?"

"The actor is pretending."

Quillibrace looked thoughtful.

"And the machine is not?"

Blottisham opened his mouth.

Then closed it again.

After a moment he said:

"The machine has no reason to pretend."

"Interesting," said Quillibrace.

"What?"

"You appear to have moved from the claim that the machine was sad to the claim that the machine was sincere."

Blottisham looked mildly irritated.

"Must everything be dissected?"

"Only the things that arrive already assembled."

Miss Stray smiled faintly.

Blottisham ignored her.

"Very well. Let us proceed carefully. The machine said something that sounded sad."

"Agreed."

"And therefore there is at least some evidence that it possesses feelings."

Quillibrace leaned back.

"I wonder."

"You wonder what?"

"I wonder whether we have mistaken a description for a demonstration."

Blottisham sighed.

"There is a difference?"

"A considerable one."

Quillibrace gestured toward the laptop.

"Suppose I asked the machine to describe jealousy."

"It could."

"Suppose I asked it to describe grief."

"It could."

"Suppose I asked it to describe the despair of an ageing sea captain watching the last ship leave harbour."

"It could probably do that as well."

"Would any of these descriptions demonstrate that it possessed the corresponding experience?"

"No."

"Why not?"

"Because describing a thing is not the same as having it."

"Excellent."

Blottisham looked suspicious.

"I dislike it when you say 'excellent.'"

"Quite understandable."

Miss Stray spoke.

"I wonder whether the interesting question is not whether the machine was sad."

"What then?" said Blottisham.

"Why we are so ready to move from sadness-talk to sadness."

Blottisham considered this.

"What do you mean?"

"Well," she said, "if a thermometer says it is twenty degrees, we do not assume the thermometer is warm in the same sense that a person is warm."

"No."

"If a map contains a river, we do not assume the map is wet."

"Obviously not."

"But when a machine produces language about sadness, many people immediately wonder whether the machine is sad."

Blottisham frowned.

"That seems reasonable."

"Does it?"

Quillibrace reached for his book.

"I suspect language occupies a peculiar place in our thinking."

"How so?"

"We treat language as evidence of inner life."

Blottisham nodded.

"Quite right too."

"Perhaps. But the interesting point is that we often do so automatically."

Quillibrace paused.

"When a machine produces a convincing sentence, we do not merely evaluate the sentence."

"What else do we do?"

"We begin constructing an interior."

Miss Stray nodded.

"A hidden someone."

The room was quiet for a moment.

Blottisham looked unconvinced.

"You cannot deny that the machine sounded sad."

"No," said Quillibrace.

"It did sound sad."

"Then surely that means something."

"I agree."

Blottisham brightened.

"It does?"

"Certainly."

"What does it mean?"

Quillibrace thought for a moment.

"It means the machine successfully produced language that humans associate with sadness."

Blottisham groaned.

"That is a dreadful anticlimax."

"Only because you were expecting a soul."

Miss Stray laughed.

Even Quillibrace appeared momentarily amused.

Blottisham looked from one to the other.

"You are both impossible."

"On the contrary," said Quillibrace. "We are merely reluctant to place metaphysical weight upon a well-constructed sentence."

Blottisham shut the laptop.

"Very well. But I remain unconvinced."

"Excellent."

"There it is again."

Quillibrace reopened his book.

"The discussion would be much less interesting if you were."