The History of an LLM — I. When the Past Becomes the System

An LLM has a history.

But where is that history?

It is tempting to look for it in memory: conversations remembered, documents stored, events recorded. But there is another possibility.

Perhaps a system can have a history without containing a record of its history.

Training provides the first clue.

An LLM is exposed to an enormous history of linguistic activity. That history does not remain inside the model as a collection of remembered texts. It changes the organisation of the model itself.

The past becomes parameters.

What happened during training alters the probabilities available to the system later. What the model can produce has been shaped by what it encountered before.

The history has become organisation.

This is different from remembering an event.

A model does not need to retrieve a particular sentence from its training in order for that training to matter. The past can persist as a changed disposition: a tendency to respond one way rather than another, to make some continuations more available than others.

History has become possibility.

And this gives us a useful way to think about learning more generally.

A system learns when something that happened to it changes the organisation through which something else can happen to it later.

The important thing is therefore not whether the past is stored as an explicit representation.

It is whether the past has changed what the system can become next.

An LLM makes this unusually visible because its history is partly written into its parameters. But the principle is broader.

A scar carries the history of an injury.

A nervous system carries histories of interaction.

A genome carries histories of selection.

A landscape carries histories of geological and biological activity.

In each case, the past survives by changing the present organisation.

So perhaps the deepest question about an LLM's history is not:

What does the model remember?

It is:

What has happened to the model that has changed what it can now do?

Once we ask that question, history stops looking like something stored behind the present.

It becomes part of the organisation of the present itself.

And therefore part of the organisation of the future.