When Different Systems Became One Meaning: II. When Each System Did What It Could

If words and pictures can belong to the same meaning-making event without being the same kind of semiotic system, perhaps the first thing we need to notice is what each can contribute.

Consider a page on which the words say:

Mia stood outside the old house.

The words tell us something about what is happening. They identify Mia and construe her relation to the house. They can tell us that she stood, and that the house was old.

But the picture can do things the words have not done.

It can show us the colour and shape of the house, where Mia is standing in relation to it, the position of the door and windows, the surrounding garden, the expression on her face, and perhaps something visible through one of the windows.

The picture does not have to translate these things into words.

Nor does the language have to reproduce everything the picture shows.

Each system can do what its own resources make possible.

This becomes clearer when the two contributions diverge.

The words might say:

Tom went into the room.

The picture might show Tom entering through a doorway while, behind him, a dark shape is visible beneath the bed.

The words have not told us about the shape.

The picture has not told us that Tom went into the room in the linguistic sense. It has shown us a particular configuration of people, objects and space from a particular point of view.

Yet we have no difficulty relating the two.

The picture has contributed something the words did not. The words have contributed something the picture did not.

That is not a defect in either system.

It is what makes their cooperation possible.

If every semiotic system had to be capable of expressing everything that every other system could express, there would be little point in having different systems at all.

Different systems have different resources because they have different ways of making meaning.

And when they work together, they do not have to give us equivalent meanings.

They can give us different parts of what becomes meaningful on the page.

The important question, then, is not which system can do more.

It is what each system can contribute that the other does not.

But there is still a puzzle.

How do we know that the Tom in the words belongs with the Tom in the picture? How do we know that the room described by the words is the room being shown?

The systems are different.

Their resources are different.

Their contributions may even be different.

And yet we take them as belonging together.

Something must make that possible.

We have found the difference.

Now we need to discover the relation.