When Different Systems Became One Meaning: III. When One System Took Up the Other

We have two different semiotic systems.

The words say:

Tom went into the room.

The picture shows Tom entering a room. It also shows something the words have not mentioned: a dark shape beneath the bed.

The two systems have contributed differently.

But we do not leave their contributions separate.

When we read the words, we take the Tom in them to be the person shown in the picture. We take the room to be the room represented there. The picture, in turn, becomes relevant to how we understand what the words are saying.

One system has, in a sense, taken up the contribution of the other.

But this does not mean that the picture has interpreted the words, or that the words have interpreted the picture.

The picture remains a picture.

The words remain words.

What changes is the relation we construe between them.

The meaning-maker can relate the contributions of the two systems because they have been presented as belonging together.

This is why multimodal meaning is not simply a matter of adding one meaning to another.

Suppose the picture shows Tom entering the room and the words say:

Tom had never been here before.

The picture cannot, by itself, show us that this is the first time Tom has ever been there.

The words cannot, by themselves, show us exactly what the room looks like.

But when we take the two together, each changes what the other makes relevant.

The room we see becomes the room Tom has never visited before.

Tom's first visit becomes the visit we can see.

The contribution of one system becomes part of the meaning of the other.

This does not require the two systems to share a grammar.

It does not require one system to be translated into the other.

And it does not require some third semiotic system hidden between them.

What it requires is that their contributions can be related.

The reader does not first construct two completely separate meanings and then add them together.

The meanings are construed in relation.

That gives us a more precise way of thinking about what it means for different semiotic systems to work together.

They do not have to become the same.

They have to become relevant to one another.

But that raises a further question.

What makes it possible for the reader to treat these different contributions as belonging together in the first place?

The answer cannot lie entirely inside either semiotic system.

Perhaps we need to look beyond the systems themselves—to the situation in which they are functioning.

And that is where context enters the picture.

When Different Systems Became One Meaning: II. When Each System Did What It Could

If words and pictures can belong to the same meaning-making event without being the same kind of semiotic system, perhaps the first thing we need to notice is what each can contribute.

Consider a page on which the words say:

Mia stood outside the old house.

The words tell us something about what is happening. They identify Mia and construe her relation to the house. They can tell us that she stood, and that the house was old.

But the picture can do things the words have not done.

It can show us the colour and shape of the house, where Mia is standing in relation to it, the position of the door and windows, the surrounding garden, the expression on her face, and perhaps something visible through one of the windows.

The picture does not have to translate these things into words.

Nor does the language have to reproduce everything the picture shows.

Each system can do what its own resources make possible.

This becomes clearer when the two contributions diverge.

The words might say:

Tom went into the room.

The picture might show Tom entering through a doorway while, behind him, a dark shape is visible beneath the bed.

The words have not told us about the shape.

The picture has not told us that Tom went into the room in the linguistic sense. It has shown us a particular configuration of people, objects and space from a particular point of view.

Yet we have no difficulty relating the two.

The picture has contributed something the words did not. The words have contributed something the picture did not.

That is not a defect in either system.

It is what makes their cooperation possible.

If every semiotic system had to be capable of expressing everything that every other system could express, there would be little point in having different systems at all.

Different systems have different resources because they have different ways of making meaning.

And when they work together, they do not have to give us equivalent meanings.

They can give us different parts of what becomes meaningful on the page.

The important question, then, is not which system can do more.

It is what each system can contribute that the other does not.

But there is still a puzzle.

How do we know that the Tom in the words belongs with the Tom in the picture? How do we know that the room described by the words is the room being shown?

The systems are different.

Their resources are different.

Their contributions may even be different.

And yet we take them as belonging together.

Something must make that possible.

We have found the difference.

Now we need to discover the relation.

When Different Systems Became One Meaning: I. Different Systems, One Situation

Open a children's picture book and you may find something quite ordinary: words on one page, a picture beside them.

The words might say:

Mia stood outside the old house.

The picture might show Mia standing beside a weathered house, with its door, windows, garden and surrounding landscape all visible.

The words and the picture are not doing the same thing.

Language has its own resources for construing experience. It can identify participants, construe processes and circumstances, locate events in time, and relate one experience to another.

The visual system has different resources. It can show shape, colour, spatial arrangement, relative position, appearance and what can be seen from a particular point of view.

Nor do we need to suppose that the picture is secretly a kind of language. It does not need a grammar like the grammar of the words in order to participate in the meaning of the page.

And yet we do not experience the words and the picture as two unrelated events.

We understand Mia in the words as the girl we can see. We understand the old house as the house represented in the picture. The two systems are different, but what they are contributing is related.

Something has brought them together.

The obvious answer is that they are about the same situation. But that only moves the question back one step.

How can two different semiotic systems construe different aspects of a situation and still become part of the same meaning-making event?

The answer cannot simply be that they say the same thing. Often they don't.

And it cannot be that one system has been translated into the other. The picture remains a picture; the words remain words.

The interesting thing is precisely that difference does not prevent coordination.

Two semiotic systems can remain different while participating in the same act of meaning.

So perhaps the question is not how one system can become like another.

It is how different systems can become relevant to one another.

And that is where we should begin.