When Mathematics Became a Proposition: IV. When One Statement Became Many

We began with one equation:

x + 2 = 5

To solve it, we produced another:

x = 5 − 2

And then another:

x = 3

There are now three clauses.

But they do not constitute three independent statements.

They form one continuing mathematical act.

The first clause gives us the problem.

The second reorganises the relation.

The third gives us the solution.

Each new clause takes up what has already been established and makes a further claim about it.

This is why solving an equation is more than a sequence of calculations.

It is a clause complex.

The successive clauses are successive elaborations of the first. The second is acceptable because it follows validly from the first. The third is acceptable because it follows validly from the second.

The process therefore has a peculiar kind of continuity.

We are not changing the proposition arbitrarily.

We are continually restating the same mathematical relation in a new form.

The equation changes, but what it construes as mathematically equivalent is preserved.

And this is where the interpersonal dimension begins to become difficult to ignore.

Each clause is a proposition.

A proposition is something that can be stated.

But each new proposition also makes itself answerable to the one before it.

x + 2 = 5

x = 5 − 2

x = 3

The second statement implicitly says:

This follows from what we have just established.

The third says the same.

There is no need for another speaker to reply.

No one has taken a conversational turn.

The mathematician is still speaking in one continuous turn.

Yet the propositions are not simply sitting beside one another. Each one is constrained by the validity of the preceding one.

This is different from ordinary dialogue.

In conversation, one participant can make a statement and another can accept it, reject it, question it or respond to it in some other way.

In solving an equation, the successive statements are produced by the same participant.

There is no exchange of turns.

But there is still something that can be tested.

A transformation can be valid or invalid.

A step can follow from what has already been established—or fail to follow.

The mathematical discourse therefore carries its own requirement of validity from one clause to the next.

We do not need to call this an internal dialogue.

The more immediate fact is simpler.

One statement has become many, while remaining one continuing mathematical argument.

And that gives us a new question.

What exactly does it mean for one mathematical statement to be valid in relation to another?

Because once validity enters the picture, we are no longer dealing only with what a mathematical proposition means.

We are dealing with the status of that proposition in relation to other participants who may accept it, challenge it, or demand that it be demonstrated.

The equation has begun to talk back.

Not by taking another turn.

But by making every further turn answerable to what has already been said.

When Mathematics Became a Proposition: III. When the Unknown Became the Identified

Suppose we begin with:

x + 2 = 5

We know there is an unknown quantity in the equation.

But x is not yet the whole of the Identified.

The identifying relation is between:

x + 2 and 5

The expression x + 2 is what is being identified with 5.

To find the value of x, we transform the equation:

x + 2 = 5

x = 5 − 2

x = 3

At first glance, this looks like a sequence of calculations.

But something more interesting is happening.

Each step produces another clause.

And the clauses progressively reorganise the relation being construed.

In the first equation, the quantity containing the unknown is:

x + 2

In the second, the unknown has been isolated:

x

And in the final equation:

x = 3

x has become the Identified.

3 has become its Identifier—the value that identifies what x is.

The same movement can be seen across the metafunctions.

At the beginning, x + 2 is the Theme of the clause and the Identified/Token of the identifying relation.

By the end:

x = 3

we have:

Theme ^ Rheme

x ^ = 3

and:

Identified/Token ^ Identifier/Value

x ^ 3

The unknown has become the thing being talked about, the thing being identified, and the Subject of the proposition.

Its value has become what is said about it, what identifies it, and the Complement of the clause.

The solution is therefore not simply a number.

It is a new clausal configuration in which the unknown has been brought into the position where its value can be stated.

This also explains why solving an equation is not simply a matter of manipulating symbols until a number appears.

Each transformation has to preserve the mathematical relation being construed.

The new clause must remain valid in relation to the preceding one.

That is why:

x + 2 = 5

can become:

x = 5 − 2

and then:

x = 3

but not just any arbitrary expression.

The transformations are constrained.

Something has to remain true as the equation changes.

And now we can see something that was hidden in the apparently simple notation.

Solving an equation is a process in which one clause is progressively elaborated into another until the unknown can be identified.

The final equation does more than give us an answer.

It construes the answer as the value of the unknown.

The unknown has become the Identified.

And that raises a new question.

If every step in solving is another clause, what kind of relation holds between all those clauses?

They are not separate statements.

They form one continuing mathematical argument.

And that is where the interpersonal dimension begins to come into view.

When Mathematics Became a Proposition: II. When an Equation Became a Clause

Take a very simple equation:

x = 3

We usually look at it mathematically.

x is an unknown quantity.

3 is its value.

But if mathematical discourse is a register of language, we can ask a different question:

What kind of clause is this?

The first thing we notice is that the equation has a textual structure.

It is organised around a starting point and what is said about that starting point:

Theme ^ Rheme

x ^ = 3

The equation begins with x. The value 3 is what is said about it.

The same equation also has an ideational structure.

It construes an identifying relational process:

Identified/Token ^ Identifier/Value

x ^ 3

The unknown is identified with its value.

But the equation has an interpersonal structure as well.

It functions as a proposition, and the proposition has the structure of a statement:

Subject ^ Finite ^ Predicator ^ Complement

The mathematical relation is therefore not simply a relation between two quantities.

It is being construed as something that can be stated.

This is easy to overlook because the mathematical notation is so compact.

If we expanded the equation into words, we could say:

x equals 3.

Now the linguistic organisation becomes more visible.

x functions as Subject.

-s provides the Finite.

equal functions as Predicator.

3 functions as Complement.

The equation compresses this grammatical structure into a symbolic form.

But the grammar has not disappeared.

It has simply become extraordinarily condensed.

And this gives us an interesting convergence.

The same two elements participate simultaneously in different metafunctional structures:

	x	3
Textual	Theme	Rheme
Ideational	Identified / Token	Identifier / Value
Interpersonal	Subject	Complement

The equation is therefore not ideational meaning with a little interpersonal meaning added afterwards.

Its textual, ideational and interpersonal organisations are different perspectives on the same clause.

That matters because it changes how we should think about mathematical notation.

An equation does not merely construe a relation between quantities.

It enacts that relation as a proposition.

And a proposition is something that can be given as information.

It can be stated.

It can be accepted.

It can be questioned.

It can be challenged.

For the moment, the important point is simply that we have found the clause hiding inside the equation.

But something peculiar happens when we try to solve it.

The equation we began with is not yet the equation we want.

We have to transform it.

And every transformation produces another clause.

That is where mathematics begins to become even more interesting.

When Mathematics Became a Proposition: I. When Mathematics Became Language

Mathematics looks like a language.

It has symbols, conventions, specialised vocabulary and rules for combining its expressions. It can state propositions, construct arguments and produce texts that can be read and understood by people who know how its resources work.

And yet mathematics is not a natural language.

English, Mandarin or Arabic are natural language systems that have evolved as part of human social life. Mathematics is different. Its symbolic resources have been deliberately designed and progressively developed for particular kinds of meaning-making.

So what is mathematics?

One answer is that mathematics is a field of the culture as semiotic system.

It is a specialised domain of human meaning-making.

And when that field is construed through language, something rather interesting happens.

We do not need to suppose that mathematics is a separate language.

Mathematical discourse is a register of language.

More precisely, it is a register of language, not a register of a language.

That distinction matters.

Mathematics can be expressed through English, French, Japanese or any other natural language. The mathematical register is not a specialised variety of English, for example. It is a specialised configuration of the resources of language associated with the field of mathematics.

The symbols of mathematics are part of that specialised meaning-making practice.

So are the linguistic expressions that accompany them:

equals, greater than, therefore, let x be, if, then, suppose, hence.

Mathematics may look as though it has escaped from language into a world of pure symbols.

But the symbols do not stand outside language.

They participate in a larger registerial organisation through which mathematical meanings are construed.

This becomes particularly clear when we look at an equation.

Take:

x = 3

It is tempting to regard this as simply a mathematical relation: an unknown quantity has the value three.

But it is also a clause.

It has a textual organisation.

It has an ideational organisation.

And it has an interpersonal organisation.

In other words, the equation is not merely representing something.

It is saying something.

That is where the peculiar status of mathematics becomes especially interesting.

Mathematics is a designed system of symbolic resources.

But mathematical discourse is language.

And once we recognise that, an equation is no longer quite the innocent piece of notation it first appeared to be.

It has entered the grammar.

And the grammar has more to tell us.

When Different Systems Became One Meaning: V. When Different Systems Became One Meaning

We began with something ordinary: words and a picture on the same page.

The words remained words.

The picture remained a picture.

Neither became a little version of the other.

And yet, as we have seen, something happens when they meet.

Each system contributes what its own resources make possible.

The words can tell us that Tom has never been in the room before.

The picture can show us Tom entering it.

The picture can also show things the words have not mentioned. The words can construe things the picture cannot directly show.

Their contributions do not have to be equivalent.

They do not even have to be complete on their own.

What matters is that they can become relevant to one another.

We can take the Tom in the words to be the Tom in the picture.

We can take the room described in the words to be the room shown in the picture.

We can understand the visual scene in the light of what the words tell us, and the words in the light of what the picture shows.

The two systems remain different.

But their meanings no longer stand apart.

They have become part of the same contextual configuration.

This is why multimodal meaning is not simply a matter of adding meanings together.

Something happens between the contributions.

The picture does not merely provide an extra piece of information.

The words do not merely provide another.

Together they construe a situation that neither system was construal of in isolation.

And we do not need to imagine a third system mediating between them.

The meaning emerges from their relation within the same meaning-making event.

This also changes the question we should ask about multimodality.

We do not need to ask whether every semiotic system has a grammar like language.

Nor do we need to ask which system is doing the “real” meaning-making.

We can ask instead:

How do different semiotic systems become relevant to one another in the making of meaning?

Sometimes the answer is that they contribute different aspects of the same situation.

Sometimes one makes something possible for the interpretation of the other.

Sometimes neither is enough for the particular meaning the situation requires.

But in each case, the systems can remain what they are.

They do not have to merge.

They do not have to become equivalent.

They only have to participate together in the same contextual configuration.

The words remain words.

The pictures remain pictures.

And yet, together, they can become one meaning-making event.

Different systems.

Different resources.

One situation.

One meaning.

When Different Systems Became One Meaning: IV. When the Same Context Held Them Together

We have seen that different semiotic systems can contribute differently to the same meaning-making event.

We have also seen that their contributions can become relevant to one another.

But what makes that possible?

Consider again the picture book.

The words say:

Tom had never been here before.

The picture shows Tom entering a room.

The words and the picture belong to different semiotic systems. They have different resources, and neither one contains the meaning of the other.

Yet they are not simply placed beside one another.

They are functioning in the same situation.

The words and the picture are both participating in the same act of making sense of what is happening to Tom.

This gives us a different way of thinking about context.

Context is not a box surrounding the words and the picture.

Nor is it an external mechanism that tells each semiotic system what to do.

It is the semiotic environment in which their contributions become relevant.

The context is not one thing, and the words and picture another thing, with the context standing between them.

Rather, the situation can be construed at different levels of symbolic abstraction.

We can look at what is happening as a configuration of meanings, or at the semiotic resources through which those meanings are made.

The two perspectives belong together.

This is why we can say that context and the semiotic systems that realise it are construed together.

The picture and the words are different ways of participating in the same contextual phenomenon.

The picture gives us Tom entering the room.

The words tell us that this is somewhere he has never been before.

Neither contribution has to be converted into the other.

Their relation comes from the fact that both are functioning in the same meaningful situation.

This also explains why different semiotic systems do not need a common grammar in order to work together.

They do not need to become structurally identical.

They need to be able to participate in the same context.

And because they participate in that context, their different contributions can become related.

Context, then, does not stand above the semiotic systems and coordinate them from outside.

It is part of the same meaning-making phenomenon of which the systems are different realisations.

The words remain words.

The picture remains a picture.

But they are no longer unrelated meanings placed side by side.

They belong to the same contextual configuration.

And once we see that, the final question becomes possible.

How can different semiotic systems, remaining different, become one meaning?

That is what we need to ask next.

When Different Systems Became One Meaning: III. When One System Took Up the Other

We have two different semiotic systems.

The words say:

Tom went into the room.

The picture shows Tom entering a room. It also shows something the words have not mentioned: a dark shape beneath the bed.

The two systems have contributed differently.

But we do not leave their contributions separate.

When we read the words, we take the Tom in them to be the person shown in the picture. We take the room to be the room represented there. The picture, in turn, becomes relevant to how we understand what the words are saying.

One system has, in a sense, taken up the contribution of the other.

But this does not mean that the picture has interpreted the words, or that the words have interpreted the picture.

The picture remains a picture.

The words remain words.

What changes is the relation we construe between them.

The meaning-maker can relate the contributions of the two systems because they have been presented as belonging together.

This is why multimodal meaning is not simply a matter of adding one meaning to another.

Suppose the picture shows Tom entering the room and the words say:

Tom had never been here before.

The picture cannot, by itself, show us that this is the first time Tom has ever been there.

The words cannot, by themselves, show us exactly what the room looks like.

But when we take the two together, each changes what the other makes relevant.

The room we see becomes the room Tom has never visited before.

Tom's first visit becomes the visit we can see.

The contribution of one system becomes part of the meaning of the other.

This does not require the two systems to share a grammar.

It does not require one system to be translated into the other.

And it does not require some third semiotic system hidden between them.

What it requires is that their contributions can be related.

The reader does not first construct two completely separate meanings and then add them together.

The meanings are construed in relation.

That gives us a more precise way of thinking about what it means for different semiotic systems to work together.

They do not have to become the same.

They have to become relevant to one another.

But that raises a further question.

What makes it possible for the reader to treat these different contributions as belonging together in the first place?

The answer cannot lie entirely inside either semiotic system.

Perhaps we need to look beyond the systems themselves—to the situation in which they are functioning.

And that is where context enters the picture.

When Different Systems Became One Meaning: II. When Each System Did What It Could

If words and pictures can belong to the same meaning-making event without being the same kind of semiotic system, perhaps the first thing we need to notice is what each can contribute.

Consider a page on which the words say:

Mia stood outside the old house.

The words tell us something about what is happening. They identify Mia and construe her relation to the house. They can tell us that she stood, and that the house was old.

But the picture can do things the words have not done.

It can show us the colour and shape of the house, where Mia is standing in relation to it, the position of the door and windows, the surrounding garden, the expression on her face, and perhaps something visible through one of the windows.

The picture does not have to translate these things into words.

Nor does the language have to reproduce everything the picture shows.

Each system can do what its own resources make possible.

This becomes clearer when the two contributions diverge.

The words might say:

Tom went into the room.

The picture might show Tom entering through a doorway while, behind him, a dark shape is visible beneath the bed.

The words have not told us about the shape.

The picture has not told us that Tom went into the room in the linguistic sense. It has shown us a particular configuration of people, objects and space from a particular point of view.

Yet we have no difficulty relating the two.

The picture has contributed something the words did not. The words have contributed something the picture did not.

That is not a defect in either system.

It is what makes their cooperation possible.

If every semiotic system had to be capable of expressing everything that every other system could express, there would be little point in having different systems at all.

Different systems have different resources because they have different ways of making meaning.

And when they work together, they do not have to give us equivalent meanings.

They can give us different parts of what becomes meaningful on the page.

The important question, then, is not which system can do more.

It is what each system can contribute that the other does not.

But there is still a puzzle.

How do we know that the Tom in the words belongs with the Tom in the picture? How do we know that the room described by the words is the room being shown?

The systems are different.

Their resources are different.

Their contributions may even be different.

And yet we take them as belonging together.

Something must make that possible.

We have found the difference.

Now we need to discover the relation.

When Different Systems Became One Meaning: I. Different Systems, One Situation

Open a children's picture book and you may find something quite ordinary: words on one page, a picture beside them.

The words might say:

Mia stood outside the old house.

The picture might show Mia standing beside a weathered house, with its door, windows, garden and surrounding landscape all visible.

The words and the picture are not doing the same thing.

Language has its own resources for construing experience. It can identify participants, construe processes and circumstances, locate events in time, and relate one experience to another.

The visual system has different resources. It can show shape, colour, spatial arrangement, relative position, appearance and what can be seen from a particular point of view.

Nor do we need to suppose that the picture is secretly a kind of language. It does not need a grammar like the grammar of the words in order to participate in the meaning of the page.

And yet we do not experience the words and the picture as two unrelated events.

We understand Mia in the words as the girl we can see. We understand the old house as the house represented in the picture. The two systems are different, but what they are contributing is related.

Something has brought them together.

The obvious answer is that they are about the same situation. But that only moves the question back one step.

How can two different semiotic systems construe different aspects of a situation and still become part of the same meaning-making event?

The answer cannot simply be that they say the same thing. Often they don't.

And it cannot be that one system has been translated into the other. The picture remains a picture; the words remain words.

The interesting thing is precisely that difference does not prevent coordination.

Two semiotic systems can remain different while participating in the same act of meaning.

So perhaps the question is not how one system can become like another.

It is how different systems can become relevant to one another.

And that is where we should begin.

When Two Semiotic Systems Met: VI. When Two Systems Became One Event

We began with something ordinary.

Words on a page.

A picture beside them.

Two different semiotic systems.

Yet we experienced them as one thing: a page in a story.

By now, we have seen why that matters.

The words can tell us something the picture does not.

The picture can show us something the words do not.

Sometimes they overlap.

Sometimes one adds something the other leaves unsaid.

Sometimes neither, on its own, gives us quite the meaning that emerges when we put them together.

But through all of this, the two systems remain distinct.

The words do not become pictures.

The pictures do not become words.

The visual system does not acquire a grammar simply because we can describe what it means in language.

Nor does language cease to be language because a picture is present beside it.

Something else has happened.

They have become part of the same event of meaning.

This may seem like a small distinction.

It isn't.

If we imagine that every semiotic system must operate independently, we are left asking how one system can somehow transfer its meanings to another.

If we imagine that every system must be organised like language, we start looking for grammars where there may be no grammar to find.

But neither is necessary.

Different semiotic systems can remain what they are and still participate together.

The words contribute what words can do.

The picture contributes what pictures can do.

The reader relates those contributions.

And the meaning of the page emerges through that relation.

So perhaps the important question is not:

How does one semiotic system become another?

Nor even:

What is the grammar of each mode?

It may be:

How do different semiotic systems become relevant to one another?

A picture book gives us a particularly clear example because we can see the two systems sitting there together.

But the same question must arise whenever different ways of making meaning are brought into relation.

What makes their contributions belong together?

What allows us to construe them as parts of one event rather than as unrelated acts of meaning?

We have not answered that question.

And perhaps we should not answer it yet.

For the moment, it is enough to notice what we have discovered.

Two semiotic systems do not have to merge in order to participate together.

They do not have to share a grammar.

They do not have to become versions of one another.

They only have to meet in a situation where their different possibilities for meaning can become relevant to one another.

The words remain words.

The pictures remain pictures.

But together, they can become something neither was doing alone:

one meaning-making event.

When Two Semiotic Systems Met: V. When Neither System Was Enough

The picture shows a girl standing at the edge of a forest.

A narrow path disappears between the trees.

The words say:

She knew she had to go on.

Either one could stand alone.

The picture could be meaningful without the words.

The words could be meaningful without the picture.

So neither system is inadequate.

And yet, for the meaning of this particular page, neither is enough on its own.

The picture gives us the girl, the forest and the path.

It shows us where she is.

It shows us what lies ahead.

But it does not tell us that she knows she must continue.

The words tell us that.

Yet the words alone do not give us the forest, the path disappearing into the trees, or the girl's position at its edge.

They tell us that she has to go on.

The picture lets us see what going on means here.

Something happens when we bring the two together.

The girl we see is the girl who knows.

The path we see is the path she must follow.

The forest becomes more than a setting.

It becomes the situation in which her decision has significance.

The meaning of the page is therefore not simply the meaning of the words plus the meaning of the picture.

Something is happening between them.

Each gives us something the other does not.

And the reader relates those contributions.

This is a different kind of relationship from simply saying the same thing twice.

The picture does not translate the words.

The words do not translate the picture.

Nor is one merely an illustration of the other.

They remain two different semiotic systems, with different resources for making meaning.

But they have been brought into a relationship in which their meanings can be construed together.

That relationship matters.

Without the picture, we would know that she has to go on, but not see the world through which she must go.

Without the words, we would see a girl facing a forest, but not know that she has recognised an obligation to continue.

Together, we can construe something neither has expressed on its own.

And this suggests that the meaning of a multimodal event cannot always be located inside one semiotic system or the other.

Sometimes the meaning emerges from their coordination.

That does not require a new semiotic system between them.

It does not require language to become visual, or the visual system to become linguistic.

It requires only that different semiotic resources can be brought into relation.

The picture has not become words.

The words have not become a picture.

But on this page, neither is doing the whole job alone.

Perhaps that is the deeper significance of two semiotic systems meeting.

They do not have to become one.

They only have to become meaningful together.

When Two Semiotic Systems Met: IV. When Words Said What the Picture Couldn't

The picture shows a girl standing beside a closed door.

We can see the girl.

We can see the door.

We can see that she is outside it.

But the words say:

She had never been inside before.

Now the words have taken us somewhere the picture could not take us on its own.

The picture can show us the girl standing outside the door.

It cannot simply show us that this has never happened before.

That is not something visible in the scene.

It belongs to the girl's experience.

The words can tell us about that experience directly.

This is not because language is somehow more powerful than the picture.

It is because the two systems have different resources.

The picture can make meanings available through what it depicts: people, objects, places, spatial relations, appearances, actions.

Language has resources for construing experience in other ways: including what has happened before, what might happen later, what someone knows, remembers, believes or expects.

So the picture gives us the girl and the door.

The words give us something about the girl's history.

And again, we do not experience these as two unrelated messages.

The girl we see becomes the girl who has never been inside.

The door we see becomes the door she has never crossed.

The visual and linguistic meanings become related within the story.

This is important because it shows that the relationship between semiotic systems is not simply a matter of one system supplying what the other has already expressed.

Sometimes the contribution is genuinely different.

The picture makes one set of meanings possible.

The words make another set possible.

Together they allow us to construct something larger.

And the relationship can work in either direction.

In the previous case, the picture told us something the words had not said.

Here, the words tell us something the picture has not shown.

Neither system has failed.

Neither is incomplete in itself.

They simply have different possibilities for meaning.

This also gives us a reason to be cautious about asking whether a picture has the same meanings as language.

It may have meanings that can be interpreted linguistically.

We can say what a picture shows.

We can describe what it depicts.

But the fact that language can construe a visual meaning does not mean that the visual system itself possesses the linguistic resources with which we describe it.

The girl does not become linguistic because we can say girl.

The picture does not acquire a tense system because we can say had never been.

The two systems remain distinct.

What changes is what becomes possible when they are brought together.

A picture can show us where the girl is.

Words can tell us where she has been.

A picture can show us what is visible now.

Words can tell us something about what has happened before.

Neither has to become the other.

They only have to meet.

And when they do, the meaning-making possibilities of the page become larger than those available to either system alone.

When Two Semiotic Systems Met: III. When the Picture Said More

The words say:

Tom went into the room.

The picture shows Tom standing just inside the doorway.

But it also shows something else.

Behind the curtain, almost hidden from view, there is a large black shape.

Tom has not noticed it.

We have.

Suddenly, the picture is doing more than illustrating the words.

The words tell us that Tom went into the room.

The picture tells us something the words have not told us: that something is waiting behind the curtain.

And that changes the meaning of the page.

We now know something Tom does not know.

The story has acquired a possibility that neither the words nor the picture alone necessarily establishes.

This is a simple example, but it shows why we should be careful about saying that the picture merely repeats the language.

It doesn't.

The visual system has resources for making meanings that are not simply linguistic meanings expressed in another form.

It can show us the arrangement of things in space.

It can show us what is visible and what is hidden.

It can place two events before us at once.

It can give us the expression on a face, the direction of a gaze, the distance between people, the atmosphere of a place.

The words could describe these things, of course.

But that is different.

Language can say what the picture shows. The question is whether it has said it.

Here, it hasn't.

The picture has contributed something of its own.

And something else has happened.

We don't simply add the picture's information to the words as though making a list.

We interpret the two together.

Tom went into the room.

Something is behind the curtain.

Tom does not know.

The reader does.

The relationship between the two semiotic systems has therefore created a possibility in the story.

The picture has not merely supplied another expression of a meaning already present in the words.

It has changed what the words mean for us.

Or perhaps more precisely, it has changed what the whole page makes possible for us to understand.

This is why it is useful to resist the idea that multimodality is simply a matter of translating one semiotic system into another.

If the picture were merely another way of saying Tom went into the room, we could treat the two resources as equivalent.

But they are not equivalent.

The picture has its own possibilities.

The words have theirs.

And when they meet, one system can sometimes take the meaning-making somewhere the other has not gone.

That does not make the picture more important than the words.

It makes the two systems different.

The interesting question is therefore not which system carries the meaning.

It is how their different possibilities become part of the same meaning-making event.

The picture has said more.

And because it has, the page now means more than the words alone could have given us.

When Two Semiotic Systems Met: II. When One Meaning Had Two Resources

Suppose the words on a page say:

The little boy opened the door.

And suppose the picture shows exactly that.

A little boy.

A door.

The door opening.

At first, it seems that we have the same meaning twice.

The words tell us what happened.

The picture shows us what happened.

But look more closely.

The words tell us that the person is a little boy. They tell us that he opened the door.

The picture can show us what the boy looks like, where he is standing, what the door looks like, what lies beyond it, and how he is opening it.

The two systems are not necessarily saying the same thing.

They are making different contributions to what we understand.

This matters because it is tempting to think of a picture book as a verbal message accompanied by an illustration.

That would make the picture secondary.

But the picture does not have to be an illustration of something the words have already said. It can contribute meaning in its own right.

Nor does this mean that the words and the picture are two independent messages that happen to occupy the same page.

Something else is happening.

The reader relates them.

The boy in the words becomes the boy in the picture.

The door that is opened in the wording becomes the door we can see.

The linguistic description and the visual representation become parts of the same situation we are construing.

So perhaps we need to distinguish two things.

There can be one meaning with two resources.

And there can be two expressions of the same meaning.

They are not the same.

If the words say the little boy opened the door and the picture shows a little boy opening the door, there is certainly some overlap. But the overlap does not exhaust what either system contributes.

The picture may tell us that the door is enormous.

The words may tell us that the boy is frightened.

Neither has to reproduce the other's contribution.

What we experience is not simply a word meaning plus a picture meaning.

We experience a boy, a door, an action, a place, perhaps a mood and the beginning of a story.

The meaning has been assembled from more than one semiotic resource.

And this gives us a more precise way of thinking about what happened when words met pictures.

They did not merge.

They did not become a single system.

They remained different.

But the meaning-making event gave them something to do together.

Perhaps that is why the question Which one is saying it? is sometimes the wrong question.

The more interesting question is:

What is each one making possible that the other is not?

Once we ask that, the picture book begins to look rather different.

It is no longer a linguistic message with pictures attached.

It is a meaning-making event in which more than one semiotic resource can contribute to what the reader understands.

And sometimes, one of those resources can tell us something the other never could.

That is where things become interesting.

When Two Semiotic Systems Met: I. When Words Met Pictures

Open a children's picture book and something very ordinary happens.

There are words on the page.

There is a picture.

We read the words. We look at the picture. Somehow, we understand what is happening.

It seems almost too obvious to be worth asking how.

After all, the words tell us the story and the picture shows us what is happening.

But that last sentence already assumes something rather interesting.

It assumes that words and pictures are participating in the same meaning-making event.

The words belong to one semiotic system. The picture belongs to another. They do not have the same resources, and they do not organise meaning in the same way. Yet on the page they are not strangers.

Consider a page showing a child standing before a large red door.

The words might say:

Mia stood outside the old house.

The picture does not simply repeat those words. It gives us the house, the door, Mia's position, the surrounding landscape, perhaps her expression and the way she is standing.

Nor does the picture replace the words. The words tell us that this is an old house, and that Mia is standing outside it. Those are linguistic meanings.

We do not need to decide that the picture is somehow saying the same thing in another language.

It isn't.

The picture has its own resources.

The words have theirs.

What matters is that both have been brought together on the page.

This changes the question.

Instead of asking, What is the grammar of the picture? we can ask a simpler question:

What is each semiotic system contributing?

And then another:

What happens when those contributions meet?

The answer may be more interesting than either system considered on its own.

The picture can show us something the words do not say.

The words can tell us something the picture cannot show in the same way.

Sometimes they appear to say almost the same thing. Sometimes one adds something to the other. Sometimes the picture quietly changes how we understand the words.

Yet the page does not become two separate messages.

We experience it as one event.

That is the small puzzle with which we can begin.

Words and pictures have not become the same semiotic system.

They have not acquired a common grammar.

They have simply met.

And perhaps the first thing we need to understand about multimodal meaning is not how one system becomes like another, but how different semiotic systems can participate together in making meaning.

The next question is therefore not whether words and pictures mean the same thing.

It is what happens when one meaning has more than one semiotic resource at its disposal.

When Other Semiotic Systems Met Language: IV. When Language Was No Longer the Only Prototype

We began with a traffic light.

It seemed almost too simple to be theoretically interesting.

Red realises STOP.

There is no grammar between the colour and its meaning. Yet the meaning can function as a command. The traffic light can function as Speaker and Sayer. Its signal can enter into an exchange with a human participant.

And when the driver puts the signal into words, the grammar comes from the driver.

The traffic light does not need to become language in order to participate in a social semiotic world.

This leaves us with a broader question.

If language is the only semiotic system we know that has a grammatical stratum in the full linguistic sense, why do we so readily expect other semiotic systems to have one?

Part of the answer is obvious.

Language is our most powerful resource for talking about meaning. We can use it to describe a painting, explain a diagram, interpret a gesture or map a building. Language provides an extraordinarily rich vocabulary for making the organisation of other semiotic systems explicit.

It is therefore natural that language should become a prototype for theorising them.

And sometimes that may reveal genuine structural analogies.

But the traffic light suggests a useful caution.

We should distinguish between a semiotic system being organised like language and a semiotic system being interpreted by language users.

Those are different relationships.

A system may have its own organisation without having a grammatical stratum. Its meanings may nevertheless be taken up by language, described through language, and even function within exchanges involving language users.

This opens a larger possibility.

Perhaps semiotic systems do not form a hierarchy in which language provides the fully developed model and everything else is a partial version of it.

Perhaps they form an ecology.

Different systems may organise meaning in different ways. Some may require grammatical organisation; others may not. Some may be routinely translated into language; others may combine with language without being translated. Some may be modelled fruitfully through linguistic concepts, while others may become more intelligible when we begin from their own organisation.

Language would remain extraordinarily important.

But it would no longer have to be the prototype against which every other semiotic system is measured.

The traffic light gives us a remarkably small example of what that might mean.

It has meaning without grammar.

It can participate in interpersonal exchange without grammar.

And a language user can construe its meaning grammatically without importing grammar into the traffic-light system.

Perhaps, then, the question for a general theory of multimodality is not:

What is the grammar of this mode?

Nor even:

Does this mode have a grammar?

Perhaps the prior question is:

What kind of semiotic organisation does this system require in order to do the work it does?

Once we ask that question, grammar becomes one possibility rather than the inevitable answer.

And that may leave us with a rather different conception of multimodality.

Not a collection of lesser languages, each waiting to have its own grammar discovered.

But an ecology of semiotic systems, differently organised, entering into relations with one another — sometimes with language, sometimes through language, and sometimes without it.

The traffic light was never a little language.

We only needed it to be meaningful.

And perhaps that is where we should have started.

When Other Semiotic Systems Met Language: III. When We Gave Other Semiotic Systems a Grammar

Once we notice that language can supply the wording for meanings made in other semiotic systems, another question arises.

Why have we so often described those systems as though they had grammars of their own?

The answer is not difficult to see.

A great deal of work in systemic functional linguistics has extended linguistic concepts to other semiotic systems: visual art, architecture, diagrams and many others. Halliday and Matthiessen described such systems as being constructed, in some cases, “in the image of language itself” — and therefore capable of being modelled on language as a prototype.

There is an important qualification in that formulation.

They can be described as if they had grammar and semantics.

That as if matters.

It recognises that the grammar may be a model we bring to the system rather than a grammatical stratum that the system possesses in the same sense that language does.

This is not a trivial distinction.

When we analyse a painting in terms of composition, information value, salience and framing, for example, we may be identifying real organisational principles in the visual system. But we are also describing those principles through concepts that have been developed within a theory of language.

The same question arises with architecture, gesture, diagrams and other semiotic resources.

Are we discovering their own semiotic organisation?

Or are we discovering how useful it is to model that organisation through the prototype of language?

Sometimes the two may amount to much the same thing.

But they need not.

The traffic light gives us a simple case in which the distinction is difficult to ignore.

We can describe its organisation perfectly well without inventing a grammar:

red → STOP

We can then describe what happens when that meaning enters language:

STOP → Stop!

The second relation belongs to the linguistic system.

There is therefore a danger in moving too quickly from:

“We can describe this semiotic system grammatically”

to:

“This semiotic system has a grammar.”

Those are not necessarily equivalent claims.

And yet there is a good reason for the first kind of description.

Language is the semiotic system through which we can make explicit, compare and theorise meanings made in other systems. Its categories give us powerful tools for discovering patterns that might otherwise remain difficult to articulate.

So modelling another semiotic system in the image of language is not simply a mistake.

It may be a productive theoretical strategy.

The question is what kind of claim we are making when we do it.

Are we proposing that the other system has an organisation analogous to language?

Are we using linguistic categories as an analytical model?

Or are we claiming that the other system actually contains a grammatical stratum corresponding to the lexicogrammar of language?

Those are different propositions.

The traffic light suggests that we should keep them apart.

It can mean without grammar.

It can enter an exchange without grammar.

And its meanings can be expressed in grammatical wording without grammar ever becoming part of the traffic-light system itself.

Perhaps, then, the interesting question is not whether other semiotic systems can be given a grammar.

It is what we learn about them when we do.

When Other Semiotic Systems Met Language: II. When Language Supplied the Grammar

In the first post, we saw that a traffic light can make meaning without having a grammar of its own.

But we also noticed something curious.

When a driver sees red, it is almost irresistible to put the meaning into words:

Stop.

The traffic light supplies the meaning.

Language supplies the wording.

That distinction is easy to overlook because the two events can happen so quickly that they seem like one.

The traffic light expresses STOP.

The driver construes that meaning through language as Stop!

The grammar belongs to the driver.

This gives us a rather different way of thinking about the relation between language and other semiotic systems.

We don't necessarily need to imagine that a non-linguistic system contains a grammar corresponding to the wording we can give to its meanings. The language user may simply be bringing the resources of language to bear on another semiotic system.

The same thing happens whenever we put something non-linguistic into words.

A diagram can be described in a sentence.

A gesture can be glossed verbally.

A piece of music can be described as sad, urgent or triumphant.

A map can be explained in language.

In each case, language can provide a grammatical construal of something that does not itself have to possess a grammatical stratum.

The important question, then, is not necessarily:

What is the grammar of this other semiotic system?

It might instead be:

How does a language user construe what this other system means?

The distinction is subtle, but it changes what we are looking for.

Consider the traffic light again.

Its system gives us:

red → STOP

The driver's language can turn that into:

Stop!

But we should not work backwards from the wording and conclude that Stop! must therefore be structurally present inside the traffic light.

The wording is an achievement of the linguistic system.

This also explains how the traffic light can function as a Sayer without becoming linguistic.

The traffic light can project STOP.

The driver can supply the wording.

The two semiotic systems therefore cooperate without having the same organisation.

And perhaps this is one of the things that language makes possible for linguate beings.

We can take meanings from other semiotic systems and bring them into language. We can describe them, interpret them, classify them, argue about them, and respond to them in words.

Language does not merely provide another mode of meaning-making.

It can provide a grammar for talking about meanings made elsewhere.

That possibility may be easy to miss when we ask whether other semiotic systems have grammars of their own.

Perhaps sometimes the grammar we are looking for is not there.

Perhaps it is ours. 

When Other Semiotic Systems Met Language: I. When a Semiotic System Didn't Need a Grammar

A traffic light is a semiotic system.

It has expressions and meanings. Red, amber and green are expressions; STOP, STOP IF SAFE and PROCEED are their contents.

red → STOP

It is a perfectly functional system.

But there is something it doesn't have.

It doesn't have a grammar.

There is no lexicogrammatical stratum between red and STOP. The traffic light does not combine smaller meaningful elements into larger ones. It does not inflect, rank, or organise clauses. It has no words from which sentences can be constructed.

And yet it can participate in meaning remarkably effectively.

Red can mean STOP.

More than that, STOP can function as a command. A driver can construe the traffic light as saying Stop! and respond accordingly.

So how can this work if the traffic light has no grammar?

Perhaps the answer begins with something fundamental about language.

For Halliday, grammar is not simply one more resource among those available to a semiotic system. It is the distinctive stratum of language: the level at which meanings are organised into wording.

The traffic light does not need such a stratum because it is not doing what language does.

Its own semiotic organisation is sufficient for the meanings it needs to express.

This is easy to miss because, as language users, we can immediately put its meanings into words.

We see red and think:

Stop.

We see green and think:

Go.

But the availability of linguistic wording does not mean that the wording was already present in the traffic-light system.

The traffic light gives us STOP.

Language gives us Stop!

That distinction may matter more generally.

When we encounter a non-linguistic semiotic system, it is tempting to ask where its grammar is. How does it combine elements? What are its clauses? What are its equivalents of nouns, verbs and sentences?

But perhaps that is sometimes the wrong question.

A semiotic system does not necessarily need grammar simply because it has meaning.

It needs whatever organisation is required for the meanings it makes.

For a traffic light, content and expression are enough.

And that leaves us with an intriguing possibility.

Perhaps some semiotic systems do not have a grammar of their own because they don't need one.

Perhaps what makes this seem strange is not the semiotic system.

Perhaps it is us.

We are language users, so when something means something, we are extraordinarily good at putting it into words.

The traffic light may never have needed a grammar.

We may simply have been supplying one.

When a Traffic Light Became a Sayer: V. When a Traffic Light Became a Social Participant

We began with a simple observation:

Red means stop.

But a traffic light is more than a system in which colours have meanings.

Its meanings are simultaneously ideational and interpersonal. STOP construes a material process, but it also functions as a command. The traffic light can therefore function as Speaker, addressing a driver as Addressee.

Ideationally, the traffic light can also function as Sayer in a verbal process:

The traffic light signals [STOP].

What is projected is not the wording Stop!

The traffic light has no lexicogrammatical stratum. Its own semiotic system consists of content and expression:

red → STOP

A driver, however, has language. The driver can construe the traffic light's content through linguistic wording:

Stop.

The wording belongs to the driver's language, not to the traffic light.

This resolves the apparent paradox with which we began.

A traffic light can say STOP without having a word for stop.

It does not need to be a linguistic system in order to participate in a linguistic social world.

Its signal can function as a command. Its content can be projected in a verbal process. Its meaning can be directed towards another participant. And the other participant can respond.

The traffic light therefore participates in an exchange without becoming a human conversationalist.

It does not need consciousness.

It does not need language.

It does not need a grammar of its own.

It needs only a semiotic system whose meanings can enter into a socially organised exchange.

That may be the more general lesson hidden inside those three familiar colours.

A semiotic system can do more than represent the world.

It can participate in the organisation of social action.

And sometimes the thing that says STOP does not need to know the word at all.

When a Traffic Light Became a Sayer: IV. When a Signal Entered an Exchange

So far, we have followed the traffic light from meaning to saying.

But a traffic light does not say something into an empty world.

Its signals are directed towards someone.

Red makes the command STOP. Green makes the command PROCEED. Amber makes the command STOP IF SAFE.

And the driver can respond.

The response need not be verbal. The driver can stop, proceed, or, in circumstances where the command is not undertaken, continue moving.

This is an exchange.

The traffic light initiates a move; the driver responds to it.

Nothing about this requires the traffic light to be conscious or to deliberate about what it will say next. The exchange is built into the social organisation of the traffic system. The light has been designed to make particular meanings available at particular moments, and drivers have learned how to respond to them.

The signal therefore does more than convey a meaning.

It has a social function.

And this helps explain why the interpersonal meaning of STOP matters. If STOP were merely information, the driver could receive it without being called upon to do anything. As a command, however, it places an expectation on the Addressee.

The same content can therefore be viewed from several angles:

red realises STOP

STOP construes a material process

STOP enacts a command

the traffic light functions as Speaker and Sayer

the driver functions as Addressee

None of these descriptions replaces the others.

They are different ways of seeing the same semiotic event.

And the event does not happen only once.

The light changes.

Green becomes amber. Amber becomes red. Red eventually becomes green again.

The meanings therefore occur in a socially organised sequence. A signal has its place in relation to the signals that precede and follow it.

This is particularly clear with amber. Its meaning cannot be understood simply as the meaning of a third colour alongside red and green. Its significance depends on its position in the changing sequence and on the circumstances in which the driver encounters it.

The traffic light is consequently not just a code that a driver deciphers.

It is a semiotic system that takes part in an exchange.

And that may be the most important discovery so far.

We began with three colours and three meanings.

We found a system of content and expression.

Then we found ideational and interpersonal meaning.

Then we found a Speaker and a Sayer.

Now we find an exchange: a semiotic system makes a move, and another participant responds.

The traffic light has not become a human conversationalist.

But it has become a participant in a social process.

So what kind of participant can a non-linguistic semiotic system be?

When a Traffic Light Became a Sayer: III. When a Traffic Light Became a Sayer

We have called the traffic light a Speaker.

Now we can ask the ideational question: what kind of participant is a traffic light in the process of saying?

Halliday and Matthiessen use saying in a deliberately broad sense. A verbal process does not require a human being with a voice. Something can function as a Sayer whenever it puts out a signal that participates in symbolic exchange.

A notice can tell us to keep quiet.

A watch can tell us the time.

And a traffic light can tell us to stop.

So we can construe the traffic light as Sayer in a verbal process:

The traffic light signals STOP.

But now we have to be precise about what is being projected.

It is not the wording Stop!

The traffic light has no lexicogrammar of its own. Its semiotic system consists simply of content and expression:

red → STOP

The content STOP is what the traffic light's expression realises.

When that content is construed as a verbal process, the traffic light can therefore be understood as projecting a material process:

The traffic light signals [STOP].

And the same applies to amber:

The traffic light signals [STOP IF SAFE].

Here STOP is the material process, while if safe specifies the condition under which it applies.

The linguistic wording comes later — and from somewhere else.

A driver who encounters the red light can construe its meaning in language:

Stop.

But those words belong to the driver's linguistic system. They are not hidden inside the traffic light.

This is the point at which the distinction between the traffic light and language becomes important.

The traffic light does not need a grammatical stratum in order to say something in this broad symbolic sense. Its own semiotic system can supply the content. A human participant, whose semiotic system does include language, can supply the wording.

So the traffic light can be a Sayer without being a speaker in the ordinary sense.

It can say STOP without saying Stop!

And now the two perspectives line up.

Interpersonally, the traffic light is a Speaker making a command to an Addressee.

Ideationally, it is a Sayer in a verbal process projecting a material process.

The traffic light has not acquired language.

Something more interesting has happened.

A non-linguistic semiotic system has become capable of participating in symbolic exchange.

When a Traffic Light Became a Sayer: II. When a Meaning Became a Command

In the first post, we saw that STOP is the content realised by red.

But STOP is not only something that can be represented.

It is something that someone is being asked to do.

This is the interpersonal side of the traffic light.

When the light turns red, the traffic light is not merely presenting the meaning STOP to the driver. It is making a command.

The traffic light is therefore functioning as Speaker, and the driver as Addressee.

This does not mean that the traffic light has become a person. Speaker and Addressee are roles in an exchange. The traffic light can occupy the first role without possessing a human voice, consciousness or language.

The important thing is what its signal is doing.

Red realises STOP.

STOP functions as a command.

The two descriptions are not competing accounts. They are different metafunctional construals of the same semiotic event.

Ideationally, STOP is a material process.

Interpersonally, STOP is a command.

And the two meanings are not added together after the fact. They belong to the same content.

This helps explain something that otherwise seems rather odd about traffic lights.

We can say:

Red means stop.

But in an actual traffic situation, we naturally understand something stronger:

Stop.

The first describes the relation between expression and content within the semiotic system.

The second construes what that content is doing in an exchange.

A traffic light therefore does not merely have a meaning. Its meaning is organised so that it can function as a move directed towards another participant.

And this brings us back to the word saying.

If the traffic light is a Speaker, then perhaps it can also be understood as a Sayer.

But that seems to create a problem.

A Sayer is involved in a verbal process. And verbal processes normally involve language.

What, then, does the traffic light actually say?

Not Stop! — because those words belong to language, and there are no words inside the red light.

So perhaps the traffic light is saying something without saying it in words.

The next step is to find out what that could mean.

When a Traffic Light Became a Sayer: I. When Red Meant Stop

Everyone knows what red means.

At a traffic light, red means stop. Green means proceed. Amber means stop if safe.

It seems simple enough. Three colours, three meanings.

But what kind of meaning is STOP?

A traffic light is a small semiotic system. Its expressions are red, amber and green; its contents are STOP, STOP IF SAFE and PROCEED.

We can put the relation very simply:

red realises STOP

The colour is the expression; STOP is the content.

Nothing linguistic is required for this relation. The traffic light does not contain the word stop, or even an abbreviated version of the clause Stop! There is no hidden grammatical stratum between the colour and its meaning.

The traffic light has its own semiotic organisation.

But STOP is not quite like the meaning of an ordinary piece of information.

A traffic light doesn't merely tell us that stopping is relevant. When the light turns red, it tells someone to stop.

That introduces another dimension of meaning.

STOP is not only something represented. It is something demanded.

In other words, the content of the traffic-light system is doing more than one kind of semiotic work. STOP can be understood ideationally as a material process. But in the situation in which the traffic light operates, it also functions interpersonally as a command.

The same content is doing both.

And this changes the question.

We began with:

What does red mean?

Perhaps the more interesting question is:

What is the traffic light doing when red realises STOP?

It isn't merely displaying a colour.

It is saying something.

But the traffic light has no words.

So what could it mean to say that a traffic light says stop?

That is where our little three-colour semiotic system becomes much more interesting.