A word can tell us something.
An image can show us something.
Music can alter how we feel or anticipate something.
A gesture can make something happen.
But what happens when several of these resources are present at once?
We do not simply receive several messages and add them together.
The modes can change one another.
An image can alter the meaning of music.
Music can alter the meaning of an image.
Words can make an image ironic.
An image can make words ambiguous.
A silence can change the force of a sentence.
Meaning can emerge between modes.
This is the distinctive problem of multimodality.
No mode does everything
Different semiotic modes have different possibilities.
Language can distinguish, classify, sequence and explain.
Images can organise spatial relations and present several elements simultaneously.
Music can organise temporal expectation, affect and participation.
Gesture can make meaning bodily and interactive.
Space can establish relations of proximity, separation, orientation and movement.
A culture can therefore draw upon different resources for different kinds of construal.
The modes do not need to say the same thing.
Their differences are precisely what makes their combination powerful.
Meaning can be distributed
Suppose someone says:
"I'm fine."
The words may appear straightforward.
But imagine that the person is visibly shaking.
Now the body changes our construal of the words.
Or imagine the same sentence accompanied by an affectionate smile.
The meaning changes again.
The wording has not changed.
The other mode has changed the possibilities for interpreting it.
Meaning is therefore not necessarily located within one mode.
It can be distributed across a configuration of modes.
Music changes the image
This is particularly striking with music.
Imagine a character entering a dark room.
The image tells us where we are.
But suppose the music has already established a sense of danger.
We now watch the room differently.
Nothing in the image has changed.
Our construal of it has.
The music has therefore contributed to the meaning of the visual scene without depicting anything itself.
This is one of the most important lessons from Music Before Meaning.
Music can matter through its own organisation and then be recruited into a larger semiotic configuration.
It does not need to become a language in order to contribute to meaning.
A triumphant disaster
The most revealing example may be the one we have already encountered.
Imagine footage of a natural disaster accompanied by triumphant music.
The image presents catastrophe.
The music evokes triumph, celebration or victory.
What results?
Perhaps irony.
Perhaps dark humour.
Perhaps absurdity.
Perhaps propaganda.
Perhaps an unsettling mixture of celebration and horror.
But where does that meaning reside?
Not in the music alone.
The music is not intrinsically ironic.
Not in the image alone.
The image is not intrinsically ironic.
The irony emerges from their relation.
Each mode changes the possibilities for interpreting the other.
Contradiction is productive
This means that multimodality does not require agreement.
Indeed, disagreement between modes can be extraordinarily productive.
A beautiful image can accompany disturbing music.
A comic scene can be accompanied by solemn music.
A character can speak sincerely while the soundtrack suggests deception.
A cheerful song can accompany images of suffering.
The tension forces the audience to construct a relation between the modes.
The resulting meaning may be more complex than either mode could produce alone.
Contradiction can therefore be a meaning-making resource.
Film makes this visible
Film is particularly useful because it deliberately combines so many semiotic resources:
- image,
- dialogue,
- music,
- sound effects,
- gesture,
- movement,
- lighting,
- editing,
- spatial organisation.
The film does not simply place these resources beside one another.
It composes them.
A close-up followed by a musical change means something different from the same close-up accompanied by silence.
The same piece of music can mean something different when played under different images.
The same image can mean something different when placed in a different sequence.
The meaning belongs partly to the configuration.
Editing creates relations
Editing shows this particularly clearly.
Two shots can mean little in isolation.
Placed beside one another, they can create a new relation.
A face.
A photograph.
A landscape.
The viewer may construe memory, longing, recognition or association.
The individual shots have not changed.
Their relation has.
Editing therefore creates meaning through juxtaposition.
It allows one sign to modify the construal of another.
In that respect, montage is structurally related to what we have already seen in metaphor.
Something becomes meaningful through its relation to something else.
A film can create its own conventions
Film does not merely draw upon conventions inherited from a wider culture.
It can create conventions within itself.
A particular musical motif can become associated with a character.
A colour can become associated with memory.
A recurring visual pattern can become associated with danger.
A camera movement can acquire a particular significance.
At first, the audience may not understand the association.
The film repeats it.
The pattern becomes recognisable.
The audience learns the system.
A meaning potential has emerged within the film itself.
This is a miniature example of semiotic evolution.
The score becomes a sign
Consider a musical motif associated with a character.
The first appearance may simply be music.
The second establishes an association.
The third strengthens it.
After several appearances, the motif can occur without the character being visible.
The audience anticipates the character.
Perhaps the character is about to appear.
Perhaps the character is being remembered.
Perhaps the character is absent.
Perhaps the music is being used ironically.
The motif has become a sign within the film.
But it remains music.
Its rhythm, harmony, timbre and temporal organisation still matter in their own right.
The symbolic function has been added to its existing affordances.
Meaning can move across modes
This is why it is useful to say that meaning can be recruited across modes.
A musical pattern can acquire significance through an image.
An image can acquire significance through a musical pattern.
A word can acquire irony through a visual contradiction.
A gesture can give emotional force to language.
A place can provide meaning to an object.
The modes do not simply transmit meaning from one place to another.
They can transform one another's meaning potential.
Myth was multimodal too
Film is a modern example, but the underlying phenomenon is not modern.
Before cinema, cultures could already combine:
words, images, music, gesture, objects, places and ritual.
Myth is one of the clearest examples.
A story can give an image a significance it would not otherwise have.
The image can make the story memorable.
A ritual can enact the relation.
Music can accompany it.
A sacred place can give the whole configuration a setting.
The meaning does not belong to one component.
It belongs to the network of relations among them.
This is why it can be misleading to isolate an ancient image and ask what it "means" without asking what other modes once accompanied it.
A cave is not just a wall
Imagine a prehistoric cave.
There may be images on the walls.
But there is also darkness.
Movement.
Sound.
Echo.
Light from lamps or torches.
The physical route through the space.
The people who enter.
The actions they perform.
The stories they tell.
Perhaps music or vocalisation.
Whatever the original meaning of the images, it may have been distributed across this larger environment.
The image may never have functioned as an autonomous "artwork".
It may have been one component in a multimodal semiotic ecology.
Newgrange and distributed meaning
The same caution applies to monuments such as Newgrange.
A carved pattern cannot easily be separated from the stone on which it appears, the architecture around it, the movement through the passage, the changing light and the temporal cycles that structure the site.
The meaning of the carved form, whatever its precise historical interpretation, may have emerged partly through its relations with these other modes.
Meaning can therefore be distributed not only across different media but across material, spatial and temporal organisation.
Modes do not translate one another
There is a temptation to say that music "expresses the same meaning" as words, or that an image "translates" a story.
Usually it does something more interesting.
Each mode offers its own affordances.
Music can make a relation temporal and affective.
Images can make relations spatial.
Language can make them explicit and classificatory.
The modes therefore do not merely repeat one another.
They reconstrue one another.
A story may become an image.
An image may become a musical theme.
A musical theme may change how we experience the story.
The meaning changes as it moves between modes because the mode changes what can be made salient.
From one relation to a configuration of relations
We can now make the point more abstractly.
A sign is itself a relation.
A multimodal composition places several sign relations into relation with one another.
Meaning therefore becomes a relation among relations.
The image relates to what it depicts.
The music relates to its own musical potential and cultural associations.
The dialogue relates to the linguistic system.
The editing relates one scene to another.
The film organises these relations together.
The resulting meaning is not reducible to any one of them.
This is why multimodality is not simply a matter of adding channels.
It creates new possibilities of construal.
The repertoire of the audience matters
There is another important participant in the process: the audience.
The modes do not arrive in a cultural vacuum.
Viewers bring repertoires of meanings.
They have learned musical conventions.
Visual conventions.
Genre conventions.
Narrative expectations.
Cultural associations.
A familiar melody may immediately evoke a particular tradition.
An unfamiliar viewer may hear only its musical properties.
The same visual reference may be obvious to one audience and invisible to another.
The film provides a structured field of possibilities.
The viewer's repertoire determines much of what can be readily construed within that field.
The meaning emerges from their interaction.
But interpretation is not arbitrary
This does not mean that every viewer can make the film mean anything whatsoever.
The film constrains possible construals.
The conventions of its modes constrain possibilities.
The sequence of images constrains the narrative.
The music establishes expectations.
Cultural repertoires make some interpretations much more probable than others.
Meaning is therefore neither completely encoded nor completely invented.
It emerges from a coupling between semiotic affordance and participation.
A mode can change what another mode can do
This may be the central insight of multimodality.
The presence of one mode can change the meaning potential of another.
Music can make an image ominous.
An image can make music triumphant.
Words can make a colour symbolic.
A narrative can make an object sacred.
A ritual can make a sound significant.
A sign can therefore acquire possibilities through its relations with other signs.
The semiotic system becomes richer through combination.
Meaning becomes compositional
Once modes can interact, meaning becomes increasingly compositional.
A single sign has a range of possible construals.
Another sign narrows or expands that range.
A third transforms the relation again.
The whole configuration acquires meaning that no component possessed independently.
This is why complex cultural forms can become extraordinarily expressive.
They are not simply carrying more information.
They are creating more relations among possibilities.
The film score reveals something about music
This brings us back to our starting point.
We began by observing that music can matter without meaning.
Film shows that this does not prevent music from becoming an important participant in semiosis.
A musical passage can matter through its own organisation.
Its cultural history can give it further significance.
A film can place it in relation to images and words.
The resulting configuration can make it meaningful in a new way.
The music does not have to become an autonomous symbolic system.
It can become a semiotic participant.
This is an important distinction.
And it reveals something about meaning
The same example tells us something larger.
Meaning need not originate within a single medium.
A culture can distribute meaning across modes.
It can combine the different affordances of language, image, music, gesture and space.
The resulting meanings can be richer than any of the individual modes.
This gives semiotic evolution another dimension.
It is not only the growth of what can be meant.
It is the growth of how meaning can be organised across different resources.
The world becomes representable in many ways
Once modes can make meaning together, a culture can construe the same phenomenon in multiple forms.
A story can become an image.
An image can become a song.
A song can become a film.
A film can recontextualise the song.
A ritual can enact the story.
The same meaning can circulate across modes while being transformed by each.
This creates a richly interconnected symbolic environment.
Meanings can travel.
But they do not travel unchanged.
Each mode gives them a new form.
The next step
We have now reached an important point in our argument.
Things matter.
Some differences become signals.
Some signals become signs.
Meanings become resources for further meanings.
Relations of meaning can take material form.
And now different modes can enter into relations that generate meanings none could produce alone.
The semiotic world has become compositional and multimodal.
But once meanings can be composed, preserved and transmitted, they can also begin to reorganise the world in which they emerged.
A story can make a place sacred.
A symbol can create an identity.
A metaphor can change what a community notices.
A song can become an emblem.
A film can change how we hear music.
Meaning does not remain on the side of representation.
It feeds back into value, attention and action.
So the final question is:
What happens when meaning begins to change what matters?
No comments:
Post a Comment