A shark swims towards the camera.
We watch it.
Then the music begins.
The shark suddenly seems more threatening.
The image has not changed. The animal is the same. The camera has not moved. Only the soundtrack has changed.
Yet the way viewers construe the shark changes with it. In the well-controlled experiment we began with, participants who saw the same shark footage accompanied by ominous music subsequently evaluated sharks more negatively than participants who saw the footage with uplifting music or silence. The study therefore provides a particularly clear demonstration that music can alter the interpretation of visual material.
That simple result opened a much larger question:
How does music participate in meaning when music and image are different modes?
The answer we have developed is that the music does not need to become a language before it can become meaningful.
Something subtler happens.
Music brings its own potential
Music already has an organisation of its own.
Rhythm.
Tempo.
Timbre.
Dynamics.
Harmony.
Melodic contour.
Repetition.
Expectation.
These can matter without representing anything.
A rhythm can coordinate bodies.
A harmonic tension can create anticipation.
A timbre can alter affect.
A recurring phrase can become memorable.
A familiar style can invoke a cultural history.
Music therefore brings to the film a substantial potential for mattering.
The film does not create this potential.
It recruits it.
The picture changes the music
But recruitment is not one-way.
The image changes how we hear the music.
A melody can seem tender beneath one image and sinister beneath another.
A triumphant musical gesture can become grotesque over destruction.
A familiar passage can become comic when the visual situation changes.
The notes have not changed.
Their semiotic position has.
The image supplies a context in which the musical possibilities are differently organised.
So the relationship is reciprocal:
music changes the construal of image;image changes the construal of music.
This is the first central discovery of the series.
Congruence and incongruence
Sometimes the modes broadly agree.
Threatening images receive threatening music.
Romantic images receive tender music.
Triumphant events receive triumphant music.
Such congruence can make a scene fluent.
But disagreement can be even more revealing.
A cheerful song over violence.
A solemn piece over comedy.
Triumphant music over catastrophe.
The difference creates a gap.
And the gap can become meaningful.
The audience begins to ask, often without consciously formulating the question:
Why have these things been put together?
The answer may be irony, satire, estrangement, absurdity, commentary or something else.
The incongruence is not necessarily a failure of integration.
It can be the very means by which the film makes meaning.
Expectation is the hidden foundation
This depends upon something we have encountered throughout our investigations: structured possibility.
A musical continuation is surprising because other continuations were possible.
A cinematic convention can be violated because we have learned the convention.
A musical motif can become significant because it has appeared before.
Silence can matter because music was expected.
Incongruence becomes noticeable because congruence was available.
The film therefore works with a field of expectations.
A particular audiovisual combination becomes meaningful partly because it is one selection among alternatives.
This is the same broad logic by which an instance becomes significant within a potential.
From coupling to sign
Repeated audiovisual associations can produce something new.
A musical theme accompanies a character.
It returns.
The association becomes recognisable.
It returns again.
The audience learns the relation.
Eventually the music can appear without the character and still invoke them.
This is the transformation from multimodal coupling to semiotic convention.
The Jaws motif provides a classic example. The 2016 shark study itself refers to the Jaws theme as a case in which repeated pairing between music and the shark can eventually allow the musical figure to evoke the shark even when the animal is absent.
The movement is:
music + image → association → repetition → recognition → sign
The music has not become a language.
It has become a sign within a particular semiotic system.
The absent appears in sound
Once the musical sign has been established, it can operate without the image.
A character can be absent but remain musically present.
A past event can be recalled.
A future event can be anticipated.
A relationship can remain active in the narrative even after one of its participants has disappeared.
The score can therefore extend the present across time.
It can make the absent relevant.
It can make the future felt before it happens.
It can make the past present without showing it again.
Music becomes a carrier of narrative memory and anticipation.
The score becomes part of the story
At this point, the score is doing more than setting a mood.
It can alter our understanding of character.
It can establish point of view.
It can suggest what a character does not know.
It can foreshadow.
It can create irony.
It can connect apparently unrelated events.
It can make a scene appear to be memory rather than present action.
It can cause a later revelation to transform the meaning of an earlier musical occurrence.
The score has become part of the organisation of the narrative itself.
The question is no longer:
How does the music make us feel?
but:
What does the music make this event mean within the story?
The film as a semiotic ecology
By this point, it becomes difficult to treat a film as a collection of separate channels.
There is:
image, music, dialogue, sound effects, gesture, movement, lighting, editing, silence, space and cultural convention.
Each has its own possibilities.
But the important thing is that they interact.
The image can change the music.
The music can change the image.
Dialogue can make music ironic.
Editing can connect two otherwise unrelated signs.
Silence can become significant because sound was expected.
A recurring colour can acquire meaning because of a musical motif.
The film therefore behaves less like a bundle and more like an ecology.
Its resources affect one another.
Their combinations create new possibilities.
The meaning of the whole is not simply the sum of the meanings of its parts.
Meaning emerges between modes
This is the central theoretical conclusion.
Meaning does not always reside within an individual semiotic resource.
It can emerge between modes.
The shark is not intrinsically frightening because ominous music is played beneath it.
The music is not intrinsically ironic because catastrophe is shown beneath it.
The meaning arises from the relation.
The modes bring different potentials.
The film places those potentials into a configuration.
The viewer construes the resulting relation.
Meaning therefore belongs partly to the organisation among resources.
This is why multimodality is not merely additive.
It is compositional.
The viewer participates
The viewer is part of this ecology.
We bring cultural and semiotic repertoires.
We know musical conventions.
We recognise genres.
We remember earlier scenes.
We learn motifs.
We anticipate narrative developments.
The film does not simply place meanings into our minds.
It provides a structured field of possibilities within which we participate.
Indeed, a film can teach us new conventions as we watch it.
A theme begins without a fixed significance.
Repeated association establishes one.
We learn it.
The film has changed our repertoire.
By the end, we can construe something that was unavailable to us at the beginning.
The work has therefore not simply communicated meaning.
It has helped create a new potential for meaning.
Meaning is historical
The film-score problem also reveals that meaning develops through time.
A motif heard for the first time is not the same as the motif heard for the fifth time.
Its significance depends upon its history.
A later scene can transform an earlier occurrence.
A theme can become ironic through repetition.
Its absence can become significant.
Its return can produce recognition.
Meaning accumulates.
The film becomes a history of its own semiotic relations.
The score is one of the devices through which the film can remember itself.
Music remains music
All of this nevertheless leaves one important distinction intact.
The music does not cease to be music when it becomes meaningful within the film.
Its rhythm remains rhythm.
Its harmony remains harmony.
Its timbre remains timbre.
Its temporal organisation remains active.
The symbolic function does not replace those properties.
It is added through participation in a larger system.
This is why we should resist both extremes.
Music is not simply a private emotional stimulus with no semiotic significance.
But neither must we turn music itself into a miniature language in order to explain its role in film.
The more interesting claim is:
music can be recruited into semiosis while retaining its own organisation and its own forms of mattering.
Meaning is not contained in the score
We can now return to the question we began to ask midway through the series:
What does the score mean?
Perhaps this is the wrong question.
A better one is:
What does the score become capable of meaning through its relations with the rest of the film?
That formulation changes everything.
A musical passage does not arrive with its filmic meaning already contained inside it.
Its possibilities are expanded and reorganised by the image, narrative, editing, cultural context and the history of its previous appearances.
The score becomes meaningful through participation.
The deeper lesson about meaning
And now the film-score problem returns us to the broader theory developed in From Mattering to Meaning.
We began there with the proposition:
Something can matter without meaning.
The film score confirms it.
Music already matters.
The film recruits that mattering.
Through relations with other modes, the music acquires symbolic functions.
Those functions can become conventional.
They can become narrative resources.
They can become part of the film's larger semiotic ecology.
And eventually the meaning feeds back into the music itself.
A familiar theme now matters partly because of what it means.
We therefore have:
mattering → meaning → new mattering
The circle has returned.
The score as a model of semiosis
The film score therefore gives us a small but revealing model of how semiosis can evolve.
A resource has its own organisation.
It matters in its own right.
It enters a new system.
Its possibilities are recruited.
Relations become stable.
Repetition creates convention.
Convention creates portability.
The sign becomes part of a larger system.
The system creates new meanings.
Those meanings alter what the original resource matters as.
This is not peculiar to music.
The same pattern can occur with images, gestures, objects, places, sounds and other modes.
Film simply gives us an unusually clear laboratory in which to observe it.
The final formulation
We can now answer the question with which the series began:
How does music make a film mean?
Not by translating itself into words.
Not by carrying a fixed emotional label.
Not by simply decorating the image.
It does so by bringing its own potential for mattering into relations with the film's other semiotic resources.
Those relations can reinforce.
Contradict.
Foreshadow.
Recall.
Identify.
Reframe.
Create irony.
Establish narrative structure.
And, through repetition, create new signs.
The resulting meaning is therefore neither simply in the music nor simply in the image.
It emerges from the organisation of their relations.
What the score has taught us
The score has taught us something larger than a theory of film music.
It has shown us that:
a thing need not be a symbolic system in isolation in order to participate in symbolic meaning;
a mode can bring its own forms of mattering into another semiotic system;
meaning can emerge between modes rather than within one mode alone;
repeated multimodal relations can become semiotic conventions;
and once established, those meanings can feed back into what the original resources matter as.
This gives us a more dynamic conception of semiosis.
Meaning is not a substance attached to forms.
It is something that emerges, stabilises, travels, transforms and feeds back.
And the shark swims on
We began with a shark moving towards us.
The music made the shark more threatening.
Then we discovered that the shark could make the music more threatening too.
The two became coupled.
The coupling could become conventional.
The convention could survive the image.
The music could evoke the shark in its absence.
The musical sign could enter narrative relations.
The film could become a semiotic ecology.
And the meaning produced by that ecology could change how the music itself mattered.
A remarkably simple scene therefore revealed a much larger principle:
Meaning can grow from the relations among things that already matter.
That is perhaps the deepest lesson of the film-score problem.
We do not begin with meaningless sound and manufacture meaning out of it.
We begin with organised forms of mattering.
We place them into new relations.
The relations generate new possibilities.
Some of those possibilities become meaningful.
And once they do, they can return to the world from which they emerged and change what matters there.
So perhaps the deepest formulation is the simplest:
Music can make a film mean because music already matters.
The film does not erase that mattering.
It recruits it.
And in doing so, it gives mattering another life — a semiotic life, a narrative life, a cultural life.
The score becomes part of the story.
The story becomes part of the meaning of the score.
And the meaning that emerges belongs, finally, to neither alone.
It belongs to the relation.