Imagine a shark swimming towards the camera.
Nothing particularly unusual happens. The animal moves through the water. We watch it approach. We see its body, its movement, its environment.
Now add music.
The same shark appears more threatening.
Remove the music, and the threat seems to recede.
It is tempting to say that the music has simply supplied an emotion: the shark was frightening, and the music made us feel frightened. But that description misses something more interesting.
The music has not merely accompanied what we see.
It has changed what we see.
That is the problem we will investigate in this series.
How can music participate in the meaning of a film when music and image are different semiotic modes?
The experiment
There is unusually good experimental evidence for the effect.
In 2016, Nosal, Keenan, Hastings and Gneezy published a study in PLOS ONE examining the effect of background music on viewers' perceptions of sharks. Participants watched the same 60-second footage of swimming sharks, but with different soundtracks: ominous music, uplifting music, or silence. Across three experiments, participants who saw the sharks with ominous music subsequently evaluated them more negatively and less positively than those who saw the same footage with uplifting music or silence. The researchers also included audio-only controls, finding that the difference was not simply an effect of listening to the soundtrack by itself.
The experimental design is important.
The sharks did not change.
The camera did not change.
The footage did not change.
What changed was the music.
And the viewers' responses to what they had seen changed with it.
The researchers consequently concluded that background music can affect how viewers interpret documentary footage of animals.
There is an important historical detail here. The experiment did not use John Williams's famous music from Jaws. The ominous and uplifting tracks were taken from documentary music, including the soundtrack of Blue Planet.
That makes the result particularly useful for us.
The effect does not depend upon a famous shark theme that audiences have already learned to associate with danger.
Something more basic is happening.
The music changes the construal
What, then, has the music changed?
Not the shark itself.
Not the visual information provided by the film.
What it changes is the way the visual information is construed.
The same movement can be understood differently depending upon the larger configuration in which it occurs.
A shark swimming towards us can be construed as:
an animal behaving normally;
or:
a threat approaching its prey.
The visual information does not determine the distinction by itself.
The music helps establish which possibilities become salient.
This is a much more interesting claim than "music creates an emotional mood".
It suggests that music can participate in the construction of the viewer's interpretation of the image.
But what does the music itself mean?
Here we encounter a difficulty.
It would be easy to say that ominous music means danger.
But that is not quite right.
Music certainly has the capacity to produce tension, expectation, arousal, anticipation and affective response. It also has cultural histories and conventions that can make some musical configurations ominous, triumphant, tender or unsettling.
But those capacities do not make music a semiotic system in the same sense as spoken language.
A musical passage does not need to contain a lexical equivalent of danger.
Its contribution can be less determinate and more open.
Perhaps this is why the shark experiment is so interesting.
The music does not seem to replace one proposition with another.
It alters the field within which the visual scene is construed.
Music before meaning
This takes us back to a distinction we have already encountered.
Music can matter before it means.
A rhythm can organise bodies.
A melody can create expectation.
A change in harmony can create tension.
A timbre can alter affective disposition.
A recurring musical form can become associated with memory.
None of this requires a symbolic reference.
Music already has its own potential for mattering.
The film does not have to turn music into a miniature language before it can use music meaningfully.
It can recruit the things music already does.
The score can bring temporal expectation, affective orientation, cultural association and memory into relation with what is being shown.
This suggests a hypothesis we will return to repeatedly:
A film can recruit the existing potential of music for mattering into a larger system of meaning.
The image does not remain unchanged
This also forces us to reconsider a common picture of the film score.
We often speak as though the image carries the information and the music supplies atmosphere.
The image tells us what is happening.
The music tells us how to feel about it.
Sometimes that is approximately true.
But the shark experiment suggests that the relationship can be more intimate.
The music can influence not merely our feelings about a visual event, but our construal of the event itself.
The shark does not first appear as a fully determined object and then receive an emotional label from the soundtrack.
The soundtrack helps determine which aspects of the visual event become salient.
The viewer does not simply see:
shark + fear
but construes:
shark as threat.
That little word — as — is important.
The music participates in the construal.
A second line of evidence
The shark study is not an isolated finding.
Earlier research on music and visual perception found that music-induced mood can bias visual perception itself. In one experiment, participants listening to happy or sad music were better at detecting facial expressions congruent with their induced mood, and their false visual detections were also biased in the same direction. The researchers concluded that music-induced mood could influence visual processing and even the content of conscious visual experience.
More recent neuroscientific work has similarly investigated how consonant and dissonant music can influence interpretation during film viewing, finding evidence that auditory information can contribute to the processing of visual and social information.
The details of these studies differ.
But they converge on an important point:
what we hear can affect how we see.
That is already enough to make the conventional distinction between "soundtrack" and "picture" look rather inadequate.
The score does not simply decorate the image
Suppose a filmmaker shows an ordinary landscape.
With one musical treatment, it seems peaceful.
With another, ominous.
With another, nostalgic.
With another, comic.
The landscape remains visually identical.
Yet it becomes a different semiotic object.
The music has helped determine what sort of scene we are watching.
This is why "mood music" is both useful and misleading as a description.
Useful, because music certainly can alter mood.
Misleading, because the result can extend beyond the viewer's internal feeling.
The music can alter the construal of the represented world.
And the relationship can work in reverse
We should resist turning this into a one-way theory in which music acts upon passive images.
The image can also change the way we hear the music.
A musical passage that sounds melancholy in isolation may become tender when paired with a particular image.
The same passage may sound sinister beneath another.
A triumphant musical gesture can become grotesque when placed over destruction.
A quiet melody can become threatening when something terrible is happening just outside the frame.
The relation is therefore reciprocal.
Music changes the construal of image, and image changes the construal of music.
This is where we begin to approach the genuinely multimodal problem.
What belongs to the music?
We should therefore resist asking too quickly:
What does this music mean?
Perhaps a better question is:
What possibilities does this music make available when it enters the film?
It can create expectation.
It can establish tension.
It can invoke memory.
It can suggest genre.
It can alter affect.
It can establish associations.
It can make one interpretation more probable than another.
These are resources.
The film can recruit them.
And when it does, they enter into a larger semiotic configuration.
The resulting meaning is not necessarily a property of the music alone.
The score is recruited, not translated
This distinction may become central to the series.
When music participates in a film's meaning, it is tempting to say that the music has been translated into a visual meaning.
But that is not really what happens.
The music remains music.
Its rhythmic, harmonic, timbral and temporal properties remain active.
The image remains an image.
Its spatial, visual and representational properties remain active.
The film places them in a relation.
Meaning can then emerge from that relation.
The music has not been translated into the image.
It has been recruited into a larger semiotic system.
The Jaws complication
This brings us to a fascinating further complication.
John Williams's music from Jaws has become culturally associated with the shark and with approaching danger. The 2016 shark study itself invokes Jaws as a classic example of leitmotif: repeated pairing between the music and the shark can allow the musical figure eventually to evoke the character even when the character is absent.
Here something different has happened.
The music first participates in a multimodal relation:
music + shark → a particular construal
But through repetition, the association can become sufficiently established that the relation works in the opposite direction:
music → shark
The image is no longer necessary.
The music can invoke what is absent.
This will be one of the major questions of the series.
From interaction to convention
We can therefore already see at least two different phenomena.
The shark experiment demonstrates that music can change the construal of visual material.
The Jaws example points toward something further: repeated audiovisual association can allow music to acquire a more conventional symbolic function within the film's cultural afterlife.
These should not be collapsed.
The first concerns multimodal interaction.
The second concerns semiotic stabilisation.
Between them lies a fascinating developmental problem.
How does something that initially participates in meaning through its relation to another mode become capable of carrying some of that significance when the other mode is absent?
The question that opens the series
We therefore have an empirical phenomenon that is simultaneously simple and profound.
The shark remains the same.
The music changes.
The viewer's construal changes.
But the music does not need to "say" anything about sharks in isolation.
Something else is happening.
A musical resource that already possesses capacities for mattering enters into a relation with a visual resource.
The two constrain one another.
A new construal emerges.
And if the relation is repeated, it may become part of the semiotic repertoire through which later viewers encounter the image.
That gives us the central problem of the series:
How does a film recruit the potential of music into the meaning of an image?
And perhaps the deeper question is:
When meaning emerges between modes, where does that meaning reside?
Is it in the music?
In the image?
In the viewer?
In the cultural conventions surrounding both?
Or is it, more interestingly, in the relation among them?
That is where we will go next.
Because before we can understand what music makes a film mean, we need to ask a more basic question:
What does music bring to the picture in the first place?
No comments:
Post a Comment