Multimodal Horizons: A Relational History of Semiotic Complexity: 4 Notation, Script, and Symbolic Codification

The emergence of notation and script represents a pivotal moment in the relational history of multimodal semiotics. Where gesture, image, and vocalisation had previously modulated relational fields directly, codified symbols introduce stable, recombinable, and transportable forms that can interact across time, space, and modality. Writing, musical notation, and other systems of codification extend the reflexive potential introduced by language, allowing semiotic relations to be objectified, transmitted, and systematically recombined.

Halliday’s account of language stratification illuminates this process. With the content plane folded into semantics and lexicogrammar, meanings can stand for other meanings. Notation harnesses this reflexive capacity: a grapheme, a note, or a sign becomes a token within a structured system, encoding relations between actions, events, or concepts. Codified symbols do not replace embodied participation or performative enactment; rather, they mediate, amplify, and distribute these semiotic potentials. A musical score, like a written narrative, enables temporal extension, cross-modal interaction, and cumulative construction of meaning.

The development of script and notation also transforms the semiotic ecology itself. Social coordination, previously grounded in immediate bodily and affective alignment, can now operate across communities separated by distance and time. Codification scaffolds collective memory, rehearsal, and innovation, creating fields in which interdependent modes of meaning-making — linguistic, visual, gestural, and auditory — can cohere and interact. This makes multimodal systems more recursive, generative, and historically extended, yet always grounded in the social, affective, and temporal substrates of human life.

A critical feature of notation is its capacity for abstraction and recombination. Symbols, once standardised, can be manipulated independently of immediate context, allowing performers, readers, or interpreters to instantiate meanings in novel configurations. Musical notation, for example, transforms ephemeral sound into a recombinable semiotic resource, permitting layering of rhythm, harmony, and gesture across ensembles, generations, and geographies. Similarly, written narrative allows plots, themes, and motifs to be reused, recombined, and iteratively extended, producing relational fields of unprecedented complexity.

Yet the reflexive codification introduced by notation is never divorced from participation. Scores are performed, texts read aloud, diagrams enacted. Codified symbols modulate rather than replace human coordination, social alignment, and affective resonance. The power of notation lies in its ability to bridge immediacy and abstraction, enabling relational patterns to persist, be transmitted, and be transformed while remaining embedded in human semiotic life.

In sum, the emergence of notation and script illustrates a crucial evolutionary stage in multimodal semiotics. By harnessing the reflexive potential of language, codification creates portable, recombinable, and historically extended semiotic resources, allowing humans to scaffold relational worlds, extend temporal reach, and integrate modalities. Far from a mere technical innovation, notation represents a profound amplification of semiotic agency, transforming the ways humans can construe, transmit, and co-individuate meaning across space, time, and media.

Multimodal Horizons: A Relational History of Semiotic Complexity: 3 Visual Semiotics: Image as Relational Extension

With the reflexive capacities introduced by language, visual modalities could enter a new semiotic terrain. Images, once primarily indexical traces or participatory enactments, now had the potential to interact with symbolic content, aligning with the stratified semantic structures language afforded. A painted figure, a carved motif, or a patterned design could no longer be seen as merely reflecting immediate action; it could stand in relational alignment with meanings beyond the present act, participating in broader, temporally extended semiotic fields.

Early visual semiotics emerges from the coalescence of bodily experience, social coordination, and emergent symbolic capacity. Cave paintings, abstract motifs at Newgrange, Aboriginal Dreaming designs, and Native American sand drawings exemplify this interplay. They are not simply representations of the visible world; they are relational enactments, shaped by attention, sociality, and emerging symbolic structures. Language enables these visual forms to construe relations, encode temporal sequences, and mediate meanings, transforming participation into co-individuated semiotic fields.

The reflexivity introduced by stratified language allows cross-modal reinforcement: image and speech, mark and ritual, object and enactment, begin to interact within a semiotic ecology. The painted bison, the handprint silhouette, the spiral motif — these are tokens within an emergent relational system, aligning perception, memory, and potential action. The capacity for metaphor in language mirrors the potential for symbolic extension in image, creating a scaffold for multimodal semiotic complexity.

Crucially, this extension does not render the non-semiotic substrate obsolete. Bodily coordination, affective resonance, and temporal entrainment remain central, grounding visual forms in participatory experience. Images act as interfaces between lived reality and symbolic structuring, mediating action, perception, and interpretation. Their relational power derives from alignment rather than representation, from co-individuation rather than passive reflection.

Over time, visual semiotics becomes reflexively integrated with other modes. Gesture, performance, ritual, and spatial organisation interact with visual forms, producing temporally extended, relationally coherent semiotic fields. The potential for abstraction and recombination grows, enabling innovation in narrative, notation, and symbolic architecture. The emergence of perspective, formalised motifs, and codified symbolic conventions reflects the ongoing stratification and reflexivity first catalysed by language.

In sum, visual semiotics, when viewed relationally, is an extension of human participatory and symbolic capacities. Images do not merely depict; they mediate relational fields, enable temporal layering, and co-individuate meaning across modalities. Their evolution demonstrates how human semiotic systems, scaffolded by language, expand the horizons of possibility, integrating perception, action, memory, and symbol into the growing complexity of multimodal life.

Multimodal Horizons: A Relational History of Semiotic Complexity: 2 Language as the Semiotic Catalyst

Language, as Halliday elucidates, represents a revolution in the structuring of meaning, not merely a system of labels for pre-existing phenomena. The stratification of its content plane into semantics and lexicogrammar enabled the first true junctional phenomena: lexical and grammatical metaphors, where a wording realises both its congruent and metaphorical meanings. This reflexive fold allowed humans to relate meanings to meanings, opening a new horizon for semiotic interplay across modes.

Before this reflexive stratification, gesture, vocalisation, and material artefacts functioned primarily as indexical or participatory acts. Each mode modulated relational fields directly, enacting alignment among participants without standing for anything beyond the immediate interaction. Language introduced a new layer of abstraction, permitting objectification of meaning. A sign could now denote a type rather than an instance; a pattern could encode a relation rather than a single act. This enabled cross-modal extension: gestures could now interact with symbolic content, images could be interpreted in relational terms, and early notation could coordinate temporal sequences with semantic precision.

Crucially, language did not replace the non-semiotic substrate identified by Edelman. Social coordination, affective value exchange, and temporal entrainment remained fundamental. Rather, language amplified and reconfigured these relational potentials. It permitted semiotic fields to become recursive, recombinable, and reflective, providing a scaffold upon which multimodal interactions could be conceptually extended and temporally layered.

Through this catalytic role, language enabled multimodal semiotics to evolve from participation to symbolic interplay. Early ritual, theatre, and performance are visible manifestations of this process: vocalisation, gesture, costume, and spatial arrangement interact to create temporally extended, relationally coherent fields of meaning. Each mode enhances the other, producing synergistic patterns that exceed the capacities of any single mode.

The reflexive power of language also introduced historical depth. Symbolic relations could now be recorded, transmitted, and recombined, allowing humans to construct semiotic systems that were both temporally and socially extended. A myth told in speech could be echoed in image; a gesture could be codified into notation. Language, therefore, acts as a semiotic catalyst, mediating between the non-semiotic substrate of participation and the emergent, cross-modal complexity of multimodal systems.

In sum, the stratification of language transformed the landscape of human semiotic potential. It provided the junctional architecture for meanings to interact reflexively, creating the conditions under which multimodal semiotic systems could cohere, diversify, and innovate. Language catalyses multimodal semiotics by enabling abstraction, cross-modal integration, and recursive extension — a foundation upon which the rich complexity of human symbolic life is built.

Multimodal Horizons: A Relational History of Semiotic Complexity: 1 Foundations of Multimodality: From Gesture to Symbol

Before there were words, there were gestures, traces, and enactments — ways of participating in and modulating relational fields rather than representing them. Early humans, like other social species, coordinated attention, intention, and affect through bodily movement, posture, and vocalisation. These proto-semiotic behaviours had a content plane and an expression plane, as in many species, but the content was not yet stratified: meaning and expression were co-occurrent, inseparable, and lived in the act itself.

The emergence of language proper, with its stratified content plane (semantics and lexicogrammar), transformed these proto-semiotic capacities. As Halliday describes, the junctional phenomenon of metaphor allowed a wording to realise both its congruent and metaphorical meanings, establishing a token–value relation within semantics. This reflexive fold — meanings standing for other meanings — created the preconditions for symbolic extension across modes: images, objects, ritual acts, and early notation could now interact with language in novel, cross-modal ways.

Gestures and enactments, previously indexical and participatory, began to acquire potential for symbolic extension. A hand movement could become a sign; a mark on the ground could encode a sequence of actions; a painted figure could stand for a type rather than merely enact it. Early multimodal semiotics was thus grounded in participatory alignment, scaffolding social cohesion, affective flow, and shared attention — the non-semiotic substrate identified in Edelman’s value systems. Language, emerging within this substrate, provided a reflexive anchor, enabling meanings to relate to other meanings, and ultimately enabling cross-modal hybridisation.

From this foundation, human semiotic practice began to coalesce into layered systems: gesture, image, and vocalisation interacting within social, temporal, and affective networks. Each mode enhanced the potential of others, producing relational patterns that could be iterated, recombined, and reflected upon. In these early multimodal fields, humans were not simply communicating; they were modulating worlds, co-individuating patterns of attention, affect, and potential action.

The key insight from a relational perspective is that multimodality did not emerge as a system of representation. Instead, it emerged as a system of relational alignment, later augmented by symbolic stratification. Gesture, artefact, and proto-image coexisted with emergent language, forming the matrix in which semiotic complexity could flourish. This foundational interplay set the stage for all subsequent developments in writing, performance, notation, and mediated communication.

In sum, the beginnings of multimodal semiotics are participatory, relational, and emergent, grounded in human sociality and bodily coordination. Language introduces reflexivity, creating the conditions for symbolic interplay across modes, but the affective, temporal, and relational substrate remains primary. Understanding this substrate is essential: it reveals that multimodal semiotic complexity is not a given, but an achievement of co-individuation across modes, bodies, and worlds.

The Sound of Value: A Relational History of Music: 9 Afterword: Music as Relational Technology and Semiotic Laboratory

The journey through the relational history of music reveals a continuously unfolding interplay between social coordination and semiotic abstraction. From the earliest rhythmic entrainment and melodic gestures to polyphony, improvisation, and notated composition, music demonstrates how humans extend, explore, and amplify relational potential across temporal, affective, and social dimensions.

Music is first and foremost a technology of relational alignment. It modulates attention, synchrony, and affective flow among participants, sustaining collective engagement and the circulation of value. Edelman’s conception of value systems highlights this foundation: musical experience structures affective and social exchanges, enabling coordination, cohesion, and shared resonance without invoking explicit denotation. The field of music is therefore inherently participatory, temporal, and emergent, grounded in human interaction and communal attunement.

Simultaneously, music has evolved into a laboratory for semiotic and symbolic exploration. Notation, theory, and improvisational practice allow reflection, recombination, and trans-temporal experimentation. Polyphony, counterpoint, and harmonic design create abstract relational patterns that extend beyond immediate experience, providing vehicles for recursive attention, anticipation, and creativity. Here, semiotic structures are tools for manipulating relational complexity, expanding the scope of human experimentation while remaining anchored in social practice.

The dual trajectory of music — social and semiotic — demonstrates the co-emergence of non-semiotic participation and symbolic abstraction. Social engagement provides the substrate for meaning-making, while semiotic techniques enable reflection, recombination, and the elaboration of temporal and relational horizons. Music, in this sense, is both experience and experiment, practice and theory, medium and method.

Viewed through a relational ontology, music exemplifies how humans extend their capacity to construe, modulate, and transform relational worlds. It is a mirror of collective life, reflecting and shaping attention, affect, and coordination; it is also a laboratory of innovation, permitting new configurations of interaction, structure, and temporal organisation. Each note, phrase, and ensemble interaction is simultaneously social gesture and semiotic probe, a point of alignment, divergence, and creative possibility.

In conclusion, the relational history of music affirms that the human engagement with sound is not merely representational or expressive in a semiotic sense, but fundamentally participatory, relational, and experimental. Music actualises the intersection of collective life and symbolic potential, offering a field in which humans can explore the dynamics of interaction, the modulation of value, and the expansion of temporal and relational horizons. As both technology and laboratory, music embodies the infinite generativity of relational worlds, revealing the power of sound to structure experience, shape collective life, and open new dimensions of possibility.

The Sound of Value: A Relational History of Music: 8 Music, Emotion, and Social Value: The Non-Semiotic Foundations Revisited

While notation, theory, and improvisation situate music within the semiotic plane, its deepest roots remain non-semiotic, affective, and social. Music in itself does not stand for meanings; it modulates attention, synchrony, and relational alignment, shaping the flow of social interaction and the circulation of value among participants. Edelman’s framework of value systems provides a lens for understanding this: music orchestrates the exchange of affective significance, coordinating collective experience without invoking symbolic reference.

Rhythmic entrainment exemplifies this principle. Heartbeats, footsteps, and bodily gestures resonate within groups, creating temporal alignment and facilitating mutual orientation. Melodic contours, dynamic shifts, and harmonic tension guide affective flow, establishing peaks, resolutions, and shared moments of intensity. These processes mediate social cohesion, influence decision-making, and scaffold coordinated action, even in the absence of explicit denotation or semiotic content.

Music’s power lies in its capacity to amplify relational and temporal dynamics. Unlike language, which symbolises and abstracts experience, music operates directly within the field of participation, shaping interaction through temporally extended patterns of alignment and divergence. Its efficacy emerges not from referentiality but from the sensitive modulation of social and affective currents — a technology of relational attunement that humans have refined across cultures and epochs.

The semiotic innovations of notation and theory enhance but do not supplant this foundation. They allow reflection, recombination, and cross-temporal communication, yet the social and affective substrate remains indispensable. Even the most abstractly structured compositions — polyphonic fugues, complex jazz improvisations, or digitally mediated soundscapes — are experienced and sustained within networks of social attention, bodily coordination, and affective resonance.

Viewed relationally, music exemplifies a dual ontology:

1. Non-semiotic sociality — the modulation of attention, affect, and value exchange through embodied and temporal patterns;

2. Semiotic extension — the symbolic representation, codification, and manipulation of musical patterns via notation and theory.

This duality explains music’s enduring significance. It anchors human interaction, amplifies affective and attentional flow, and scaffolds collective life, while simultaneously extending human capacities for abstraction, experimentation, and temporal projection. Music is thus both practice and symbol, social technology and laboratory of relational potential.

In sum, the relational history of music shows how affective, social, and symbolic layers co-emerge, interact, and transform one another. Music actualises the potential for coordinated social experience, provides a semiotic canvas for reflective creativity, and amplifies the human capacity to modulate, explore, and reconfigure relational worlds. It demonstrates that human culture is not merely a matter of representation, but of participation, alignment, and the ongoing negotiation of value — the very processes that underpin both social cohesion and the emergence of symbolic thought.

The Sound of Value: A Relational History of Music: 7 Notation, Improvisation, and Reflexive Performance: Music as Symbolic Experimentation

The duality of music — social coordination and semiotic abstraction — reaches a sophisticated stage when notation meets improvisation. Notation preserves patterns, formalises relationships, and provides a stable symbolic substrate. Improvisation activates these patterns, generating novel relational configurations within performance. Together, they exemplify music as reflexive experimentation, where sound, time, and relational alignment are simultaneously structured, enacted, and transformed.

Improvisation relies upon embodied mastery of rhythmic, melodic, and harmonic patterns. Musicians navigate these structures, anticipating outcomes, responding to fellow performers, and dynamically modulating affective and attentional flow. Here, the non-semiotic foundations of music — entrainment, synchrony, and collective modulation — are actively repurposed for semiotic ends: improvisers manipulate motifs, sequences, and textures as objects of reflection and recombination, producing meaning-like effects without invoking explicit denotation.

Notation provides a cognitive scaffold: it enables musicians to treat sound events as discrete, manipulable units, to experiment with relationships that extend across time and ensemble. Improvisation becomes a dialogue between constraint and freedom: the semiotic forms of theory and notation delineate possibility spaces, while real-time performance allows for emergent creativity within those spaces. This interplay mirrors Halliday’s account of metaphor as a junctional phenomenon: the congruence of structure and inventive deviation produces novel configurations of relational significance.

The reflexivity of performance is crucial. Musicians are not merely enacting predetermined sequences; they are simultaneously participants, analysts, and co-creators. Each gesture resonates socially, emotionally, and temporally, shaping collective experience while exploring the bounds of semiotic representation. Music becomes a laboratory of relational possibility, testing, modulating, and extending patterns of social and symbolic interaction in real time.

From a relational perspective, this interplay demonstrates how non-semiotic and semiotic layers interweave dynamically. Social entrainment sustains the flow of interaction; semiotic structures allow reflection, recombination, and cross-temporal continuity; improvisation mediates between the two, generating novel configurations of alignment and divergence. The music itself becomes a medium for exploring relational potential, a field in which attention, affect, and coordination are consciously and creatively modulated.

In sum, the convergence of notation, improvisation, and reflexive performance illustrates music’s unique position at the intersection of sociality and symbolism. It is simultaneously a technology of collective regulation, a semiotic system for representing sound, and a laboratory for exploring relational complexity. Through this interplay, music enables humans to experiment with worlds of interaction, extend temporal and social horizons, and actualise potentials that are neither fully representational nor purely social, but intrinsically both.

The Sound of Value: A Relational History of Music: 6 Harmony and Polyphony: Relational Complexity Amplified

With the advent of notation and theory, music enters a space where multiple lines of sound can be conceived, manipulated, and coordinated independently. Harmony and polyphony exemplify the amplification of relational complexity, extending the social and temporal scaffolds of early rhythm and melody into multi-dimensional textures that challenge perception, expectation, and affective alignment.

Harmony is the simultaneous juxtaposition of pitches, creating vertical relationships that modulate tension, resolution, and affect. Polyphony, in contrast, is independent melodic lines interacting over time, a temporal layering that demands both coordination and differentiation. Both phenomena exemplify how music can construe relational patterns across multiple dimensions: pitch, time, intensity, and timbre, while remaining grounded in the non-semiotic social substrate of collective engagement.

These developments deepen music’s capacity to phase attention and expectation. Participants — whether performers or listeners — track concurrent patterns, anticipate resolutions, and negotiate relational tensions across multiple strands. The field of experience becomes nested and recursive: individual lines, harmonic clusters, and contrapuntal textures are simultaneously independent and interdependent, creating rich, emergent relational dynamics.

Polyphony also magnifies music’s temporal horizon. Independent lines can extend across measures, movements, or entire compositions, requiring sustained attentional engagement and the ability to mentally integrate multiple streams of relational information. The social analogue is clear: just as music can coordinate multiple voices in abstract interaction, human groups can coordinate multiple actors, agendas, and relational networks over extended periods.

Harmony and polyphony demonstrate how semiotic tools — notation and theory — expand the potentialities of non-semiotic musical practice. By abstracting and formalising pitch relationships, music becomes capable of combinatorial creativity: composers can generate textures, counterpoint, and harmonic progression independently of performance, while performers and listeners can apprehend complex structures as coherent, meaningful, and emotionally compelling. Yet the emotional and social grounding remains: tension and resolution in sound mirror relational dynamics in social life, resonating through the exchange of attention, arousal, and affective engagement.

Viewed relationally, harmony and polyphony are meta-structural innovations. They extend the dimensionality of collective experience, scaffold multi-layered coordination, and enable recursive modulation of relational states. They exemplify music’s dual nature: firmly anchored in social interaction and value circulation, yet capable of semiotic abstraction, reflection, and anticipatory creativity.

In sum, the emergence of harmony and polyphony marks a new stage in the relational history of music. Building on rhythm, melody, and synchrony, these innovations multiply the possibilities of social coordination, attention modulation, and aesthetic experience. They demonstrate how non-semiotic and semiotic layers interweave, amplifying human potential to construe, modulate, and reflect upon relational worlds — making music a vehicle for both collective life and symbolic imagination.