Multimodal Horizons: A Relational History of Semiotic Complexity: 1 Foundations of Multimodality: From Gesture to Symbol

Before there were words, there were gestures, traces, and enactments — ways of participating in and modulating relational fields rather than representing them. Early humans, like other social species, coordinated attention, intention, and affect through bodily movement, posture, and vocalisation. These proto-semiotic behaviours had a content plane and an expression plane, as in many species, but the content was not yet stratified: meaning and expression were co-occurrent, inseparable, and lived in the act itself.

The emergence of language proper, with its stratified content plane (semantics and lexicogrammar), transformed these proto-semiotic capacities. As Halliday describes, the junctional phenomenon of metaphor allowed a wording to realise both its congruent and metaphorical meanings, establishing a token–value relation within semantics. This reflexive fold — meanings standing for other meanings — created the preconditions for symbolic extension across modes: images, objects, ritual acts, and early notation could now interact with language in novel, cross-modal ways.

Gestures and enactments, previously indexical and participatory, began to acquire potential for symbolic extension. A hand movement could become a sign; a mark on the ground could encode a sequence of actions; a painted figure could stand for a type rather than merely enact it. Early multimodal semiotics was thus grounded in participatory alignment, scaffolding social cohesion, affective flow, and shared attention — the non-semiotic substrate identified in Edelman’s value systems. Language, emerging within this substrate, provided a reflexive anchor, enabling meanings to relate to other meanings, and ultimately enabling cross-modal hybridisation.

From this foundation, human semiotic practice began to coalesce into layered systems: gesture, image, and vocalisation interacting within social, temporal, and affective networks. Each mode enhanced the potential of others, producing relational patterns that could be iterated, recombined, and reflected upon. In these early multimodal fields, humans were not simply communicating; they were modulating worlds, co-individuating patterns of attention, affect, and potential action.

The key insight from a relational perspective is that multimodality did not emerge as a system of representation. Instead, it emerged as a system of relational alignment, later augmented by symbolic stratification. Gesture, artefact, and proto-image coexisted with emergent language, forming the matrix in which semiotic complexity could flourish. This foundational interplay set the stage for all subsequent developments in writing, performance, notation, and mediated communication.

In sum, the beginnings of multimodal semiotics are participatory, relational, and emergent, grounded in human sociality and bodily coordination. Language introduces reflexivity, creating the conditions for symbolic interplay across modes, but the affective, temporal, and relational substrate remains primary. Understanding this substrate is essential: it reveals that multimodal semiotic complexity is not a given, but an achievement of co-individuation across modes, bodies, and worlds.