BERLIN GESTURE CENTER
Interdisziplinäres BGC-Kolloquium
Freitag, 10. November 2006

The Optic, Haptic, and Acoustic Dimensions of Gestures: Evolutionary Significance and Methodological Implications

Paul Bouissac
(University of Toronto, Victoria College)

The scientific study of gestures has been so far concerned with “what” and “how” rather than with “why” questions. This field of inquiry is presently subdivided in several paradigms concerned with definitions, classifications and descriptions of gestures as visually perceived phenomena. The theory of signs and communication and functionalist linguistics provide most of the implicit models through which gestures are conceptualized and analyzed. Within these epistemological limits, a high degree of sophistication has been reached in understanding, for instance, the interface between gesture and speech and in investigating cultural repertoires.
The functions and meanings of gestures are principally construed in these research endeavors as pertaining to visually perceived movements of the upper limbs and the hands, and the identifiable patterns they form in space. Such patterns can be labeled or coded for the purpose of macro-analysis. When reference is made to the multimodality of communication, it is usually meant that the acoustic channel of speech is combined with the visual channel of gestures, and possibly with other sensory modalities such as olfaction or touch in order to produce a complex meaning.
But gestures themselves can be multimodal and include acoustic dimensions, for instance, as an essential part of their meaning in the sense that this meaning results from both a visually perceived hand movement and the sounds it produces. Many examples come to mind such the clapping of the hands which conveys a range of degrees of appreciation depending on the pace and intensity of the clapping, and the level on which it is performed with respect to the body. Another appreciative gesture consists of knocking on a table as is customary in Germany. Other examples include the Chinese gesture of thanks (tapping on the table with the tip of the fingers), the snapping of the fingers (with a variety of communicative functions), the rubbing of the hands, the clicking of the fingernails, the hitting of teeth with a fingernail, a fist crashing in the opposite palm, or even the pounding of the chest with one’s two fists in imitation of a gorilla’s dominance display. All these gestures can intentionally vary the acoustic output to the point that it can sometimes be reduced to zero in order to undermine its accepted meaning.
A similar observation can be made with respect to the haptic (touch) modality which is inherent in many gestures. Most obvious are embracing, shaking hands, tapping on the arm or shoulder, or holding hands. Another striking example for a Western observer is the touching of the feet or the knees as a greeting gesture toward a superior that is commonly practiced in India. In all these cases there are many degrees of duration and intensity of contact, each of which conveys a differential meaning not only to the interactants but also to the observers. In fact several highly significant gestures can be shown to be the result of haptic inhibitions or from ritualization. Since touch always implies a contact between two surfaces, all haptic gestures necessarily also have an acoustic dimension: shaking hands, high fives, rubbing or tapping on the shoulder can be performed with meaningful sound variations.
From the above remarks, it is possible to heuristically formulate the hypothesis that the notion of gestures as purely visual phenomena is a convenient abstraction, and that gestures are essentially multimodal and involve the complete range of sensorial modalities including the olfactive channel in addition to the haptic and acoustic ones. Indeed all movements of the upper limbs contribute to the dispersion of pheromones, natural and artificial scents, and to the circulation of environmental smells. For instance, the handling of smoke coming from the combustion of tobacco or other plants offers a range of social behaviors that are highly meaningful through their capacity to operate certain degrees of fusion or fission among the interactants.
From a methodological point of view, the above remarks should lead to more inclusive observations than is generally the case. The foregrounding of the optic qualities of gestures reflects the importance for humans of the visual medium in the processing of information coming from the environment, but it is also in part due to the recording convenience of drawings, photographs, and film. This does not mean that haptic, olfactive and acoustic gestural information is not processed by the brain with great attention to differential qualities. But their meanings are not made fully explicit through conceptual models and lexical categories, and are assigned sometimes to intuitions that are difficult to articulate such as whether trust or distrust is conveyed by a particular gestural interaction. These dimensions come to our full awareness in the metalanguage of cross-cultural communication. For instance, a Chinese handshake is characterized haptically by the absence of muscular contraction, a feature that in some western cultures is interpreted as a marked lack of friendliness, but where, conversely, an excessive contraction can “crush” the hand that is seized as a sign of aggression or dominance in the guise of politeness.
Most scientific endeavors are based on methodic phenomenological descriptions that transform raw information into standardized data, and on the understanding of the processes that account for the data. The scientific study of gestures requires a precise anatomic knowledge and a good understanding of the neuro-physiological processes which sustain the appropriate performance of gestures and the adaptive integration of multimodal information. But no scientific undertaking is complete until it can move from the “what” and “how” questions to the “why” question. The next remarks will outline a tentative theory of the origin of gestures.
Gestures are a function of the upper limbs whose morphology has evolved millions years ago. The paleontology of the tetrapods (four limb organisms) is fairly well known through the abundance of fossils that make it possible to retrace their successive adaptations to aquatic, terrestrial and arboreal environments. This is not a unilinear process since there is evidence that evolutionary pressures drove some organisms back to the previous environment from which they had emerged. Primates are tetrapods who adapted to arboreal life, and the structure and physiology of their limbs are evidence of this adaptation, as are the characteristics of their visual system. It does not seem plausible that the gestural communication was the primary source of the natural selections that molded their upper limbs although it may have played a reinforcing role in as much as they were social species. With Homo erectus, a new adaptation to terrestrial mobility, bipedalism, had evolved, whose causes and consequences are still debated. The point of interest with respect to the evolutionary account of Homo is that gestures exploit a dynamic and articulatory potential of the limbs that obviously evolved as adaptations which were not primarily communicative.
It is indeed generally accepted that the common ancestor of apes, monkeys and humans was a relatively small mammal which had adapted to arboreal environment. The identification of this common ancestor is the object of debates among paleontologists because only scant evidence is presently available. The most ancient fossil that may qualify for this ancestral status, Eosimias sinensis, lived some fifty million years ago and weighted about one hundred grams. Another, more recent arboreal species, Proconsul africanus, had an average body mass of nine kilograms and lived some twenty million years ago. The latter is known through a very large number of fossils that yield considerable information concerning its ecology and behavior. Limbs and hands were crucial for catching preys or picking fruit, processing food and feeding, moving, grasping, climbing and keeping balance, for attack and defense, holding to each other as infants or mates, grooming and interacting directly through touch. An arboreal environment is usually dense and does not allow the visual distance of open space. Furthermore the basis of support can be very narrow and considerably constrains proxemic variations within group interactions.
Evolution is very conservative even though small genetic variations can make a great difference. This is obvious as far as biology is concerned but not less true of behavior. Humans have inherited from their evolutionary lineage a set of capacities that were highly adaptive in various niches within the arboreal environment but do not necessarily fit perfectly all environments to which they have simply “exapted”. A case in point is the visual system with its range of color discriminations and the sense of perspective that is a part of the package. Recent research has shown that the complex neuronal computation of the sense of perspective is accurate only within a certain angle and over a certain distance for the purpose of aiming either in order to catch a proximal object or to hit a distal object with a projectile. The range within which this computation is optimal corresponds to what can be expected from an organism which needs to negotiate arboreal space rather than open space such as the savannah. It is well known that humans are poor visual evaluators of distances beyond a certain range through vision alone. Regarding chromatic perception, humans inherited also a capacity to discriminate colors that is congruent with the needs of a predator of fruit, insects, seeds, leaves, birds and other arboreal resources.
The vast repertory of skilled movements with which natural selection had endowed Homo during his long evolution in arboreal environments was recycled as he became exclusively an erect terrestrial organism. Some fossil behaviors, such as the grasping reflex, have been preserved. This is also true, for instance, of gestures relating to bonding, affiliation, and alliance making; seduction and pre-copulation behavior; challenge, dominance and submission. In human social behaviors, all these movements pertain to the hands and involve actual or implied touching. Or displaced touching when the contact is made with an object rather than a person and becomes fully ritualized. Even communicative gestures that are purely performed “in the air” exhibit features that bear the mark of haptic communication: for instance, the deceleration that is required for the “soft landing” process since catching or touching an object necessitate precise neuronal computation of distance and velocity. Errors in doing so can impair the hands, and a tree dwelling primate cannot survive long without functional hands.
Regarding communication, an arboreal environment can be assumed to select acoustic signals since the visual range is limited and constrained by branches and foliage. “Phatic” communication, territorial signaling, bluffing in conflict, etc., must rely on sounds rather than visual displays, and non-vocal sounds are produced by gestures. This earlier “semiotic layer” in human communication has been reactivated in the newly created audio environment of telecommunication…until, of course, technology made visual displays available again to the interactants.
Among the consequences of bipedalism for the further evolution of Homo from H. erectus to H. sapiens is the freeing of the hands from the demanding locomotion functions. It has been the locus of many innovations, in particular the construction of artificial niches and tools to meet the demands of the new open space environment. But until the very recent urban explosion, the gestural requirements for reproductive and associative group formation and maintenance that primates evolved over at least sixty million years, remained relevant to the survival of Homo sapiens. From this point of view it could be claimed that most social gestures are fossil behaviors which, indeed do not require language for being understood and are cross-culturally observed where they are not suppressed by cultural taboos.
An intriguing question concerns the conservation of organs which evolved under a particular set of environmental constraints and are preserved in a totally different environment. It can be assumed that the new environment has sustained the results of earlier natural selection by providing some additional constraints. In the case of gestures, thermoregulation and social displays can be construed as factors in the conservation of the anatomy of the upper limbs when they ceased to be vital for locomotion. Therefore, it could be considered that the human gestural endowment evolved new purely semiotic functions (in addition to thermoregulation) without losing its more primitive functions (minus locomotion once infants start walking). This evolutionary perspective should be kept in mind when raising the issue of how gestures relate to articulate language.

Selective bibliography

Beard, Chris (2004). The Hunt for the Dawn Monkey: Unearthing the Origins of Monkeys, Apes, and Humans. Berkeley: University of California Press.
Berthoz, Alain (2000). The Brain’s Sense of Movement. Translated by Giselle Weiss. Perspective in Cognitive Neuroscience. Cambridge: Harvard University Press.
Burton, Robert (1976) The Language of Smell. London: Routledge & Kegan Paul.
Kendon, Adam (2004) Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press.
Langdon, John (2005). The Human Strategy: An Evolutionary Perspective on Human Anatomy. Oxford: Oxford University Press.
Mountcastle, Vernon (2005). The Sensory Hand: Neural Mechanisms of Somatic Sensation. Cambridge: Harvard University Press.
Parker, Andrew (2003). In the Blink of an Eye: How Vision Sparked the Big Bang of Evolution. New York: Basic Books. [See also his Seven Deadly Colours, 2006]
Stanford, Craig (2003). Upright: The Evolutionary Key to Becoming Human. New York: Houghton Mifflin.
Stoddart, Michael (1990) The Scented Ape: The Biology and Culture of Human
Odour. Cambridge: Cambridge University Press.
Vogel, Steven (2001). Prime Mover: A Natural History of Muscle. New York:
Norton.
de Waal, Franz (1989). Peacemaking among Primates. Cambridge: Harvard University Press.
Walker, Alan & Shipman, Pat (2005). The Ape in the Tree: An Intellectual and
Natural History of Proconsul. Cambridge: Harvard University Press.
Zimmer, Carl. (1999). At the Water Edge: Fish with Fingers, Whales with Legs,
and How Life Came Ashore but Then Went Back to Sea. New York: Simon &
Schuster.

��