SEMIOTICS AND THE SCIENCE OF MEMORY

Paper presented at the conference on Semiotics and the European Heritage (Dresden, February, 1999)
Paul Bouissac (University of Toronto)

In epistemological matters the notion of heritage is double edged. Undoubtedly, historical research can trace back the development of concepts, models and methods and bring to light some patterns of dynamic continuity through filiation or contagion. But, more often than not, the most interesting achievements in the pursuit of human knowledge comes from sudden shifting of perspectives and counter-intuitive evidence that succeed in overcoming the force of inertia of intellectual traditions. From this point of view, heritage can be equated more with epistemological hinderance than with scientific advances. Naturally, for those who consider it to be a mere doctrine, semiotics can appear to have been formed by successive layers of commentaries and by school debates and exercises derived from the authoritative words of some early fountainheads. Ultimately, a doctrine can only progress through further endoctrination that conserves and increases the "heritage". But, if the semiotic project, as many modern thinkers understand it, has some legitimate claim to scientific status and epistemological relevance, it must be prepared to undergo paradigmatic shifts and confront cognitive revolutions. It cannot develop in isolation from the dynamic of the sciences, even if it positions itself on a different level. This, however, is a risky epistemological position. 
Semioticians have often raised the question of the epistemological status of the sciences. But what about the epistemological status of semiotics itself? Is it not all too often taken for granted by semioticians that their discipline provides a privileged vantage point? Is this really so? What kind of knowledge does semiotic inquiry produce? What does semiotic knowledge consist of? Are the methodologies used by semioticians reliable? These questions are in order for whoever tries to explore the interface between semiotics and the sciences.

1. SEMIOTICS AND THE SCIENCES
For the sake of heuristic simplification, we can consider that there are at least four basic ways of acquiring knowledge, that is, meaningful information either in the form of solutions to well defined problems or counterintuitive discoveries that bring forth new ways of interpreting data and the life experience in general. The great majority of those involved in research are likely to agree that these four ways of acquiring knowledge include: 
(i) the construction of problems based on the state of knowledge in a particular domain and the invention of methods to solve these problems.For instance it is known that some experienced events are somewhat represented in the human brain and remain accessible to consciousness only for a limited time after which they fade away, while some others are stored in a manner such that they remain accessible practically for a life time. Thus, neuropsychologists distinguish working memory (that ensures the conscious binding of the parallel and successive stages of a complex task), short-term memory (that lasts from a few hours to a few days) and long-term memory (that persists over years and decades). These various kinds of memory can be selectively impaired by brain traumas and diseases. Therefore it can be assumed that either the storage processes or the accessibility processes, or both, are supported by different neurological networks and architectures. Consequently, neuroscientists design experiments in order to obtain evidence toward a solution to the precise problems that can be formulated with respect to which specific cognitive deficits can be correlated with which functional part(s) of the brain. 
(ii) Another way to obtain information is by reasoning and argumentation that build virtual models either through a calculus which is blindly pursued to its ultimate consequences, or a systematic metaphorical extension of patterns across apparently distant domains of experience. A good example of this is Gabriel Tarde's elaboration of a nominalistic model of collective behaviour based on imitation and his extension of epidemiological models to the understanding of languages and other semiotic systems as social phenomena (Tarde 1903). More recently, a similar reasoning lead evolutionists such as George Williams (1966), Richard Dawkins (1976) and Terrence Deacon (1997) and social scientists such as Cavalli-Sforza and Feldman (1981) and Dan Sperber (1996) to formulate counterintuitive hypotheses purporting to explain cultural emergence, diffusion and transformation through the biological notions of parasitism and contagion. Similarly, Levi-Straussian structuralism introduced a new vision of cultures by pushing to their conceivable limits the models extrapolated from structural linguistics (Levi-Strauss 1963). 
These two ways of producing information are deliberate and controlled. They usually operate in complementary manner. The latter may lead to formulating precise problems such as the question of how cultural knowledge is represented in the human brain. Is it through digital algorithms or prototypical analogical models? Is its storage content specific? Cognitive neurologists contend that cultural information is acquired, stored and accessed in a manner that is different from episodic memory (working, short- or long-term) and various people have given various names to cultural memory, some calling it "semantic memory", some "generic knowledge" or "general knowledge of the world" (see Tulving 1995). Another question is whether this kind of memory is a sort of procedural memory -- that is, the memory that support skills, habits, all that can be defined as instances of "knowing how" rather than "knowing that" --. or is implemented through specific processes in a distinct architecture. 
(iii) A third way of acquiring knowledge is through chance discovery, also called serendipity, the surprising occurence of an empirical event that is beyond the scope of rational expectations as defined by a state of knowledge in a given domain. Short of probing at random, there cannot be a cost-efficient method that produce serendipity. But, in spite of its unpredictability, serendipity is far from being a negligible aspect in the edification of scientific knowledge. If cognitive neuroscientists now tentatively distinguish at least five kinds of memory (namely, 1-procedural or non-declarative, 2-episodic or personal, event memory, 3-perceptual priming memory, 4-primary, short-term or working memory, 5-semantic memory), it is because these categories emerged from surprising observations ( e. g. ,Blakemore 1977). Striking examples have come from pathological cases that showed the selective impairment of only one of the five kinds of memory that are currently distinguished as a result of these observations rather than as result of pure reasoning. Earlier clinical categorisations such as dementia or amnesia are now replaced by more refined cognitive pathological categories, and several explanatory models are competing for confirmation either through case studies or through non invasive observations.
(iv) Finally, an important source of information comes from an approach to research that is called "meta-analysis". It consists of reading through a large number of specialised scientific publications, selected among the published literature in one or several domains of inquiry, and of relating the partial results within a more encompassing model than the ones that are held by the various specialists concerned. In so doing, results are assessed in view of whether or not they are congruent with a particular hypothesis, and thus support or weaken an argument. If data appears to contradict each other, it may be that some data are artefacts of the experiments or that the model needs to be changed in order to accomodate those seemingly incompatible data. It happens indeed that trough meta-analysis some unexpected patterns emerge which lead to new theories. 
How does today's semiotics fare with respect to these four ways of acquiring knowledge? Putting aside the part of semiotic discourse that consists of mere doctrinal exegesis of speculative texts written in a more or less remote past, it would seem that semiotics is an epistemological enterprise that, until now, has relied upon argumentation and meta-analysis rather than upon experimentation and serendipity. While there is always a more or less latent expectation of revolutionary discovery in the sciences, the epistemological horizon of today's semiotics is fairly redundant and lacks a sense of risk. Information comes more in the relative form of reconfigurations or formalizations of established knowledge than in the form of radical paradigmatic shifts. This remark is not meant to question the validity of semiotics with respect to the construction of scientific knowledge. On the contrary, argumentation and meta-analysis are essential parts of even the most specialised empirical research since any design of experiments is necessarily based on some form of argument derived from past experimental results, but this is done usually within the confines of a disciplinary culture or subculture. For instance, neuropsychologists currently test the treshold of facial recognition in patients affected by various kinds of neurological lesions (e. g. , Bruce et at 1992). Clinical data led them to assume that primate faces constitute a particular perceptual input that tends to override other inputs and focus attention, and that the memory bank of familiar faces is handled by specific brain architectures and circuitry. For instance, experiments in this domain consist of testing the speed of recognition of a visual pattern as a face using as inputs various versions of a drawing or of a photograph representing a human face (degree of schematization or disorganization, unusual orientation, different patterns of light and shadow, scale of chromatic saturation, etc.) both in normal and brain-impaired patients. Other experiments test the recognition of familiar faces (kins, historical figures or contemporary celebrities) (Moscovitch et al. 1997). The punctual results thus obtained can become significant only through wide-ranging meta-analyses of similar published experiments, as well as psychological and ethnological reports concerning the importance of facial recognition and categorisation in social transactions from an evolutionary point of view. It is known, for instance, that monkeys recognize individual infants in their band and behave toward each of them in view of their past experience with the infants' mothers, that is, what is a particular mother's rank and whether they have with her a history of mutual support or aggression (Cheney and Seyfarth 1990; Kummer 1995). However, meta-analysis across disciplines is extremely rare in the sciences, and the role of semioticians in this respect is crucial. Even if they tend to jump to unwarranted conclusions, or to reduce a wealth of data to the few abstract categories that their particular brand of semiotics has dreamed, the epistemological dynamic thus created can only feed further speculations, argumentations and experiments. Naturally, this can be true only as long as semioticians develop and cultivate interfaces with the researchers in the sciences, and do not lock themselves within a solipsist formal system or a mythical grand narrative.

2. THE CHALLENGE OF MEMORY
Considering that semiotics takes as its main object of inquiry systems of signs that are learned (languages, cultural codes, social discourse, etc.), it is surprising that so few semioticians so far have shown a marked interest in the science of memory. Moreover, although most semiotic models that have been produced to date imply some form of constitutive duration over time -- let them be associative, mimetic, intertextual, processual, dialogical, dialectical, and so on -- the issue of their memory foundations has been generally obfuscated by considerations bearing upon their logical consistency. We can say that, all in all, memory is taken for granted in semiotic speculations as we take for granted the oxygen we breathe. Unfortunately, the memory that is taken for granted is a fallacious representation that is conceived, in mediaeval manner, as a faculty of the mind, together with imagination, emotion, reason, volition and the like. Semiotic models construct a kind of virtual universe to which common sense knowledge and thought experiments lend a degree of credibility. Semiotic models are indeed often introduced and delivered through a rhetoric of philosophical persuasion and the way some of these models have spread among fairly large constituencies owes a great deal to the charisma of a few individuals and the institutional pressures they create. Like sects, some semiotic models offer a theory of everything rife with tautological predications and self-fulfilling prophecies. They lack the capacity of constructing a horizon of ignorance, that is, to formulate real problems that can be solved, so as to provide the means of eliciting true information (Bouissac 1992). Fortunately, semiotics fosters since its early beginnings a critical capacity that it can apply to itself as well to other epistemological constructs. It is within the purview of the semiotic project to critically raise the issue of why memory is so conspicuously absent from contemporary mainstream semiotic discourse.
Memory has been a topos of western philosophical discourse at least since Plato. If innate ideas constitute a sort of ontological memory, recoverable through anamnesis, signs are only shadows of shadows and what is learned and remembered through sensorial experience can only be accidental and superficial. The relative significance of these two kinds of memory -- ontological and accidental -- in Aristotle and Augustine is endlessly debated in the Middle Ages. For instance, Richard Fishacre and his disciple Robert Kilwardby (ca 1215-1279) pursued this debate by explicitly distinguishing two types of memory (Popkin 1999:239-241). Such a distinction is based on impression, reasoning and argumentation rather than upon psychological evidence. Later philosophers, relying on both logical reasoning and psychological evidence provided by controlled introspection or other empirical observations, will propose different kinds of distinctions, always pointing to the fact that memory is not a simple, wholesome faculty but a complex, more or less diversified set of competencies. For instance, Bergson (1914) distinguishes "habit-memory" (the capacity of remembering something one has learned) from "pure memory" or "recollection" (the capacity of representing in the present something one has experienced in the past). Taking issue with Russell's logico-philosophical views on the relationship of memory to knowledge (1921), Ryle (1949) points out that when we use the verb "remembering" we may mean any of the following different senses: "retaining", "memorizing", "recognizing" or "recollecting". However, these distinct processes remain for him "aspects" of a single knowledge property. The details of this continuing philosophical discussion of memory is documented informatively in von Leyden (1961). With the emergence of experimental psychology (the first laboratory was founded in Leipzig by Wundt in 1879) memory became a prime target of systematic investigations (e.g. Ebbinghaus 1885), with contrasting new approaches brought about, as time unfolded, by Bartlet (1932), Lashley (1950) and Penfield (1975), to name only a few. 
This brief, sketchy and patchy excursus into the history of the science of memory in the context of European research institutions underlines the strangeness of the conspicuous absence of concern for the science of memory in contemporary European semiotic discourse. 

3. SIGN PROCESSES AS MEMORY PROCESSES
While keeping in mind that the development of a science of memory is an on-going process, three possible bridges or anchor points between semioticians and the researchers who investigate the various forms of memory can be suggested : (a) a reevaluation of the notion of semiosis in view of current neuropsychological knowledge concerning memory. (b) a reconsideration of semiosis in view of the development of evolutionary psychology. (c) a critical questioning of the communication model that pervades semiotics in view of alternative models provided by biological theories of imitation and contagion. 
(a) Semiosis and memory
The notion of semiosis now pervades the semiotic discourse. It is used sometimes in a technical sense in relation to Peirce's sytem of thought, sometimes it refers more casually to the action of signs as opposed to a static vision of sign structures. In spite of these frequent uses semiosis remains a rather vague notion that minimally includes the idea of directionality, transitivity, mediation, transformation and, more generally, dynamism. But even if the (intensional) definition lacks precision, there is no shortage of examples. At least as far as primates, including humans, are concerned, semiosis is a process that is not conceivable in the absence of a brain. The state of knowledge in the cognitive neurosciences may be still short of a definitive answer to many problems, but there is nevertheless a wealth of recent discoveries which should allow semioticians to go beyond simplistic notions such as semiosis in their efforts to understand the processes that involve signs. A phenomenological description of any semiosic event reveal that all the memory systems which have been elucidated during the last few decades must be factors in such processes. Reconceiving semiosis in view of these memory systems reveals parameters which remain conceptually invisible in the model as long as semiosis is understood as a general, all-purpose competence of the mind. 
Let us take, as a typical act of semiosis, the reading of a multi-media message such as a comic strip ( Gubern 1998) or a joke that involves a gesture as its punchline (Sebeok forthcoming). The syntactic and pragmatic dimensions of such messages require the mental capacity of holding their simultaneous and immediate successive components in the unified structure of the task. But the acts of reading or joke understanding demand that other cognitive resources be available, some coming from the knowledge accumulated in the relatively recent past (e. g. , recent political events), some belonging to a stock of data that have been stored for so long in the memory of the decoder that he / she does not remember when or how he / she acquired this knowledge. This applies to both the "knowledge of" (for instance the list of capital cities in the world) and the "knowledge how" (for instance how to read). Often, partial information such as the beginning of a sentence will trigger the automatic completion of a proverb, for instance, or the mention or vision of an object will trigger an association of a paradigmatic or syntagmatic kind. Looking at memory as a mere general competence that can be taken for granted overlooks the complex synergy of semiosis and its reliance on the memory sytems that have been independently fine-tuned by evolution under distinct and specific environmental constraints.
Working memory, short term memory, semantic memory, long term memory , procedural memory, priming memory have been distinguished by neuropsychologists not for the sake of classification but because there is cumulative evidence that these events are supported by distinct brain architectures and circuitry since each of these functions can fail while the others continue to operate. Semiotics has not paid enough attention to the dysfunctions of communication and sense making . Roman Jakobson is the exception, although it is now recognized that he may have jumped too quickly to generalisations (Jakobson and Halle 1956). What neurologists traditionally called dementia can be shown to be specific semiosic dysfunctions caused by various impairments of memory systems. Semioticians would learn a lot about semiosis if they were teaming up more efficiently with cognitive neuroscientists who try to make sense of the fine-grained pathologies of human cognition, which until recently were lumped together under the gross category of deep amnesia or dementia. A meta-analysis of clinical case studies by semioticians would undoubtedly yield data relevant to a better understanding of semiosis. 
(b) Evolutionary psychology and the science of memory
Early empirical research on memory assumed that the human brain was a tabula rasa and that the faculty of memory could be better tested with arbitrary, nonsensical sounds or patterns. This is what did Ebbinghaus in 1885, following Gustav Theodor Fechner's psychophysical methods for the study of the "higher mental processes" (Ebbinghaus 1964). The idea that associations can be created at will with the help of the proper method or through the open-ended happenings of experience remains at the base of the semiotic approaches that emerged at the turn of the XXth century. But it took time for alternative perspectives to gain scientific credentials. The most enlightening is perhaps the British psychologist Frederick Bartlett (1887 - 1969), a professor of experimental psychology at the university of Cambridge whose Remembering: A study in experimental and social psychology (1932) demonstrates what we would call now a marked semiotic sensitivity in as much as his experiments take into account the study of the conditions of organic and mental functions. Rather than aiming at a mere analysis of abstract, all-purpose mental structures, he tested memory with material that is of interest to humans as a species (let it be through linguistic or visual input). Bartlett aknowledged the evolutionary constraints that must have moulded the various memory competencies. In animal ethology, Konrad Lorenz ( )focused on a similar sort of constraints by considering patterned behaviour as a result of the same evolutionary laws that created organs if only because, even if the brain evolved a general competence to learn, this competence remains determined by the law of evolution, in the same manner as an omnivorous organism is only relatively omnivorous. From the contemporary point of view of evolutionary memory, memory cannot be a general competence that would be the result of a mere general plasticity of the brain. Each memory system must have evolved under specific selection pressures and, consequently, must be content-specific (Gallistel 1995; Desimone 1995).
A case in point is working memory, that is, the capacity to hold in awareness a bundle of relevant information with respect to a particular task or event. There are two sets of constraints on this system: first the limits of the sensorial input (i. e. , the limits of the sensorial apparatus of Homo sapiens) and the limits of the capacity of the memory system both in range and duration (this capacity varies among individuals but has absolute limits that preclude the simultaneous consideration of a large informational set. The maximum capacity can be understood as being sufficient for survival in the physical and social contexts in which it has evolved. Another case in point is that some kind of information cannot be recorded but are forgotten beyond their echoic or iconic resonance as if the scrachtpads, as some psychologists call these, were automatically erased by the system (Horowitz and Wolfe 1998; Ward 1998). Information that cannot be construed as meaningful is as interesting for semiotics as information that is construed as meaningful. Likewise for meaningful information that is not remembered, or remembered for a limited time. 
(c). Imitation as memory
The dominance of the communication model in semiotics both in its functionalist and its technological forms has reduced imitation to a special case of either forms: mechnical replications and functional equivalences. Imitation has been the focus of attention almost exclusively as an intentional, psychological, goal-oriented behavior, mostly in the context of aesthetics under the name of mimesis or more generally as an instance of iconism. In the same way, mimetism has been treated as a special case of animal signaling. But the notion of imitation as a general process through which behavior of any sort spread among organisms of one or several species has been the object of scant attention. Perhaps this is because contemporary semiotics has developed under the umbrella of individualistic psychology and continues to carry an implicit teleological ideology of free subjects from whom emanates intentional communicative behavior (the psychoanalytical perspective enhances rather than mitigates this focus on the individual). However since Gabriel Tarde's revolutionary theory of imitation, that stood as a nominalistic alternative to the sociological model propounded by Emile Durkheim, imitation and its obvious reliance on memory has been the object of isolated speculations adumbrating a different semiotic paradigm. One of the most serious problem raised by semiotic theories based on communication is that they does not fit well with evolutionist perspectives (Bouissac 1993). Many phenomena labeled as communicative are better described as imitative. It seems that this stream of speculation is now coming of age following its effective popularisation by Richard Dawkins who recognized his debt to George Williams ' ideas when he coined the word meme. Since then, a powerful movement has developed under the name of memetics whose relevance to the main issues of semiotics is obvious. Terrence Deacon's The Symbolic Species (1997) offers an innovative integration of the concept of meme as parasite in an explicitly semiotic theory of the origin of symbols and languages. It amounts to a Copernician revolution in the understanding of signs and semiosis in as much as signs are conceptualized as agents, rather than passive tools, that exploit the human brain as a resource for their replication. Like in any parasitic relationship, the meme-brain coevolution presupposes that the initial resource -- in this particular case, a memory resource --evolved under independent evolutionary constraints. This counter-intuitive theoretical vista can open the way to formulating hypotheses that could be tested in the context of what could be called "wet" semiotics -- that is, semiotic research conducted in the neurological clinical laboratory -- in contrast with "dry" semiotics, or "armchair semiotics" (Bouissac 1998a and b). 

CONCLUSION
Obviously, I have relied in this paper on argumentative and meta-analytical strategies. The advantage of this combined approach is that it allows external information to constrain and control speculation and thus to avoid the pitfalls of purely subjective evidence and thought experiments. However it is not either without dangers: (i)reasoning is on the one hand subject to cognitive illusions, and, on the other hand, it is to a larger extent than usually thougth, historically conditioned by a particular zeitgeist or episteme. To perceive and appreciate information sometimes require that we "unthink" basic knowledge we take for granted. (ii) The value of meta-analysis depends of how complete is the literature that is perused. Not only is the selection of the corpus under the dependence of the individual preconceptions of the researcher but information comes prepackaged so to speak by disciplinary gatekeepers. Moreoever, this information is couched in specialized languages often hard to decipher, and the experimental and theoretical landscape of a vast and diversified domain like the neurosciences is fast changing. With respect to the particular topic that has been the focus of this paper, two multidisciplinary scientific journals offer a wide array of research papers among which appear fairly regularly some articles relevant to the cognitive neurosciences, in particular to memory. These are Nature and Science. Should a breakthrough occur in the understanding of memory, or memes, it is more than likely that it would be reported in their pages. The Annual Review of Neuroscience and The Journal of Cognitive Neuroscience are also reliable sources of information for whoever wants to keep an eye on developments in the field. There also appears once or twice a decade a collective volume that provide state of the art knowledge in the neurophysiology of cognitive functions (e. g. , Cazzaniga 1995). But, more importantly, there exist at least two journals devoted to memory research in relation to domains akin to semiotics: Memory and Cognition and Memory and Language. Both offer articles very relevant to the sort of problems that have been indicated in this paper. 
To conclude on an optative note, it would make sense for IASS / AIS to endeavour to create a journal titled Memory and Semiotics, whose function would be to develop much further the interface between semiotics and psychology (Evolutionary Psychology, Neuropsychology, Cognitive Psychology, Computational Psychology, and whatever other subspecialties and emerging paradigms that may appear in the near future). This would provide both a focus and a forum for many younger researchers interested in constructing productive interfaces between the rich speculations of semiotics and the methods of the empirical sciences..

REFERENCES