Abstract
This paper introduces dyadic brain modeling – the simultaneous, computational modeling of the brains of two interacting agents – to explore ways in which our understanding of macaque brain circuitry can ground new models of brain mechanisms involved in ape interaction. Specifically, we assess a range of data on gestural communication of great apes as the basis for developing an account of the interactions of two primates engaged in ontogenetic ritualization, a proposed learning mechanism through which a functional action may become a communicative gesture over repeated interactions between two individuals (the ‘dyad’). The integration of behavioral, neural, and computational data in dyadic (or, more generally, social) brain modeling has broad application to comparative and evolutionary questions, particularly for the evolutionary origins of cognition and language in the human lineage. We relate this work to the neuroinformatics challenges of integrating and sharing data to support collaboration between primatologists, neuroscientists and modelers that will help speed the emergence of what may be called comparative neuro-primatology.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Introduction
This paper is intended to highlight the promise of the emerging field of comparative neuro-primatology and to propose informatics tools and interdisciplinary directions that will open up new avenues of research for ethology and neuroscience. Computational modeling, and specifically dyadic/social brain modeling, can be used to integrate, extend, and test theories from both neuroscientific and behavioral fields. However, there are considerable theoretical and practical challenges to building realistic neuro-computational models of social behavior – both capturing the social elements of behavior, and making the most of the limited data that is currently available.
To illustrate the challenges of this integrative modeling approach, we propose a computational model based on the gestural communication of great apes. Gesture – in particular gesture acquisition – provides an excellent case study in social brain modeling because it raises issues that would be problematic for modeling any social behavior. For example, how do the brains of interacting agents process shared events differently? How do agents respond to behavioral changes in others, and how are these changes reflected in brain activations and/or adaptive synaptic wiring? Are there dedicated neural structures or pathways for social interaction, or do primates largely rely on domain-general regions for social cognition? Focusing on gesture also grants us empirical purchase as ape gestural behavior has been studied both experimentally and observationally, and manual action production and recognition systems in monkeys are fairly well characterized at the neural level. Finally, the added learning component of gesture acquisition forces us to consider both immediate and lasting changes in the neural organization of behavior.
The learning process we discuss – ontogenetic ritualization – has been proposed as a mechanism through which great apes may acquire new communicative gestures through the mutual shaping of action, resulting in a stable, but non-arbitrary gestural form. Modeling the process of ontogenetic ritualization provides several distinct challenges that must be confronted. The model must be able to account for the fact that ontogenetic ritualization is (i) a dynamic process in which (ii) multiple individuals process and respond to the interaction differently, while the interaction itself (iii) changes and develops over time.
Constructing a model of the dyadic interactions of the social brain requires integration of data across multiple datasets, methodologies, and disciplines, and thus places unique demands on informatics tools and resources. Data management tools and techniques for integrating resources must focus on efficiently navigating questions of homologies between species, finding the appropriate granularity of data for modeling projects, and producing simulation results specific enough to test existing frameworks and offer novel hypotheses. We highlight the need for new and more integrated resources for researchers operating in these highly interdisciplinary fields, and offer new suggestions and challenges for the neuroinformatics community.
Primate Social Behavior
All animals face the challenges of finding and obtaining food, water, shelter, and suitable mates while, at the same time, minimizing injury from competitors or predators. For social animals, these physical challenges arise in an abstract network of social relationships that often impact an individual’s success, and which must therefore be tracked, fostered, and exploited. The need for such socio-cognitive abilities likely provided a strong selection pressure that helped shape both brain structure and cognitive skill in the primate lineage (Byrne and Whiten 1988; Dunbar 1998; Sallet et al. 2011). The study of the primate brain has only begun to explore the neural correlates of these socio-cognitive abilities, but new developments in brain imaging and neurophysiological designs allow neural activity to be measured in both human and non-human primates during social interaction. For this paper, we focus on non-human primate – henceforth “primate” – data, but recognize the substantial insights that can be gained from human lesion and neuroimaging studies (Adolphs 2010; Amodio and Frith 2006; Shilbach et al. 2012). Combining insights from neural and behavioral studies promises to greatly increase the scope of the questions that may be addressed.
Ape Gesture
Great apes – gorillas, bonobos, chimpanzees and orang-utans – are proficient at copying manual actions, though their skills in this area are limited relative to those of humans (Byrne and Russon 1998; Dean et al. 2012). The ability to acquire manual skills through observation of conspecifics likely plays a role in the development of group-specific behaviors in both wild and captive populations. ‘Local traditions’ (behaviors restricted to particular populations or subgroups of individuals) involving the presence or variation of manual actions, such as tool use, food processing, and grooming, have been reported in both wild and captive great apes (Byrne 2004; Hobaiter and Byrne 2010; van Schaik et al. 2003; Whiten et al. 2001), and provide evidence that apes can develop ‘cultures’ of behavior (Whiten et al. 1999).
Surprisingly, manual gestures do not show the same levels of inter-group variability as manual actions do. Studies of ape gesture consistently report that the majority of gestures are either “species typical” (i.e., used by members of a species regardless of what geographic site they inhabit), or are idiosyncratic and therefore produced by only a single individual—and presumably recognized by at least one other (Genty et al. 2009; Hobaiter and Byrne 2011; Liebal et al. 2006; Pika et al. 2003). There have been some reported differences in gesture form or use between apes at different research sites, but the predominant pattern is one of similarity across sites, with most of the gestures observed at site A also observed at sites B and C. The gestural repertoires of individual apes at different sites typically overlap as much as those of apes at the same site (Call and Tomasello 2007). Additionally, gestural repertoires typically vary more strongly between age classes than between sites – with juveniles displaying largely different repertoires from adults. The proportion of apes using a particular gesture does vary between site – at some sites a gesture will be used by the majority of individuals, while at others it will only be used by a few – but it does not vary much (Genty et al. 2009; Hobaiter and Byrne 2011). There are some exceptions to the ‘species-typical or idiosyncratic’ characterization of gesture use. A few group-specific gestures have been observed in orangutans (Cartmill 2008; Liebal et al. 2006), gorillas (Genty et al. 2009; Pika et al. 2003), chimpanzees (Nishida 1980), and bonobos (Pika et al. 2005). These studies suggest the existence of group-specific gestures that may result from social learning (Arbib et al. 2008), though some have argued that reports of relatively infrequent gestures observed only in one group may simply be an artifact of under-sampling (Genty et al. 2009). However, if a gesture is used frequently in one group and rarely or never in another, a strong case can be made for a local ‘gesture culture.’
A significant problem with this approach of comparing gesture use across sites is that unless all data are collected and coded according to the same criteria, gestures at multiple sites may vary in how they are defined and recorded. This, in turn, may lead to inaccurate estimates of the repertoire overlap between groups. Comparing gestures across multiple sites and species is crucial to understanding gestural communication in primates, and new informatics approaches to integrating data gathered at different sites are needed to make significant progress in this field. We return to this issue in the final section.
Primate Neurophysiology
We are concerned with building a bridge between ape social behavior and its underlying neural circuitry. However, while there are data sets on single cell recordings in monkeys (especially in macaques), no such data are available for apes (although brain imaging data are now becoming available). Thus, our strategy is indirect, extending our understanding of brain processes in monkeys to construct a framework for modeling the social behavior of apes. In this section we will describe neurophysiological studies on macaques that have linked neural activity to both manual behaviors and cognitive abilities. In a later section we review a key set of computational models linking vision and action and which describe the neurophysiological data. We examine the assumptions and limitations of these models and ask: “what properties must be added to macaque models to support brain modeling of ape (social) behavior?”
Primate neurophysiology has been used to address some aspects of manual and social behavior in monkeys, but the designs have traditionally relied on ‘passive’ designs that do not require interaction between the subject and another individual. For instance, the research on ‘mirror neurons’ has always been passively social in that neural responses could be elicited by observing the performance of other individuals (di Pellegrino et al. 1992; Gallese et al. 1996) rather than through interaction. Neurons were found in premotor (and later in parietal) areas whose activity during an individual’s own performance of a particular action was found to be similar to the activity in response to observing another individual – usually a human researcher – performing a more-or-less similar action. In this way, mirror neurons can be driven by social variables, but the experimental designs do not require the monkeys to differentially ‘use’ this information, and so cannot assess how these responses affect downstream targets.
These passive designs can be contrasted with explicitly social or ‘interactive’ designs, more recently developed, that require the subject to directly interact with other entities, whether computer agents (Lee et al. 2005; Seo et al. 2009; Seo and Lee 2007) or conspecifics (Azzi et al. 2011; Chang et al. 2012; Fujii et al. 2008; Yoshida et al. 2011, 2012). These interactive designs have led to new insights into how the brain is organized to process specifically social information, how this information affects downstream targets, and how interaction between a monkey and another agent places unique demands on processing structures within the brain. For instance, responses in medial parts of frontal cortex, in and around pre-SMA, have been found to be ‘other’ responsive neurons – instead of firing both when an action is done by one’s self and when observing another, as in the above ‘mirror neurons’, these only fire during observation of another’s actions (Yoshida et al. 2011). Orbito-frontal cortex (OFC) neurons, recorded in monkeys playing interactive games, revealed modulations encoding social influences on motivation and reward processing (Azzi et al. 2011). OFC and anterior cingulate (ACC) neurons, in a separate but similarly ‘interactive’ study, were shown to differentially process how rewards were allocated between others and oneself, with ACC gyrus appearing important for processing the ‘shared experience’ of rewards (Chang et al. 2012). Together, these and other data demonstrate that social behaviors are becoming increasingly accessible to neurophysiological study in monkeys, and not just in a ‘passively social’ sense, but within tasks demanding back-and-forth exchanges. Additionally, the emergence of functional monitoring via PET, fMRI and other neuroimaging techniques adapted to non-human primates is most promising. These have the double advantage of being non-invasive, while being of the same ‘format’ of the most used techniques in humans, easing comparison of data across species.
For example, non-invasive EEG and ERP studies have recently been applied to the production and comprehension of vocal communicative behaviors in apes (Hirata et al. 2011; Ueno et al. 2008). These techniques complement the emerging use of PET in functional brain monitoring (Parr et al. 2009; Taglialatela et al. 2011), and eye-tracking, an indirect measure for attentional processing (Kano and Tomonaga 2009) in apes. Combining functional data with mathematical techniques to understand these indirect measures of brain activity in terms of neural firing allows researchers to ‘convert’ data between domains of analysis, including making non-invasive functional data more compatible with neuro-computational analysis (fMRI: Arbib et al. 2000; PET: Arbib et al. 1995; ERP: Barrès et al. 2013). All the above methods can be combined with the use of structural imaging techniques such as DTI (Hecht et al. 2012; Ramayya et al. 2010; Rilling et al. 2008), MRI (Sakai et al. 2011), and other imaging, anatomic and cytoarchitectonic methods comparing primate brains (Hopkins et al. 2010; Keller et al. 2012; Rilling et al. 2011; Schenker et al. 2010). Given the difficulty in directly assessing brain function in apes, it is necessary for researchers to use existing data from all available techniques to develop more complete models of primate neural processing during social behavior.
In this paper, we present the design of a brain-based conceptual model – to be followed with a fully implemented computational model elsewhere – aimed at testing a proposed learning process through which great apes may develop manual communicative gestures. Despite a dearth of direct neural data for gestural communication in great apes (Taglialatela et al. 2011), we have several reasons to focus on gesture acquisition as a test case for modeling social cognition. Firstly, the proposed learning process – ontogenetic ritualization – rests on repeated interactions between pairs of individuals, thus demanding a direct treatment of social interaction. Secondly, computational modeling of primate manual gesture intersects with a growing body of work on the mirror system and has implications for understanding the origins of human language. The Mirror System Hypothesis (MSH: Arbib 2010, 2005, 2008, 2012) makes explicit claims about brain function evolution throughout the hominid line, and the concomitant capacity for social learning and flexibility in intentional communication, that made the human brain ‘language-ready’. Although others have adopted a neuro-evolutionary approach to communication (e.g., Aboitiz 2012; Corballis 2002; Deacon 1997), MSH is unique in explicitly grounding the evolutionary account in the computational description of macaque neural processing (including ‘mirror neuron’ systems) and ape behavior (including gesture). It is within this MSH framework that we approach our case study, emphasizing the computational description of brain function to formalize hypotheses on gesture acquisition.
In order to properly contextualize our proposed model, we first outline the claims of ontogenetic ritualization, and then provide details on primate brain mechanisms known to be important for manual and social tasks (especially those formalized in computational models). We then describe our model – a conceptual analysis of the proposed process of ontogenetic ritualization – and discuss those features important for the field of ‘dyadic/social brain modeling’. Finally, we consider the impact of incorporating observational, experimental, and computational approaches in the study of the social brain, and conclude with a discussion of issues related to data management and sharing that will support future interdisciplinary collaborations.
Ontogenetic Ritualization
Ontogenetic ritualization (OR) is the proposed process of ritualizing movements of ‘effective’ actions (those that directly alter the behavior of other individuals) into communicative signals aimed at eliciting particular responses in others (Tomasello and Call 2007; Tomasello and Camaioni 1997). During this process of ritualization, a movement such as shoving another out of the way becomes ritualized over time into a ‘nudge’ as the actor learns that only the beginning of the movement is necessary to elicit the desired behavior in the recipient, and as the recipient learns to respond to the gesturer using only the initial movements of the action. However, according to this process, the actor and recipient form different associations resulting from their respective roles in the interactions – indeed, the recipient may only be able to perceive but not produce the gesture, and vice-versa (Genty et al. 2009). The degree to which OR plays a role in the acquisition of ape gestures is debated (Genty et al. 2009; Perlman et al. 2012; Tomasello and Call 2007). Here, we do not take a strong stance on whether OR is the main acquisition mechanism for ape gesture, but we do note that OR could explain those species-typical (and not just idiosyncratic) gestures whose relation to species-typical actions is readily derivable through the OR process. We propose a model of the cognitive and neural changes that, we hypothesize, could support OR. It is our hope that such modeling work will make it possible to identify the conditions under which OR is a plausible mechanism for gesture acquisition, while simultaneously generating hypotheses for new behavioral and neuroimaging experiments that test social and communicative behaviors more broadly.
The process of ontogenetic ritualization is described by Call and Tomasello (2007) as proceeding in three steps (Fig. 1, left):
-
(1)
Individual A performs behavior X (not a communicative signal), and individual B consistently reacts by doing Y
-
(2)
Subsequently B anticipates A’s overall performance of X by starting to perform Y before A completes X
-
(3)
Eventually, A anticipates B’s anticipation and produces an initial portion of X in a ritualized form XR in order to elicit Y
Of particular relevance to social brain modeling is that this is a dyadic learning process – it requires differential learning in the brains of A and B, which reflects the changing patterns of interaction between them throughout the ritualization process.
Ontogenetic ritualization is thought to underlie the development of some human gestures, but the process in humans differs in some important ways from the ape process we focus on here (Fig. 1, right). The palm-up ‘gimme’ gesture or the ‘arms up’ gesture in which an infant raises his arms to indicate a desire to be picked up are good examples of ritualized human gestures (Bruner et al. 1982; Clark 1978). Initially, these types of gestures occur only in the immediate context of the actions they are derived from – a 9-month-old infant will use the ‘arms up’ gesture only when an adult behaves as though she is about to pick him up (perhaps only following the adult’s contact under the arms of the child). Over time, however, the gestures become more removed from these particular contexts so that a 13-month-old infant might use the gesture according to his own desires to request rather than facilitate being picked up (Lock 2001). Thus, ‘arms up’ emerges as a sign used with communicative intentions.
In human infants, this ritualization process may be more accurately described as assisted ritualization (Zukow-Goldring and Arbib 2007) because the adult recipient is monitoring and reinforcing seemingly communicative behavior in the infant, and in many cases the adult already has an idea of what the final gestural form should look like (XR in Fig. 1, right) – because, for example, it already exists in the culture. Thus, the process and speed of ritualizing an action into a gesture becomes driven by the recipient.
In the case study we consider, we restrict ourselves to the first interpretation of ontogenetic ritualization as a naïve interactive process though which a sign emerges, rather than a process in which a sign is shaped by a more knowledgeable individual. It may be the case that experienced primates play a more active role in shaping the behavior of others, as humans are known to do (see Ferrari et al. 2009), but here we focus on the simpler, unassisted version of ontogenetic ritualization in which each participant is naïve as to what the final form of the gesture will be.
Action, Perception and Cognition in the Brain
To fully represent the changes in the brain of each participant during ontogenetic ritualization, our model must minimally incorporate brain structures critical for (i) the visuo-motor control necessary for action and gesture, (ii) recognizing and responding to the actions of others, and (iii) motivating social interactions between conspecifics – as well as considering how learning affects each. We now review some known primate brain systems for visually-guided grasping, action-recognition, and decision-making, and outline their proposed computational properties. In a later section, we will suggest how these brain mechanisms supporting praxic actions directed at changing the physical state of an object could provide a basis for extended circuitry that also supports communicative actions (e.g., gestures) directed at changing the behavior of a conspecific.
Visually-Guided Grasping
The FARS model (Fagg and Arbib 1998) has been offered as a computational description of manual visuo-motor coordination in the macaque brain. FARS describes the fronto-parietal reach-to-grasp production circuitry macaques use when they manually grasp objects (so called reach-to-grasp actions), based on neurophysiological data. Parietal structures on the dorsal stream extract the ‘affordances’ of the world relevant to the grasp (i.e., the physical and spatial properties of the object to be grasped) and forward these to premotor cortex for selection of an appropriate grasping action (Jeannerod et al. 1995). A ventral object-recognition path allows prefrontal structures to select an appropriate motor program when working memory or task structure provides relevant constraints. The model additionally invokes interaction between prefrontal cortex and the basal ganglia when a sequence of actions is required to complete the overall task. This computational description of primate manual control – well supported by contemporary accounts of brain function (Cisek 2007; Cisek and Kalaska 2010) – can help frame our model of gesture learning. It is important to point out, however, that such an ‘affordance-driven’ description must be complemented with a description of the control structures participating in guiding hand motions without explicit physical targets, as would occur during the performance of visually intransitive gestures (as opposed to tactile gestures like the ‘nudge’ example discussed previously).
Action-Recognition
The MNS, for Mirror Neuron System (Oztop and Arbib 2002), and MNS2 (Bonaiuto et al. 2007) models build off of the computational description of manual action in FARS to describe the recognition component of the mirror neuron responses. These models have suggested that mirror neurons learn their property of ‘action parity’ – responding similarly for production and recognition – by learning the visual trajectory of the hand in relation to objects for actions already in the agent’s repertoire – combining signals of visual feedback during the course of generating an action with the efferent motor commands controlling that action (Oztop and Arbib 2002). These models formalize how neurons in parietal and premotor regions can learn to recognize a range of movements associated with a given reach-to-grasp action, and illuminate how action recognition in macaques may be supported by these parietal-premotor circuits. In our analysis of ape gesture learning, the MNS class of models provides sufficient machinery for recognizing affordance-driven actions like reach-to-grasp – crucial, as we will see, for the learning that must occur in the initial stages of ritualization. However, again because of the model’s emphasis on the relation of the hand to an explicit physical target, the MNS models (like the FARS model for action generation) are not flexible enough to account for the movements associated with known ape gestures. Thus, the MNS model for the macaque must be extended by additional visual-processing machinery to recognize movements not explicitly directed towards objects, and likewise for FARS (see Fig. 2). And while monkey reaching and grasping behavior has been long studied (Georgopoulos et al. 1981; Jeannerod and Biguer 1982; Rizzolatti et al. 1987; Taira et al. 1990), non-human primate gestural control has not (though for studies of, e.g., apraxia in humans, see: Buxbaum et al. 2005; Petreska et al. 2007; Rothi et al. 1991). This gap can be partially bridged by generating testable hypotheses derived from computational models (e.g., how do apes maintain approximate visual form when no explicit targets are available?).
Decision-Making
In order for an animal to adjust its actions to respond to the immediate environment, it must be able to evaluate contextual and motivational information and select an appropriate action from its repertoire on the basis of that information. For an animal to adapt its actions to environmental variables over time, the neural system must also be sensitive to the outcomes of its past actions. Reinforcement learning has been a successful framework for describing this type of adaptation – particularly when considering the decisions and actions of goal-directed, reward-driven agents (Sutton and Barto 1998). Crucially, estimates of the ‘value’ in performing particular actions in particular contexts are learnable, even when no explicit positive or negative reinforcement is received until some time in the future – after the completion of further actions. These estimates of the value of particular actions predict how an agent will learn and act when it encounters similar circumstances in the future.
The Augmented Competitive Queuing (ACQ) model (Bonaiuto and Arbib 2010) places reinforcement learning mechanisms alongside MNS mechanisms, in the context of making decisions in the face of changing environments, goals and, crucially, skills. This allows actions to be evaluated in a particular context for executability – the availability of affordances that allow the given action – as well as desirability – the expected (future) reinforcement following that action (e.g., motivational components for decisions). This model predicts that actions will be chosen opportunistically: the next action chosen will be that which is most desirable among the set of executable actions. This separation of decision variables into contextual and motivational components – and their ultimate integration – is supported by the neurophysiology of decision-making (Watanabe 2007; Watanabe and Sakagami 2007).
In the ACQ model, visual feedback analysis (mirror neuron system activity) of one’s own actions determines whether the action achieved its goal, and whether its execution resembled some other action (the apparent action). On this basis, reinforcement learning can update the executability of the intended action and the desirability of the self-observed action (whether intended or apparent). In this way, an agent uses an evaluation of current context based on traces of past experiences to estimate the effectiveness of different possible actions. This, in fact, fits the observed patterns of great ape gesturing, in which apes choose gestures based on their goals, the immediate social context, and their past interactions with their partner (Cartmill 2008; Hobaiter and Byrne 2011; Liebal et al. 2004a).
Given that computational models of neural circuitry for visually-guided grasping, action-recognition, and adaptive decision-making exist, our model of ape gesture acquisition need not be constructed de novo. The FARS, MNS, and ACQ models, along with insights drawn from other models – robotic (Chaminade et al. 2008; de Rengerv et al. 2010), and neural (Bullock and Grossberg 1988; Caligiore et al. 2010; Demiris and Hayes 2002) – provide a circuitry framework upon which neuro-computational models of gesture acquisition may be based. Moving from simpler single-agent models into more complex, social brain modeling may necessitate a re-evaluation of previous models, and require ‘extensions’ to these models to more closely capture brain function. It is here that neuroinformatics tools could provide crucial insights into (i) model benchmarking – what can a model do or explain and what can’t it do? (ii) representation of data at the appropriate ‘level’ – does it explain dynamic time-courses, or sequences of discrete decision events? and (iii) comparing predictions derived from model simulations with empirical results from behavioral studies or neurophysiology.
Case Study: Developing the Gesture ‘Beckon’ via Ontogenetic Ritualization
We now consider an analysis of the progressive changes in brain and behavior that would need to occur during the proposed process of OR. We do this by constructing a hypothetical sequence of interactions between a mother and child ape that could lead to the emergence of ‘beckoning’ as a gesture used by the child to get the mother to approach. This gesture, or variants of arm-extended ‘approach’ gestures, has been observed in several ape species (Cartmill 2008; Pika and Liebal 2012; Pika et al. 2003; Tomasello and Call 1997), though it is not clear how it is acquired. Our aim is not to claim that this specific gesture is learned in this way, but to use this example to help clarify both the types of interactions and the neural changes that would be necessary to support the general transition from action to gesture via OR. Our model is conceptual, not a fully implemented computational model (though the latter is an ongoing research goal). The conceptual model serves to make general points about gestural acquisition through OR, and offers a framework to analyze existing behavioral data from a neuro-computational perspective.
Where Call and Tomasello (2007) describe ontogenetic ritualization with the above 3-step formula, we offer a finer-grained analysis using 6 stages to highlight the distinct learning processes that we expect to occur in the Mother (M) and Child (C) as the child’s pulling action is ritualized into a beckoning gesture. We then walk through the neural changes that seem necessary to support the behavioral changes at each stage and discuss the challenges in modeling the changes in mother and child at each stage.
Proposed Behavioral Changes in Mother (M) and Child (C) During OR of a Beckoning Gesture
-
Stage 1)
C reaches out, grabs, and tugs on M, causing M to move towards C as a response.
-
Stage 2)
C reaches out, grabs, and begins to tug on M, and M quickly moves towards C.
-
Stage 3)
C reaches out and makes contact with M, and M quickly moves towards C.
-
Stage 4)
C reaches out towards M, attempting to make contact, but M responds before contact is made.
-
Stage 5)
C reaches part way towards M, and M responds by moving towards C.
-
Stage 6)
C gestures towards M and M responds to this ritualized gesture by moving towards C.
It is our belief that such a finer-grained process, when pegged to behavioral and neural changes in each agent, presents a more appropriate framework with which to compare or benchmark a computational model, while still remaining consistent with Call & Tomasello’s description of the overall pattern of interaction.
-
Stage 1)
Child reaches out, grabs, and tugs on Mother, causing Mother to move towards Child as a response.
Since our example is meant to illustrate the salient steps in all plausible cases of ontogenetic ritualization, the key for stage 1 is that the actor is able to achieve his desired goal directly through physical manipulation of the recipient. For this initial period of interaction, neither participant has any prior expectations of the other’s behavior.
Child
In order for the child to achieve his goal, his only option is to physically manipulate the mother to bring her into physical contact with himself (that is, we assume no latent gestural form for this goal). He orients towards the mother, identifying appropriate surfaces for grasping to pull. He reaches out, grasps a part of her body, and initiates pulling on the mother. The pulling force begins the movement of the mother towards the child, and after enough tension, the mother complies and moves closer to the child. The neural machinery required to coordinate this sequence of actions can be fully described by the FARS model of visual control of grasping discussed above, which will serve as a benchmark for the child’s behavior in subsequent stages.
Mother
Throughout this example, we will assume that the mother is motivated to complete her child’s request (not always the case!) and that her attention is appropriately oriented towards the child, allowing her to process the child’s actions visually as well as haptically. Assuming that the mother is attending to the child, her mirror neuron/action-recognition system would register the ‘reach-to-grasp’ followed by ‘pull’. A key property of mirror neuron firing, captured by the MNS models, is that mirror activity often signals recognition of the observed actions before the action is completed. However, the mother’s response to, as distinct from recognition of, the child’s action has not yet been established. Associative learning mechanisms establish this connection between the child’s action and the response ‘approach.’ Importantly, this association must be retained as the child’s action changes form over the OR process. We also note the need for ‘social’ motivation to fulfill the goals of the child or to prioritize physical proximity – a motivation shared by both agents.
-
Stage 2)
Child reaches out, grabs, and begins to tug on Mother, and Mother quickly moves towards Child.
In this step, both individuals experience an adaptive change in behavior in real-time and begin to alter their expectations of the other’s actions. For the child, (i) he need not pull as hard once the mother begins to comply, suggesting feedback modulation of his on-going action, and (ii) he forms the expectation that the mother will be increasingly compliant. For the mother, she learns that given contextual considerations – similar play conditions, perhaps, and/or perceived emotional state – and her child’s grasp-pull action, she is rewarded (socially) for moving herself to his side.
Child
The child’s intention is to reach out, grab and physically move the mother near him. However, following his grasp, his mother becomes more compliant and begins the movement towards his side. The child perceives the mother’s movement as beginning to satisfy his goal and acts less forcefully on the mother as she responds to his action. This further refines his expectations of his mother’s likely response. In future interactions, he will expect that less force is needed to achieve his goal.
At this stage, we encounter the problem of how recognition of someone else’s actions can affect the ongoing execution of one’s own actions – a general concern for social brain modeling. Here, the child, as in step 1, expects a full ‘reach-to-grasp-to-pull’ action is necessary to achieve his goal, but as he begins his tug on the mother, the mother responds by ‘completing’ the action. Recognition of the early success of the action must be able to modulate the child’s ongoing behavior in such a way that his action can be modified either (i) by reducing the force he pulls on the mother, as in this step, or (ii) by interrupting and even extinguishing the action mid-trajectory (as we describe below). Such sensitivity to changing perceptual variables during grasping behavior has been explored in a computational model of how the reach, grasp, and their coordination may be affected by perturbations in the size and location of grasped objects (Hoff and Arbib 1993), which use on-line feedback to modulate what might otherwise have been a feed-forward movement. Thus, insights from other models may guide our own model development.
Mother
Following contact, the mother moves towards the child, easing the tension on her arm. The association between the child’s action and the approach response is rather weak at this point, and can only be triggered by proprioceptive contact and mechanical tension as a complement to the visual-form representation established by the mirror neuron system. As in stage 1, the MNS models of monkey action recognition provide an explanation for the mother’s recognition of the child’s actions, but are unable to provide a clear description of the effects of this recognition – a problem we explore below. Future models of action recognition must address the role of multisensory integration in the recognition process more thoroughly. The MNS2 model (Bonaiuto et al. 2007) characterized the audio-visual neurons seen experimentally in Kohler et al. (2002), and showed how associative learning mechanisms may link acoustic cues with the visual form of actions. A key for a model of OR would be extending this to visual-haptic cues (see Fig. 2).
-
Stage 3)
Child reaches out and makes contact with Mother, and Mother quickly moves towards Child.
Child
The child’s attempt at grasping and pulling the mother remains the same as stages 1 and 2, with the exception that he becomes increasingly sensitive to the mother’s anticipatory response, having in the past two stages come to expect a ‘completing’ response. In stage 3, as he begins to make contact with the mother, the mother’s response appears consistent to his expectation, and he aborts the second half of the action sequence: the pull on his mother. As we see in Fig. 3, however, such a process may be described at different levels of representation – discrete and continuous, or ‘event’ and ‘trajectory’. Models of reaching and grasping (e.g., Bullock and Grossberg 1988; Fagg and Arbib 1998) emphasize the dynamic unfolding of the behavior and how certain elements (the positions of joints, perhaps) vary continuously in time. Models of learning and decision-making (e.g., Bonaiuto and Arbib 2010; Botvinick et al. 2009) on the other hand, emphasize the serial structure of decisions as discrete events. Both levels may be helpful in understanding brain function, and in fact the brain appears to utilize both (see: Averbeck et al. 2002; Campos et al. 2010; Georgopoulos 2002; Sawamura et al. 2002). The challenge for neuroscientists is to understand how both may coordinate behavior, and how best to represent these descriptions in models.
Mother
Visual recognition of the child’s reach-to-grasp action, coupled with contextual cues and the proprioceptive contact as above, is sufficient for the mother to select an appropriate response consistent with the child’s goals. This stage represents the terminal phase of proprioceptive cues involved in training the recognition-response linkage – in future stages visual recognition alone suffices to initiate the response.
-
Stage 4)
Child reaches out towards Mother, attempting to make contact, but Mother quickly responds before contact is made.
Here, we have reached the point where both (i) the child has learned that a ‘reach-to-grasp-to-pull’ action is not necessary (though perhaps unsure about the extent to which he must contact and attempt to manipulate the mother), and (ii) the mother has learned that (attempted) grasps to her arm may signal an opportunity for social bonding. Note that whereas the child’s learning consists largely in tuning his forward expectations of the mother’s behavior, the mother’s learning consists in mapping the recognition of the child’s actions to behavioral responses that satisfy mutual goals.
Child
The child at this point has learned that incomplete versions of a ‘reach-to-grasp-to-pull’ action can be used to achieve his goal, and so only intends to make minimal contact. Here, the child’s attempted action should still be seen as transitive, directed at a surface. This will be the starting point for the last two stages, in which the actions become increasingly removed from orientation towards a specific surface, and instead the hand’s movement pattern in space becomes the most salient element, resulting eventually in a ritualized, intransitive gestural form.
Mother
By stage 4, the mother has robustly linked visual recognition of the child’s ‘reach-towards-body’ action with the ‘move-towards-child’ response, and can effectively fulfill the child’s goal without even minimal haptic cues. The key to this stage of the ritualization process is that visual form alone is now sufficient for the mother to respond. Subsequent stages serve to train the mirror neuron/action-recognition system to recognize the now visual-only ‘proto-beckoning’ act in shorter and more reduced forms.
-
Stage 5)
Child reaches part way towards Mother, and Mother quickly responds by moving towards Child.
Child
This stage is unique in that now the child no longer intends to physically interact with the mother, but instead acts only in a way sufficient to elicit the appropriate response. Crucial for the child, though, is that his intransitive performance must appear visually similar enough to his transitive performance that the mother can recognize it and respond appropriately. This would suggest that proprioceptive and visual feedback signals from the past transitive episodes trained a forward internal model – a model of an appropriate trajectory through space – that now can sufficiently control the limb in an apparent ‘proto-beckoning’ action. Whereas previous models of object-directed motor control (e.g., FARS) neglected intransitive performance, we see that models of gesture acquisition may not. Instead, as Fig. 2 shows, additional machinery must be added to recognition and production processes, informed by analyzing behavioral and/or functional data.
Mother
The mother is able to recognize, in mid-trajectory, the motion of the child’s arm, and to respond appropriately. In this stage, the mother’s action-recognition system would need to begin to respond to smaller and smaller portions of the action. Just as the child must maintain visually similar performance in the absence of explicit targets (e.g., mother’s arm), the mother must be able to recognize the child’s actions absent such contextual cues. Without sufficient ‘overlap’ in trajectory, the putative visual training would be unable to maintain the link between recognition and response in the mother.
-
Stage 6)
Child gestures towards Mother, and Mother responds to this ritualized gesture by moving towards Child.
A ritualized form of the gesture emerges. While previous stages have highlighted the changes that would allow both mother and child to progress towards using and recognizing an intransitive gestural signal, stage 6 represents a more stable communicative form that the child can continue to use in future interactions.
Child
The child intends the mother to come to his side, and performs the ritualized gesture. As a result of continued interaction with his mother, he may learn that the gesture is more or less effective over various distances or in different contexts. The child maintains an association between the gesture and the original action, and is able to substitute the action for the gesture if the gesture is not effective at achieving the child’s goal.
Mother
The mother sees and recognizes his gesture, and responds in a manner that fulfills his goal. Again, we note that the mother would still respond accordingly to the original effective action.
Summary of the Ontogenetic Ritualization of ‘beckon’
The ‘beckoning’ gesture that would result from this 6-stage process can be seen as a truncated (and modified) version of the instrumental act of reach-to-grasp-to-pull. While this case study illustrates general behavioral changes that would occur in the ‘naïve’ ontogenetic ritualization process as it has been discussed in the literature, we do not argue that this particular gesture is necessarily derived in this way. In fact, we note that any model of the acquisition of a gestural form is likely to have idiosyncratic features. What we have described may be seen as a generic analysis of ritualizing effective actions, and not wholly specific to ‘beckoning’. For example, ‘beckoning’ as described in the literature involves a sweeping of the hand or curling of the fingers towards the body. This requires additional machinery beyond what we have described here – perhaps to anticipate the recipient’s movement towards oneself, or perhaps merely to distinguish the form from other similar gestures (e.g., ‘palm-up give’) – just as a ‘nudge’ gesture, or ‘arms-up play’ gesture would require unique mechanisms specific to each gesture. Also, this transition from action to gesture may take several forms – including phenotypic changes – nevertheless elements of the stages we propose are likely to be central to any ritualization processes that yields gestural forms.
Discussion
Computational models have been important in understanding neural control of behavior, from object and action recognition (Bonaiuto et al. 2007; Deco and Rolls 2004; Oztop and Arbib 2002), to saccadic eye control (Dominey et al. 1995; Silver et al. 2012), and visual control of grasping (Fagg and Arbib 1998). Such models have even made useful contributions to our understanding of higher order cognitive skills (O’Reilly and Frank 2006; Rougier et al. 2005). However, our proposed analysis of ontogenetic ritualization presents several unique challenges to the brain modeling community.
Challenges for Dyadic Brain Modeling
The first challenge for social brain models is simply being social. Few brain models to date have incorporated explicitly social tasks that are central to primate behavior and cognition. The Mental State Inference model (MSI) is perhaps the first explicit instance of multiple brains in simulation, and simulates the manual performance of one individual being observed by another. The MSI model is based in large part on the MNS models of action recognition and suggests mirror responses are a single part of an extended ‘internal model’ that serves to decode the intentions of others (Oztop et al. 2005). However, there is no explicit interaction between the agents in the model, and the observations do not affect the subsequent behaviors of the observer – that is, there is no ‘task’. We are not able to predict how observation would affect future performance, and we are still lacking any consideration of interaction between the brains. Thus, the MSI model is a ‘passively social’ model in the same vein as the MNS and MNS2 models (and most of the work on mirror neuron neurophysiology).
A few ‘interactive’ dyadic models have been put forth, but they often lack the neural specificity found in models of passively social or purely instrumental tasks. Taking an interactive approach, Steels and colleagues have modeled multiple interacting agents in ‘language learning’ games, showing interesting results for ideas of grammar learning and cultural transmission (e.g., Steels 2003). However, Steels’ multi-agent simulations lack ‘brains’, and instead describe agents with simplified mechanisms that are highly task-specific. The dynamics of the interactions are enlightening, but say little about the specific brain processes involved in interactions between primates. Other models simulate neural dynamics between interacting agents, but do not engage questions of the computational properties of detailed brain circuitry during a specific task (Dumas et al. 2012).
Our analysis of ontogenetic ritualization diverges from previous work incorporating purely observational social dimensions in that it is an explicitly interactive process of social learning. Social learning encompasses all learning that is modulated by the actions of another individual (Galef and Laland 2005), though different kinds of learning may be distinguished, for example, in the degree to which available environmental information may be processed to influence future behavior (Acerbi et al. 2011). Dyadic learning, like ontogenetic ritualization, might be considered to be an ‘interactive’ form of social learning in that the learner must interact with (rather than just observe) another individual for learning to proceed. (Indeed, both agents are ‘learners’ in ontogenetic ritualization.) In these cases, brain models must show how distributed patterns of neural activation in each individual affect their behavior and how socially-influenced learning processes in the brain given rise to adaptive changes in behavior.
A further challenge for social brain modeling concerns the fact that the motivations underlying social behaviors may not be the homeostatic motivations (hunger, thirst) that are often used in simulations of behavior. During social interactions, animals may perform the same task, but with differing motivations – for example, preferring social information over food rewards (Deaner et al. 2005; Klein et al. 2008). Brain modeling lacks serious consideration of these differing motivational drives during behavior (especially social behavior), and rarely incorporates reward-modulated or motivational responses in neural network models (Arbib and Bonaiuto 2012; Guazzelli et al. 1998). This failure to incorporate motivational elements becomes more consequential when modeling social tasks, which require both navigating the multi-dimensionality that is ‘motivation’ (Berridge 2004) and incorporating the perception of motivations and intentions in partners, thought to be a large driver of cognitive skill evolution in primates (Byrne and Whiten 1988).
Another challenge is to address the debate over the extent to which neural networks and cognitive modules can be said to be ‘domain general’ or ‘domain specific’, and how these systems would interact, especially with respect to social cognition. As a simple example, face-selective neurons have been described in specific regions of temporal cortex (Barraclough and Perrett 2011). Using more complex interactive designs, (Yoshida et al. 2011) describe medial frontal neurons that appear to respond exclusively to social variables (at least for the task studied), and may suggest certain functional specialization within especially frontal and prefrontal regions. Anterior Cingulate Cortex gyrus (ACCg) and Anterior Cingulate Cortex sulcus (ACCs) dissociate, in a lesion study, in their recruitment in response to social variables, with ACCg important for social valuation (Rudebeck et al. 2006) – though see (Chang et al. 2012) for a more complex result. Lastly, the LIP mirror neuron responses in monkeys suggest that ‘integration’ regions like PPC can represent both social and non-social information important for decision-making (Shepherd et al. 2009). Together, these and other data must be analyzed to identify whether and which structures can be said to be ‘social-domain specific’, and how such regions would interact with wider neural systems (Fig. 4).
Linking Neuroinformatics to Gestural and Behavioral Datasets
The dyadic brain approach to social behavior challenges the neuroinformatics community to provide resources to integrate data from neurophysiological and behavioral studies of primates in a way that could provide new insights into the cognitive and structural changes underlying the evolution of primate (and human) communication. We review discipline-wide concerns for managing data, including:
-
Primate behavioral data
-
Primate brain imaging data
-
Macaque neurophysiological data
-
Comparative neuroanatomical data
-
Model simulation results
Behavioral Data Management
Researchers in biological and biomedical sciences have made significant advances in constructing searchable databases and have tackled the challenges of standardizing and archiving data in a range of fields, including within neuroscience (see companion articles). Though these approaches could not be transferred verbatim to data in comparative cognition, the challenges inherent to linking studies and identifying patterns in data are common to all integrative databases and should be used to inform future efforts to consolidate data across studies of primate cognition.
Tomasello and Call (2011) raised concerns about the isolation of individual studies in primate cognition, particularly in relation to gesture studies in the great apes. Differences as to what qualifies as a ‘gesture’ and how gestures are coded and defined lead to significant differences between the conclusions of different studies of the same species, and can reflect local traditions in research groups (Cartmill and Byrne 2011). The potential of drawing erroneous conclusions based on single studies or only studies from a single research group underscores the importance of establishing a consistent ontology of primate social behavior and developing resources for managing, integrating and sharing behavioral data.
The level of comparison or ‘granularity’ of the searchable data between studies is particularly important. Definitions of behaviors or functional responses differ between studies and these differences often make it difficult to directly compare the results of studies without explicitly accounting for differences in methodologies. For example, one paper that surveyed multiple groups of gorillas found a total repertoire of 33 gestures, most of which were shared between zoos (Pika et al. 2003). Another paper reported 102 gestures, also from a survey of multiple zoos (Genty et al. 2009). The differences between the repertoire sizes reported in these two studies are not the result of group-specific gestures or cultural variation between sites. Rather, they result from granularity differences in the researchers’ gesture definitions. The first study defines gestures by the predominant movement involved, but does not typically draw distinctions between gestures based on the limb or hand shape used. Thus all examples of hitting a surface with a hand would be considered a “slap ground” gesture. In the second study, however, the limb and hand shape used are considered part of the gesture definition so “knock object,” “punch object,” “slap object 1-handed,” and “slap object 2-handed” are all recorded as separate gestures. These differences in the granularity of definition could result in the same set of observations yielding drastically different summary results (a potential problem not just for behavioral, but also neural datasets).
On the one hand, it is necessary to accurately record those methodological differences that make direct comparison of results between studies difficult at the moment. On the other hand, it is not practical to fully recode primary source data from different studies according to the same guidelines so that it can be easily pooled. Allowing individual variation in the coding systems not only removes a substantial barrier to contributing data to a collective database, it also allows coding systems to be appropriately tailored to the differences between species’ communication systems. For example, one frequently coded behavior is whether an ape waits for a response from the recipient before giving up or attempting another gesture. This measure of ‘response waiting’ is used as an indication of intentional communication since it signals that the gesturer expects a particular response from the recipient. Since primate species differ in temperament and energy levels, the length of time that suggests waiting for a response is likely to differ. The amount of time thought to indicate response waiting in a low-energy species might be far too long for a high-energy species with a shorter attention span. In this case, it would be better to ignore the differences in definition of response waiting between studies since those differences account for temperament variation and allow the same behavior to be measured across species. Though variation of definitions within a species is likely to cause problems (as in the gorilla gesture example), allowing definitions to vary by species facilitates direct comparison between studies by bringing the cognitive ability rather than the temperamental differences to the forefront.
Longitudinal data are especially valuable because they allow us to ask direct questions about the development of gestures over time, but longitudinal studies in apes are rare and time intensive. Indirect questions about gesture development may be asked by comparing individuals of different age classes between different sites to identify developmental trajectories in gesture use. Incorporating longitudinal data of gesture in the same individuals into a cross-study/cross-site database would be invaluable to the field because the integration of cross-sectional and longitudinal data would allow researchers to ask more sophisticated questions about development within a species, and facilitate comparative studies.
Neural Data Management
Resources for managing functional and neuroanatomical data provide a strong backbone for research in social brain modeling. BrainMap <http://www.brainmap.org/> and Brede <http://neuro.imm.dtu.dk/services/jerne/brede/> are tools for managing and performing meta-analyses for functional neuroimaging data (and see companion articles in this volume), and resources like BrainLiner <http://brainliner.jp/> offer a platform for managing and standardizing neurophysiological data. As non-invasive functional brain monitoring in apes becomes more available, specific resources may have to be developed tailored to the needs of researchers. For neuroanatomical data, the NeuroHomology DataBase (Bota and Arbib 2001), for example, was developed to offer researchers the tools to investigate the relationship between similar brain structures in different species. However, newer techniques like diffusion tensor imaging (DTI) now allow researchers to ask novel questions in a non-invasive design. Recent comparative DTI analyses, for instance, suggest significant differences in fiber pathways linking regions in parietal, temporal and frontal areas between modern primates – specifically between macaques, chimpanzees and humans (Hecht et al. 2012). The results suggest an increase in connectivity between STS and inferior parietal regions – moving from macaques, then to chimpanzee, and finally to humans with robust connectivity – together processing the visual form of movements. These and other neuroanatomical studies may support, for example, model hypotheses regarding connectivity between kinematic-processing structures and action recognition and other structures (see Fig. 2). As these data become more prevalent, efficient ways to handle and link these data with functional and neuro-homology databases becomes more important.
Model Result Management
Software designed for computational neuroscience is widely available (e.g., NEURON; http://www.yale.edu/neuron) and code repositories like ModelDB (http://senselab.med.yale.edu/modeldb) offer researchers ways to share code. These resources and others can often be linked or ‘federated’ to offer access to data from other systems, as the Brain Operations DataBase (BODB) does. BODB (http://bodb.usc.edu/bodb; and see companion articles) currently allows linkages to data sources ranging from neuroanatomical datasets of monkey and human, to functional imaging sets like those offered by BrainMap (<brainmap.org>). BODB also offers tools for managing Summaries of Empirical Data (SEDs) with the goal of facilitating work in computational neuroscience. The SED format is designed to be at the appropriate ‘level’ to offer challenges to existing ideas for brain function, and flexible enough to be understood both in relation to other data, and in relation to specific models of the brain, allowing direct comparison between model simulation results and existing (or future) empirical work against which the simulations can be benchmarked. However, as the above analyses have shown, model benchmarking becomes much harder when the behavior studied – gesture, for example – has differing operational definitions and levels of description.
Ideally, integration should be possible at multiple levels of representation. One study may want to ask how manual gestures are used in different age groups and integrate this information with what is known about primate brain systems involved in action recognition. Another study may focus on the ability to respond to the gaze of potential recipients by using a visual vs. tactile gesture, and may be concerned only with the neural representation of gaze awareness and not the gesture type. Flexibility for future integration and expansion is key. For example, BODB currently offers tools for managing behavioral data, and has the possibility of integrating its functionality with other, more specific databases. It would be possible then to create links between BODB and a future database of primate gesture research, thus enabling a platform to manage behavioral and neuroscientific data. Still, existing resources within neuroscience need more structuring, as the examples of non-human primate brain imaging suggest.
Establishing links between collections of neural and behavioral data and allowing searches to span and connect data in different fields would transform our ability to ask questions about the evolution of cognition, brain, and behavior. Though the creation of integrative databases holds great promise for researchers, barriers to participation in a collective database must be minimized. The simplicity and power of the built-in tools for adding and managing data in a database greatly impact the likelihood that people will contribute data and use the database to conduct research. The behavioral and cognitive sciences are becoming increasingly interdisciplinary and advances in our understanding are more likely to be made by comparing across studies and disciplines than by individuals working on isolated datasets using a unidirectional approach.
In this paper we have argued that social brain modeling is a promising field with potential to combine and extend the insights gained from the neural and behavioral sciences. We use gesture, and specifically the proposed learning of gesture via ontogenetic ritualization, as a test case for the construction of this integrative modeling approach. We focus on gesture because it incorporates social features that are problematic for modeling (e.g., different processes in the signaler and receiver, goal attribution, recognition of social variables like attention, and flexible deployment), but also because it allows us to build on existing models of the production and perception of manual action. As our proposed model of ontogenetic ritualization illustrates, integrating ethological data with models grounded in neural detail offers the possibility to ask targeted questions about social learning and cognition and to make testable predictions about behavioral outcomes – and ultimately to help unravel questions about development and evolution. However, substantial challenges remain. We believe that many of these challenges require innovative new informatics approaches, like the construction of searchable databases that would allow integration of data across studies, fields, and methodologies. We call for a concerted interdisciplinary effort between primatologists, neuroscientists, and computational modelers to consider new collaborative approaches to the integration and maintenance of both raw and summarized data. Even small steps into this interdisciplinary terrain promise to transform the research landscape from isolated studies to richly collaborative conversations, and to open up powerful new approaches to very old questions.
Information Sharing Statement
This article was made feasible through various online resources, including university-maintained journal access.
References
Aboitiz, F. (2012). Gestures, vocalizations and memory in language origins. Frontiers in Evolutionary Neuroscience, 4.
Acerbi, A., Tennie, C., & Nunn, C. L. (2011). Modeling imitation and emulation in constrained search spaces. Learning & Behavior, 39, 104–114.
Adolphs, R. (2010). Conceptual challenges and directions for social neuroscience. Neuron, 65, 752–767.
Amodio, D. M., & Frith, C. D. (2006). Meeting of minds: the medial frontal cortex and social cognition. Nature Reviews Neuroscience, 7, 268–277.
Arbib, M. A. (2005). From monkey-like action recognition to human language: an evolutionary framework for neurolinguistics. The Behavioral and Brain Sciences, 28, 105–124. discussion 25–67.
Arbib, M. A. (2008). From grasp to language: embodied concepts and the challenge of abstraction. Journal of Physiology, Paris, 102, 4–20.
Arbib, M. (2010). Mirror neurons and language. Annual Review of Anthropology, 40, 257–273.
Arbib, M. A. (2012). How the brain got language: The mirror system hypothesis. New York: Oxford University Press.
Arbib, M. A., & Bonaiuto, J. J. (2012). Multiple levels of spatial organization: world graphs and spatial difference learning. Adaptive Behavior, 20, 287–303.
Arbib, M. A., Bischoff, A., Fagg, A. H., & Grafton, S. T. (1995). Synthetic PET: analyzing large-scale properties of neural networks. Human Brain Mapping, 2, 225–233.
Arbib, M. A., Billard, A., Iacoboni, M., & Oztop, E. (2000). Synthetic brain imaging: grasping, mirror neurons and imitation. Neural Networks: the Official Journal of the International Neural Network Society, 13, 975–997.
Arbib, M. A., Liebal, K., & Pika, S. (2008). Primate vocalization, gesture, and the evolution of human language. Current Anthropology, 49, 1053–1076.
Averbeck, B. B., Chafee, M. V., Crowe, D. A., & Georgopoulos, A. P. (2002). Parallel processing of serial movements in prefrontal cortex. Proceedings of the National Academy of Sciences, 99, 13172–13177.
Azzi, J.C.B., Sirigu, A., & Duhamel, J-r. (2011). Modulation of value representation by social context in the primate orbitofrontal cortex.
Barraclough, N. E., & Perrett, D. I. (2011). From single cells to social perception. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 366, 1739–1752.
Barrès, V., Simons, A., & Arbib, M. (2013). Synthetic event-related potentials: a computational bridge between neurolinguistic models and experiments. Neural Networks, 37, 66–92.
Berridge, K. C. (2004). Motivation concepts in behavioral neuroscience. Physiology & Behavior, 81, 179–209.
Bonaiuto, J., & Arbib, M. A. (2010). Extending the mirror neuron system model, II: what did I just do? A new role for mirror neurons. Biological Cybernetics, 102, 341–359.
Bonaiuto, J., Rosta, E., & Arbib, M. (2007). Extending the mirror neuron system model, I. Audible actions and invisible grasps. Biological Cybernetics, 96, 9–38.
Bota, M., & Arbib, M. A. (2001). The NeuroHomology database. Neurocomputing, 38–40, 1627–1631.
Botvinick, M. M., Niv, Y., & Barto, A. C. (2009). Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition, 113, 262–280.
Bruner, J., Roy, C., & Ratner, N. (1982). The beginnings of request. Children’s Language, 3, 91–138.
Bullock, D., & Grossberg, S. (1988). Neural dynamics of planned arm movements: emergent invariants and speed-accuracy properties during trajectory formation. Psychological Review, 95, 49–90.
Buxbaum, L. J., Kyle, K. M., & Menon, R. (2005). On beyond mirror neurons: internal representations subserving imitation and recognition of skilled object-related actions in humans. Brain Research. Cognitive Brain Research, 25, 226–239.
Byrne, R. W. (2004). Culture in the great apes: using intricate complexity in feeding skills to trace the evolutionary origin of human technical prowess. Philosophical Transactions of the Royal Society, Series B, Biological Sciences, 362, 577–585.
Byrne, R. W., & Russon, A. E. (1998). Learning by imitation: a hierarchical approach. The Behavioral and Brain Sciences, 21, 667–684. discussion 84–721.
Byrne, R., & Whiten, A. (1988). Machiavellian intelligence: Social expertise and the evolution of intellect in monkeys, apes, and humans. Oxford: Claredon Press.
Caligiore, D., Borghi, A. M., Parisi, D., & Baldassarre, G. (2010). TRoPICALS: a computational embodied neuroscience model of compatibility effects. Psychological Review, 117, 1188–1228.
Call, J., & Tomasello, M. (2007). The gestural communication of apes and monkeys. New York: Lawrence Erlbaum Associates.
Campos, M., Breznen, B., & Andersen, R. A. (2010). A neural representation of sequential states within an instructed task. Journal of Neurophysiology, 104, 2831–2849.
Cartmill, E. A. (2008). Gestural communication in orangutans (Pongo pygmaeus and Pongo abelii): A cognitive approach. Ph.D. Dissertation St. Andrews: University of St. Andrews.
Cartmill, E. A., & Byrne, R. W. (2011). Studying non-human primate gesture from the perspective of the individual In G. Stam, M. Ishino (Eds.), Integrating gestures (pp. 15–26). Amsterdam: John Benjamins.
Chaminade, T., Oztop, E., Cheng, G., & Kawato, M. (2008). From self-observation to imitation: visuomotor association on a robotic hand. Brain Research Bulletin, 75, 775–784.
Chang, S.W.C., Gariépy, J.-F., & Platt, M.L. (2012). Neuronal reference frames for social decisions in primate frontal cortex. Nature Neuroscience, 16, 243–250.
Cisek, P. (2007). Cortical mechanisms of action selection: the affordance competition hypothesis. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 362, 1585–1599.
Cisek, P., & Kalaska, J. F. (2010). Neural mechanisms for interacting with a world full of action choices. Annual Review of Neuroscience, 33, 269–298.
Clark, E.V. (1978). Strategies for communicating. Child Development, 49(4), 953–959.
Corballis, M. C. (2002). From hand to mouth, the origins of language. Princeton: Princeton University Press.
de Rengerv, A. D, Boucenna, S., Andry, P., & Gaussier, P. (2010). Emergent imitative behavior on a robotic arm based on visuo-motor associative memories. In IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1754–1759).
Deacon, T. W. (1997). The symbolic species: The co-evolution of language and the brain. New York: WW Norton.
Dean, L. G., Kendal, R. L., Schapiro, S. J., Thierry, B., & Laland, K. N. (2012). Identification of the social and cognitive processes underlying human cumulative culture. Science, 335, 1114–1118.
Deaner, R. O., Khera, A. V., & Platt, M. L. (2005). Monkeys pay per view: adaptive valuation of social images by rhesus macaques. Current Biology, 15, 543–548.
Deco, G., & Rolls, E. T. (2004). A Neurodynamical cortical model of visual attention and invariant object recognition. Vision Research, 44, 621–642.
Demiris, Y., & Hayes, G. (2002). Imitation as a dual-route process featuring predictive and learning components: a biologically-plausible computational model. In K. Dautenhahn, C. Nehaniv (Eds) Imitation in Animals and Artifacts (pp. 327–361). Cambridge, MA: MIT Press.
di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V., & Rizzolatti, G. (1992). Understanding motor events: a neurophysiological study. Experimental Brain Research, 91, 176–180.
Dominey, P. F., Arbib, M. A., & Joseph, J. P. (1995). A model of corticostriatal plasticity for learning oculomotor associations and sequences. Journal of Cognitive Neuroscience, 7, 311–336.
Dumas, G., Chavez, M., Nadel, J., & Martinerie, J. (2012). Anatomical connectivity influences both intra- and inter-brain synchronizations. PloS One, 7, e36414–e14.
Dunbar, R. I. M. (1998). The social brain hypothesis. Evolutionary Anthropology: Issues, News, and Reviews, 6, 178–190.
Fagg, A. H., & Arbib, M. A. (1998). Modeling parietal-premotor interactions in primate control of grasping. Neural Networks: the Official Journal of the International Neural Network Society, 11, 1277–1303.
Ferrari, P. F., Paukner, A., Ionica, C., & Suomi, S. J. (2009). Reciprocal face-to-face communication between rhesus macaque mothers and their newborn infants. Current Biology, 19, 1768–1772.
Fujii, N., Hihara, S., Nagasaka, Y., & Iriki, A. (2008). Social state representation in prefrontal cortex. Social Neuroscience, 4(1), 73–84.
Galef, B. G., & Laland, K. N. (2005). Social learning in animals: empirical studies and theoretical models. BioScience, 55, 489–89.
Gallese, V., Fadiga, L., Fogassi, L., & Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain, 119, 593–609.
Genty, E., Breuer, T., Hobaiter, C., & Byrne, R. W. (2009). Gestural communication of the gorilla (Gorilla gorilla): repertoire, intentionality and possible origins. Animal Cognition, 12, 527–546.
Georgopoulos, A. P. (2002). Cognitive motor control: spatial and temporal aspects. Current Opinion in Neurobiology, 12, 678–683.
Georgopoulos, A. P., Kalaska, J. F., & Massey, J. T. (1981). Spatial trajectories and reaction times of aimed movements: effects of practice, uncertainty, and change in target location. Journal of Neurophysiology, 46, 725–743.
Guazzelli, A., Corbacho, F. J., Bota, M., & Arbib, M. A. (1998). Affordances, motivation, and the world graph theory. Adaptive Behavior, 6, 435–471.
Hecht, E. E., Gutman, D. A., Preuss, T. M., Sanchez, M. M., Parr, L. A., & Rilling, J. K. (2012). Process versus product in social learning: comparative diffusion tensor imaging of neural systems for action execution-observation matching in macaques, chimpanzees, and humans. Cerebral Cortex.
Hirata, S., Matsuda, G., Ueno, A., Fuwa, K., Sugama, K., et al. (2011). Event-related potentials in response to subjects’ own names: a comparison between humans and a chimpanzee. Communicative & Integrative Biology, 4, 321–323.
Hobaiter, C., & Byrne, R. W. (2010). Able-bodied wild chimpanzees imitate a motor procedure used by a disabled individual to overcome handicap. PloS One, 5, e11959–e59.
Hobaiter, C., & Byrne, R.W. (2011). The gestural repertoire of the wild chimpanzee. Animal cognition, 14(5), 745–767.
Hoff, B., & Arbib, M. A. (1993). Models of trajectory formation and temporal interaction of reach and grasp. Journal of Motor Behavior, 25, 175–192.
Hopkins, W. D., Coulon, O., & Mangin, J. F. (2010). Observer-independent characterization of sulcal landmarks and depth asymmetry in the central sulcus of the chimpanzee brain. Neuroscience, 171, 544–551.
Jeannerod, M., & Biguer, B. (1982). Visuomotor mechanisms in reaching within extra-personal space. In D. J. Ingle, R. J. W. Mansfield, & M. A. Goodale (Eds.), Advances in the analysis of visual behavior (pp. 387–409). Cambridge: The MIT Press.
Jeannerod, M., Arbib, M. A., Rizzolatti, G., & Sakata, H. (1995). Grasping objects: the cortical mechanisms of visuomotor transformation. Trends in Neurosciences, 18, 314–320.
Kano, F., & Tomonaga, M. (2009). How chimpanzees look at pictures: a comparative eye-tracking study. Proceedings of the Royal Society, series B, 276, 1949–1955.
Keller, S. S., Deppe, M., Herbin, M., & Gilissen, E. (2012). Variability and asymmetry of the sulcal contours defining Broca’s area homologue in the chimpanzee brain. The Journal of Comparative Neurology, 520, 1165–1180.
Klein, J. T., Deaner, R. O., & Platt, M. L. (2008). Neural correlates of social target value in macaque parietal cortex. Current Biology, 18, 419–424.
Kohler, E., Keysers, C., Umilta, M. A., Fogassi, L., Gallese, V., & Rizzolatti, G. (2002). Hearing sounds, understanding actions: action representation in mirror neurons. Science, 297, 846–848.
Lee, D., McGreevy, B. P., & Barraclough, D. J. (2005). Learning and decision making in monkeys during a rock-paper-scissors game. Brain Research. Cognitive Brain Research, 25, 416–430.
Liebal, K., Call, J., & Tomasello, M. (2004a). Use of gesture sequences in chimpanzees. American Journal of Primatology, 64, 377–396.
Liebal, K., Pika, S., & Tomasello, M. (2004b). Great ape communicators move in front of recipients before producing visual gestures. Interaction Studies, 5, 199–219.
Liebal, K., Pika, S., & Tomasello, M. (2006). Gestural communication of orangutans (Pongo pygmaeus). Gesture, 6, 1–38.
Lock, A. (2001). Preverbal communication. In J. G. Bremner & A. Fogel (Eds.), Blackwell handbook of infant development (pp. 379–403). Oxford: Blackwell.
Nishida, T. (1980). The leaf-clipping display: a newly discovered expressive gesture in wild chimpanzees. Journal of Human Evolution, 9, 117–128.
O’Reilly, R. C., & Frank, M. J. (2006). Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Computation, 18, 283–328.
Oztop, E., & Arbib, M. A. (2002). Schema design and implementation of the grasp-related mirror neuron system. Biological Cybernetics, 87, 116–140.
Oztop, E., Wolpert, D., & Kawato, M. (2005). Mental state inference using visual control parameters. Cognitive Brain Research, 22, 129–151.
Parr, L. A., Hecht, E., Barks, S. K., Preuss, T. M., & Votaw, J. R. (2009). Face processing in the chimpanzee brain. Current Biology, 19, 50–53.
Perlman, M., Tanner, J. E., & King, B. J. (2012). In S. Pika & K. Liebal (Eds.), A mother gorilla’s variable use of touch to guide her infant: Insights into iconicity and the relationship between gesture and action (pp. 55–73). Amsterdam: John Benjamins.
Petreska, B., Adriani, M., Blanke, O., & Billard, A. G. (2007). Apraxia: a review. Progress in Brain Research, 164, 61–83.
Pika, S., & Liebal, K. (Eds.). (2012). Developments in primate gesture research. Amsterdam: John Benjamins.
Pika, S., Liebal, K., & Tomasello, M. (2003). Gestural communication in young gorillas (Gorilla gorilla): gestural repertoire, learning, and use. American Journal of Primatology, 60, 95–111.
Pika, S., Liebal, K., & Tomasello, M. (2005). Gestural communication in subadult bonobos (Pan paniscus): Repertoire and use. American Journal of Primatology, 65, 39–61.
Ramayya, A. G., Glasser, M. F., & Rilling, J. K. (2010). A DTI investigation of neural substrates supporting tool use. Cerebral Cortex, 20, 507–516.
Rilling, J. K., Glasser, M. F., Preuss, T. M., Ma, X., Zhao, T., et al. (2008). The evolution of the arcuate fasciculus revealed with comparative DTI. Nature Neuroscience, 11, 426–428.
Rilling, J. K., Scholz, J., Preuss, T. M., Glasser, M. F., Errangi, B. K., & Behrens, T. E. (2011). Differences between chimpanzees and bonobos in neural systems supporting social cognition. Social cognitive and affective neuroscience, 7(4), 369–379.
Rizzolatti, G., Gentilucci, M., Fogassi, L., Luppino, G., Matelli, M., & Ponzoni-Maggi, S. (1987). Neurons related to goal-directed motor acts in inferior area 6 of the macaque monkey. Experimental Brain Research, 67, 220–224.
Rothi, L. J. G., Heilman, K. M., Affairs, V., & Centre, M. (1991). A Cognitive Neuropsychological Model of Limb Praxis. Cognitive Neuropsychology, 8(6), 443–458.
Rougier, N. P., Noelle, D. C., Braver, T. S., Cohen, J. D., & O’Reilly, R. C. (2005). Prefrontal cortex and flexible cognitive control: rules without symbols. Proceedings of the National Academy of Sciences of the United States of America, 102, 7338–7343.
Rudebeck, P. H., Buckley, M. J., Walton, M. E., & Rushworth, M. F. S. (2006). A role for the macaque anterior cingulate gyrus in social valuation. Science (New York, N.Y.), 313, 1310–1312.
Sakai, T., Mikami, A., Tomonaga, M., Matsui, M., Suzuki, J., et al. (2011). Differential prefrontal white matter development in chimpanzees and humans. Current Biology, 21, 1397–1402.
Sallet, J., Mars, R. B., Noonan, M. P., Andersson, J. L., O’Reilly, J. X., et al. (2011). Social network size affects neural circuits in macaques. Science, 334, 697–700.
Sawamura, H., Shima, K., & Tanji, J. (2002). Numerical representation for action in the parietal cortex of the monkey. Nature, 415, 918–922.
Schenker, N. M., Hopkins, W. D., Spocter, M. A., Garrison, A. R., Stimpson, C. D., et al. (2010). Broca’s area homologue in chimpanzees (Pan troglodytes): probabilistic mapping, asymmetry, and comparison to humans. Cerebral Cortex (New York, N.Y.: 1991), 20, 730–742.
Seo, H., & Lee, D. (2007). Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game. The Journal of Neuroscience: the Official Journal of the Society for Neuroscience, 27, 8366–8377.
Seo, H., Barraclough, D. J., & Lee, D. (2009). Lateral intraparietal cortex and reinforcement learning during a mixed-strategy game. The Journal of Neuroscience: the Official Journal of the Society for Neuroscience, 29, 7278–7289.
Shepherd, S. V., Klein, J. T., Deaner, R. O., & Platt, M. L. (2009). Mirroring of attention by neurons in macaque parietal cortex. Proceedings of the National Academy of Sciences, 106, 9489–9494.
Shilbach, L., Timmermans, B., Reddy, V., Costall, A., & Bente, G. et al. (2012). Toward a second-person neuroscience. Behavioral & Brain Sciences. (in press).
Silver, M. R., Grossberg, S., Bullock, D., Histed, M. H., & Miller, E. K. (2012). A neural model of sequential movement planning and control of eye movements: item-order-rank working memory and saccade selection by the supplementary eye fields. Neural Networks, 26, 29–58.
Steels, L. (2003). Evolving grounded communication for robots. Trends in Cognitive Sciences, 7, 308–312.
Sutton, R., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge: MIT Press.
Taglialatela, J. P., Russell, J. L., Schaeffer, J. A., & Hopkins, W. D. (2011). Chimpanzee vocal signaling points to a multimodal origin of human language. PloS One, 6, e18852–e52.
Taira, M., Mine, S., Georgopoulos, A. P., Murata, A., & Sakata, H. (1990). Parietal cortex neurons of the monkey related to the visual guidance of hand movement. Experimental Brain Research, 83, 29–36.
Tomasello, M., & Call, J. (1997). Primate cognition. New York: Oxford University Press.
Tomasello, M., & Call, J. (2007). Intentional communication in nonhuman primates. In J. Call & M. Tomasello (Eds.), The gestural communication of apes and monkeys (pp. 1–15). Mahwah: Lawrence Erlbaum Associates.
Tomasello, M., & Call, J. (2011). Methodological challenges in the study of primate cognition. Science, 334, 1227–1228.
Tomasello, M., & Camaioni, L. (1997). A comparison of the gestural communiction of apes and human infants. Human Development, 40, 7–24.
Ueno, A., Hirata, S., Fuwa, K., Sugama, K., Kusunoki, K., et al. (2008). Auditory ERPs to stimulus deviance in an awake chimpanzee (Pan troglodytes): towards hominid cognitive neurosciences. PloS One, 3, e1442–e42.
van Schaik, C. P., Ancrenaz, N., Borgen, G., Galdikas, B., Knott, C. D., et al. (2003). Orangutan cultures and the evolution of the material culture. Science, 299, 102–105.
Watanabe, M. (2007). Role of anticipated reward in cognitive behavioral control. Current Opinion in Neurobiology, 17, 213–219.
Watanabe, M., & Sakagami, M. (2007). Integration of cognitive and motivational context information in the primate prefrontal cortex. Cerebral Cortex, 17(Suppl 1), i101–i109.
Whiten, A., Goodall, J., McGrew, C., Nishida, T., Reynolds, V., et al. (1999). Cultures in chimpanzees. Nature, 399, 682–685.
Whiten, A., Goodall, J., McGrew, W. C., Nishida, T., Reynolds, V., et al. (2001). Changing cultural variation in chimpanzees. Behaviour, 138, 1481–1516.
Yoshida, K., Saito, N., Iriki, A., & Isoda, M. (2011). Representation of others’ action by neurons in monkey medial frontal cortex. Current Biology, 21, 249–253.
Yoshida, K., Saito, N., Iriki, A., & Isoda, M. (2012). Social error monitoring in macaque frontal cortex. Nature Neuroscience, 15, 1307–1312.
Zukow-Goldring, P., & Arbib, M. A. (2007). Affordances, effectivities, and assisted imitation: caregivers and the directing of attention. Neurocomputing, 70, 2181–2193.
Acknowledgments
We thank the participants of the Workshop on Action, Language, and Neuroinformatics at the University of Southern California for their helpful discussion and insights and thank J. G. Foster for his comments on the manuscript. Part of this work was funded through NSF grant number 0924674, Michael A. Arbib, Principal Investigator.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Gasser, B., Cartmill, E.A. & Arbib, M.A. Ontogenetic Ritualization of Primate Gesture as a Case Study in Dyadic Brain Modeling. Neuroinform 12, 93–109 (2014). https://doi.org/10.1007/s12021-013-9182-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12021-013-9182-5