Early naturalists described the behavior of the animals that they observed in terms of the sights, sounds, and smells that the animals produced while interacting with one another and treated these separate but simultaneously produced displays as integrated performances (Darwin 1871, 1872; Huxley 1916). As the study of animal communication shifted from these holistic descriptions to more experimental studies, researchers began to focus on each signal separately or may have neglected one signal entirely. The signal chosen often reflected constraints on our ability to quantify and manipulate the structure of a particular signal rather than the importance of one signal relative to the others. The courtship displays of birds, which often combine visual displays and sounds, are an excellent example. The study of sound and the ability to manipulate sounds preceded our ability to capture, quantify and manipulate visual signals by several decades (Baker 2001; Oliveira et al. 2000), hence the disparity in the body of literature focusing on bird song compared to that focusing on the visual displays of birds. The recent resurgence in interest in these simultaneous signals (referred to as “multimodal signals”) has markedly increased our understanding of the function and design of these behaviors (for reviews see Rowe 1999; Candolin 2003; Partan and Marler 2005; Hebets and Papaj 2005; Otovic and Partan 2009; Uetz 2010). However, there are several areas of research that are under-represented in the current literature. The focus of this paper is to suggest areas of multimodal signaling that would profit from further empirical study and to provide a new heuristic using 3D surface response plots that will aid in the understanding of new and existing areas of research. Understudied areas include: (1) the importance of variation within and between signals in different modalities to signal function and signal efficacy; (2) interactions between signaling modality and signaling environment; and (3) greater empirical exploration of the selective pressures that may have shaped multimodal signals. Lastly, we examine multicomponent multimodal signals.

Terminology

A signal can be broadly defined as any aspect of the signaler that affects the behavior of the receiver (Bradbury and Vehrencamp 1998). This definition encompasses both static signaling components and dynamic signaling components. Static signal components, such as plumage coloration, body size or non-volatile scents, are an enduring part of the signaler. These cannot be turned “off”. Dynamic signal components have a limited duration and require an action by the signaler to initiate (turn “on”) and to terminate the signal. These signals can include any action detectable by a receiver such as the production of sounds, visual movements, volatile scents or electrical discharges. The term “modality” is used to describe the sensory system with which a signal is produced by the signaler and perceived by a receiver. Signals that can be detected by two or more of the receiver’s sensory systems (e.g., auditory and visual systems) are referred to as multimodal signals (Otovic and Partan 2009). In contrast, multiple signals within the same modality, such as the blue body coloration and display movements of Sceloporus lizards, which would be detected using the same sensory system (Quinn and Hews 2010), are often referred to as “multicomponent signals” (Rowe 1999). However, it is important to recognize the existence of “multicomponent multimodal signals”. These combine multiple signals within a sensory modality with signals from other modalities. There are many examples of species that combine color badges, display movements and sounds, such as red-winged blackbirds (Agelaius phoeniceus), jungle fowl (Gallus gallus), and squirrel treefrogs (Hyla squirella).

Classifying multimodal signals

Central to our understanding of the form and function of multimodal signals is an understanding of the interaction between the two modalities. Previous research has demonstrated that interactions between signal modalities can alter signal detectability and signal discrimination (Rowe 1999; Hebets and Papaj 2005). Furthermore, the combined multimodal signal may also generate unique receiver behaviors not elicited by the unimodal components alone (e.g., mate recognition in crayfish, Austropotamobius pallipes (Acquistapace et al. 2002) and in aggressive interactions in dart-poison frogs, Epipedobates femoralis (Narins et al. 2003)).

Receiver responses are often used to determine the function and information content of multimodal signals. Signals that elicit the same type of response to each modality are referred to as redundant (“backup” signals, Johnstone 1996). These signals transmit the same information in each modality and can be further divided based on different receiver responses (Partan and Marler 1999; 2005; Partan 2004). Equivalent signals evoke the same responses singly or combined. Enhancement signals elicit a higher level of response when combined such that the two signals may have an additive effect (i.e., a slight increase over the individual response levels or summation of the two effects) or a multiplicative effect wherein the combined response exceeds the sum of the individual responses. In contrast, nonredundant signals can convey different information in each modality and the combined signal can alter the information content. The two signals may elicit two distinct responses, one may overshadow the other, the combined signal may elicit a new response not seen as a response to either modality alone, or one may modulate the response to the other (referred to as independence, dominance, emergence, and modulation respectively, sensu Partan and Marler 1999).

The most common method of testing the function and efficacy of multimodal signals is to present a receiver with each unimodal signal separately and to then compare these responses to the effects of the combined multimodal signal. The benefit of using these “cue-isolation” experiments is that they allow for independent manipulation of signal parameters in a specific modality. In these experiments, the individual components are treated in a binary manner as either present or absent and the exemplars used typically represent an average display in each modality. While this provides a general categorization of the signals, it does not allow for full exploration of the natural signal production and receiver responses to variation in each modality.

To address this limitation, we have developed the use of 3D surface plots to visualize the many different ways in which signaler performance in two sensory modalities might combine to determine overall receiver response. This approach takes into account the natural variation in performance within and between each modality to determine their relative contribution to signal function and signal efficacy. This heuristic is intended to facilitate interpretation of signal production and receiver responses. It is a complementary approach to the traditional “cue-isolation” experiments and may provide more in-depth insight into the parameters that determine multimodal signal function and signal efficacy by revealing receiver preferences for specific signal parameters.

We have also identified several research areas to which the heuristic can be profitably applied. These areas have received some research attention but the more detailed exploration that this heuristic provides is essential as the field of multimodal signaling advances.

Impact of environmental variability on multimodal signals

Despite the growing interest in multimodal signals, the study of multimodal communication still lags behind our understanding of the factors that affect unimodal signals. One critical gap in the current literature is research on the effect of environmental noise on multimodal signals. The signaling environment and behavioral plasticity of the signaler and receiver have been shown to interact to determine the overall signal efficacy (Endler and Théry 1996; Heuschele et al. 2009; Wilgers and Hebets 2011) and the affect of environmental “noise” on unimodal signals has been extensively studied for most modalities (e.g., color (Endler 1992), visual displays (Ord et al. 2010), acoustic signals (Catchpole and Slatter 1996), chemical signals (Zimmer-Faust 1991)). Previous research has demonstrated that many animals exhibit specific adaptations in signal production to overcome signal masking in one modality (Klump 1996; Brum and Zollinger 2011). For example, Old World monkeys (Sinnott et al. 1975) and New World monkeys (Brumm et al. 2004) increase the intensity of their vocalizations when background noise is increased. Fowl (G. gallus) increase the amplitude of their distress calls in response to increased background sound, an involuntary response known as the Lombard effect (Brumm et al. 2009).

In contrast, how environmental noise affects signalers production of multimodal signals has received less attention. Of greatest interest may be whether signalers exhibit flexibility in their signal production in response to interference in one or both modalities. In some species, expression of the two signals is obligatory [“fixed” (Smith 1977)] whereas in others, the modalities can be flexibly combined [“fluid” or “free” (Smith 1977; Wickler 1978)]. Fixed signals always co-occur. The modalities are physiologically linked such that the performance of the signal in one modality is dependent on the other modality (e.g., the production of sonation through wing stridulation in club-winged manakins (Machaeropterus deliciosus, Bostwick and Prum 2005)). In fluid signals, the performance in one modality may be partially or entirely decoupled from the other (e.g., visual and acoustic courtship displays in fowl (G. gallus, Smith et al. 2011)). This raises the question of how environmental noise in one or both modalities may affect the production of each signal modality.

Flexibility in signal production in response to noise in one modality has been demonstrated in the courtship displays of male wolf spiders, Schizocosa ocreata (Gordon and Uetz 2011). These displays are multimodal, consisting of visual and seismic signals. The signaling habitat consists of leaf litter, in which seismic signals transmit well, and hard or loose substrates, which dampen the seismic signals. Males used significantly more visual signals on substrates that attenuated seismic signals (such as soil and rocks), however, the vibrational signal remained unchanged. Thus males of this species are increasing their signaling effort in the modality in which there is less interference, whilst maintaining effort in the modality that is attenuated and less likely to be detected by conspecifics. The heuristic described herein provides a framework for testing more broadly if changes in signal production in one or both modalities in response to environmental noise are common across different taxa.

The heuristic can also be used to answer the complementary research question of how variation in signaling effort between modalities affects the receiver’s response. In the case of the wolf spider, females did not exhibit a shift in preference and instead were still more receptive to the males courting on substrates that transmitted the vibrational signal. However, research in other species reveals that as the signal-to-noise ratio increases in one modality the receiver may weight more heavily the signal in the second modality (Ernst and Banks 2002). For example, humans rely on both the visual and auditory modalities to understand speech, however, more attention is given to the visual modality when the signal in the auditory modality is degraded (Massaro 1998). Gray squirrels (Sciurus carolinensis) exhibit a similar shift in their responses to the multimodal warning signals (visual tail flicks and vocalizations) of conspecifics. Partan et al. (2010) demonstrated that each component was sufficient to evoke antipredator responses. However, in the presence of anthropogenic acoustic masking, squirrels appear to change their weighting of the two signals, relying more heavily on the visual signal in the presence of auditory interference. These results demonstrate the importance of examining the relative effect of each modality on the receiver’s response. The heuristic is designed specifically to allow researchers to visualize these types of interactions between receiver responses to changes in each modality.

Costs and benefits of multimodal signals

Multimodal signals have been shown to enhance signal detectability (Guilford and Dawkins 1991; Guilford and Dawkins 1993; Rowe 1999). Signals that can be perceived by more than one sensory modality may be more likely to exceed the receiver’s signal detection threshold and therefore increase the probability of signal reception (Rowe 1999; Candolin 2003). However, communication often occurs within a matrix of intended and unintended receivers (referred to as “eavesdroppers”, Peake 2005) and greater detectability may increase the costs of signaling by attracting eavesdroppers such as competitors, predators or parasites (Burk 1982; Endler 1991; Rosenthal et al. 2001; Partan and Marler 2005). Research on wolf spiders (Araneae: Lycosidae) provides an excellent example of the potential benefits and costs of the increased detectability of multimodal signals. Uetz (2000) showed that courtship displays in wolf spiders (S. ocreata) that include signals in more than one modality are often more effective at attracting females than unimodal signals, which should benefit the male by increasing his mating opportunities. However, the same signal parameters that increase signal detectability to the female may also increase the likelihood of the signaler being detected by predators (Roberts et al. 2007). In this study, predators responded more quickly to the male’s combined visual and seismic courtship signals than to visual alone. Using a different species of wolf spider (Schizocosa uetzi), Hebets (2005) demonstrate that females attending to a multimodal courtship display were more likely to be captured by a simulated predator. These results demonstrate the one of the potential costs of multimodal signals to both the sender and the intended receiver. The performance of a multimodal signal will likely reflect the opposing selective forces of the benefit of communicating with conspecifics and the costs of increased predation risk (Johnstone 1998). While the effect of eavesdroppers on signal design has been well studied in unimodal signaling (see McGregor 2005), there is a significant need for comparable studies, like the above example, of multimodal signals. The heuristic presented below may aid in visualizing the affect of eavesdroppers on signal design.

Mapping relationships between signals

The goal of this heuristic is to facilitate interpretation of the many ways in which signals in different modalities (or multiple signals within the same modality) may contribute to signal function and signal efficacy. The heuristic also provides a framework in which to examine the interactions between each signal and its signaling environment as well as potential selective pressures on multiple signals from the intended receivers’ and potential predators’ sensory capabilities. For simplicity, we will outline the data collection steps using the example of a visual and vocal courtship multimodal signal, including several examples of surface response plots showing the interaction between signal modalities for this hypothetical courtship display. We then outline other applications of this heuristic.

The first step in using this heuristic is to determine the natural variation within and between each modality for the multimodal signal of interest. Precise quantitative descriptions of the characteristics of both modalities are needed to determine whether specific variables within each of the two sensory modalities co-vary systematically. Highly correlated signal dimensions in different modalities may provide “redundant” information that potentially enhances signal performance when receivers face constraints on signal processing, either because of sensory or perceptual limitations or because of environmental constraints (e.g., noisy conditions). Conversely, if different modalities were to vary independently, then each could potentially provide different information to receivers (Partan and Marler 1999), supplementing that available in the other modality and increasing the effective bandwidth of the signal. Once the structure of each signal has been determined, a correlation matrix of the relationships among signal attributes, both within and across modalities, can be derived. Measurement of these parameters is necessary to create realistic tests to examine the receivers’ overall responses to the individual modalities.

The second step is to establish a common metric that will enable comparisons of the relative contribution of each modality to the overall receiver responses. This is a challenge because of the lack of common physical units (i.e., there is no a priori way to create comparable scales for acoustic characteristics, in hertz, and rate of movements, per second) thus the two metrics should be chosen based on characteristics that are most likely to affect receiver responses. In a courtship display, for example, the rate of performance is most commonly associated with female preferences (Ryan and Keddy-Hector 1992) and therefore may be a useful parameter in designing a common metric (e.g., rate of notes per min and number of displays per min). The receiver responses can then be analyzed based on playbacks in which the performance rate in each modality is manipulated in units based upon estimated natural distributions, presenting mean values, +1sd, +2sd, −1sd, and −2sd. This approach, which was originally developed for the analysis of acoustic signals (Gaioni and Evans 1986), provides a good overall assessment of female response relative to the range of male signaling behavior that would be experienced in nature. If distributions of performance rate deviate substantially from normal, then the median, 75 percentile, 95 percentile, 25 percentile, and 5 percentile values, which define similar probabilities relative to the population as a whole, should be used. Both scaling methods generate a matrix of 25 (5 × 5) possible unique combinations of performance rate in the two modalities.

The goal is to generate similarly realistic exemplars within each modality by presenting rate-appropriate levels for all other intra-modality dimensions of performance. Appropriate values would be available from the analyses conducted in the first step, which can be used to create a correlation matrix of natural variation for each parameter and the relationships among them. These data can be used to derive playbacks that take into account the signal parameters that co-vary in both modalities. Using natural variation within a modality is preferable to holding all related parameters at an arbitrary constant (e.g., the sample population mean) and simply increasing or decreasing the rates without regard to co-variation within a modality. For example, individuals that are physiologically capable of calling at a higher rate may also have lower fundamental frequency and a narrower frequency bandwidth (i.e., more tonal quality of call) than population-typical individuals. If simulated calls are titrated away from the mean call rate using fixed values for other call parameters, they would move quickly outside the bounds of natural performance overall, introducing a confound of increasing abnormality. Attention can then be focused on testing female responsiveness to combinations of acoustic and visual signals, both when these aspects of performance are congruent and when they are opposed (e.g., a male calling at a high rate but with an improbably slow visual display rate), across a wide range of values.

Once the data are collected for each cell, a response surface plot can be generated using variation in the signaling parameters of each modality represented along axes defining the floor (X- and Y-axis) and variation in receiver response represented vertically (Z-axis). This will create a unique, multi-dimensional, quantitative map revealing the integration of the two modalities. The shape of resulting response surfaces can accommodate almost every extant hypothesis predicting how levels of performance in different modalities might interact to drive receiver behavior. Some of the more obvious possibilities are mapped out in Fig. 1. We use the general term “performance” here to reflect variation in production of the signals (e.g., rate of calls and rate of movements or concentration of pheromones and rate of wing beats) because the heuristic can be applied to any two simultaneous signals. We use the generic term “response” to refer to the behavior of interest elicited by the signals (e.g., latency to orient, duration of behavior). Figure 1a represents a multimodal signal in which the receiver response is insensitive to variation in signal performance in each modality. The receiver response is the same whether both modalities are two standard deviations above the average performance or one modality is two standard deviations below and the other modality is two standard deviations above the average performance. Therefore, this multimodal signal has equal effects in both modalities, which suggests that the two signals act as a “backup” against environmental interference (“redundant equivalence” sensu Partan and Marler 1999). Figure 1b reveals that changes in the performance of modality A have no effect on the response to modality B (“dominance” sensu Partan and Marler 1999). Note that this response curve suggests that receivers respond most strongly to signal performance that is two standard deviations above the sample population average, which suggests a receiver preference for signalers capable of the highest level of performance. Figure 1c shows an additive effect in each modality up to a maximum level of response. Figure 1d shows a threshold effect for one modality, below which the receiver does not respond to either modality. This is similar to an “emergent” signal, wherein a new response is generated by the combination of signals but it not seen to one signal alone (Partan and Marler 2005). Once the signal performance is above the threshold, there is an additive effect (each incremental increase in performance results in an equal increase in the receiver response but only to one modality). Figure 1e reveals that the interaction between the two modalities has multiplicative effect on receiver response (in this example the response is four times higher when modality A and B are two standard deviations greater than the response to modality A and B at the sample population mean). There are many more subtle possibilities, all of which can be accommodated.

Fig. 1
figure 1

Surface plot of receiver responses to variation in signal performance in each modality X-axis and Y-axis are plotted using the mean and ±2 standard deviations of signaler performance for the specific modality. Z-axis shows the receiver’s level of response to each cell of the 25-cell matrix. Hypothetical surface plots representing: a receiver response insensitive to difference between the two modalities (“redundant equivalence” sensu Partan and Marler 1999); b changes in the performance of modality A have no effect on the response to modality B (“dominance” sensu Partan and Marler 1999); c an additive effect in each modality up to a maximum level of response; d threshold effect, below which the receiver does not respond to either modality. Once the signal performance is above the threshold, there is an additive effect (each incremental increase in performance results in an equal increase in the receiver response but only to one modality). e Interaction between the two modalities has multiplicative effect on receiver response

Full size image

Applying the heuristic

We have used rate of courtship performance of a visual and vocal multimodal signal as an example for ease of interpretation. However, the variation in each modality can be used for any number of factors that may reveal signal function or affect signal efficacy. As Wickler (1978) observed, the signals the receiver perceives may be very different from the original transmission. He pointed out that nearby receivers might perceive a multimodal signal whereas more distant receivers would experience a unimodal signal. This is due to the differential signal propagation of the two modalities through the physical environment (Uetz 2000; Elias et al. 2010). The majority of the previous research has focused on the active space of signals in a single modality (e.g., Lohr et al. 2003 (auditory); Ord et al. 2010 (visual); Weissburg 2011(chemical)), rather than on how the interaction between the two modalities affects the active space of the signal. This represents a critical gap in the current research. The heuristic can be used to examine the difference in the active space of the two modalities by systematically degrading or intensifying each signal and testing the receiver’s responses. Figure 1c shows a hypothetical response map for this scenario. As signal intensity increases in both modalities, there is a linear increase in the receiver response, up to a maximum possible level of response. This suggests that the signal in one modality could be severely degraded but the receiver response would be unaffected, meaning that the signals are redundant.

Examining the impact of environmental variability on multimodal signals

This heuristic can also aid in the interpretation of the effects of the signaling environment on signal production in both modalities. It can reveal if signalers exhibit flexibility in their signal production in response to environmental noise. Using the example of visual and acoustic environmental noise on a visual and vocal multimodal signal, the researcher would first measure environmental noise in both modalities (e.g., movement of vegetation due to the effect of wind (e.g., Peters et al. 2002) and sound levels of leaf movement and wind in the environment (e.g., Slabbekoorn 2004)). The researcher would then test the signaler under varying environmental conditions in each modality. The environmental conditions and signaler performance could then be graphed with visual noise (e.g., background vegetation movement) on the X-axis, acoustic noise (e.g., wind generated sound) on the Y-axis and two Z-axes (one for the signal performance in each modality). If the signaler were insensitive to background noise in either modality, then the plot would be flat at the population average. If the modalities co-vary together, then the data in both of the Z-axes would show the same shape. However, if the modalities vary independently, then shapes would reveal the type and extent of signal flexibility for each modality as signalers switched their signaling effort to the channel with less interference.

Visualizing the costs and benefits of multimodal signals

Since most communication occurs within a matrix of intended and unintended receivers and it has been demonstrated that eavesdroppers (such as conspecific competitors) and “cue-readers” (such as predators) alter the costs associated with signaling (Peake and Brown 2006), our approach may provide insights into how these unintended receivers may have shaped the form of multimodal signals. By testing systematically the responses of the intended receiver to a range of signal levels in each modality, we can determine the best overall signal form for detectability by conspecifics. We may then conduct a similar test using a predator as the receiver and then compare the predator’s response curve to the conspecific’s responses, and both of these to the typical signal performance. Differences in these curves would suggest trade-offs between the costs and benefits of conspicuousness in each modality.

Assessing multicomponent signals

The proposed heuristic can also be profitably applied to multiple signals within the same modality (“multicomponent signals”; Rowe 1999). This approach can easily be used to distinguish between the possible explanations for phenomenon such as the presence of multiple secondary sexual characteristics in males (Johnstone 1995, 1996). Møller and Pomiankowski (1993) proposed three possible explanations why males of many species have multiple ornaments: multiple messages, redundant signals, and the unreliable signal hypothesis. If each ornament reflects one property of overall mate quality (i.e. multiple traits provide multiple messages), then the receivers are predicted to integrate information from each component in their evaluation of overall signaler quality. Multiple messages should reduce the receiver’s uncertainty about the specific qualities of the potential mate. For instance, one ornament may reflect developmentally mediated traits and another may indicate the signaler’s current condition. These qualities may be weighted differently by the receiver (e.g., current condition may be more important than developmental conditions in mate choice). The shape of the response curve would reflect these relative rankings. In contrast to multiple messages, redundant signals both provide the same information, so receivers need only assess one signal to evaluate signaler quality. The predicted response to each component would be the same (e.g., Fig. 1a). However, there may be a degree of error in the assessment of each ornament; therefore, evaluating multiple ornaments could increase the accuracy of the receiver’s assessment of the signaler’s quality (e.g., Fig. 1c wherein females respond most to males with preferred traits that are match compared to mismatched traits). These first two hypotheses assume that all signal components are salient but differ in specifying how the signals interact. However, in some species that have multiple ornaments, one or more of the ornaments may appear to be irrelevant for mate choice and thus require an alternative explanation. The unreliable signal hypothesis suggests that seemingly non-functional, but still potentially costly, secondary sex characteristics are maintained because they enhance the efficacy of another signal that is directly assessed during mate choice (Candolin 2003; Galván and Sanz 2008). Under these conditions, the response curve for female receiver’s choice based on the interaction of the two ornaments would be similar to Fig. 1b with the female receptivity to the males’ signals plotted on the Z-axis. The unreliable signal would have no effect on the receiver’s mate choice. However, the speed of response would reveal a different story, wherein receivers respond more quickly when the unreliable signal is present than when it is absent or stronger compared to weaker. A response curve with latency plotted on the Z-axis would then reflect how changes in the unreliable ornament alter the efficacy of the signal.

Multicomponent multimodal signals

Many species combine both multiple signals within a sensory modality and multiple signals across modalities in their communication. This type of signaling is common to many species of birds, which perform elaborate displays involving movements, color badges or ornaments, and songs simultaneously. A classic example is the tidbitting display of male jungle fowl (G. gallus), which combines multiple visual ornaments, movements and sounds (Davis and Domm 1943). Previous research has shown that the interaction between the ornaments and visual movements changes the speed of the receiver’s response (Smith et al. 2011). There is also an interaction between the visual display and the acoustic signal (Smith and Evans 2008, 2009). However, the interaction between all three signals has never been tested. This example highlights the need for studies into multicomponent multimodal signals.

This type of interactions can be mapped by plotting the three signals on the X-, Y-, and Z- axes and using color to indicate the level of receiver response. Programs, such as MathWorks Matlab, provide subroutines that are designed to plot this type of data (Jayaraman 2006).

Technical considerations

Experimental techniques

Improvements in technology should facilitate the application of this heuristic, particularly in the study of acoustic and visual multimodal signals. Researchers have long had the ability to record vocalizations, manipulate specific parameters of the signal, and conduct playback experiments using sounds (Baker 2001). One example is the use of modified formant frequencies using PLOSA-based algorithms in programs such as Praat (Boerma and Weenink 2005) in studies on the affect of body size on receiver response. This modification changes the apparent vocal tract length, and therefore apparent size of the animal, without changing any other signal parameters (i.e., duration, fundamental frequency, harmonics, and intensity). The use of video playback lagged behind acoustic studies until research in the 1980s revealed that video images could elicit natural responses from birds and spiders (Clark and Uetz 1990; Evans and Marler 1991). At the time, VHS camcorders and the standard definition televisions provided only 240 lines of resolution, but even with this low resolution, video was sufficient to replace a live companion (Evans and Marler 1991). For example, experiments revealed that Burmese red jungle fowl (G. gallus spadiceus) could recognize the feeding movements of a companion in the video and discriminate these movements from other types of movements (McQuoid and Galef 1993). Furthermore, preferences acquired from video sequences, such as for a particular color of food bowl, can transfer to their real equivalents (McQuoid and Galef 1993). The current high-definition camera and television technology provide approximately five times more detail than those pioneering video playback studies (Smith and Evans 2008, 2009). This greater resolution provides enough detail that birds will even attempt to take food items directly from the screen (Smith and Evans 2008). Advances in computer technology have also created the ability to directly manipulate visual signals using 3D animation (birds, Watanabe and Troje 2005; Smith et al. 2009; lizards, Van Dyk and Evans 2008). Computer animation facilitates the study of how individual signal components contribute to signal function and efficacy. The development of realistic animal “robots” allows researchers to test multimodal signals in the field (Patricelli et al. 2002; Goth and Evans 2004; Rundus et al. 2007; Taylor et al. 2011). In addition, the use of optic-flow algorithms allow for the quantitative analysis of visual motions (Peters et al. 2002; Elias et al. 2006). These algorithms create similar outputs to those used in the analysis of the acoustic signal structure and can be analyzed using similar statistical techniques. Together, these techniques allow individual aspects of signaling movements to be quantified and more easily manipulated. When combined with established methods for analyzing and manipulating acoustic signals, this should enable researchers to create stimuli that precisely match the range of signaling levels in both modalities needed to implement this heuristic.

Data collection and matrix preparation

Understanding the signaling environment, which includes the physical environment as well as the social environment, is central to our understanding of the factors that have shaped multimodal communication (Bradbury and Vehrencamp 1998). As with the study of unimodal signals, it is hence desirable to collect data on signal production under natural conditions as well as under the controlled conditions of a laboratory setting. This allows the researcher to have confidence that the range of signaling performance and environmental conditions have been captured. Examining signal production in the laboratory setting then allows the researcher a greater degree of control to test the affect of variation on receiver responses. Data collection for this heuristic should be no more challenging than for studies applying the traditional classification methods or testing signal efficacy since quantification of signal variation is required for both types of studies.

Measuring receiver responses

The method of testing receiver response will depend largely on the sample population available. One concern of repeated testing of the same individual is that the subject may stop responding or change its response because of prior experience (Martin and Bateson 1993). Since the aim of the heuristic is not to statistically test the difference between each level of performance in each modality to every other level (e.g., the combination of −2sd in the visual modality and +2sd in acoustic modality compared to the mean in both modalities), it may be desirable to randomly assign an individual to a subset of signal combinations. Presenting a subset of stimuli to each individual represents a compromise between the within-subject control of presenting all stimuli to every individual and the problem of reduced response due to habituation. Other controls, such as testing individuals that are of similar age, reproductive status, social status (if applicable) and condition (e.g., body size, body coloration) can reduce the between-subject variability. However, care must be taken in drawing inferences to the wider population of receivers.

Conclusions

Multimodal signals are the product of interactions between the signalers’ and receivers’ sensory systems, the evolutionary signaling environment and the immediate conditions surrounding a signaling event. To understand these interactions requires a flexible approach that allows the full range of variation in signal properties to be tested. This new heuristic is a complementary approach to the traditional “cue-isolation” experiments. It will enable further insight into the factors that affect the relative receiver response to each modality. It may also shed light on the underlying processes that have shaped on each modality.