Keywords

Introduction

Everything that we formerly electrified we will now cognitize. (Kelly 2014)

By any account, humanity’s rate of technological progress has been breathtaking: in 1969, the single, inadvertently prophetic word “login” became the first message ever to travel between two connected computers. Today, 10 billion devices routinely access a vast cloud of near-ubiquitous knowledge and connections; by 2020 the number is projected to reach up to 75 billion (Riggins et al. 2015).

Meanwhile, the field of cognitive science, progressing in intimate parallel with computing technology, has facilitated major advances in our understanding of brains, minds, and their constituent operations. As a result, today we stand at the cusp of producing a Cognitive Society , where knowledge-based cognitive processes, natural and artificial, underlie the functions around which human activity is organized.

In many respects, the materials required to forge a society where interconnected objects are integrated into real-time thinking are already in place: the quasi-unlimited knowledge store of the Internet, the burgeoning speed and complexity of available computational power, and the growing scope of online devices are likely to accelerate the rate at which our society changes. These devices will interact with the environment, react to events, and anticipate outcomes faster than human awareness at both individual and global levels.

In fact, we already live in an incipient Cognitive Society. The rise of cognitive science and neuroscience has given study of the brain a cultural, rather than just scientific, significance (Olds 2015). It is commonplace to talk casually about the “pleasure center” or “visual center” of the brain, reflecting a popular acceptance of the neuroscientific principle of modular neurocognitive functions. Cognition as reflected in brain function has gained traction in the form of neurophysiological measurements as evidence in legal cases, albeit controversially so, with neuroimaging scans interpreted as indicating deception or psychopathy (Gazzaniga 2008). At a clinical level, neurodevelopmental disorders with cognitive components, such as schizophrenia and autism spectrum disorder, are now widely accepted as originating in disordered brain function rather than, for instance, psychological trauma alone. Thanks largely to the explosion in available computational power, artificial systems incorporating cognitive principles or functions have also become increasingly prevalent. Far from being an abstraction or far-off prediction, artificial intelligence in various forms has embedded itself into daily life, particularly so for any person equipped with an Internet connection. Deep learning algorithms (LeCun et al. 2015) can now recognize objects and places, read, identify voices, and even predict human memory .

Reflecting this convergence of technological and cognitive progress , our working definition of a Cognitive Society includes a ubiquity of convergent cognitive technologies that are leveraged to enhance human decision-making, well-being, and public health. What does it mean to live in a Cognitive Society? How can we think about the implications of a vast range of human-technology interactions in cognitive terms? What does life in such a society look like, both currently and in the speculative future? In this chapter we will address these questions with illustrative scenarios and the framework of a cognitive envelope to conceptualize some of the consequences of human and artificial cognitive interactions.

The Cognitive Envelope

Cognition in the broadest sense is both a straightforward and elusive concept. Intuitively we think of cognition as thinking – “the ultimate brain function” (Robbins 2011). In the context of artificial systems, cognitive computation must be both fast and complex. To speak concretely about the implications of cognitive computation to the Cognitive Society, we propose to sketch a cognitive envelope that places the broad concept of cognition within pragmatic dimensions of time and space.

Human thought and action operate on a wide range of time scales: an individual episodic memory , for example, may take fractions of a second to retrieve, seconds to select from among others, minutes to write down, and a lifetime to forget. The concept is not new to cognitive science: in building a case for a universal theory of cognitive architectures, Allen Newell (1990) divided human activity into four “bands” – biological, cognitive, rational, and social – spanning 12 orders of temporal magnitude between 100 μs and several months. For Newell, the range from about 10 ms to 1 s was key for basic cognitive processes and was thus labeled the cognitive band. As Newell himself pointed out, these boundaries were approximate; for our purposes, the bulk of cognitive psychology and neuroscience experiments place critical cognitive processes in this range, up to several seconds.

Cognition has a spatial as well as a temporal scale. This notion is common within the field of embodied cognition, which posits, e.g., that cognition is situated in relevant real-world contexts, optimized for motor action, and sometimes “offloaded” to the environment (Wilson 2002). It also finds traction in the neuropsychology literature, with evidence of distinct cortical networks supporting different behaviorally relevant realms. For example, space within arm’s reach has a different behavioral relevance, and thus likely a different cognitive role, than does space at an unreachable distance (e.g., Previc 1998). Notably, the various models reviewed by Previc (1998) and others tend to limit space for interaction to a radial distance of a few tens of meters. For our purposes, the intuition to extract from this body of research is that space matters to cognition and that the interactions for which cognition is most relevant tend to occur on the order of 101 m or less.

We can thus concisely circumscribe cognitive processes in a two-dimensional logarithmic space, centered on the familiar scales of 1 s and 1 m, as shown in Fig. 1. This also allows us to examine how technology modifies those constraints and how artificial cognitive systems might compare when doing similar processing.

Fig. 1
figure 1

Illustration of the temporal (in log seconds) and spatial (in log meters) bounds of the cognitive envelope in relation to other processes in the world tied to space and time. The framework facilitates thinking about the kind of technology that would bring specific processes within the cognitive envelope. Labeled arrows indicate examples of technology that mediate transfer of these processes into the typical human cognitive envelope. The broad envelope of cognition is centered on the familiar scales of 1 s and 1 m

Full size image

The cognitive envelope provides a framework to the idea of untapped potentials for systems, natural and artificial, to evolve or enhance human reach. Today, the ubiquity of Web-enabled devices allows people to send a much larger amount of information than even a few years ago. Consider the examples in the four quadrants surrounding the human cognitive envelope in Fig. 1. In various ways, each of them is brought closer to the typical cognitive envelope by convergent technology. The point at the upper right indicates the rough parameters of a cross-country flight in the United States – about 6 h to cover some 4100 km – and represents the minimum practical time to transport physical items or people over this distance. Many of the exchanges that once required travel are mediated by modern communication technologies that bring conversations, images, documents, and so on into a recipient’s immediate reach in nearly real time. Illustrated on the upper left, health tracking mobile apps and wearable monitors, for example, can generate and maintain an ongoing record of personal metrics such as steps walked, heart rate, or sleep patterns. The data is not just available to healthcare providers but to users directly. The aggregation of historical data over days, weeks, or months can be more useful than any given instantaneous measurement but presents an impractically tedious sustained attention task to perform manually. Thus, the cognitive load of repeated, regular measurements over long periods of time is transformed into data available at a glance to a device within easy reach.

Many processes, both natural and artificial, operate at short time scales unavailable to conscious perception , i.e., below the cognitive envelope. Represented by the lower left example in Fig. 1, an artificial robot system comprising a three-fingered hand and high-speed camera can achieve a perfect winning record against a human in repeated games of rock-paper-scissors (Katsuki et al. 2015). The implications of this seemingly innocuous example are profound: the robot can perceive the human player’s gesture and react accordingly in less time than it takes the human player to complete her own move. Thus, a game premised upon unpredictable decisions, driven essentially by chance, becomes wholly deterministic. In this way, small interactions between humans and artificial systems take on a fundamentally different character from a perspective inside versus outside the cognitive envelope. Finally, at lower right, sub-perceptual speed of processing can span large distances as well. As part of a research program in neural prostheses at Duke University, a monkey in the United States was able to remotely control a walking robot in Japan using implanted neural electrodes (Cheng et al. 2007). The signals traveling from the monkey’s brain reportedly reached the robot, over 11,000 km away, 20 ms before arriving at the monkey’s own leg. Thus, through this high-speed fiber-optic connection, an artificial motor system on the opposite side of the planet was integrated into the monkey’s own cognitive envelope as she controlled the robot using her own motor cortex and visual feedback from a video feed. This demonstration illuminates the possibility that technology can enable cognitive-level operations (in this case, deciding to initiate or stop a motor movement) even across distances otherwise inaccessible to real-time cognitive interaction.

While the cognitive envelope framework is illustrated along two salient dimensions, cognitive operations are necessarily complex and likely to exist in a high-dimensional space. Yet a computer performing a billion floating-point operations in 1 s is not automatically equivalent to a human performing a cognitive act in 1 s. Thus, a third dimension could capture complexity or “cognitive capacity,” some measure of not just the time and spatial scales of cognitive processes but of their sophistication.

Operationalizing cognitive capacity, especially into a meaningful single dimension, is difficult at best. However, intuitively, a Cognitive Society should be able to characterize, to some extent, the relationships between artificial systems, biological cognition, and the common principles underlying them. As with the two-dimensional cognitive envelope presented above, a three-dimensional model provides an intuitive representation of the space that cognitive processes inhabit and that human-technology interactions can traverse.

We can speculate on the possible expressions of cognitive capacity. For example, information such as the time of day, the distance to an obstacle, or the number of people in a crowd is difficult to estimate quickly unaided, but does not comprise very different operations from what a human would conduct over a longer time scale. A head-up or other augmented-reality display would therefore present this information into a user’s cognitive envelope across time and space, but not capacity. By contrast, humans have many well-documented limits on cognitive capacity: remembering or visually tracking more than a handful of moving items simultaneously will tax a typical person to the point of near-certain errors. Cognitive tasks such as mental rotation or continuous attention, critical to monitoring surveillance, defense, or medical imaging equipment, are also subject to systematic performance limitations. Wearable or prosthetic artificial devices without such limitations could, for example, bring a 20-object tracking capacity, occurring over the same time and space scale as tracking three objects, into a user’s cognitive envelope via the capacity axis.

In the next decade, we will likely witness an era where technology will compress or expand time, space, and capacity, to bring remote information into our cognitive envelope. Transformations like these are among the most direct embodiments of the oft-heard sayings that “the world is shrinking” or that “life is speeding up.” In a Cognitive Society , the technologies mediating these distortions will become increasingly pervasive, and the consequences of leveraging them, positive and negative, must be taken into account. In introducing the cognitive envelope, we saw the effects of some current technologies, such as Internet ubiquity, biometrics, connected devices, computer vision (and other artificial intelligence) algorithms, and neuroprosthetic interfaces. In the next section, we explore their implications for coming generations of technology merging cognitive and computational principles.

Expanding the Cognitive Envelope

The future is already here—it’s just not very evenly distributed. – William Gibson

Smartphones, wearable devices, and other technology continue to provide ever closer and more abundant human-technology interactions in daily life. The act of using an interface itself becomes a bottleneck; hence, efforts to make interfaces more efficient, ergonomic, and “natural” – autocompleting – form fields, search predictions, natural language voice interfaces, gesture recognition, and so on. We might say that a goal of user interfaces is to bring technological operations near to, or deeper within, our cognitive envelope. Recent examples of wearable technology exemplify this trend. Google Glass, Microsoft HoloLens, and similar peripherals project a virtual overlay of data onto the visual world, effectively integrating themselves with the user’s perception .

Perceptual Realm: Sensory Prosthetics and Substitution

Beyond integrating with the user’s senses, becoming that sensory input is the function of sensory prosthetics and substitution devices (SSDs), a class of peripherals whose main goal is to ameliorate the consequences of sensory loss, typically blindness or deafness.

Sensory prostheses attempt to reproduce sensory input lost to injury, disease, or abnormal development. For example, a neuroprosthetic device may capture a visual image and emulate the elicited photoreceptor signals in the case of retinal injury or degenerative disease. The signals would then be transmitted to intact cells using electrodes or optogenetic stimulation (e.g., Nirenberg and Pandarinath 2012). The principle is straightforward, though in practice it is extremely difficult to mimic the complex neural patterns into which sensory information is transduced. Still, currently available sensory prostheses offer crude approximations to the visual functions they replace.

In contrast to sensory prosthetics intended to reproduce the lost sensory input, SSDs operate by converting visual input, such as that from a camera, into a preserved sensory modality (Bach-y-Rita et al. 2005). SSDs for blind persons typically present a visually captured environment in auditory or tactile format. With training, SSDs can be used as aids in navigation and object perception (Maidenbaum et al. 2014). Crossmodal neuroplasticity is amenable to sensory substitution : the brain reorganizes functionally to process nonvisual input in traditionally visual processing regions (e.g., Amedi et al. 2007; Merabet and Pascual-Leone 2010). Importantly, ongoing neuroplasticity in adulthood would also serve the function of maintaining sensory and motor functions, known to weaken with aging.

The research community in assistive technology via sensory substitution is recognizing a persistent gap between laboratory-based advances in sensory substitution or maintenance and the widespread usage of such devices in the general population. Put simply, virtually no sensory substitution devices for low-vision and blind persons pass the filter from research labs into real-world usage (Elli et al. 2014). Researchers have begun to identify factors in design and application that could bridge this gap (Maidenbaum et al. 2014), and as with retinal prosthetics, a detailed understanding of not just raw sensory pathways, but how the brain represents incoming sensory information is critical to this advance. Understanding the relationship between sensory information and neural representation will be important not just for devices designed to reproduce or substitute for human sensory functions, but eventually for advanced sensory augmentation as well – the technologically mediated expansion of existing, rather than missing, sensory capabilities (Di Pino 2014).

More generally, researchers in fundamental neuroscience have begun to characterize population-scale patterns of information in the brain at a level of abstraction that can be compared across vastly different types of data and species. While this approach has only very recently been applied to the study of sensory loss and brain plasticity, it may prove a critical platform for the convergence of engineering with fundamental cognitive neuroscience research .

Cognitive Realm: A Mnemonic Neuroprosthesis

Within the framework of the cognitive envelope, the logical extension of a sensory substitution interface would be one in which data is accessed or communicated as fast and effortlessly as internal thoughts. Personal electronics and contemporary sensory prostheses are still crude when measured against this standard (a seed of this technology can be seen when our phones and calendars remind us what to do), but an environment in which such interfaces are commonplace would surely constitute one strong convergence technology aspect of an advanced Cognitive Society.

Although we are far from a neuroprosthetic that would encode and retrieve memories as fast as, or faster than, a human brain (but see Berger et al. 2011, for a rat hippocampal prosthesis), such a device could alter the perceived flow of time, as in the rock-paper-scissors robot example (Katsuki et al. 2015). Potentially this could also improve decision-making by allowing more computations to be available in less time than before.

One of the most dramatic differences between natural and artificial cognition may involve the perception of time : subjectively perceived time may exist differently in an artificially modified mind. To put time in perspective, an artificial device producing one cognitive operation per nanosecond could “experience” 30 years’ worth of subjective time in one objective second. Because the relationship between the speed of mental operations and time perception is not clear, how and if an artificial device would alter time perception itself is an open question.

Nevertheless, neuropsychology work demonstrates that distortion of perceived time over several orders of magnitude can occur in the human brain, especially as a consequence of injury or disease. For example, in addition to deep amnesia, patients with bilateral damage to their hippocampi, a neural structure important to memory, also experience compressed time, temporal disorientation, and an inability to predict their own futures (Dalla-Barba and La Corte 2013). The most striking example is Henry Molaison, the famous patient H.M., who became amnesic after his hippocampi were surgically removed to save his life. H.M. is best remembered for his memory loss, but psychophysical data suggest that he may have also experienced an extreme form of time compression, in which a year for us may have corresponded to three subjective hours to him (Richards 1973).

While the hippocampus subtends many memory and spatial cognition functions, including the ability to place information into a temporal context, it is also a fountain of cognitive youth: neurogenesis, the continuing addition of new neurons in the adult brain, allows the storage of new experiences (Aimone et al. 2006). Importantly, this turnover generated by the dentate gyrus, a subregion of the hippocampus, is barely affected by age, with older adults producing almost as many new neurons as young adults. The impact of neurogenesis on cognitive functions is a subject of active debate, with unanswered questions including the potential costs of adding new units to a fully developed network (Mongiat and Schinder 2014). However, models suggest that increasing the number of codes in a system would increase memory capacity and reduce interference between existing memories. Importantly, changing time perception , for instance, by increasing the number of codes and/or the speed of access to information, is within the brain’s plasticity capacity, which suggests that the shape of our cognitive envelope may very well adapt to new technological influences.

Social Realm: Interconnected Devices and People

Beyond individual augmentation, the emerging network of interconnected, Web-enabled devices – the so-called Internet of Things (IoT) – is poised to connect people and artificial systems to an unprecedented degree, influencing everyone, everything, and everywhere. Far-off technology will become commonplace: personalized medicine will be a highlight of the Cognitive Society, possibly in the form of drugs tailored to individual genomes and body-based sensors that monitor vital signs. Beyond the self, one significant net effect of IoT to the economic reality of the Cognitive Society will be a massive reduction of waste: for instance, today, with ride-sharing companies like Uber, car supply and fares are dynamically updated based on demand. Waste and error reduction will impact everything, as it currently impacts lean management, automated inventory, and responsive supply chains. The movement of people, goods, ideas, knowledge, and information will be guided by accurate information, winkling out errors and waste associated with decay, an intrinsic part of “inventory.”

At the center of the amplified connectivity realm, through the proliferation of smart sensors and massive data centers, is the “connected individual” who will use IoT as a platform to extend her sensory environment. The result is much more than the sum of its augmented parts: it is an extended self, able to act into a larger scope of the temporal and spatial physical world, changing our perception , and so reality, of what intuitive physics, causal reasoning, and determinism are.

A striking aspect of the Internet of Things is that it will be invisible to the naked eye and dauntingly to people’s naked awareness. Most external devices (e.g., phones, wearables, personal computers, monitoring systems, etc.) will be communicating with us and on our behalf, facilitating interactions between physical and virtual worlds at a pace far exceeding the capacity of a human brain. A result of hyper-dynamic regime changing faster than consciously followed is an illusion of continuity to the human brain, which may be countered by devices seeing, hearing, sensing, and informing us outside the standard bounds of the human cognitive envelope.

In this hyper-dynamic world, human mental resources, or attention, will become the scarce and limited resource, probing the human cognitive envelope to reshape in order to deal with information at stretched spatial and temporal scales.

The Cognitive Society: A Society of Knowledge

We envision that a Cognitive Society would take seriously the principles on which individual human cognition is based. This suggests that such a society would value the acquisition of knowledge to create new knowledge and incorporates the principles of cognition, some described here, in its devices and functions.

Understanding cognition on an individual level facilitates communication between natural (i.e., brains) and artificial systems, resulting in improved interfaces, devices, and even neuroprosthetics for healthy as well as injured or disabled people. Neurally inspired algorithms in search engines and computer vision systems already play an important role in present-day efforts to organize information. Ultimately, understanding and applying cognitive principles at a societal level will bring about positive policy changes in education, health-related, and legal systems, which have always tried to account for the drivers of human behavior but have lagged behind the state of the art in understanding those drivers.

Several other outcomes may emerge as the shared knowledge of a Cognitive Society becomes increasingly comprehensive, reflecting a greater diversity of sources. Importantly, such knowledge enables enhanced and more precise predictions . Future events will become predictable at larger spatial and temporal scales, through the collective “cognition” of the devices that connect our independent experiences. This may in turn facilitate increased individual cognitive capacity, as outsourcing tasks to artificial systems frees up cognitive bandwidth. In short, the IoT may have the potential to enable a sort of cognitive genesis in which individual minds find themselves enhanced with novel abilities, extending the reach of their cognitive envelope in time, space, and capacity. Finally, greater knowledge will also be beneficial for understanding the goals and beliefs of other individuals and cultures (i.e., theory of mind). As a much greater fraction of the population will share the same living legacy and as convergent technologies allow people to connect over larger bands of spatial and temporal scales, the bonds of common experience and empathy will transcend geographical constraints.

Conclusion

The examples and scenarios discussed here illustrate that the principles and practices of a Cognitive Society are not just speculations on an uncertain future but extrapolations of the present. These signals can be found in individuals and swaths of populations with access to the resources that living in a Cognitive Society demands. Yet they have been slow to distribute vertically to the levels of policy and horizontally to all corners of society. Thus, pockets of such a society exist but in embryonic form.

Additionally, the above examples underscore that we are still far from the synergistic interactions envisioned in a mature Cognitive Society. Some of the technical hurdles are clear: as the number of connected devices explodes, the infrastructure supporting them will be taxed. Neuroscientifically, the computational principles of cognition and perception , crucial to successful interaction with artificial devices, remain incompletely understood. Culturally, there remains a shortfall in education, despite increasing inroads made by cognitive science. This is partly because neuroscientific knowledge has lagged behind enthusiasm in reaching the rest of society, but more fundamentally, because cognitive science and cognitive neuroscience are themselves still maturing fields. One consequence is that research findings may be incorporated into popular opinion and education but in oversimplified, misinterpreted, or simply incorrect ways. For example, popular opinion has largely embraced a “left-brain/right-brain” dichotomy between analytical, logical reasoning and creative, emotive thinking – a vast exaggeration of empirically supported interhemispheric differences (e.g., Kaufman 2013). A more recent and general phenomenon has been what critics call “neurobabble” – the excessive invocation of brain activation to explain phenomena that do not require such an explanation (e.g., McCabe and Castel 2008). In the case of sensory substitution devices for blindness, a telling detail is that despite decades of research and development, none have yet reached widespread use and distribution. They require refined engineering as well as an understanding of cognitive and perceptual principles at a fundamental level. On both counts further research is needed.

The many research fields comprising cognitive science must be able to deliver relevant, ecologically valid, and conclusive research findings to serve as the basis for policy decisions. In this way, a fully realized Cognitive Society will incorporate principles of cognitive science at an advanced level, to all members of society, in a critical and self-correcting fashion. In other words, to paraphrase William Gibson, it will distribute the future evenly.