Keywords

1 Introduction

Multisensory data representation is a complex and complicated endeavor, in part because of our extensive use of data visualization. Given the use of visual media in data creation and collection, the dominance of data visualization may come as no surprise. This form of data representation is pervasive to the point that imagining data outside of a predominantly visual context to many may be an interesting artistic exploration in aesthetics, rather than a deep inquiry into the politics of information and interface. However, new research into the possibilities of data sonification (auditory data representation) [1] and more recently data haptification (tactile data representation) [2] have seen a dramatic increase in the last two decades. Furthermore, efforts to sonify and haptify data thoughtfully weave aesthetic exploration into transdisciplinary research approaches that investigate important questions of what it means to understand, witness, and be in relation with data. Questions such as: How are we creating stories about our world through the use of data? What stories are best told in a multisensory context? How can multisensory data representations impact users’ data understanding and experience? The goal here is not to argue for a reworking of sensorial priority, nor is it to claim that tactile or auditory representations must be researched in isolation from vision to ‘catch up’ with visualization research. The aim is instead to consider how we might begin to understand our other senses’ role in building data representations that allow us to interface with data’s affective and cultural contexts. Given that we experience our world through all our senses, I am more interested in creating work that thoughtfully includes touch and sound within a multisensory framework. To do this I employ collaborative practices within an arts-based research approach to better understand how we work with touch and sound as compositional tools. Furthermore, I encourage approaches that build frameworks for understanding touch and sound without reducing them to a set of standards or best practices.

In this paper I will discuss my philosophy and approach to designing multisensory data representations and interfaces with a specific focus on haptic media. This discussion extends my research in embodied practices for interaction design [3] and non-taxonomic methods for designing touch [4]. The broader philosophical discussion articulated throughout this paper will be augmented with design examples from my work Vibrant Lives, a collaboratively designed haptic/sonic infrastructure that gives people a real time sense of how much data they output from their personal mobile devices [5]. Methods cultivated in Vibrant Lives extend into my current research designing multisensory data representations of archival data with a focus on haptic aesthetics. In this paper, I discuss novel methods for representing large, complex datasets in multisensory formats that are both immersive and legible.

2 Transdisciplinary Approaches to Sensory Research

Comprehending the complex web of haptics research requires a broad understanding of the practices, trends, politics, and ethical dilemmas manifested by engaging haptics as an area of study, particularly within user experience. This is in no small part because we are in a time where how we create haptic media determines the processes and protocols we will effectively use to define the field. In my work, I address problems with designing haptic interfaces that reinforce sensory isolation, efficiency, utilitarianism, knowledge acquisition, and bodily control. Whether or not these are the explicit design intentions, it’s worth articulating that such priorities are embedded in the cultural and methodological fabric of many contemporary user design processes, particularly those with empirical roots and aspirations toward adoption by industry. As such, the following section discusses tensions between scientific sensory research methods and haptic design. Furthermore, I highlight a growing body of haptics research that extends beyond empirical design methods into the areas of media studies, digital music, dance and movement practices, philosophy, and sociology to name a few. The purpose here is to highlight some key issues and topics that inform my own work in multisensory data representation design, including the following: 1) the haptic subject; 2) touch as political; 3) co-formed knowledge; and 4) arts-based research methods.

2.1 The Haptic Subject

The transformation of touch into a conscious target of empirical study and computational mediation lead to the emergence of what Parisi calls the haptic subject.

This haptic subject embodies the self-conscious efforts of scientists, engineers, and marketers made to transform touch, as they sought to give tactility a new utility in a political economy of sensations vital to a society with a growing dependency on the circulation of information through sensing bodies [6, p. 4].

The haptic subject is marked by a need to control, reproduce, and commodify touch as a concept or even object that can live outside of the subjective lens of any one person, community, culture, or society. To do this, researchers aspire to transform touch into an acultural commodity that can be controlled by scientific methods and tools. Efforts to simplify or standardize touch are pursued so that findings may be shared and reproduced; however, such efforts also pressure users to normalize their touch experience to that which can be defined within the scientific parameters bound to the haptic subject. Furthermore, as Classen points out, attempts by science to define touch speaks more to the culture of science than to any possible standard features of touch. [7, p. 4] Research working toward the haptic subject is often articulated as a necessary response to participating in contemporary society as it relates to validation, valuation, and access to monies and resources. Returning again to Parisi, he discusses the technologizing of touch as including the following interrelated developments:

[…] the institutionalized and formalized knowledge production networks that rose up around touch, the new intellectual and financial resources funneled into the study of touch, the training and regimentation of tactility demanded by the new machines, and the motivations – explicit and implicit – of the various researchers who set themselves to work at the immense challenge of bringing touch under the control of scientific and technical apparatuses [6, p. 10].

The collective scientific hopes for a haptic subject speaks to a yearning for touch research to be validated within the constructs of empirical epistemology, which has very specific assumptions about how knowledge and information should be organized and handled as a utility largely decoupled from personal, subjective, cultural, and communal readings of touch experience.

2.2 Touch as Political

The decoupling of touch from its connection to lived experience is what Erin Manning describes as a process of normalization in which bodies become “stabilized within national imaginaries in preordained categories”. [8, p. xv] Politics are often affiliated with the workings of governments, but the term “politics” can also be used to describe decision-making and enforced organizational control over a group of members. Considering touch within a dialogue of politics, Manning points to the inherent political nature of engaging and mediating touch as a static, finite object or thing. In this, the haptic subject props up the politically organized, collective consciousness of science–prioritizing that which can be rendered visible, logical, quantifiable, and categorizable. Conversely, that which eludes, evades, or escapes the bounds of the haptic subject is rendered unnecessary, invaluable, irrelevant, and even problematic. This not only includes physiological experiences of touch, but subjective and cultural understandings of touch that do not fit within the empirical framework and related political imaginary of the haptic subject. Normalizing here is also considered a form of policing, not only of our sense of touch, but of what bodies are, can be, and should do. How we choose to ask users to physically engage with an interface expresses our own beliefs about how bodies should act and behave.

Further complicating the politics of mediated touch is how haptic technologies are appropriated across fields and design practices. Haptic technologies such as tactile transducers and force feedback devices are often adopted by designers working outside the device’s originally intended application. Teoma Naccarato and John MacCallum discuss the ethical dilemmas posed when appropriating technologies in their work with biomedical sensors [9]. Their paper investigates the “ethical and aesthetic implications of the appropriation of biomedical sensors in artistic practice” and discusses the pitfalls of appropriating any form of technology from one field for use in another [p. 1]. Their work highlights the importance of acknowledging that the transition of a technology outside its intended domain is not only an appropriation of technologies and their functional uses, but also an appropriation of cultural practices specific to the original discipline in which it was used. This cultural appropriation happens whether we recognize it or not, and can result in a wide array of consequences, some of which deeply impact user experience. Here, they suggest adopting a critical appropriation:

Critical appropriation involves the process of intentionally and explicitly deconstructing the ontology of technologies in order to rebuild them with and through a value system shared by all participants in the collaboration [p. 6].

Building new interface designs means that we are not only responsible for our own design intentions, but we must also be aware of how we are introducing the design goals of those who made the technologies we use. While Naccarato and MacCallum do not explicitly talk about the haptic subject, they observe a similar socio-political transition in how biosensors are being used within the design process that reinforce a need to control and police the bodies observed. In their observations, the results can be deeply impactful for those whose bodies do not conform to normalized interaction settings, as is seen in this example from their paper:

As Tom improvised with the particular movement patterns and qualities available in his body, the software had trouble tracking his gestures, and as such, kept losing him. The choreographer repeatedly asked Tom to “just stand still”, so that the tracking system could calibrate his skeleton. It was not possible for Tom to “just stand still”. Jokingly, but perhaps also with a hint of frustration, the choreographer said to Tom “you broke my system”. Because Tom’s particular movements did not conform to the expectations of the software (as preset by the human designers), his body was literally invisible to the tracking system. Despite the choreographer’s desire to market his motion-tracking system for people with disabilities, he created a program that was very limited in its capacity to process—never mind embrace—bodily differences [p. 5].

2.3 Co-formed Knowledge

Knowledge is co-formed in the relationship between those in the exchange. One does not inform the other. One is not static while the other transforms. Both transform and become new. Returning to Manning, she writes, “I accept this paradox and offer Politics of Touch not as a reading of what touch is, but as an exploration of what might happen if we are willing to direct our thinking toward movement, toward a relational stance that makes it impossible to pin down knowledge but asks us instead to invent”. [8, p. xvi] What does it mean to design a user experience from this perspective? For me, it means the following: 1) creating experiences that acknowledge the mutually transformative act of exchanging touch and its vast impact on a shared experience between users and interface; 2) designing tactile interfaces as an invitation to co-create meaning; and 3) engaging the design process as an act of facilitation rather than as a series of commands. It is impossible to fully control, impact, manipulate, or know the outcome of a mediated touch experience. In explicitly naming its impossibility, I recognize I am not stating something novel. Rather, I name the futility of controlling touch to emphasize that to design with the intention of inciting control is not only futile, but potentially an act of violence.

Susan Kozel highlights the potential for violence in her thoughtful reflection on performing in Paul Sermon’s installation work, Telematic Dreaming [10]. In this installation performance, two separate rooms with beds were projected upon each other using livestream camera feeds. Kozel performed on one bed and audiences were invited to interact with her on the other bed so that each body was projected onto the other space. [p. 439] While the interactions between Kozel and audience members were digital, she describes the physical responses she had to various forms of digital touch including pain when being virtually punched, sexual intimacy when digitally caressed, and the threat of violence when virtually accosted with a knife. Kozel’s work demonstrates a very clear co-formed techno-mediated exchange between herself and other. Similarly, I suggest that touch experiences between users and digital interfaces can elicit similar visceral, physical responses.

2.4 Arts-Based Research Methods

Artists and artistic methods have been longtime contributors to design methodology. More specifically, somatics and first-person, arts-based research methods are widely engaged within cognitive science and human computer interaction to augment the rise in embodiment research within these fields [11]. Contributing to this, artists have engaged haptics within various contexts. Musician Lauren Hayes designs haptic interfaces that explore haptic/acoustic relationships that extend beyond a simple reinforcement of touch as an extension of sound [12, 13]. Her work is informed by musicians such as Eric Gunther and Sile O’Modhrain [14], and Kaffe Mathews and Lynn Pook [15] who similarly explore haptic aesthetics in relation to sound and music. Within dance and somatic practices Susan Kozel [16] and Thecla Schiphorst [17] explore first-person design approaches to haptic media in performative wearable technology design. When creating her work soft(n) Schiphorst explored a “somaesthetics of touch” to investigate a poetic approach to touch design supported by somatic practices and first-person arts-based research approaches in designing mediated touch experiences.

A poetics of interaction supports a somaesthetics framework because it acknowledges that meaning is simultaneously constructed on multiple levels: conceptual, experiential, material, and computational (or technological). Meaning derives from our experience and the imaginative interplay between our self and our environment [18, p. 2432].

Arts-based approaches to design, particularly those informed by self-study, first-person methods, and somatic practices by their very nature resist the haptic subject. By integrating personal experience into the design process, subjective human experience becomes entangled with the resulting design. This work is not without structure or repeatable methods, rather it focuses on methods for facilitating shared exploration and reflection through the use of workshops, open questions, dialogue, and iterative refinement. For example, when creating whisper[s] Schiphorst held a series of workshops that focused on various sensory experiences such as hearing one’s own body or making physical contact with another participant [19]. The purpose here, “was to explore how people pay attention to their own body states and share those states with others in a space [19, p. 226].” Users were asked to improvise and interact with various props and objects and then respond to open-ended questions such as, “What did it feel like?” [19, p. 227] Similarly, Loke and Khut engaged workshop participants in activities specifically informed by the somatic practice Feldenkrais and used this work to create design workshops in which people somatically explored interactive systems [20]. Building from these somatically-informed, arts-based design methods, my own work engages in various models of facilitated exploration, reflection, and discussion – all of which influences the resulting design choices.

2.5 Moving Toward Multisensory Data Representation

I envision my work as an ongoing effort to refine a process of design inquiry, rather formalize a specific haptic infrastructure. The results of which are not standardized systems, but a stronger protocol for designing faciliatory touch experiences between users and data. In my design process, I consider many aspects of touch, not only including the vibrotactile actuators themselves, but also the material, shape, and texture of the interface and the user’s physical action of approaching the interface. My interest in bringing touch into the domain of data representation is twofold. Firstly, I am interested in haptic aesthetics that are not explicitly concerned with replicating real-world touch experiences, which is a common feature of contemporary haptic design [21]. Secondly, I wish to explore the potential for haptic data representations that make the cultural contexts, stories, and people embedded within the data more intimate, palpable, and emotionally resonant to users. This second point amplifies the import of engaging haptics outside of the haptic subject and recognizing the power of touch politics. In my case, I see the intimate, personal, cultural, and political resonances of touch as an asset to my work, rather than a hinderance.

3 Vibrant Lives

My research began with a 4-year exploration of haptic data through the project Vibrant Lives, which includes a suite of unique, custom-designed, vibrotactile interfaces that give audiences a real-time experience of their own personal data output [5]. The project was constructed through a series of intimate workshops and unique performance installations, conducted with my core collaborators Jaqueline Wernimont and Eileen Standley. User interfaces include wearable devices, hanging tapestries, and large sculptures that engage audiences in personal data output (see Fig. 1).

Over the course of four years, Vibrant Lives evolved into multiple, unique installations that offer various bodily, spatial, auditory, material, and social relationships between people, interface, and data. Across all iterations, we used a software/hardware system specifically designed for the project that reinterprets people’s mobile phone data output as auditory and vibrotactile feedback. Simply described, as users produce more data through their mobile phone activities the sound volume and vibrotactile sensation increases. Conversely, when their activities produce let data output, the sound and sensation decrease. The custom system includes vibrotactile actuators, a sound-producing smartphone app, and custom server software. Together, our haptic infrastructure allows people to feel and hear their data output in real time. (see Fig. 2) A detailed description of the technical infrastructure is discussed in a previous publication [5]. Throughout the design of Vibrant Lives, I conducted several individual and collaborative workshops exploring haptic aesthetics and data representation, including workshops at the following events and locations:

  • ACM Tangible, Embedded and Embodied Interaction Conference, March 2019.

  • Dance Program at Davidson College, February 2018.

  • Alliance of Women in Media Arts and Technology Conference, February 2018.

  • ART Lab at University of New Mexico, November 2017.

  • Synthesis Center at Arizona State University, November 2016.

  • Signal/Noise: FemTechNet Conference on Feminist Pedagogy, Technology, & Transdisciplinarity, April 2016.

The haptic workshops served as an open forum for user exploration, discussion, and participant documentation and observation.

3.1 Initial User Study

Conceptually grounding Vibrant Lives was a question of how to give users a very real sense of the massive amounts of data their mobile devices output in a given moment. Also embedded within our line of inquiry was a critical investigation of how and why companies and governments find user data acquisition to be a useful and necessary activity. Given the gravity of a discussion about personal data acquisition, we knew we were asking people to move into a space of possible discomfort and unease. Our first user study included guided discussions about the initial interface design [5]. Once users connected to our wireless network and opened the custom mobile phone app, I asked them to set aside their phones as I led them through a series of verbal prompts meant to heighten their sensory and spatial awareness using proprioceptive, kinesthetic, and cutaneous cues [22]. After I completed verbal prompts, users returned to their phones, ran the app in the background, and experienced the feeling of data output via the portable/wearable tactile transducer. The app ran in the background as they conducted various mobile phone activities. During and after user exploration, I asked the following questions:

  • As you navigate, what do you notice?

  • What stands out to you?

  • What do you feel (from the device) and how does that make you feel?

  • What thoughts come to mind?

  • How does this inform you/make you feel about your own data output?

  • How does this experience inform you/make you feel about your mobile phone?

  • What does this make you want to do (if anything)?

  • What part of this experience was the most impactful for you?

  • What will you take away from this experience?

Fig. 1.
figure 1

The images represent the various interfaces created for Vibrant Lives. The top-left image shows the hanging crocheted tapestry, the bottom-left image is of three wood/fabric sculptures, and the right image is of the portable/wearable tactile transducer users connected to their mobile phone.

Full size image
Fig. 2.
figure 2

This image shows the system used for all iterations of Vibrant Lives. Users connect to our custom server via our wireless network. Their data output is then captured and sent back to their mobile device using our mobile phone application. The mobile phone application creates a data sonification, which is transformed into vibrotactile stimulation using the tactile transducer.

Full size image

User responses were permeated with sadness, frustration, confusion, surprise, and curiosity as to who is gathering their data, how, and why. A full overview of the study is available in an earlier publication [5]; however, I highlight this work because it greatly impacted how we moved forward with our various installation designs. We thought deeply about what it meant to foster environments of curiosity and compassion rather than spaces that reinforced the themes of surveillance and governmental monitoring so easily elicited by the project. This is where visual elements became critical. We integrated bright colors, playful, improvisatory dance performances, and surprising use of other tangible media such as torn paper, dust, and household objects to reinforce our conceptual themes without reinforcing the negative and oppressive undertones of the tactile infrastructure. The visual elements that augmented our haptic infrastructure were devised during movement workshops involving various contributors that flowed in and out of the project.

During the first movement workshop we brought in a group of dancers to engage with cyber security experts and digital humanists. We discussed packet sniffing and packet capture software uses, devised various physical gestures and choreographies, and composed touch-based exercises that elicited the characteristics and behaviors of various data acquisition protocols. This process created a suite of rich, mutually understood physical, tactile, and visual metaphors we then used to anchor the performative and visual elements. This process helped us create interactions that avoided some of the common sociocultural affiliations with vibrotactile interfaces, such as relaxation or erotic pleasure. To see video of the performative elements of the work, visit the online video documentation [23].We also learned the importance of including performers whose job was to “host the space” by verbally and nonverbally inviting users to touch and physically interact with the haptic data infrastructures. This helped users drop into the experience and consciously focus on their own touch experience with the haptified data.

3.2 Sculptures and Shared Data Experiences

The first public exhibition of Vibrant Lives invited users to feel their personal data output by holding or wearing a tactile transducer while navigating a multi-room installation. The exhibition was in conjunction with a dance concert at Arizona State University, and the Vibrant Lives audience was mostly comprised of concertgoers. During the exhibition, we noticed that while users were able to navigate the installation together, the solitary nature of the wearable/portable haptic interface kept people from interacting. (see Fig. 1). This created a quiet space in which individual people were absorbed in their own experience. We would see and hear people talk to each other at times, but only on occasion or with prompting. Also, several users shared their discomfort with having the mobile phone app on their phone when exiting the exhibition. All audience members were assured that the Vibrant Lives project was not saving any of the data captured through the custom server, but our assurances did not assuage the visceral experience of feeling one’s own data output. The act of having to download and use the mobile app on a personal device was its own act of political touch. We learned from the first installation that this performative gesture of touch was incredibly powerful, prompting users to uninstall the app quickly after navigating the installation. We decided to make the mobile app a unique feature of the first installation. In later iterations, the project’s software/hardware infrastructure stayed relatively the same; however, we transitioned from engaging individual data to working with the collective data output of every device connected to our custom sever. The group data vibrated larger objects that could be touched by multiple people. This includes a series of large sculptures and tapestries that allow for a shared experience of data output. The transition from individual to collective data haptification meant users did not need to download the mobile phone app, only connect to our server network. Logistically, removing the additional mobile phone app step made it easier for users to move in and out of the installation experience.

Vibrant Lives: Data Play.

The second iteration of Vibrant Lives included a suite of three wood sculptures upon which we attached large bass shakers called ButtKickers. (see Fig. 3) We worked with artist Bobby Zokaites to design and construct the sculptures, which were exhibited at large, family-friendly festivals and outdoor events. The sculptural nature of Vibrant Lives: Data Play allowed for multiple users to sit, stand, and lie on the same structure. This collective experience meant people could co-witness haptic data output and more easily dialogue about their shared experience. We often saw families sit together and collaborate to discover different methods for interacting with the installation. The soft, curvy design of the sculptures also invited a playful interaction that offered many possibilities of physical contact with various bodily surfaces. We intentionally created shapes that leveraged a cultural familiarity with benches, ottomans, and chairs without offering obvious, flat surfaces. The precarity of this design choice hinted at the possibility of sitting but invited other possibilities, which many users explored. Some of the activities we saw were rolling, sliding, lying down, and climbing.

Fig. 3.
figure 3

The left images show the ButtKickers attached inside the sculptures. The right images show users collectively and individually engaging with the completed interfaces.

Full size image

Additionally, our reference to a socially familiar object meant that people would often happen upon the haptics themselves, sitting down and then expressing surprise when they felt sculptures vibrate. Many users expressed subtle joy in discovering the dynamic haptic feature of the sculptures, which prompted them to seek out information about the sculptures’ intent. We always kept an installation host on hand to answer questions, but users would most often find each other and willingly share their knowledge about how the installation worked. The opportunity for self-discovery and collective knowledge sharing across users were strong attributes of Vibrant Lives: Data Play; however, the sculptures also quickly became jungle gyms for family audiences – particularly when the installation saw large waves of people. In these moments, the busy activity of the space decreased the likelihood that people would spend time with the haptic data.

From our conversations with users, we learned that people were fascinated by the conversation of data output and surprised by the haptic representation of personal data. Users most often expressed a sense of excitement, enthusiasm, and play. Overall, the affective response to Vibrant Lives: Data Play was much more positive and joyful than the first installation. User response was fostered by the ease in which they happened upon a collective experience of the data, which allowed for users to maintain a sense of agency in their own experience and discovery process. The affective user response is also due to the event environment. Users were often with other family and friends enjoying an outdoor festival. This was a very different environment than the initial installation, which was an indoor exhibition connected to a contemporary modern dance concert. Finally, users were able to come and go from the installation without downloading the mobile app, which avoided some of the concerned response we received from earlier users.

Vibrant Lives: The Living Net.

Our final iteration was Vibrant Lives: The Living Net, which involved a vertically hung crocheted tapestry visually designed to look like a generic data visualization. (see Fig. 4). Tactile transducers were crocheted into the nylon cord, which transmitted vibration across the tapestry. Our goal with the installation was to create a piece that could grow and evolve over the duration of an exhibition. During exhibitions, I continued to crochet the net across various objects and surfaces in the space, increasing its size and reach. (see Fig. 4). This performative element served two purposes. First, it enlivened the haptic interface by giving it sense of growth and spread, which reinforced previous user observations about the pervasive, creeping nature of their own data. Second, it allowed me to be nearby as the installation host without idly standing and waiting for people to ask questions. Users appeared to be comfortable with my presence, even though the installation was mostly preformed in quiet gallery-like spaces. In our first performance of Vibrant Lives: The Living Net, I invited dance performers to improvise and transform the space by moving and reshaping objects. This prompted users to also rearrange and transform the space. In later iterations, we invited users to leave personal objects behind and annotate their contribution on a tag. We would crochet the objects into the net, essentially leaving visual traces of those who touched the installation. (see Fig. 4).

Fig. 4.
figure 4

The top-left image shows the net after being crocheted across the room during live performance. The bottom-left image shows the tactile transducer crocheted into the net. The right images show users’ personal objects crocheted into the net.

Full size image

The flat, two-dimensional nature of the net combined with the personal objects meant people most often assumed the piece was meant to be a visual sculpture until invited to touch or lean upon it. These moments of host invitation resulted in many rich discussions, but also impacted the sense of personal agency to create one’s own touch experience. Users often referred to me as the host for the “correct” way to experience the work, ultimately predefining the interaction. This was something we were seeking to avoid in our design process. To help users discover the haptic elements, we used speakers to amplify the data sonification that drove the tactile transducers. The sound was very effective in inviting users to touch the tapestry for much longer than when it was not present. I also noticed that the reinforced sound amplified the haptic experience. When the sound would increase, users would perform visceral responses through facial expressions of surprise, a deep inhalation, and/or a slight postural receding from the installation. Conversely, as sound volume decreased, users would soften in their stance, at times leaning toward the installation.

3.3 Haptifying Data Archives

The initial work with Vibrant Lives provided many rich, multilayered experiences from which to consider haptic design. As described in Sect. 2, data haptification is a relatively new concept. Since people are generally accustomed to looking at data, there is a nuance to providing enough context so that people can settle into a haptic data experience without providing so much information that people don’t feel the need to touch data to feel satisfied with their own understanding of how it works. We found that if a haptic data experience was over explained, either by an installation host or accompanying signage, people were less likely to physically engage with the work. Also, what initial information we provide and how we provide it drastically impacts user experiences. Fortunately, in the case of Vibrant Lives the relationship between personal data and haptic sensation was tightly coupled to users’ physical interactions with their own mobile devices. Meaning, when people engage with their mobile devices and simultaneously feel vibration, there is a very clear pairing of the user’s physical gesture and the haptic output. It is worth noting here that sculpture installations still required users to use connect their mobile device to our network to generate data output. While the haptified data was a sum of all mobile data moving through our custom server, users were often still able to feel their individual spikes in activity. We learned about the power of this implicit gesture/haptic pairing when we began working with archival datasets.

In the case of archival data, the data is still about people and human activity, but it is not directly connected to user activity. The separation of data from the physical gesture of a user adds another layer of complexity and requires a more thorough investigation of how we contextualize the haptic/data relationship through a multisensory experience.

3.4 Haptic Archive Workshops

I have conducted a few small experiments with archival datasets along with my collaborators Jaqueline Wernimont and Eileen Standley. We used various sonified archival datasets to resonate tactile transducers connected to unique materials and objects. (see Table 1). None of the installations were visually encoded with explicit details about the dataset parameters used to create haptic feedback. As such, users typically needed a thorough verbal overview of the dataset parameters before they could drop into the haptic experience. In this, we were compelled to verbally describe how the data was rendered over time, much like describing a graph. Once users had an understanding of the data, they could experience it in haptic form. However, for several users, the simple verbal description of the data itself was enough to satisfy their curiosity. As a result, some users would spend little time with the haptic data before moving onto the next installation. This was particularly true for installations in which the structure, texture, and gesture toward the haptified object did not elicit any strong metaphors for how the data should be understood.

Table 1. Dataset/object pairings.
Full size table

The most compelling objects were the white beachballs and the sandbox. The white, inflated beachballs resonated with data from the Eugenic Rubicon project, which represents the over 20,000 voluntary and forced sterilization recommendation records from the state of California from 1920–1960 [24]. Most workshop participants chose to hold the balls near their chest or abdomen with their hands or arms so that data was felt across the upper torso and limbs. The use of a large circular object conjured many metaphors and images across users. Given the size and shape of the ball, it fostered connections to fertility, pregnancy, and the loss of the ability to bear children by those forcibly sterilized. For some, this experience was quite powerful and helped them connect to the data; however, for others our form of data representation was considered too intimate, exploitative, and even a misrepresentation of the data itself. Some participants also found the use of beachballs inappropriate because they were objects of play. In these cases, users were not able to decouple their experience from the cultural connotations of what a beachball is and is used for. The Eugenic Rubicon installation led to many deep discussions about who the data is for, who should have access to it, and how to appropriately represent data.

The small sandbox represented global warming data, which sonified and haptified global warming trends over time. Participants would rest or dig their hands into the sand to feel the data. The use of sand was a very clear metaphor for global warming, inciting images of deserts and dry land. Some users would almost caress the sand or find themselves shaping it in ways pleasing to them. One person said they felt it necessary to take care of the sand as they felt the data. Like the ball installation, users found the sandbox compelling because the installation’s physical materials intuitively connected to the data itself.

The initial workshop explorations helped us understand what to consider when haptifying archival data within a multisensory installation. The first is the importance of intentionally pairing the physical elements of the installation with the data itself. In this, we learned not only to consider size, shape, structure, and texture of the physical installation elements, but also: 1) user’s personal and cultural familiarity with the objects and materials, 2) likely user gestures for entering into, exiting, and sustaining touch, and 3) discrete visual and aural information for contextualizing the dataset parameters. These considerations and how I use them are highly contingent upon the user group and larger environment in which the data representation is situated.

4 Discussion

The design of tactile data representations surfaces many questions that are not only specific to touch but elicit deeper questions about how we are using data as a form of storytelling. Offering data as touch heightens awareness of the people and human infrastructures present enmeshed with the data. In the case of Vibrant Lives, users became more attuned to the unnamed people who were collecting their data, often asking questions such as, “Who wants my data and why?” During our first installation, one participant said she already knew her data was being collected by companies, but in touching her data, she felt far more implicated by her decision not to change her behavior or do something about it. This points to the livingness of the data installation as it takes on qualities of movement. Haptic data evokes the materialist theories of Jane Bennett, who inspired our initial conceptual work on Vibrant Lives. In this, our work amplifies the ways in which non-living things like our devices and data are themselves “quasi-agential” forces that shape lived experience [25]. In all iterations of my work, users were not so much focused on discrete data points or static renderings of the data, as one might see in a static data visualization, rather users focused on the overall changes in haptic sensation over time, noting major spikes and drops in sensation. These major shifts would become places for inquiry and discussion, mostly of what the data represented and why such representations were meaningful to various stakeholders producing, using, and consuming the data.

4.1 Ethical Data Representation

Initially, my research goal was to understand the ethical implication of using haptic aesthetics in data representation. This is a key area of my research and something I continue to engage throughout my work. However, moving my work toward data archives surfaces many questions related to data representation. The first is, how are we treating the people embedded within the data, and do we have the right to work with the data at all? Working within Vibrant Lives, the data we used throughout our preliminary research was inherently our own. I began to make intimate relationships with the information. When a colleague would share their data with me, it was conducted with great generosity as a gesture of self-offering. We were able to cultivate a mutually respected ethos of care between the data creator, researcher, and research process because we were all physically in the same room. However, working with archival data means engaging with people who are not in the room and thus, do not have the ability to decide if and how the data should be used. In these cases, how do we make a mutual contract of care between those within the data, the designers, and the users? How do we create ethical processes for publicly representing data when the people represented cannot participate in the ethical debate? I don’t think these questions are important just because I am working with touch, or because I continue to work with highly political and emotionally charged datasets, but the importance of these questions is more palpable because I am working with haptics. I feel the implications differently, and I am ever-ruminating on the ethical implications of data representation as a form of storytelling and narrativizing information. Discussions such as this would have perhaps been merely an intellectual exercise rather than a serious consideration prior to beginning my haptic research. Now, I am deeply considering ethical questions of access and use before moving forward with future archival datasets.

5 Conclusion

In this paper, I discuss my philosophy and approach to designing multisensory data representations and interfaces with a specific focus on haptic media. This includes designing for both real-time data and data archives. I discuss the importance of resisting the haptic subject and thus separating users from their personal and cultural understandings of touch. I also articulate why I consider touch design as an inherently political act in which knowledge is co-formed. The results of my design work suggest that the physical materials imbued with haptic data impact how users interpret the data itself. When designing physical infrastructures for data haptification, I suggest considering not only size, shape, structure, and texture of the physical installation elements, but also: 1) user’s personal and cultural familiarity with the objects and materials, 2) likely user gestures for entering into, exiting and sustaining touch, and 3) discrete visual and aural information to help contextualize the dataset parameters.

Throughout my work, my goal is to create multisensory experiences that thoughtfully generate dialogue between the information inscribed within the data and the users themselves. The process of entangling users with various data attributes imbues data representations with palpable, culturally specific elements that speak to the broader histories and social contexts of the data. I do not wish to eradicate data visualization from the design process, but to balance its impact on the user experience by deprioritizing vision as the primary means of consciously consuming and analyzing data. I do this as an embodied mediation between people and data, examining what it means to care for and create affective spaces for large datasets that are often about people but leave little trace of human experience within their representations.