1 Introduction

Emotion is not only an essential means of interpersonal communication, but also a motive to motivate psychological activities and behaviors. Linguistic studies indicate that language is to judge what we are talking about and to express opinions and emotions, rather than a means to provide facts and information. The written communication, verbal communication as well as facial expressions by humans are just as purposeful as language (Wang et al. 2019). With the advancement of computer technology and the Internet, the demand for data analysis in the digital era is on the increase, and it also creates new opportunities and challenges. Data visualization is the intuitive display of all quantitative information (Qi 2018). When emotion can be approximately quantified by computer, emotional visualization arises at the right moment. Emotion visualization is a technology that uses computer graphics to use visual coding and visual channels to output emotions into visualized results or display important data, so that users can know the emotional change trend or value gap (Bresciani 2009). Emotional visual analysis is to analyze a variety of data and output a related multidimensional data to the user to form a more comprehensive result and provide users with auxiliary decision-making.

To make the emotional state more intuitive and realize user-friendly emotional information retrieval, emotion visualization is proposed as a bridge between users and emotion analysis (Hupont et al. 2013). Just like describing one’s emotions in words, emotional visualization or visual analysis can be used for the delivery and computation of emotional information.

By investigating the related work of emotion visualization in recent years, we found a blank in the review in this field (Cernea and Kerren 2015). There are three reviews in this field. Foremost, in the survey of text-based data visualization, Kostiantyn Kucher et al. (2016) summarized the text emotion visualization technology, using the survey classification method to summarize the current 100 technologies in an interactive web page, but did not target the emotion visualization technology of other data carries out detailed classification and summary. Richard Khulusi et al. (2020) classified 129 related approaches in a review of music data visualization, divided the music work into scores and music sounds for analysis, and briefly narrated 7 cases involving emotional visualization. The article did not sort out the realization process of music emotion visualization, nor did it sort out the methods of music emotion recognition or visual coding. In the paper concerning emotional visualization in Virtual Reality (VR) immersive environment compiled by A Pinilla et al. (2020), firstly, it introduces the theories concerning emotion and affection; and secondly, it analyzes the visual findings concerning emotional response, and summarizes some of the most commonly used emotion state detection methods (Boumaiza 2015). As far as the study of emotional visualization is concerned, however, it is far from enough to summarize the one area of VR only. Compared with the content mentioned above, this paper first investigates the emotional visualization of multiple data.

A Survey on Emotional Data Visualization and Visual Analysis sources (Kim and Klinger 2018), then summarizes the visualization technology of multiple platforms, and finally focuses on the applicable visualization techniques (Du and Yuan 2020) the data types are summarized. This paper uses emotional, affective, sentimental as English keywords on Google Scholar, IEEE Xplore, Research Gate and other websites to search for emotion visualization related literature, and investigates the work in the field of emotion visualization in the past 10 years. At the same time, it also searched IEEE VIS, EuroVis, PacificVis, ACM CHI and other visualization conferences in the emotional visualization literature. The collected nearly 300 work and papers with emotional visualization tasks are sorted out, and work with repetitive or missing data acquisition steps are filtered out, and 75 representative papers are selected for analysis. The selection of work must meet three conditions. First of all, the work must be data-driven emotional visualization, which must include the extraction or collection of emotional data. Secondly, extraction techniques or visual encoding methods must be discussed in detail in the paper. Finally, emotional visualization should be output as a result (Russell 1989; Prasojo et al. 2015).

This paper categorizes and discusses the main realization process of emotional visualization. In Sect. 2, the data types of emotion visualization are analyzed. The extraction and analysis of data is the basis of emotion visualization; Section 3 focuses on the way of emotional state evaluation, and maps the data to multiple emotion types according to the emotional theory model to obtain emotional labels; then in Sect. 4, a key analysis of visual marks and visual channels, including the meaning of visual coding for emotion and data services and the meaning of each coding method, is the core of the visual work mapping scheme, which associates emotional performance with visual elements, allowing users to obtain information and trends in data more quickly and conveniently; Section 5 analyzes and discusses visualization and visual analysis methods based on different analysis tasks. Visualization makes visual analysis more effective and provides users with more accurate Analyze the results, and the user-centered visual analysis method makes complex data easier to understand. The two are fusion and symbiosis; Section 6 discusses the existing problems and challenges of emotion visualization, and discusses emotion visualization Prospects for future development trends; finally, in Sect. 7, the research work of this article is summarized.

2 Data sources

Data is the foundation of visualization. The use of visualization technology can facilitate the interpretation of data and information and simultaneously reduce the difficulty in understanding complex data. Humans habitually use images, audio, video, and text on social networks to express their opinions and share their emotions. The sentiment analysis of massive multimedia data through sentiment visualization helps one to understand human behaviors and preferences, thus playing an important role in many practical applications (Prasojo et al. 2015).

2.1 Sound data

Sound data can be divided into music and voice data.

Considering vision as the core and music as the carrier, the visualization of music emotion refers to a communication method that uses a variety of new media technologies and other communication media to interpret these emotions through pictures or images (Aljanaki et al. 2017). Visualization can strengthen the understanding of music, at the same time, play an auxiliary role in the retrieval of music information. Haro et al. (2010) used a set of automatically generated elements of music tracks to express the emotion and audio characteristics of music; they then created a cartoon character and mapped it to these elements to express the music emotion. Grekow et al. (2011) proposed a new musical emotional visualization process connecting visualization with the external system of automatic emotion detection, further enhancing the expressiveness and attractiveness of music by mapping emotional tags and drawing graphics. In another approach, Lee et al. (2016) designed an interactive theme-based music player based on music emotion data and used the underlying music characteristics to associate the player’s functional effects with shapes and colors. Further, Jeong et al. (2019) transformed the emotions formed by musical chords into dynamic works of art with shapes, colors, movements, and depth. The performance of dynamic artwork leads to an emotional experience based on synesthesia.

To further enhance the emotional impact of music, Passalis et al. (2021) proposed a deep learning method to link attribute-based music to image translation, which is used to synthesize visual stories based on the emotions expressed by songs. The generated image is intended to lead the audience to experience the same feelings as in the original song, thereby enhancing the impact of the music. The authors considered the visualization of music emotions an important means of expressing music content and emotions. Through the combination of audio and video content, the study increases users’ audiovisual enjoyment and strengthens their understanding of music emotions. Further, through visualizing music emotions, it assists users in searching for and categorizing music. After obtaining the emotional tags of the music, the common emotions are connected into an aggregate, with emotion as the tag, to assist users in the emotional screening of music or music playlists. Goto et al. (2005) and Hilliges et al. (2006) designed two digital music players with different optimization strategies based on music emotions. The former study describes a music playback interface called Musicream, which allows users to see various music fragments that are emotionally similar to those that users like, while the latter developed a new digital music player called AudioRadar to generate playlists based on the attributes of music itself and users’ emotions. In another study, Seo et al. (2019) proposed a method to automatically classify music with high precision according to the range of human emotions. Van et al. (2004) designed a music interface for browsing and navigating music on small devices, visually displaying music emotions, detecting similarities in the sound data of musical pieces, and assisting the music classification and collection function. At the same time, one can combine the visualization of music emotion with that of image emotion. To further improve the degree of correspondence between music and photographs, Chen et al. (2008) proposed a system that uses photographs to create emotion-based music visualization. The abovementioned studies enhance the emotional expression of music and enrich users’ auditory experiences by coordinating the emotions in auditory and visual content.

In recent years, the emotional visualization of speech has gradually attracted the attention of researchers. Almahmoud et al. (2020) developed a creative tool that enables users to extract voice samples from audio elements through machine learning models, identify the emotions, and subsequently display them using visualization technology. The emotional visualization of speech can also assist the rendering of the stage environment. Vryzas et al. (2017) used emotional visualization and lighting colors to assist the audience’s understanding of emotions in drama and applied the color-emotion correspondence in the emotion wheel model to render and simulate the results of stage color lighting.

2.2 Text data

Text is a means or tool for human beings to express their opinions and pass on ideas. Through the analysis, processing, and visualization of emotionally colored subjective texts, one can more intuitively understand the changes, development, and associations of emotions. In the Internet age, many forms of text exist. This section divides text data into five types: social media, news reviews, product opinions, work reviews, and log blogs.

2.2.1 Social media

Social media platforms, such as Twitter, Facebook, and Weibo, are important for people to socialize and express their emotions and opinions. Emotional data on social media can not only reveal personal opinions but also express personal emotional styles. Wu et al. (2013) proposed a visualization process that can guide the audience in analyzing social text according to the three interactive frameworks of urban emotion, related topics, and posted content and simultaneously explored visual design in the field of emotions related to public transportation. Guthier et al. (2014) automatically detected and visualized the emotional information of urban residents on geotagged posts on Twitter, while Kempter et al. (2014) developed a visual analysis system to explore trends in user emotions over time, combining the traditional timeline method with real-time dynamic animations to vividly express user emotions. By visualizing the emotions of different users toward events and comparing their reactions, the latter study summarizes the emotions expressed on social media and simultaneously tries to arouse the audience’s emotional response. In another study, Zhao et al. (2014) developed a timeline-based visual analysis tool, PEARL, to mine text information from social media and detect personal emotional styles. Further, Gaind et al. (2019) solved the problem of detecting, classifying, and quantifying different forms of text emotions, especially in the field of opinion mining, which can provide useful emotional information in a variety of ways. Visualizing and presenting text data can aid in understanding users’ sentiment styles, help them to self-reflect, allow companies to provide personalized customer service, and help in the effective tracking and analysis of opinion dissemination on social media.

2.2.2 News reviews

News reports are regarded as containing objective facts, but they hinder the audience’s understanding of and feelings about the subject matter by not providing background information or using visual and language information that may be misleading. Therefore, Wu et al. (2014) developed a visual analysis system to enable analysts to detect communication patterns on social media, collect content from comments, and convey user sentiment trends. Further, Diakopoulos et al. (2010) proposed a visual analysis tool, Vox Civitas, to help journalists and media professionals focus on large-scale social media sentiments and extract the sentimental value of news from the content. Braşoveanu et al. (2012) designed an interactive system for news flow visualization that employs visual coding to easily present multi-source social media data and arcs to adjust and enrich the interaction to help users understand the meaning of the data. To gain a deeper understanding of the events reported in the news and the public’s response, Calderon et al. (2015) proposed a time-flow-based visualization platform to analyze the trend of changes in users’ emotions after an event and evaluate its social impact. In addition, Prasojo et al. (2015) developed ORCAESTRA, a tool for visualizing user comments regarding online news articles, analyzing the sentiments expressed in each comment and dividing them into three categories—positive, negative, or neutral—to better understand different viewpoints presented in the comments and improve user experience. Further, Alhamid et al. (2017) proposed a model for understanding and analyzing social network data to measure the overall emotional attitude of online users in a region. They divided the overall sentiment of Arabic content on Twitter into three categories—positive, negative, or neutral—displayed intuitively on a map. Finally, based on the emotional feature words used by online public opinion participants, Jia et al. (2020) designed a sentiment visualization system to monitor these words and their evolution over time, reflecting the emotional tone of the online public opinion events and participants. The design and enhancement of visual effects can greatly enhance empathy toward news content.

2.2.3 Product opinions

With the rise of social media, users can freely express their opinions about products on public platforms without time constraints. Alper et al. (2011) used interactive visualization tools to help companies better understand customer reviews; they utilized the text information in public comments to design emotional visualizations and tags to categorize comment content. In addition, to improve the transparency in product quality and strengthen the relationship between a company and its customers, Gali et al. (2012) extracted the emotions in user comment banks from social media posts based on emotional strength and emotional tags and designed a general-purpose tool for visualizing the emotions in comments. Emotional visualization based on user opinions allows one to analyze users’ emotional trends and provides effective strategies for product improvement.

2.2.4 Work reviews

In addition to visualizing product reviews, researchers also visualize the emotions in music, book, and movie reviews to more clearly understand the ideas of other users. Chen et al. (2006) referred to more than 3000 reviews on the best-selling book The Da Vinci Code on Amazon.com and used visualization techniques and modeling tools to distinguish between positive and negative reviews and understand the trend of emotional changes over time. To facilitate understanding the emotional trends of users in film reviews, Ha et al. (2014) extracted film review content from the Internet and identified complex text emotions by combining multidimensional scaling (MDS) and social network graphs and used the visual mapping results to understand the emotional characteristics of each node. In another study, after collecting content about Game of Thrones from British and American news media websites and four social media platforms, Scharl et al. (2016) used trend graphs and an interactive dashboard with synchronized video analysis components to display the events and people mentioned by reporters and audiences. The emotions and their frequency, the current storyline, and the related concepts of each episode help the audience understand the plot. Topal et al. (2016) designed a movie emotion map based on audience comments, allowing users to select movies with specific emotions through different visual mappings. In relation to the music industry, Oliveira et al. (2020) developed SAMBAVis, a visualization tool that extracts the main emotion keywords from user comments on YouTube music videos and conducts sentiment analysis. Combining user comments on website content with emotional visualization can assist the website in classifying the content and ranking users’ favorability.

2.2.5 Log blogs

The text content in blogs can reflect the opinions and emotions of individuals, groups, or the public on any topic. Kuksenok et al. (2012) designed a visual text analysis system to study the emotional expression in chat logs by automatically analyzing data. In another study, Hennig et al. (2015) tracked the opinions of individuals and the public on many topics through a visualization system and performed sentiment analysis to show the emotional development of a certain related term in a particular time interval. Further, Ren et al. (2018) proposed a sentiment analysis method to analyze the sentiment labels of different sentences in blog text and designed a visual map using changes in the distance between sentences to express the intensity of emotions.

2.3 Image data

Picture data is usually converted from video content to frame-by-frame images for emotional visualization. Azcarate et al. (2005) proposed a method to recognize facial expressions in real-time video streams and sequences, converting videos into frame sequences, analyzing the emotional information, and displaying the probability of emotional tags using histograms. Focusing on human faces, Yang et al. (2019) estimated facial keypoints in Graphics Interchange Format (GIF) images, integrated these with the extracted visual features, and represented the complex emotional information in a quadrangular grid to analyze the emotions in GIFs. The visualization of emotions in image data enhances the richness and comprehensibility of the image emotion recognition system.

2.4 Video data

With the substantial increase in Internet traffic in recent years, streaming media platforms, such as YouTube, Netflix, Disney+, etc., have become increasingly popular. Traditional social media networks, such as Twitter, Facebook, etc., use emotional visualization technology to analyze video emotions and expand multimedia content, the richness of which also strengthens the empathy shown toward the video. Zhang et al. (2010) proposed an integrated system for the sentiment analysis, visualization, and retrieval of music videos (MVs): by extracting effective emotional features and analyzing user emotions, personalized MV recommendations are realized.

Dynamic facial emotion analysis is an important data source for video emotional visualization. Jin et al. (2019) proposed a new method to analyze facial expressions in images by replicating a three-dimensional face model and developing a quantitative information visualization solution to explore this type of visual data.

2.5 Mixed data

Human emotions are usually expressed in a variety of ways. Therefore, exploring multimodal emotions and their consistency is of great value in understanding human emotional expression. One can apply sentiment analysis visualization technology to mixed forms of data, such as video, audio, and text, to obtain more information about user experiences.

Zeng et al. (2019) designed an interactive visual analysis system, EmoCo, to effectively analyze the emotional consistency of facial, text, and audio patterns in a demonstration video. The combined effect of the five views in the visual analysis system enables users to quickly perceive the emotional consistency and its trend over time. Da Silva Franco et al. (2019) designed a tool, UXmood, that provides quantitative and qualitative data analysis of emotions and helps researchers evaluate user experience; it combines data from videos, audio content, interaction logs, and eye trackers and presents them visually in a dashboard on the Internet.

Using mixed data to visualize and visually analyze emotions can allow the simultaneous display of emotions from multiple data types in a system, which helps minimize errors caused during emotion analysis and calculation and reduces users’ difficulty in understanding visual content. In addition, it can reduce the data pollution and loss caused by extracting a specific type of data in the mixed data.

3 Emotional state assessment methods

After collecting and organizing the data, one can use different models of emotional theory and emotional state evaluation methods to analyze and map/visualize the data and emotional tags.

3.1 Theoretical models of emotion

Emotion can be approximately quantified as digital information; emotional visualization technology has, thus, attracted much attention. When emotions cannot be directly observed or measured, one can use visual views to approximate emotional data—that is, emotional visualization—with technological tools mainly involving the recognition and classification of emotions. Thus far, many researchers have investigated how to quantify emotions and perform sentiment analysis.

Professor Mehrabian (1997) proposed the pleasure-arousal-dominance (PAD) emotion model as early as 1974, which covers almost all the psychological dimensions of emotions and expresses all emotions through three-dimensional coordinates. The pleasure value measures the degree of emotional pleasure and judges the positive and negative states of emotions; for example, anger and fear are unpleasant emotions, while love is pleasant. The arousal value measures the intensity of emotions; for instance, madness is more intense than anger, and although both frustration and sadness are negative emotions, their intensity is relatively low. The dominance value measures the degree of emotional control; anger is a controllable emotion, while fear is uncontrollable.

Russell’s ring model (Russell 1989) is well known and often mentioned in cognitive science and psychology. As shown in Fig. 1, emotion is displayed on a two-dimensional coordinate plane using the dimensions of valence and arousal to measure the classifications of emotion in both fields of study. Valence represents the positive or negative degree of emotion, while arousal refers to its intensity. Different emotions are arranged in a circle at the end of each axis, or an emotion map can be drawn by extracting six main emotions.

As shown in Fig.  2, the psychologist Plutchik’s (2001) emotional roulette/color wheel of emotion (Plutchik model) attempts to express a form of interaction and the relationships between people’s emotions. Plutchik’s emotional elements are divided into eight categories: anger, fear, sadness, disgust, surprise, curiosity, acceptance, and joy. The emotional roulette theory constitutes a total of 24 main emotions; the closer an emotion to the inner circle, the more intense it is, and the closer the different emotional vocabulary items/colors in the circles, the more similar the emotions. The model can be represented as a three-dimensional cone, with the size of the vertical axis indicating the intensity of the emotions; there are a total of eight sectors, with two groups of emotional dimensions in the same diagonal direction and four pairs of oppositely arranged partitions. In the unfolded model, the emotions located in the white spaces are mixed and represent combinations of two adjacent basic emotions. Using the characteristics of this ring, new emotions can also be recombined from the 24 main ones.

With the help of these basic psychological models, emotional quantification has become possible in both academia and engineering, with the increasing emergence of emotional models further promoting the development of emotional visualization.

Fig. 1
figure 1

Russell’s model uses two metrics to represent each axis: valence and arousal (Kim 2020)

Full size image
Fig. 2
figure 2

Plutchik’s model to express a kind of interaction between people’s emotions and emotions (Kim 2020)

Full size image

3.2 Emotional computing approaches

Based on the selected emotional model, certain methods are used to collect data and perform emotional calculations. In the following discussion, we divide the emotional calculation process into four methods—questionnaires, behavior monitoring, electrophysiological measurements (brain-computer interfaces), and artificial intelligence—and briefly discuss their respective advantages and disadvantages.

3.2.1 Questionnaires

Personal questionnaires and reports allow participants to assess their emotional state by answering a series of questions, which is the most intuitive way of measuring emotions. For example, in a study by Derick et al. (2017), students provided emotional information related to their learning by answering a set of basic questions in an online survey, highlighting how often they felt motivated, happy, confused, frustrated, and bored in learning activities. The authors then used the Likert scale to score students’ subjective judgments of the proposed emotional visualization method and finally obtained the emotion labels. Another example involves Pesek et al. (2017) using questionnaires to collect users’ emotions toward a certain musical work, serving as the basis for their music recommendation system. Further, Van et al. (2004) used the emotions with which users had actively annotated a music list to match emotion categories to the list items and generate emotion maps. Adiletta et al. (2020) obtained different expressions of emotion by interviewing people with synesthesia. In summary, questionnaire surveys and personal reports provide the easiest way to measure emotions and require only a sheet of paper or screen with questions; no external equipment or additional calculations are needed. The disadvantage of the questionnaire survey is that the participants’ personal awareness is very strong, which sometimes affects the output results.

3.2.2 Behavior monitoring

Behavior measurement allows inferring emotional states from observable behaviors (Cernea et al. 2011), such as body movements, sound fundamental frequency monitoring, and facial expressions. Some devices, such as virtual reality ones, enable obtaining body motion data by detecting hand displacements (Arellano et al. 2008). Pion-Tonachini et al. (2015) used the HTC Vive virtual reality system (Robitaille and McGuffin 2019)—which includes a head-mounted display, two handheld controllers, and three additional Vive trackers—to capture the various actions of users in three different environments, using the approximate value of the body motion at a particular time to infer and calculate the users’ emotional state. Azevedo et al. (2017) simultaneously used motion, eye movement, face, and physiological monitoring, along with other means, to track and calculate emotions and obtain multimodal and multichannel emotional data. In monitoring users’ interactive behavior, Cernea et al. (2013b) output emotional tags by tracking the types of emotions generated from interactions with specific user interface components. Although behavior detection is a relatively simple means of evaluating emotional states, it is limited by the accuracy of the device, which, in turn, affects the accuracy of judging emotional information based on this method.

3.2.3 Electrophysiological measurements

Electrophysiological methods can be used to measure changes in body potential; examples include electromyography, electrocardiography, and electroencephalography (EEG), which record the activities of the muscles, heart, and brain, respectively. Tajadura-Jiménez et al. (2010) used electrophysiological activities and facial electromyography to determine the emotional effects of auditory evoked effects; they used the brain-computer interface to obtain cortical electroencephalograms (ECOGs) and magnetic resonance imaging data to measure brain activity. Cernea et al. (2013a) used a brain-computer interface to capture the user’s emotional valence and wake-up time (Cernea et al. 2015) and performed data analysis by combining information on the position and emotional state at the time of the interaction. Electrophysiological methods are usually used to assist techniques involving subjective judgments (Azevedo et al. 2017), such as questionnaires and behavioral testing methods, and can also verify the accuracy of the information obtained. Among them, EEG is the most commonly used method in brain-computer interfaces (Robitaille and McGuffin 2019) for several reasons: it can record a relatively large number of data points per second, it is relatively simple to use and portable, the equipment is not limited by the environment, and it can detect the dynamic emotions of the user. Further, the data obtained from EEG is very advantageous compared to other methods.

3.2.4 Artificial intelligence

With the development of artificial intelligence technology, emotional visualization researchers have started to rely more on algorithms for emotion calculation. The basic process involves establishing a simple model for emotional state assessment through a corpus, dictionary, or database from the perspective of information processing, or through the establishment of an artificial neural network, performing machine learning or deep learning methods, and forming various networks according to different connection methods used on emotional data. The approach involves the identification, calculation, and screening of information.

Natural language processing enables computers, as machines, to understand human language and can be applied to the manual labeling of groups (Hennig et al. 2015) or automatic data labeling (Diakopoulos et al. 2010). Chen et al. (2006) used the semantic role labeling (SRL) system to identify the chunk-level semantic labeling of the original parts of speech in news reports (Hanser et al. 2010), filter out the adjectives and nouns, and label them with sentiment and intensity values. Wu et al. (2013) used an improved sentiment dictionary method to calculate the user’s urban sentiment by measuring the similarity between the emotions represented by the words in the sentiment dictionary and the traffic keywords. In another study, Alhamid et al. (2017) used the Twitter interface to obtain a corpus of text content by establishing a model, using Ekman’s discrete model to label the text with emotion. Further, Ren et al. (2018) investigated an annotated emotion corpus of Chinese blog texts, dividing the corpus into three levels: the document, paragraph, and sentence levels. Each level has eight emotion categories (happiness, hatred, love, sadness, anxiety, surprise, anger, and expectation) and corresponding discrete emotional intensity values, ranging from 0.0 to 1.0, for emotion classification. Gaind et al. (2019) combined natural language processing (NLP) and a machine learning classification algorithm to extract six different types of emotions from the text. The NLP used two text features—degree words and parts of speech—and in the classification algorithm, a training set was automatically created, thereby reducing the work of manual labeling.

The application of artificial neural networks in emotion recognition is essentially a pattern recognition method, which describes, recognizes, classifies, and interprets elements or phenomena by processing and analyzing various forms of information characterizing them; they are mainly used in the emotional state assessment of text (Passalis and Doropoulos 2021), audio (Jeong and Kim 2019) and video (Paraskevopoulos et al. 2021) tasks. The basic task of sentiment analysis is to collect meaningful information units from text, audio, and video; extract and simplify the elements helpful for the analysis; combine sentiment models for value or graphic analysis; and, finally, obtain the results. Ha et al. (2019) designed an MDS map (Ha et al. 2014) and a social network diagram consisting of a network-aware movie, where the location of each node reflects the frequency of occurrence of the emotion word and an embedded network diagram from the same node. Gobron et al. (2010) used the Poisson distribution to convert vocabulary and language parameters extracted from a database into coherent valence and arousal intensity values to calculate emotion. In addition, Guthier et al. (2014) used the existence of words, labels, and emojis as features in the training phase to train neural networks for emotional state evaluation, the accuracy of which was then improved. The Surrey Audio-Visual Expressed Emotion (SAVEE) database, an emotional speech data set (Zhao et al. 2014), has also been constructed as a model to perform emotional state evaluation (Vryzas et al. 2017), making it possible to enhance the efficiency of calculations and accurately determine emotion types. Regarding sound emotion, DiPaola et al. (2006) first used original audio signals to calculate more than 60 low-level audio features and subsequently used pattern recognition to infer a set of semantic descriptors for each audio track in the set. In their study on emotion detection from musical instrument digital interface (MIDI) documents, Grekow et al. (2011) developed emotion labels, corresponding to the four main groups of Thayer’s four-quadrant emotion model, and realized sentiment classification. MIDI may be combined with measurement system analysis (MSA) and fuzzy statistical methods (Abboud and Tekli 2018), where the music file generates and outputs an emotion vector describing the six main emotions.

With the development of artificial intelligence technology, many open-source application programming interfaces (APIs) have become available on the market from which developers can choose. For example, Zeng et al. used Microsoft Azure Face to carry out face detection and emotion recognition with a programming interface; extracted emotions with the IBM Watson Tone Analyzer interface; and then, using a neural network, filtered laughter using the RAVDESS data set to detect sentiment tags, analyzing the sentiment polarity of word features through an Alchemy API in the feature extraction stage to determine the user’s sentiment.

4 Visual encoding

Visual encoding is the mapping relationship between data and visualization results and can enable users to quickly obtain information. Visual encoding consists of marks and visual channels. Marks usually include abstract geometric graphic elements, such as points, lines, areas, and bodies, while visual channels provide visual features for marks, including position, size, shape, color, hue, brightness, etc. This section mainly discusses the visual marks and channels used in emotional visualization.

4.1 Marks

As mentioned earlier, visual marks usually comprise abstract geometric graphic elements, such as points, lines, and surfaces.

4.1.1 Points

Points, as visual marks, are usually used for visual design in some standard graphics. For example, each element in a graphic can be represented by a circle (Guzman 2013), the size of which is proportional to the number of words in the work to represent the data. Some researchers use circles of different sizes (Lu et al. 2015) as marks to express the sentiment of a popular event on Twitter, or design circular (Zhao et al. 2014) emotional bubbles to reveal the emotional components related to the text. Other researchers use particle simulators (Khulusi et al. 2020) to associate each approximately circular particle with gravity and use particle changes to represent data. In addition to circles, asterisk graphics are also used to indicate the frequency of emotional words in movie reviews (Jänicke 2020) to promote the understanding of emotional strength. Further, emojis with characteristic expressions are often used as coding elements; for example, one can use emoji expressions that best reflect the characteristics of corresponding music to visualize emotional tags.

When using point elements, one can enhance the visual contrast effect by changing the size, color, etc. to help users understand the quantitative gaps between the data more effectively.

4.1.2 Lines

Lines are usually used in line and bar graphs and other basic views that use lines as visual marks in emotional visualization. Some visualization approaches use arcs as metaphors. In news flow visualizations (Braşoveanu et al. 2012), arcs do not link message nodes but rather connect the main keywords identified in online reports. In the case of visualization (Sung et al. 2016) with “time” as the filter condition, each box is connected to the time axis by a series of curves of different colors, with each curve representing an annotation. The intersection of the axes represents the time code of the comments left in a video.

Lines are mostly used to express relationships or trends. However, care should be taken to avoid using lines excessively, which may cause the mutual occlusion of lines and visual congestion.

4.1.3 Areas

Area is generally used to express a certain characteristic attribute in emotional visualization, such as frequency size, emotional intensity, etc., and can also be used to distinguish data after coloring. Torkildson et al. (2014) used the area of a river graph to distinguish events, compared the data, and used color saturation to make the data items easily distinguishable. When extracting psychoacoustic features from audio files (Baum and Rauber 2006) to cluster music, the size of the aggregated area in the heat map determines the number of music files with the same emotional tag.

Regarding emotional visualization that uses areas to perform distinguishing tasks, the common aspect is using color to enhance the visual effect, allowing users to have a satisfactory visual experience.

4.2 Channels

As mentioned earlier, visual channels provide visual features for marks, including position, size, shape, color, tone, brightness, etc.

4.2.1 Color channel

The color channel is commonly used for data visualization and is mainly divided into three elements: hue, saturation, and brightness. In the design process of emotional visualization, the emotional connotations of colors are one of the important factors, and the different properties of colors map to different forms of visual coding.

Hue Color is one of the most attractive means for users’ sense of sight, effectively adding contrast and comparison to data. Color or color attribute changes represent data changes. The emotional connotations of colors are an important element of the design process in emotional visual coding. As shown in Fig. 3, the current article organizes work that uses color channels in emotional visualization and summarizes several colors often used in emotional expression and the color preference of certain emotions.

The correspondence between emotions and colors is often the same as the synesthetic experience. When users see red, red-orange, orange, yellow-orange, yellow, brown, and other similar colors, they think of objects such as the sun, flames, and blood, which produce warmth, warmth, and warmth. Feelings such as happiness and danger lead people to feel impulsive. When users see green, blue, purple, and other similar colors, they imagine the sky, ice and snow, the ocean, and other similar elements, which produce feelings of being cold, openness, rationality, and calmness.

Positive and negative emotions are opposite. Contrasting or complementary colors are often selected in visual coding, while neutral emotions or attitudes often use neutral colors, such as gray, to avoid cold or warm colors that interfere with users. The work selected in this article concerns anger, controversy, depression, happiness, sadness, excitement, expectation, disgust, fear, surprise, trust, anxiety, affection, boredom, confusion, motivation, calm, jealousy, disdain, and other emotional classifications, mostly from Plutchik’s mood roulette or Russell’s ring model.

Saturation Changes in saturation make the emotional information based on color expression appear as visual information of different polarities. Low-saturation colors evoke a sense of depression, while high-saturation ones lead people to feel cheerful and pleasant. Adjusting the color saturation can convey different emotions under the same emotional partition or express two opposite emotions. For example, one can use grass green to express trust (Kuksenok et al. 2012); medium green, fear; blue, sadness; and cyan, surprise (Zeng et al. 2019). Color saturation can also express the intensity level of emotions (Derick et al. 2017)—stronger colors represent higher-intensity emotions—and changes in saturation can be used to distinguish the details of emotions (Guzman 2013). Similarly, one can use the saturation attribute to distinguish between the moods of music; high- and low-saturation colors (Jeong and Kim 2019) express more and less emotional music, respectively. Further, saturation can be used to distinguish different types of articles (Scharl et al. 2016), using bright colors to represent emotional articles and lower saturation to represent more realistic news reports.

Fig. 3
figure 3

By summarizing the emotional visualization work, we designed a doughnut chart to express the probability of using a certain color to express a certain emotion in works that use color to map emotions

Full size image

Brightness The brightness of colors can be used to express the intensity of emotions. Colors with high brightness lead people to think of blue skies and white clouds, and they easily produce feelings of softness, floating, and rising; colors with low brightness tend to make people think of steel, marble, and other objects, creating feelings of heaviness, stability, and landing. Braşoveanu et al. (2012) used brighter colors to express lower-value emotions. This type of coloring makes it possible to see the strong or weak semantic relationship between the highlighted words in a text more clearly. Das et al. (2012) used changes in brightness to express variations in emotional intensity, expressing positive and negative emotions by increasing and decreasing the brightness of the same color, respectively. In addition, Ha et al. (2019) used adjustments to color brightness to reflect emotional contrast. For example, dark yellow can be used to express sad emotions, and light yellow to express happy emotions. Similarly, Lee et al. (2016) used dark and light blue to express sad and happy emotions, respectively. Lower-frequency emotions are coded as lower-brightness colors, usually consistent with ocean colors, while higher-frequency emotions are coded into higher-brightness colors, such as magenta and bright yellow.

4.2.2 Size channel

The size channel is more suitable for expressing changes in the emotional attribute of a certain characteristic—such as sound sizes, valence and arousal value changes, etc.—through changes in size, stretching, or compression. The channel uses these attribute changes to convey emotional variations and simultaneously strengthens the differences/contrast in attribute data from a visual perspective. Zhao et al. (2014) used bubbles to indicate emotion and emotional intensity, where the tone and size of the bubbles represent the emotional category and its intensity level, respectively. As shown in Fig. 4, in the harmony visualization system of Grekow et al. (2011), the size of the graph is used to represent the volume of the music; the louder the volume, the larger the graph. The volume of each component of a chord is different, which will not only cause changes to the size of the figure but also lead to the stretching and narrowing of its shape.

Fig. 4
figure 4

Using the size of graphics to express the volume of music. a Through narrowing of its shape to express the lower volume. b Through the larger the size of the graph to express the louder volume (Grekow 2011)

Full size image

4.2.3 Shape channel

Visualization can use different shapes to express changes in emotions. For example, different emoji symbols represent different emotions. Jänicke et al. (2020) used emoji expressions that best reflected the characteristics of the corresponding music to design emotional labels in their visualizations. Many researchers also use common shapes to express different types of emotions. Kim et al. (2020) applied round, elliptical, and other soft-edged shapes to describe positive emotions, and squares, triangles, and other angular and sharp-edged shapes to negative emotions. As shown in Fig. 5, Grekow et al. (2011) used different graphics to express the emotions in different musical elements, such as pitch, rhythm, strength, harmony, and timbre, and comprehensively analyze the form of musical work, presenting a curve drawn in three-dimensional space, the complexity of which depends on the harmony of the chords. For example, considering that major chords are happy and minor chords are sad, the map of the major chords is simpler and cleaner, while that of the minor chords involves a complex, multilayered figure. Although the shape in the latter case is similar to the former, associating it with calmness is difficult. As depicted in Fig. 6, Jeong et al. (2019) designed a music visualization device using a lamp ball that can vibrate up and down. Light is generated using the ball’s up-and-down motion, yielding a curved shape reflecting the different intensities of emotional expression; the higher the curve frequency, the more obvious the effect of curve fluctuations, as reflected in the amplitude. The emotional lines with low arousal values are calm and display almost no fluctuations. The emotions with medium arousal values have soft fluctuations, with relatively smooth peaks and troughs, while the curves representing high-arousal emotions have obvious fluctuations, with sharp peaks and troughs, reflecting their intensity. Further, regarding the color channel, the faster the color changes of the lamp ball in the curve, the more intense the music emotion, and, conversely, the slower the color changes occur, the more gentle the music emotion becomes. With reference to the same curve, the wavelength, that is, the degree of stretching of the line, represents the volume of the sound; in low-volume music clips, the wavelength of the lines is shorter, while in high-volume clips, the wavelength is longer.

Fig. 5
figure 5

Using the shape of graphics to express the arousal of chord. a The major chords is simpler, cleaner, and more transparent. b The minor chord maps a complex, multi-layered figure, although the shape is similar, it is difficult to associate it with peace (Grekow 2011)

Full size image
Fig. 6
figure 6

Using the shape of graphics to express music which involving the two attributes of arousal and volume in music (Jeong and Kim 2019)

Full size image

4.2.4 Position channel

Position channels in visualization often express emotions through shapes. Objects that are close to each other on the plane are placed in the same category; thus, the position can be used to express different emotion classifications. When the plane uses coordinates to calibrate the emotional attributes of objects, the coordinate positions can also be used; these represent the size of the emotional attribute values of the objects. The movement of visual coding (Jeong and Kim 2019) is defined as the spatial movement and color changes that occur with changes in the rhythm. When the music tempo is slower, the visual code vibrates gently up and down according to the rhythm, while in the case of faster tempos, the visual code vibrates more intensely to express the change in rhythm. The color of the visual code also changes simultaneously with the rhythm. Fast-paced music causes colors to quickly reach a high-saturation state from a lower-saturation one, while in the case of slow-paced music, colors slowly transition from a low- to a higher-saturation state. Emotional visualization based on location changes (Zhao et al. 2014) can not only describe emotion categories but also capture changes in emotions over time and use visual coding to reveal the different components related to emotions.

5 Visual analysis methods

After completing the design of the visual mapping scheme, different visualization and interactive technologies are used to visualize or analyze different data types and emotional tags. This section discusses visualization and emotional visual analysis methods, including the basic chart; river, heat, cloud, and radar maps, etc.

Fig. 7
figure 7

Visualization and visual analysis work. A Colored river diagrams are used to analyze the impact of events (Zhao et al. 2014). B Thermal polymerization FIG emotion data vector for each visualized as four concentric circle (Guthier et al. 2014). C Using the word cloud using high frequency words to reflect the dominant emotion emotional network public opinion events (Jia and Chen 2020). D Shows the probability of the emotional type of histogram (Azcarate et al. 2005). E Radar map display different social media users in the same event in different emotions (Kerren et al. 2016)

Full size image

5.1 Basic chart

The column chart is the most basic visualization method, displaying the size of emotional intensity values according to the height of the columns and depicting the basic variables of emotional data from an intuitive visual perspective. Using histograms relating to any of a variety of content flows over time, with the y-axis displaying the number of occurrences of the x-axis intervals, Braşoveanu et al. (2012) displayed the order of appearance of the various emotional states, where the emotional changes may represent trends over time. Yang et al. (2019) also displayed the ratios of types of emotions and emotional judgment results in GIF graphs using bar graphs. The advantage of using bar graphs is that they display the value of sentiment data the most intuitively, shown as Fig. 7.

In emotional visualization technology, lines, or line charts, are used to reflect the changing trend of emotional data over time; the more data is included, the more accurately the trend is reflected. In general, the data usually presented in line charts is measured over time. Derick et al. (2017) used a timeline diagram to show the evolution of each student’s time investment during a course as well as the average time investment and emotional evolution of the whole class. If the classification label is text and represents a uniformly distributed value, a line chart should be used. Further, when multiple series are involved, using a line chart is notably suitable.

The scatter chart is used to investigate the distribution of coordinate points—values are represented by the position of points on the chart, from which one can judge whether a certain correlation exists between emotional variables or summarize the distribution pattern of coordinate points. Each emotional dimension in Derick et al. (2010)’s study has a different scatter plot associated with it, where the x-axis corresponds to the exact date and time when an emotion was displayed, and the y-axis represents the frequency value of the emotion. Van Gulik et al. (2004) used bubble sizes to indicate the number of posts; bubble colors to distinguish between data points belonging to minors and adults; and in the design of the introduction of music collections and its subsets, including the emotion, genre, year, and speed, equal angles to mark the similarities between artists in the view. Scatter charts can reflect the linear or curvilinear trend of the correlation between quantitative variables and facilitate the finding of outliers in the data.

The area chart emphasizes the degree of changes in quantities over time and can also be used to draw people’s attention to the trend in the total value. Alper et al. (2011)’s work summarizes and organizes consumer review fragments into an area chart based on the most frequently discussed main product features and bipolar emotions in the reviews. The keywords of each feature-emotion group are displayed in their corresponding areas, and the areas of the chart are used to indicate the number of comments.

5.2 River map

A river map is a special type of flow map, which is mainly used in emotional visualization technology to represent changes in events or topics over a period of time. Torkildson et al. (2014) used colorful river diagrams to analyze the emotional impact of events to facilitate numerical comparisons. Chen et al. (2014) used a river graph to display the number of emoticons and used these to indicate the levels in the graph. Each layer represents a type of emoticon and the number of each type. Zhao et al. (2014) used emotional bubbles to reveal the different components related to emotions in a river graph in order to depict the emotional profiles of social media users and summarize their changes over time. In the field of opinion analysis, Sung et al. (2011) used river graphs to analyze user opinions, where the x-axis forms the time axis, and the y-axis displays the attribute value selected by the user. Each box is connected to the time axis through a series of curves of different colors, with each curve representing an annotation. Each curve and its intersection with the video timeline representation of time leave a comment stamp. Emotional visualizations using river graphs all explore the changing trends in emotional data over time. The combination of river graphs and interactive technology can express data more accurately. Though faced with a significant amount of data and large numerical fluctuations, the river graph has an appealing visual structure, which can notably attract the attention of users while highlighting large changes in data.

5.3 Heat map

Due to its rich color changes and vivid and complete information expression, heat maps are widely used in various complex data analysis scenarios in emotional visualization technology. Ha et al. (2014) designed a heat map to depict the main emotions of netizens in film reviews by calculating the frequency of the emotional words in each review. Guthier et al. (2014) visualized the aggregate emotion vector of each data point as four concentric circles, which have the same center but different radii. The radius is determined by the number of tweets; the larger the radius of the disc, the more tweets it represents. At the same time, the concentric circles are arranged according to the relevance of the content. In the emotional heat map visualization design with “Syria” as the search term (Hennig et al. 2014), a color gradient is used to map emotions to colors on the map; the gradient transitions from dark red to white to dark green, indicating negative, neutral, and positive emotions, respectively. In the analysis of movie ratings and movie visualizations, Topal et al. (2016) combined conventional and emotional heat maps (Jänicke 2020) to express the emotions in a certain area based on the emotional content of the reviewers. The advantage of a heat map involves displaying multiple dimensions of information in the same picture. Clustering and different color schemes facilitate distinguishing different types of emotions. Heat maps can reveal differences in data, especially for very large data sets; help researchers intuitively understand the distributions or differences in data; and help in locating outliers in the actual analysis process.

5.4 Clouds

Clouds filter out a large amount of text information and only display words related to emotions. Users can directly understand the emotions of text through clouds. Guzman et al. (2013) used word cloud graphs to represent topics in long text content, where the words displayed in each word cloud belong to the same topic. Each word is generated according to the text content, and its color is used to distinguish its emotional polarity. The font size of the words indicates the frequency of occurrence of the topic represented. In Jia et al. (2020)’s study, based on the visualization of the emotional feature words of online public opinion participants, high-frequency emotional words with larger font sizes, notably those close to the center of the word cloud, reflect the dominant emotions of online public opinion events and indicate the emotional tone of the events and the participants. Siti et al. (2020) designed square and circular word cloud maps to represent different orders of magnitude of the same data, obtained from users’ use of emoticons on Twitter. The word “day” is at the core of the square word cloud diagram and appears the most frequently compared with other words; it is surrounded by other words with different frequencies. The main words displayed in the Fang word cloud have a relatively high frequency. In the circular word cloud map, the word “day” is also at the core (in the center); the words in the surrounding outer circle decrease in size in order of frequency, with the words in the outermost area of the map occurring less frequently. The word cloud image can visually display the “keywords” that appear frequently in a text, filtering out a large amount of low-frequency and low-quality text information such that the reader need only scan the text to grasp its main point.

5.5 Radar map

A radar chart is more suitable for expressing multidimensional data in emotional visualization and distinguishing data according to the multi-attribute classification criteria of sports. Kempter et al. (2014) used the emotion wheel to scale the number of tweets in their collection, expressing the width of the gray ring as the number of tweets and visualizing the emotional contours as a radar chart; that is, the width of the radar chart and the number of tweets represented in the set are proportional. A black circle is placed at the center of the radar chart to cover the low-level sentiment value and remove the remaining useless data. Li et al. (2020) added emoticons to the radar chart to indicate emotion classification and increased emotional arousal. Based on the abovementioned work, the radar chart has obvious advantages in expressing the multiple attributes of emotional data.

5.6 Other methods

The chord chart is mainly used to show the relationship between a variety of emotion types and data, and it is particularly suitable for analyzing the relationship between complex data. For example, Gali et al. (2012) used data captured online to reflect the number of social media posts in real time. Using circular elements placed side by side to compare the polarity and number of emotions, they used the size of the elements to express the number of posts per day, color to represent the emotional polarity of the display, and a curved intermediate line to connect keywords matching similar topics.

Force-directed charts are often used to express the connections between elements. Lyu et al. (2021) visualized the neural network as an interactive force-directed graph in the virtual reality system, revealing its inner workings in an experiential manner. At the same time, combined with the audibility of the data, the neural network explores the different emotional states of the user during the interaction.

The rose chart, also known as the coxcomb diagram and the polar coordinate area diagram, is suitable for comparative analysis. The angles of each fan in the rose diagram are equal, and the chart essentially involves using arc shapes with different radii to represent the data size. Arturas et al. (2020) used a rose diagram to show the correlation between emotion, biometric status, and environmental characteristics to reflect the analysis of marketing value.

The violin chart is often used to display the data distribution and probability density. This type of chart combines the characteristics of the box plot and the density map to show the distribution shape of the data. Using quantization and a violin chart, reflecting the association between emoticons and eight basic emotions, Li et al. (2019) revealed a visual coding correlation between potency and mood.

5.7 Visual analysis system

The visual analysis system is mainly used for the analysis of the association in vast multidimensional and complex emotional data. Because the information involved is relatively scattered and the data structure is not uniform, the visual analysis system can use multiple views to display data-related information, fully display the process of emotional data analysis and the trend in data changes, and provide interactive operations for users to explore and analyze the data on their own. Scharl et al. (2016) included theme management and content navigation in the interactive design of their dashboard. Users can click on a label to trigger a full-text search and select the theme displayed in the view. In another study, Kucher et al. (2016) provided two types of tabs: a single-timeline-view tab, which is used to handle any number of timeline views, and multiple-document-view tabs, which are opened when documents are retrieved within the selected time interval. The study includes a time series analysis, which was used to compare the values of targets and users, perform trend analysis, explore the text content and location mark distribution in selected documents, and export static content for users to view. In emotional visual analysis, the commonly used interactions are operations such as clicking and selecting boxes and using drop-down menus to filter information.

In their panel design of the visual analysis system, Prasojo et al. (2015) showed the link between the emotions related to a news source on the use of solar panels. They examined the emotions from user comments using text extraction, thematic orientation, and an emotional visualization panel, where the description panel provides the content analysis of the comments on the original news source. For the comprehensive analysis of multisource data, emotional tags can be explored through the visual analysis system. For example, in analyzing sentiment consistency, Zeng et al. (2019) developed a visualization system called EmoCo and designed consistency and sentence clustering views such that users can quickly become familiar with sentiment consistency and its temporal evolution, shown as Fig. 8. Further, Da Silva Franco et al. (2019) developed UXmood, which binds system data from video, audio, text, eye trackers, and interaction logs and displays them in a configurable dashboard, allowing the user to interact with the screen. The sentiment visual analysis system conducts correlation analysis on data, fully displaying the process of data analysis and the trend in the data chain.

Fig. 8
figure 8

Emoco (Zeng et al. 2019), a visual analysis system for emotional consistency

Full size image

6 Discussion

In the study process of this paper, we first need to focus on the way of emotional computing. The most common method of sentiment analysis is to express the problem as a three-way judgment task, and divide emotions into three categories: negative, neutral and positive. Sentiment analysis task can also use more complex formulas to judge the possible emotion categories involving a wider range and classify them into more diverse emotion types based on the classification scale. Next, we center on the visual marks and visual channels of visualization coding. The design of visual encoding should serve emotion and data, and every change should have a certain meaning. The mapping relationship between coding and data should enable users to obtain information and trends in data more quickly and conveniently.

Then, we conduct a classified discussion on the visualization of different analytic tasks. Finally, the data sources, emotional computing methods, visual encoding design and visualization or visual analysis of the eighty papers are described in detail.

Now, there are still some problems needing to be solved in the study of emotional visualization. The primary problem is that the source of emotional data is relatively single, and the data collection means of users are restricted by the development of collection equipment, including the questionnaire, behavior monitoring, electrophysiologist signals, brain computer interface and so on mentioned above, which all have some shortcomings. Because the subjective consciousness of some means is too strong, we cannot directly obtain the users’ emotion from the brain through the portable machine, which leads to the fact that we can only collect and process the emotional data through limited data collection means, and it is impossible to avoid the errors caused in the collection process.

What’s more, when extracting data, emotional visualization technology must use specific emotion analysis models (such as dictionary and specific corpus) to analyze and calculate within a specific scope (word level, discourse level, etc.), which also results in the limitation of emotional computing task in emotional visualization.

Likewise, the study of emotional visualization also has the problem of too high consistency of visual encoding. In the existing emotional visualization work, the design of visual encoding is relatively single. We always design the visual encoding of data from the perspective of visual perception, which causes the overuse of the three visual channels of color channels, shape channels and size channels, and results in users’ aesthetic fatigue of visual encoding design. In the future, emotional visualization can be optimized in terms of color matching, ease of use, and accessibility.

Besides, the presentation of emotional visualization is also limited to display devices, for mobile phones or computer screens can no longer bring users a more novel usage experience.Mobile phones or computer screens can no longer bring users a more novel experience. In the future, emotional visualization can combine virtual reality technology and holographic projection technology. In digital cities, smart cities, Visualization of large screens, urban brains, digital twins and other fields combine with urban emotions, road traffic emotions, social relationship diagrams, etc. to enhance the user experience of emotional visualization.

7 Conclusion

In this paper, we analyze 75 emotional visualization work and obtain the results as follows. First of all, we summarize four data sources of emotional visualization, including text data, sound data, image data, and video data. Secondly, we summarize the emotional analysis tasks from subjective detection to technical emotional analysis and position analysis, including questionnaire, behavior monitoring, brain-computer interface and AI. Thirdly, we summarize the visual encoding of emotional visualization from three aspects of point, line and area, and summarize the visual channel of emotional visualization from four aspects of color, shape, size and position. Fourthly, we summarize the advantages of visualization technologies such as heatmap, river and radar in different analytic tasks by investigating the visualization and visual analysis work of different analytic tasks. Our survey ranges from theoretical linguistics to social media and news monitoring, from text opinion mining, music perception enhancement to facial emotion exploration, and the development and utilization of a variety of visual encoding and visual channels. Furthermore, we also discuss the views on emotional visualization technology and the optimization direction in the future. We hope that the future of emotional visualization technology can be the integration of multiple perception technology and visualization technology, and emotional visualization can also give multi-level and multicategory feedback based on the vision. Besides, relying on visualization, a subjective and creative technology with high degree of freedom, emotional visualization technology should create more multi-level feedback approaches to get rid of users’ aesthetic fatigue under multidimensional data cooperation in the future.

We will continue to give attention to the field of emotional visualization in the future. We hope that our investigation may be of benefit to the researchers who study emotional visualization, and also can help practitioners or researchers in other fields who have an interest in the visualization and visual analysis of emotional data.