Keywords

1 Introduction

The Interactive Museum of Court of Audits of Paraíba State (in Portuguese, TCE-PB) is a project that merges applications, games, augmented reality environments, interactivity and audiovisual contents with the objective of making people aware and involving society in actions towards combating corruption and expanding social control. The resources are designed to lead the visitors to engage in activities focused on fighting corruption daily. The Court of Audits is responsible for monitoring public expenditures and overseeing public works.

The theoretical and methodological model Audiovisual Design (AD) proposed by Becker et al. (2017a) was the conceptual and methodological basis of interaction systems and audiovisual productions for the Museum. The model describes four roles individuals can assume while consuming audiovisual content: Audience, Synthesizer, Modifier and Producer. Each role has its enhanced levels, described as Players, when individuals engage and use all available resources. One important concept of the Audiovisual Design Model are the Interaction Triggers, namely Triggers of Action (ToA) and Triggers of Inertia (ToI). These are elements responsible for calls to action and calls to inertia and should be considered and included in each designing phase of a production. ToA may consist of visual and sound elements or even intrinsic motivations (e.g. personal and cultural) provided by the content. ToI usually are present during moments that require high levels of attentiveness. Sometimes the viewer may be unconscious of ToI, although they are central elements in storytelling and long narratives.

This article describes the production process of the audiovisual content and development of interaction strategies to engage people. We will focus on how triggers were developed considering the diversity of the Museum’s audience. The project includes around 100 min of institutional videos and content produced by high school students. The project also includes, apart from the Museum, other two visualization systems, one online and other suitable for big audiences, such as lectures and classes in auditoriums.

Each role described in the Audiovisual Design model has associated affordances to engage (ToA) or to relax (ToI). The theme of the museum is ‘Fight Against Corruption’. This way, when individuals move from the role of Audience to higher levels of engagement encouraged by ToA, they shall also develop greater awareness of control of public matters. The highest level of engagement is to produce content to be included in the Museum’s collection. ToI encourage enjoyment of the institutional videos and content produced by students. Consequently, the whole experience of producing and visiting the Museum should engage all roles described in the Audiovisual Design model.

The outcomes also depicted in this paper formulate the planning process of those interaction triggers applying a use scenario. A persona named Student interacts with audiovisual systems (audio and video content and software) using native media affordances (physical, graphic and symbolic) developed according to each Design Line provided in the DA model, based on actions, or sequences of actions, expected in each role.

This paper is structured as follows: the section two describes the Audiovisual Design model, its roles, affordances and triggers. Section three presents the system architecture, components and resources; section four describes the proposed interactions, based on a persona named Student and a use scenario. The fifth section discusses the museum’s triggers and analyses interaction strategies. Finally, section six offers some conclusions and future work suggestions.

2 The Audiovisual Design Model

The Audiovisual Design (AD) initially proposed by Becker et al. (2017a) and developed in studies of Becker et al. (2017b), Toscano et al. (2017), Gambaro et al. (2018) and Becker et al. (2018b) is a theoretical-methodological model for analysis and content development for traditional media and complex audiovisual systemsFootnote 1. The AD arises from theoretical propositions of integrating concepts and processes of media studies, (Souza 2003; Jenkins et al. 2015), and Human Computer Interaction, (Barbosa and Silva 2010; Preece et al. 2015). In order to contribute to these studies, AD develops an interdisciplinary arrangement that considers elements such as content, interfaces and multiple levels of engagement as essential to audiovisual fruition.

The model is based on the description of four roles individuals can assume when consuming audiovisual contents: Audience, Synthesizer, Modifier and Producer. The Audience’s role represents the “passive” behavior of the individual during enjoyment of audiovisual contents. The Synthesizer has the skills to compile, classify, comment, recommend and share content, thus building a digital “identity”, a profile staged in a social network. The Modifier dominates software to manipulate and recreate content, broadening the notion of engagement to appropriation. Finally, the Producers, which may correspond to a person or a group of people who create original content autonomously, independently or collaborate with large media corporations, are responsible for the design and creation of content.

In addition, each role has its enhanced level, described as Players, which corresponds to an enhancement of the individual’s actions when engaging and making the most of available resources. The Player superposes all other roles because it refers to individuals who completely use the tools available on each level. They are treated as Audience-Player, Synthesizer-Player, Modifier-Player, and Producer-Player.

The model also describes Designing Lines, which correspond to product modeling and expected actions of the roles in four aspects: Identity, Motivation, Experience and Content, as conditions to guide both audiovisual production and interaction. The design, using the Lines, can be summarized in: (a) Design with focus on the individuals, through the Line of Identity, idealized by the Producer, or Modifier, for Audience; (b) Design with focus on motivation, through the Line of Motivation, idealized by the Producer, or Modifier, for Synthesizer; (c) Design with focus on the experience, through the Line of Experience, idealized by the Producer for Modifier and Player; (d) Design through the Content Line, which is central at all times, for all roles. It is in content inferences of quality and utility are centered and the intersections between the roles occurs.

Another analysis of the Designing Lines starts from interaction, or from the fruition point of view, and considers the change of roles from an increase in engagement and interaction. The Audience identifies itself with the content and shares it, motivates itself to modify it, and then develops the experience of producing it. Since AD is a set-based model, this change of roles is not necessarily linear, and there may even be action on two or more roles simultaneously. For example, when watching a television program (Audience), the individual, with his cellphone shares comments about the show in a social network (Synthesizer) and records excerpts for a Youtube channel (Modifier or Producer). Figure 1 shows the Audiovisual Design architecture.

Fig. 1.
figure 1

(Source: Audiovisual Design Research Group).

Audiovisual Design architecture

Full size image

Considering interaction happens through interfaces, a concept relevant to Audiovisual Design is Affordance, characteristics of interfaces that are quickly identified and intended for specific uses. From this definition, the authors use the concept of Media Affordances (Becker et al. 2017b), which represent indicative elements present in traditional media and audiovisual systems, which point to meanings and means of enjoyment. Changing roles depends on a correct perception of those affordances, associated with the environment of fruition.

The affordance concept was discussed initially by Gibson (1977) and applied to the media sphere by Becker et al. (2017b), who summarized them in: (a) Physical, affordances that can be actively manipulated, with potential uses easily perceived, composed by technologies used to enjoy content; (b) Graphic, affordances present in interactive graphical interfaces implemented by software and responsible for the mediation between individuals and content; (c) Symbolic, affordances present in narrative or embedded in audiovisual content, where physical affordances are more subtle and cognitive and sensory relations depend not only on technology and interfaces, but also on understanding narrative elements.

The correct identification of possibilities and means of interaction impacts from simple reception, through propagation (Jenkins et al. 2015), and reaches actions that are not foreseen by the Producer, but which contribute to the experience of the individual (who feeds his “identity” into social media profiles) or collectively to a content-related group, such as fan communities, discussion, and technology enhancement.

The action materialized from the perception of value, or utility, of the individual on system’s affordances acts as a Trigger, or a element planned by the Producer during the interaction to reach a certain objective (consumption, propagation, participation, attention). A perceived affordance, understood as a call to a certain action, results in a trigger. For example, when entering a share button within the content, the individual in front of the system understands that a new mode of interaction and use is available and possible (Affordance). This button, in turn, acts as an Action Trigger, it influences a certain behavior in engagement with the audiovisual system.

Interaction Triggers are based on sound, visual, interface, interaction elements or even action calls on the content itself. AD basically considers two types of triggers: Action and Inertia. The first stimulates the perception of value or utility in relation to the audiovisual system in order to generate more engagement, for example, getting out of the Audience level when watching a TV show to share impressions online and become a Synthesizer. The Inertia Trigger stimulates the individual to receive the content through attention, analysis or interpretation. For example, after sharing data and interacting with a graphic interface, individuals return to a passive mode of view, just watching a TV show.

3 The “Student Citizen” System

The system described in this paper, named “Student Citizen”, is part of a project that integrates high school students into a set of activities during classes, such as video production, visit and engage in Court of Audits of Paraíba State (in Portuguese, TCE-PB) actions. Students are invited by teachers to produce videos about public matters, focusing on success stories or inefficient aspects in their school community. The audiovisual content is published in TCE-PB’s Facebook page. To publish the videos, each student describes the school’s name, city and uses predetermined hashtag to inform about the content present in each video. After this stage, conducted by students, teachers review the contents, identified by the system via the GRAPH APIFootnote 2, as shown in Fig. 2.

Fig. 2.
figure 2

Management panel

Full size image

This data collection (video, user name and school) serves two systems: Advanced and Collaborative Visualization System via Paraíba State Map and the Support System for Lectures, also called as a Custom Video Generator.

3.1 Advanced and Collaborative Visualization System

The contents are visualized in a Collaborative Map (Fig. 3), which can be accessed on line or viewed on the Interactive Table in the Museum. On line a video gallery presents the students’ productions and information about which schools are producing content. In the Museum a touch-sensitive screen contains only the map. In it the teacher defines videos to be presented and chooses a photo from the school. The system generates a personalized video, opening with images from the school, followed by the students’ videos and respective credits. The video can be shared with a QR Code, the last image of the video.

Fig. 3.
figure 3

Collaborative map

Full size image

3.2 The Custom Video Generator

TCE-PB receives high school students to visit its physical structure and the social actions carried out by it. One strategy to integrate these visitors in themes such as social control and fiscal education, a system of lecture support was developed to integrate institutional audiovisual products into the collaborative production of students, with the purpose of personalizing the experience of each class that visits TCE-PB facilities.

Applying the Audiovisual Design architecture in the development of this system, we have the following proposal: from an institutional video, which defines the Content line and conceptualizes people’s importance in supervision and monitoring public matters, videos produced by students are inserted as integral elements of the narrative (Line of Identity and Motivation), according to Fig. 4.

Fig. 4.
figure 4

Abstraction of narrative personalization.

Full size image

The personalized video, generated based on students’ data has the following structure:

  • Part 1: Contextualization on aspects of mismanagement and corruption, and how popular participation is important in combating and monitoring failures in public management. Following AD architecture, in this stage of video narrative resources (rhythm of editing, soundtrack, forms of expression, among others) have to be coherent with students spectating in such a way as to stimulate the attention and interest about the subject.

  • Part 2: Examples of social control attitudes based on student videos. Following the architecture of AD, the use of students’ videos acts as a Trigger and stimulus to attention and formation of Identity and affinity with the content.

  • Part 3: Experts explain students’ acts represent social control, a fundamental role in whole society. Following the architecture of AD, this stage of the video uses the principles of the Content and Identity Line, since the participation of the students is handled at the content level, as a partner of TCE-PB.

  • Part 4: Movie credits with names of participants (experts, school and students). This step is also a Trigger, it acts as a stimulus to attention and formation of Identity and affinity with the content.

  • Part 5: Providing QR code for video sharing. Following the architecture of AD, this step acts as an Action Trigger, forming the Identity to stimulate the student (Audience) to switch the role to Synthesizer, by sharing data in their own social media.

Other Designing Lines can be impacted by the system are Motivation and Experience. Since this personalized material from individuals’ data is made available in a file format compatible with social media (Youtube, Facebook or WhatsApp) it is possible to enable scenarios in which students can modify, edit or even remix the content.

To guarantee a scenario of enjoyment and interaction, the following functional requirements for the system were delineated: stablish connection with Facebook via API Graph; Incorporate hashtag, video, student name and school from a public post on Facebook; Insert imported data into the video project “Student Citizen”; Render all material (fixed institutional videos, customizable and data) to generate a single video file compatible with HTML5 and services like Youtube and Vimeo.

4 The Proposed Interactions

To detail the strategies of interaction and engagement, a system usage scenario based on a persona Student named Wellyngton is described below. Use scenarios are verbal narratives objectively situated, with contextual details, aiming to characterize the context of interaction and define relationships of individuals with technology. They are useful in design to visualize uses of the system in development. Personas are archetypes that represent a set of users for whom the design is done (Lowdermilk 2013; Preece et al. 2015). The focus of the scenario lies in students because it is through them the general objective of the project, related to citizen and fiscal awareness, is supposed to be reached. To create this persona and understand user requirements, a user research was conducted with 1205 high school students, using face-to-face interviews and online questionnaires.

4.1 Persona Student

Wellyngton studies in second year of high school, has lessons weekly about History, owns a prepaid internet access smartphone he uses while the daily data limit lasts. In addition, he connects to the internet using wi-fi at school, where he spends the day. Wellyngton has accounts in several social networks, but mainly uses Instagram, Youtube and Facebook. He thinks all politicians are corrupt, but is intrigued because the History teacher, Cida, spoke during a class it is everyone’s responsibility to monitor and control public spending, including those at school.

4.2 Use Scenario

In School: During the History class teacher Cida presents the Portal Student CitizenFootnote 3 (ToA 1), which has a Map of State of Paraíba with municipalities and schools. Wellyngton is very interested in videos produced by students from other schools about public management, enforcement and denunciations about unfinished public works. When teacher finishes showing videos, she asks if their class also wants to produce and include videos in the Portal. Cida then shows a movie about how to record and share videos on Facebook (ToA 2), which will then be included into Student Citizen Portal.

Wellyngton creates a group with his friends and records two videos, one about an unfinished bridge on the way home and the other about an abandoned ground next to the school where people throw garbage. They share the videos on Facebook, with hashtags describing the content and the school name (ToA 3).

In the next class, teacher opens the Student Citizen Portal, clicks on the Alagoa Grande municipality, in the name of the school Josué Gomes da Silveira and all videos produced by the class are listed (ToI 1). The class attends and discusses the origin of the problems, responsibilities and how to solve them. To compare, the teacher shows videos of students from Padre Hildo Bandeira State School, from the neighborhood, who also recorded a video on the unfinished bridge of Wellyngton’s group.

4.3 Scenario of Use Visiting the Interactive Museum

The school receives an invitation to visit the TCE-PB facilities, through the School and Citizenship Project. Wellyngton is elated to know TCE, which he has only heard about on television. At TCE students are received with snacks and juices and invited to enter the auditorium (ToI 2). A lecture begins on recycling garbage, with a TCE professional speaker. During the lecture, video clips are shown, among them those produced by school students, such as Wellyngton’s, relevant to the subject (ToA 4) and incorporated into the presentation by the speaker.

At the end of the lecture a QR Code appears on the board (ToA 5). The speaker explains that by photographing it students have access to the whole video, can copy it or share it on social networks. Wellyngton immediately photographs the QR Code and shares it in his social media profile, commenting that his video has appeared in a TCE event.

The speaker then invites Wellyngton’s class to visit the TCE Interactive Museum, where Cida begins by showing the games, virtual reality, the coworking space, and a theater to watch movies. Then the class goes close to a touch-sensitive monitor, which contains the same Student Citizen Portal Map (ToA 6). The teacher again selects the municipality and the school names. But this time there does not appear the list of all the videos produced by the class: the best videos act as “symbolic affordance” (ToA 6) and are assembled in one workpiece. Cida elucidates the videos she chose about public works and explains the responsibility on roads and bridges can be as much of the city hall as of the state and federal governments. She selects “play videos” (ToI 3), which now start with a photo and name of the school. When the video ends, teacher’s name and the students who recorded the videos also appear on a black background, such as on TV. A QR Code offers the option to share the video (ToA 7). Wellyngton leaves TCE-PB facilities talking to his colleagues about themes of next videos they will record.

5 Trigger Definition and Analysis

Methodologically, according to the Audiovisual Design framework, once established the objective to be reached with the content, the Designing Lines are defined, according to a utility perception of each role. Considering purpose and utility, media affordances are developed, resulting in triggers to promote notions of system quality, thus guaranteeing good experiences of individuals. Each Designing Line provides tools to achieve the individual’s goals.

Every change of role begins with the perception of a media affordance and is performed using triggers, which may correspond to action or to inertia. In the case of Student Citizen project, there are changes of roles proposed by the system, and the persona Student will alternate among Audience, Synthesizer and Producer. Modifier actions are focused on teachers and speakers. Table 1 describes the Triggers of Action (ToA) and Inertia (ToI) designed to change roles using the four Designing Lines: Identity (LI), Motivation (LM), Experience (LE) and Content (LC). The triggers were developed based on the use scenario and how individuals should change roles while using and interacting with the Student Citizen system, including getting contact with social control matters.

Table 1. Triggers for Student Citizen.
Full size table

Considering the purpose of this project, specifically raising awareness about social control and involving society in actions to combat corruption, individuals together play a key role. It is considered that high school students have a central role in medium and long term in this process. The main Designing Line that guides audiovisual enjoyment is Identity. Allied to the Content Line, this Line represents the contact of the individual with the premise of awareness. The stimulus to share is materialized through Motivation Line, focused on the role played by Synthesizers.

From the theoretical point of view, the Audience, when creating the identity with the theme, is encouraged through Triggers of Action to change to the role of Producer. In this role, production and sharing take place (remembering by describing the AD in form of sets, the Producer has the abilities of all antecedent roles). By enjoying videos in the classroom (Trigger of Inertia), the student, who acted as a Producer, returns to the role of Audience. It is important to consider the change from the role of Audience to Producer goes beyond engagement through identification. The stimulus to active participation is part of life experience, which, as a citizen, leads to produce contents relevant to this context. Likewise, we cannot ignore that Identity Line will also be fundamental for students who do not produce, for, identified as citizens, to be motivated to at least watch the videos.

At this point, the teacher is a mediator between objectives of the system (defined by TCE-PB) and Audience, serving as a trigger, both of action in the first moment and of inertia in the second. Already when the teacher moderates videos that compose the Portal and creates the playlist of personalized videos in the Museum, he himself acts as a Modifier. In the case of moderation, Modification happens at a macro level, since the teacher changes the whole system by selecting which videos compose the Map; in the Museum this Modification happens at a micro level, generating a video with data of the students.

A similar relationship can be established when the Audience watches the contents in the Auditorium and in the Museum. In both cases, a modified content is offered to the Audience with a trigger to share at the end (Synthesizer), represented by the QR Code. By photographing it and sharing the video, student moves from Audience to Synthesizer.

The media affordances prevail in defining triggers of this system are symbolic. Calls for action or for inertia are part of the teacher-student relationship within the classroom and in the Museum, as well as speaker and audience, in the auditorium. In addition, there are elements of awareness, central objective of the project, that lead to the action of the Audience. This process of awareness takes place at psychological and motivational levels, where perception of a possible agenda for the video can also become a Trigger of Action (as is the case of the unfinished bridge of the use scenario). The QR Codes, used in two moments, correspond to visual media affordances, centered on the video graphical interface.

6 Conclusions and Future Work

This article describes how the methodological process of Audiovisual Design was used in the production process of customized audiovisual contents for an Interactive Museum. The development of Interaction Triggers is described, based on the museum’s objectives and the visitor profile. Each role described in the Audiovisual Design model has associated affordances for Triggers of Action (ToA) or of Inertia (ToI). Role change is a central part in the production of content for complex audiovisual systems.

A methodological approach was designed starting from the objective to be attained through the content, considering different possibilities of fruition. From the objective relevant Design Lines were defined, aiming to awaken the notion of system utility (audiovisual content, interfaces and interaction). From this logical construction, relevant medium affordances have been defined, which lead to Triggers. All these steps should result in system quality, central element of usability for individuals’ satisfaction.

The research on the construction of generative audiovisual systems is still undergoing. The next parts are user tests to validate usability and engagement. As future activities, this research has two actions. The first one, theoretical and conceptual in nature, is related to an extended description of the relationship between tools needed in each Designing Line with development of media affordances and triggers. The second action consists of evaluations of software development quality and usability tests with students, teachers and employees of TCE system users. In this way, we intend to validate the development described in this article, specially aspects related to the Motivation Line, with development of engagement.