Keywords

1 Introduction

The development of audiovisual content has become an increasingly complex activity as new features of interaction, sharing, different screens and forms of visualization are becoming popular. The audiovisual experience, which has been mostly occupied by film and television, now becomes more comprehensive, ranging from Internet research to complex algorithms of recommendation, collaboration and content adaptation. One feature of this scenario is the presence and use of software to compose the experience of enjoyment of audiovisual workpieces. Expansion of computers processing power and increased access to stable network connections enable a scenario where audiovisual content producers are increasingly close to programmers, interaction designers, and data scientists.

An online video, a movie, digital games or even soap operas are examples of how representation media (audio and video) can be used to build an aesthetic and communicational experience. On the one hand, we have the discussions about technologies of capture and representation stimulated by Computer Science, on the other hand we have discussions in the field of Communication and Arts about values and meanings generated by content being exposed and watched.

From this computational and complex media context it is possible to identify discussions and experiments that dialogue with the capacity to generate audiovisual narratives from data collected from audiences. In the first cinematographic and television productions it was impracticable to think about construction of content adapted to individual interests or small groups due to the high cost of production and distribution. Nowadays, with the increase of computational processing and internet reach, this scenario becomes increasingly viable.

Based on this complex context of audiovisual media usage arises the theoretical-methodological discussion of Audiovisual Design (AD). Initially proposed by Becker, Gambaro and Ramos [1], the AD integrates aspects of Media and Reception from the Communication Science field to Human Computer Interaction (IHC) tools and concepts, with the purpose of assisting in description, analysis and creation of audiovisual systems. In a second publication the AD is characterized by the authors as an architecture of content and interaction capable of attending contexts of historicity and generativity, scenarios in which both the interaction and the user data are used by the system in offering contents [2].

This work proposes the experimental application of Audiovisual Design as the architecture and method of a generative project, a system that integrates user data and interaction for generation of new customized content. All videos are available on line and compose workpieces accessible in an interactive museum. This study is being carried out by researchers from the Audiovisual Design Research Group at the Laboratory of Interaction and Media (LIM) in the scope of a project for the State of Paraíba Court of Accounts.

2 Theoretical and Conceptual Bases

The development of computational processing capacity as well as the emergence of new sensors have enabled software to innovate ways of capturing and interpreting data to better understand what information is being captured, processed and reproduced [3,4,5,6]. Parallel to Computer Science studies it is possible to identify an increasing discussion about different ways of interaction and enjoyment, present in the community of researchers of the Media Studies field. There are scenarios of multiple means and forms of reception [7] in which systems also act as discourse partners [8] and reveal the emergence of new standards and conventions in the process of communication, representation and audiovisual enjoyment [9]. Just as computational studies approach the semantic value of digital data, producers and artists of audiovisual medium are also incorporating and using data and diverse contexts of interaction to improve their narrative and communicational premises.

2.1 Adaptation, Customization and Personalization

The terms customization, adaptation and personalization are outlined in literature in slightly different ways. The logic of customization appears in contexts of offering resources to individuals so they can adjust pre-established elements according to their interests. Adaptation is delimited mostly to technical contexts such as adaptation to network requirements, screen size or accessibility. Finally, the idea of personalization is usually associated with filtering information about audience to recommend, reposition, insert, delete, display certain content or mode of interaction [10,11,12,13,14,15].

After comprehending the idea of personalization, it becomes necessary to understand what can be personalized. The study [6] conceptualize generation of elements for personalization of audiovisual narrative experience goes through different structural levels. The most general element is the narrative, a precisely form or expression found to present a story. Story is a set of elements constitute a linear relationship. The stories are composed by plots, which are composed by dramatic events. A story can be told from a variety of narrative forms, just as the same story can have variable plots allowing the audience for different perceptions. This structuralist approach helps to understand levels at which digital systems can act in a way that impacts content and enjoyment of individuals.

3 Methodological Procedures

The architecture of Audiovisual Design (Fig. 1) is the conceptual and methodological basis of this study. The AD essentially describes four functions, or roles, individuals can assume while consuming audiovisual content: Audience, which represents a passive enjoyment of content; Synthesizer, where individuals organize, share and comment content on social networks; Modifier, which corresponds to individuals who appropriate contents and modify them; and Producer, which represents the individuals, companies and software that generate new content. Each role has its improved levels, described as Player, when individuals engage and use all available resources, or even completely change the purposes of content and system.

Fig. 1.
figure 1

Audiovisual design architecture.

Full size image
Fig. 2.
figure 2

Management panel

Full size image

In the Audiovisual Design framework interactions and relations with content occur through four Designing Lines: (a) Content; (b) Identity, which bases the personal relationship with content; (c) Motivation, which leads to engagement; and (d) Experience, which relates enjoyment to advanced use of resources, providing practical or symbolic experience with content and interfaces. This Lines are described to assist the developer of audiovisual system (content and software) to design the interactions, engagement and content of production from variables that impact the enjoyment of individuals.

Another relevant concept to AD architecture are triggers, which arise from the Designing Lines. A trigger is any element added throughout the fruition, or interaction, in order to stimulate individuals’ actions. They can come as sound or visual elements, graphic interface, interactions or even action calls done by audio or video content.

AD will be used to describe and create this project. It proposes a set of relational variables or layers that should be considered by content producers within a context of complex media use. In general, the steps that make up the development from the architecture of Audiovisual Design are:

  • Identify purpose, main uses of the system and personas;

  • Delimit Designing Lines;

  • Design Triggers and Affordances;

  • Develop the system (content and interface)Footnote 1;

  • Test the system.

The following sections detail the application of these concepts in the construction of the generative audiovisual system.

4 Results

The system described in this paper is part of a project that integrates high school students into a set of activities during classes, such as video production, visit and engage in State of Paraíba Court of Accounts (in Portuguese, TCE-PB) actions. Students are invited by teachers to produce videos about success stories or inefficient aspects in their school community. The audiovisual content is published in TCE-PB’s Facebook page. To publish the videos, each student describes the school’s name, city and uses predetermined hashtag to inform about the content present in each video. After this stage, conducted by students, teachers review the contents, identified by the system via the GRAPH APIFootnote 2, as shown in Fig. 1.

This data collection (video, user name and school) serves two systems: Advanced and Collaborative Visualization System via Paraíba State Map and the Support System for Lectures, also called as a Custom Video Generator.

4.1 Advanced and Collaborative Visualization System

The contents are visualized in the Collaborative Map (Fig. 3), which can be accessed on the web or viewed on the Interactive Table in the Museum. On the web, a video gallery presents the students’ productions and information about which schools are producing content. Already in the Museum, a touch-sensitive screen contains only the map. In it the teacher defines videos to be presented and chooses a photo of the school. At the end, the system generates a personalized video, opening with images from the school, followed by the students’ videos and respective credits. The video can be shared with a QR Code, the last image of the video.

Fig. 3.
figure 3

Collaborative map

Full size image

4.2 The Custom Video Generator

TCE-PB receives visits from high school students to know its physical structure and carried out social actions. One strategy to integrate these visitors in themes such as social control and fiscal education, a system of lecture support was developed to integrate institutional audiovisual products into the collaborative production of students, with the purpose of personalizing the experience of each class that visits the TCE-PB.

Applying the Audiovisual Design architecture in the development of this system, we have the following proposal: from an institutional video, which defines the Content line and conceptualizes people’s importance in supervision and monitoring public matters, videos produced by students are inserted as integral elements of the narrative (Line of Identity and Motivation), according to Fig. 2.

The personalized video, generated based on students’ data has the following structure (Fig. 4):

Fig. 4.
figure 4

Abstraction of narrative personalization.

Full size image

Part 1: Contextualization on aspects of mismanagement and corruption, and how popular participation is important in combating and monitoring failures in public management. Following the AD architecture, in this stage of video narrative resources (rhythm of editing, soundtrack, forms of expression, among others) have to be coherent with students spectating in such a way to stimulate the attention and interest about the subject.

Part 2: Examples of social control attitudes based on student videos. Following the architecture of the AD, the use of students’ videos acts as a Trigger and stimulus to attention and formation of Identity and affinity with the content.

Part 3: Experts explain students acts represent social control, a fundamental role in whole society. Following the architecture of the AD, this stage of the video uses the principles of the Content and Identity Line, since the participation of the students is handled at the content level, as a partner of the TCE-PB.

Part 4: Movie credits with names of participants (experts, school and students). This step is also a Trigger, it acts as a stimulus to attention and formation of Identity and affinity with the content.

Part 5: Providing QR code for video sharing. Following the architecture of the AD, this step acts as an Action Trigger, forming the Identity to stimulate the student (Audience) to switch the role to Synthesizer, by sharing the material in their own social media.

Other Designing Lines can be impacted by the system are Motivation and Experience. Since this personalized material from individuals’ data is made available in a file format compatible with social media (Youtube, Facebook or WhatsApp) it is possible to enable scenarios in which students can modify, edit or even remix the content.

To guarantee a scenario of enjoyment and interaction, the following functional requirements for the system were delineated: stablish connection with Facebook via API Graph; Incorporate hashtag, video, student name and school from a public post on Facebook; Insert imported data into the video project “Social Control”; Render all material (fixed institutional videos, customizable and data) to generate a single video file compatible with HTML5 and services like Youtube and Vimeo.

5 Analyzing Proposed Interactions

To detail the strategies of interaction and engagement, a system usage scenario based on a persona student named Wellyngton is described below. Use scenarios are verbal narratives objectively situated, with contextual details, aiming to characterize the context of interaction and define relationships of individual with technology. They are useful in design to visualize uses of the system in development. Personas are archetypes that represent a set of users for whom the design is done [16, 17]. The focus of the scenario lies in students because it is through them the general objective of the project, related to citizen and fiscal awareness, will be reached.

5.1 Persona Student

Wellyngton studies in second year of high school, has lessons weekly about History, owns a prepaid internet access smartphone he uses while the daily data limit lasts. In addition, he connects to the internet using wi-fi from school, where he spends the day. Wellyngton has accounts in several social networks, but mainly uses Instagram, Youtube and Facebook. He thinks all politicians are corrupt, but is intrigued because the history teacher, Cida, spoke during a class it is everyone’s responsibility to monitor and control public spending, including those at school.

5.2 Use Scenario

In School: During the class History teacher Cida presents the Portal Aluno CidadãoFootnote 3 (ToA 1), which has a Map of State of Paraíba with municipalities and schools. Wellyngton is very interested in videos produced by students from other schools about public management, enforcement and denunciations about unfinished public works. When teacher finishes showing videos, he asks if their class also wants to produce and include videos in the Portal. Cida then shows a movie about how to record and share videos on Facebook (ToA 2), which will then be included into Aluno Cidadão Portal.

Wellyngton creates a group with his friends and records two videos, one on an unfinished bridge on the way home and the other on abandoned ground next to the school where people throw garbage. They share the videos on Facebook, with hashtags describing the content and the name of the school (ToA 3).

In the next class, teacher opens the Aluno Cidadão Portal, clicks on the Alagoa Grande municipality, in the name of the school, Josué Gomes da Silveira, and all videos procied by the class are listed (ToI 1). The class attends and discusses the origin of the problems, responsibilities and how to solve them. To compare, the teacher shows videos of students from Padre Hildo Bandeira State School, from the neighborhood, who also recorded a video on the unfinished bridge of Wellyngton’s group.

Scenario of Use Visiting the Interactive Museum

The school receives an invitation to visit the TCE-PB facilities, through the School and Citizenship Project. Wellyngton is elated to know TCE, which he has only heard about on television. At TCE students are received with snacks and juices and invited to enter the auditorium (ToI 2). A lecture begins on recycling garbage, with a TCE professional speech. During the lecture, video clips are shown, among them those produced by school students, such as Wellyngton’s, relevant to the subject (ToA 4) and incorporated into the presentation by the speaker.

At the end of the lecture a QR Code appears on the board (ToA 5). The speaker explains that by photographing it students have access to the whole video, can copy it or share it on social networks. Wellyngton immediately photographs the QR Code and shares it in social media, commenting that his video has appeared in a TCE event.

The speaker then invites Wellyngton’s class to visit the TCE Interactive Museum, where Cida begins by showing the games, virtual reality, the coworking space, and a theater to watch movies. The class goes close to a touch-sensitive monitor, which contains the same Aluno Cidadão Portal Map (ToA 6). The teacher again selects the municipality and the school. But this time does not appear the list of all the videos produced by the class: the best videos act as “symbolic affordance” (ToA 6) and are assembled in a unitary workpiece. Cida elucidates she chose the videos about public works and explains the responsibility on roads and bridges can be as much of the city hall as of the state and federal governments. She selects “play videos” (ToI 3), which now start with a photo and name of the school. When the video ends, teacher’s name and the students who recorded the videos also appear on a black background, such as on TV. A QR Code offers the option to share the video (ToA 7). Wellyngton leaves TCE-PB talking to his colleagues about themes of next videos they will record.

6 Trigger Definition and Analysis

Methodologically, according to the Audiovisual Design framework, once established the objective to be reached with the content, the Designing Lines are defined, according to an utility perception of each role. Considering purpose and utility, media affordances are developed, resulting in triggers to promote notions of system quality, thus guaranteeing good experiences of individuals. Each Designing Line provides tools to achieve the individual’s goals.

Every change of role begins with the perception of a media affordance and is performed using triggers, which may correspond to action or to inertia. In the case of Aluno Cidadão, there are changes of roles proposed by the system, and the persona Student will alternate among Audience, Synthesizer and Producer. Modifier actions are focused on teachers and speakers. Table 1 describes the Triggers of Action (ToA) and Inertia (ToI) designed to change roles using the four Designing Lines: Identity (LI), Motivation (LM), Experience (LE) and Content (LC).

Table 1. Triggers for Aluno Cidadão.
Full size table

Considering the purpose of this project, specifically raising awareness about social control and involving society in actions to combat corruption, individuals together play a key role. It is considered that high school students have a central role in medium and long term in this process. The main Designing Line that guides audiovisual enjoyment is Identity. Allied to the Content Line, this Line represents the contact of the individual with the premise of awareness. The stimulus to share is materialized through the Motivation Line, focused on the role played by Synthesizers.

From the theoretical point of view, the Audience, when creating the identity with the theme, is encouraged through Triggers of Action to change to the role of Producer. In this role, production and sharing take place (remembering by describing the DA in the form of sets, the Producer has the abilities of all antecedent roles). By enjoying the videos in the classroom (Trigger of Inertia), the student, who acted as a Producer, returns to the role of Audience. It is important to consider the change from the role of Audience to Producer goes beyond engagement through identification. The stimulus to active participation is part of life experience, which, as a citizen, leads to produce contents relevant to this context. Likewise, we cannot ignore that Identity Line will also be fundamental for students who do not produce, for, identified as citizens, to be motivated to at least watch the videos.

At this point, the teacher is a mediator between objectives of the system (defined by TCE-PB) and Audience, serving as a trigger, both of action in the first moment and of inertia in the second. Already when the teacher moderates videos that compose the Portal and creates the playlist of personalized videos in the Museum, he himself acts as a Modifier. In the case of moderation, Modification happens at a macro level, since the teacher changes the whole system by selecting which videos compose the Map; in the Museum this Modification happens at a micro level, generating a video with data of the students.

A similar relationship can be established when the Audience watches the contents in the Auditorium and in the Museum. In both cases, a modified content is offered to the Audience with a trigger to share at the end (Synthesizer), represented by the QR Code. By photographing the QR Code and sharing the video, the student moves from Audience to Synthesizer.

The media affordances prevail in defining triggers of this system are symbolic. Calls for action or for inertia are part of the teacher-student relationship within the classroom and in the Museum, as well as speaker and audience, in the auditorium. In addition, there are elements of awareness, central objective of the project, that lead to the action of the Audience. This process of awareness takes place at psychological and motivational levels, where perception of a possible agenda for the video can also become a Trigger of Action (as is the case of the unfinished bridge of the use scenario). The QR Codes, used in two moments, correspond to visual media affordances, centered on the video graphical interface.

7 Conclusions

This article describes how the methodological process of Audiovisual Design was used in the production process of customized audiovisual contents for an Interactive Museum. The development of Interaction Triggers was described, based on the museum’s objectives and the profile of the visiting public. Each role described in the Audiovisual Design model has associated affordances for Triggers of Action (ToA) or of Inertia (ToI). Role change is a central part in the production of content for complex audiovisual systems.

A methodological course was designed starting from the objective to be reached with the content, considering different possibility of fruition of the individual. From the objective relevant Design Lines were defined, aiming to awake the notion of system utility (audiovisual content, interfaces and interaction). From this logical construction, relevant medium affordances have been defined, which lead to Triggers. All these steps should result in system quality, central element of usability for the satisfaction of individuals.

The research on the construction of generative audiovisual systems is still undergoing. The next parts are user tests for validation of usability and engagement. As future activities, this research has two actions. The first one, theoretical and conceptual, is related to an extended description of the relationship between tools needed in each Design Line with development of media affordances and triggers. The second action consists of evaluations of software development quality and usability tests with students, teachers and employees of TCE system users. In this way, we intend to validate the development described in this article, specially aspects related to the Motivation Line, with development of engagement.