Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

11.1 Introduction

The spectacular advances in the field of technology in recent years have led to new technological scenarios, among which is Ubiquitous Computing. According to Weiser [1], technology should not be explicitly shown to users, that is, it is not at the sight of the user but offers services to him in an implicit way. Among these scenarios are those denominated MDE (Multi-Devices Environment), which include digital objects and multiple devices working jointly to offer the user a specific service. In these new environments a different kind of interfaces is required, such as DUI (Distributed User Interfaces). According to Niklas Elmqvist in [2], DUI can be defined as a user interface on which its components can be distributed through one or more dimensions. These dimensions are input, output, platform space and time. Interfaces distribution in objects allows us new tangible interaction mechanisms. The term tangible user interfaces TUIs refers to user interfaces which give physical form to digital information, making the parts directly malleable and perceptible [3]. However, the following challenges should be considered: the user needs with respect to interfaces distribution in the space and in different devices in this type of environments represent a major challenge and should be performed in such a way that it is obvious for the user; some other factors, as simplicity or the learning process should also be considered. The design of new systems should not force users to learn new abilities but should perform tasks in a clear way for users, allowing them to focus on their activity by using simple interaction mechanisms.

People with cognitive disabilities usually have learning impairment and present difficulties to perform daily tasks. In order to improve their insertion in society and develop their skills they need to perform cognitive stimulation tasks. Technology is a useful tool which can offer benefits to perform those tasks. More specifically, games are activities which allow users to improve their skills and learning capacities in a fun and entertaining way. This article describes and analyzes the most important points to develop leisure MDE such as, the system architecture, the devices used, the resources available and the design of tangible distributed user interfaces which allow new interaction mechanisms. The prototype developed is called TraInAb (Training Intellectual abilities). It is a game based on collaborative environments and new technologies as: NFC and mobile devices to stimulate people with cognitive disabilities. Finally, we expose of conclusions and future work.

11.2 Related Works

Until recently, our idea about the computation was a computer where we can interact through a screen with a keyboard and a mouse. This situation is dramatically changing. Computation is being inserted into any object and device previously unthinkable. We are witnessing the integration of new environments, also called multi-device environment (MDE). These scenarios consist of multiple, heterogeneous devices distributed in the environment along with screens and other surfaces where the user interfaces can be executed. Some examples of such environments are : i-Land, [4] is an interactive system for facilitating collaboration between users through devices such as Dynawall, which is an interactive electronic wall; Coomchairs are chairs that enable computing and Interactable, an interactive table that allows interaction using touchscreen technology. WallShare [5] is a collaborative system that allows to distribute the interfaces among different devices such as mobile phones, PDAs, laptops, etc. In addition, it includes an open space to be displayed through a projector on a surface such as a wall. E-conic [13] is an application that supports multiple devices sharing information with one another. WeSpace [6] is a collaborative work space that integrates a large data wall with a multi user multi touch table, thus allowing groups to explore and visualize data. These new scenarios offer multiple advantages over computers. However, it is necessary to distribute information into different and heterogeneous devices. For this reason, the design of Distributed User Interfaces (DUI) must be taken into account. Distributed user interfaces can be displayed on different devices: phones, computers, screens, objects, etc. The interfaces that are distributed in objects are called Tangible User Interfaces (TUI) [3]. These are physical objects used as representations and controls for digital information.

There is software focused on improving cognitive abilities, specifically video games. A video game is a software programme created for entertainment and learning purposes in general. It is based on the interaction between one or more persons and an electronic device that executes the game. It is not easy to determine which game is more adequate for cognitive disability players. The barriers that people may find during the activities are complex and varied as described in the works in [7] and [8]. These studies highlight that the key element in the games must be simplicity.

“Serious Games” are games that simulate real situations for people with disabilities, such as shopping in the supermarket. The main objective is to develop the skills that can help them in their daily activities [9]. On the other hand, in Virtual Reality software using helmets, gloves and other simulators, the user may feel more immersed in the game, and it is very engaging and motivating, but the problem is the high cost of devices, and the difficulty in the use of certain devices, In addition a person is required to control the players and devices [10, 11].

The advantages offered by these systems are numerous. They enhance positive attitudes in users while being appealing and encouraging, and providing information quickly. However the system presents the following disadvantages:

  • The user needs a minimum knowledge about computer use. Not everybody can use a computer and some devices, like the mouse or the keyboard are not intuitive for people with cognitive disabilities.

  • The system requires highly specialized hardware/software which can be expensive (simulators, virtual reality).

  • In some games, impaired users may have difficulties finding specific information.

In this paper we propose a system MDE based on collaborative games to stimulate cognitive abilities. The interaction is very simple, is conducted through common objects. The system is developed with Web, NFC and mobiles technologies. In order to develop a usable and intuitive system to people with limitations, we focus has been on the distribution of interfaces in a way effective in the MDE.

11.3 Interactive Triangle

In order to design and develop an MDE scenario we considered an interactive triangle (Fig. 11.1). This is based on the following factors: Users and tasks to be performed, new technologies and devices available in an MDE scenario and tangible distributed user interfaces as an intermediary between users solutions.

Fig. 11.1
figure 1

Interactive triangle

Full size image

11.3.1 Multi-devices Environment

MDE refers to the devices and the communication among them. In the design we have to keep in mind that all available devices should be easy to use. For example the following devices: Laptop, Smartphone, Kinect, Wii, Tablet, and Projector.

11.3.2 Architecture System

The architecture is client–server mode. It allows any type of device to communicate with others through NFC and Web technologies. Tangible user interfaces incorporate an NFC tag that has written a web address that identifies the object. When the interface approaches the NFC reader (built inside the mobile device) it reads the tag information and executes the corresponding mode on the server.

The server is responsible for interpreting the data sent by the mobile device and simultaneously executing the required action in the other interfaces.

In this type of scenario the server is the main component responsible for communication among devices through an access point. This component is also responsible for the control logic, i.e. contains all services and tools necessary for the rest of devices which make up the system.

11.3.3 DUIs and TUIs

DUIs and TUIs are the links among MDE environments, the architecture and the user together with the task (the latter is an implicit factor that will be considered when designing and developing the system). The combination of this type of interfaces offers the following interaction mechanisms and distribution of information:

11.3.4 Interaction Techniques

Two different tangible interaction techniques can be found in this type of MDE environments (see Fig. 11.2):

  • Using physical objects. To interact with the system the user must simply approach the object or tangible interface to the mobile device which incorporates the NFC reader and will send the information from the tangible interfaces to the MDE system.

  • Using the mobile device: The interaction is opposite to the previous case, the user must approach the mobile device to the tangible interfaces which are located in a specific place.

    Fig. 11.2
    figure 2

    Tangible interaction. (a) Using objects. (b) Using the mobile device

    Full size image

11.3.5 Distribution Based on Mind Model

To distribute the user interfaces in the environment we have used cognitive factors as our model. And since the users’ mind models work in a distributed way, we have taken advantage of this fact to design systems which are similar to the users’ working style in their natural environment.

The mind model is a concept borrowed from psychology to explain the mechanism used by users when interacting with the real world. The users’ cognitive system is distributed and is made up of three storage areas: Sensory memory, working or short-term memory and long-term memory.

  • Sensory memory: It captures and interprets the information through the senses, eyes ears.

  • Short-term memory: It is a short-term retention store, but more importantly, it is responsible for the information encoding processes as well as for information retrieval, since the information from the LTM (long-term memory) is activated here.

  • Long-term memory: This memory system can retain information permanently and has almost unlimited capacity. The information is stored unconsciously and comes to consciousness when it is retrieved from that store. The repetition of the task is necessary so that the information can be permanently stored.

In conjunction the three types of memory work in the following manner [12]: The long-term memory is associated with the part of the user interface defined as tangible interface. This part provides physical objects, they are easy to remember and use and allow the user to interact with the system. In this way, the process that would follow the user’s mind to store the user interface on the long-term memory is as follows: firstly the user interface is visualized using the sensory memory, then it is displayed again and the image is repeated in the mind by using the short-term memory and finally when the stored image is understood, it is saved in the long-term memory.

11.3.6 Interface Distribution

The interface can be distributed in three different components similar to the user’s cognitive models. Each interface will have a different role in the system depending on the information and the task to be displayed.

  • Main Interface. This interface will use the sensory memory more often It shows the main information to be analyzed by the brain. It is identified as the main working space of applications. The user has to concentrate and pay attention but does not need to store it in memory.

  • Intermediate interface: It links the main interface and the primary interface. This information is necessary but the user does not need to remember and store it in the memory, for this reason it is associated to short-term memory.

  • Primary or tangible interface: These interfaces are in continuous use and after repetition the information is saved in the long-term memory, so the user implicitly and unconsciously uses it. They are similar to shortcuts in applications.

There are some considerations to preserve the usability in MDE systems:

  1. 1.

    Visibility of the status of the system. The system must keep users informed on the status of the system. In system which support DUIs the information is distributed in the space, and for this reason we might ask, how can we make the status of the system visible? Which device should show that so the user is not distracted? Most devices can show information in different ways, using voice, text, images, and animation. Any communication means can be used taking into account the device and its screen size.

    • Small screen devices. Images and audio will be used.

    • Large or regular screen devices. Text, image or animation are used. It is advisable that just one device emits the sound so that the user is not distracted.

  2. 2.

    It is necessary to use the user language. That is, using text and tangible interfaces based on common physical objects.

  3. 3.

    Freedom and control for the user. The user must control the system. Navigating in the interfaces must be convenient. More complex tasks must be implicitly performed, making the user believe he is working directly with the objects.

  4. 4.

    Minimizing the user memory load. The user should not memorize the action information; short-term memory load should be reduced. Keeping objects, actions and options visible is better than memorizing. Concerning physical handling, the information from the spatial disposition of the objects allows users to register their position, releasing visual attention to other objects. In this way the requirements of viso-spatial memory are reduced, thus assisting information retrieval and memorization.

  5. 5.

    Offering shortcuts (quick access ways to system functions) using objects, represented with metaphors related to reality.

  6. 6.

    Providing visual and additive guidelines to allow the user know his location and actions. In this way the user can also be guided while using the system. When it is necessary to pay attention to a specific interface, this is communicated using sounds, images or text.

11.3.7 Task Based Distribution

The user interfaces are distributed based on the tasks to be performed. There are three different types:

  • Collaborative tasks. When the task is collaborative one of its functions is displaying images, text or sound so that all users can see it and be coordinated. The main interface must be distributed in a device larger than 17 in. For example, wall projections which allow perfect visualization of the interface.

  • Individual Tasks. The distribution of individual tasks will depend on their difficulty. If the task is easy, it is enough to distribute the interfaces into the tangible objects and the mobile device. Otherwise it should be distributed into a larger one.

  • Implicit tasks. These are performed with no awareness from the user, such as internal communication among devices and the Web service.

11.4 TraInAb System

TraInAb (Training Intellectual Abilities) is an interactive and collaborative game designed to stimulate people with intellectual disabilities.

It integrates a new form of human-computer interaction. The user can interact with the system through everyday objects such as cards, toys, coins…

The collaborative system is based on the distribution of interfaces and device mobility; it offers the possibility to be used individually or by multiple users.

The functionality of the system is as follows. In the main game an interface is projected on the wall. Users with physical interfaces, i.e., the objects that integrate NFC tags, can interact with the main interface; this requires the mobile device that incorporates the NFC reader to interact with the main interface and this is necessary to bring objects to the mobile device. For example, if in the game an object must be associated with another, the user only has to bring the corresponding object closer to the mobile device, and then the system recognizes it and displays the outcome of the game (Fig. 11.3).

Fig. 11.3
figure 3

Session of cognitive-impaired users using the TrainInAb system

Full size image

11.4.1 Design Through Interactive Triangle

In order to design and develop the system we have considered the three vertices as explained in the previous section.

11.4.2 MDE Settings

Multiple devices are networked in an MDE. In this system we have used the following:

  • Smartphone: it is used to interact with the system, because of the small screen size it has just been used as an interaction device, the relevance of this device is that it offers a more natural tangible interaction style which is easy to use any size tablet could also be used, being the only requirement to incorporate NFC technology.

  • Laptop: It is responsible for displaying the main game interface. It has been chosen due to its computing power that allows us to execute quality graphics and multi-modal factor as well as offering sound, text, and graphics.

  • Projector. This device expands the size of the main interface easily. It also allows multiple players to play at the same time

  • Resources interaction, i.e. common objects that facilitate interaction with the system.

  • Tangible Menu, which has been used by therapist, parents, teachers, etc. In this case the interface allows them to control the game remotely.

11.4.3 Interface Distribution

Interface distribution in the environment was as follows:

  • Main Interface. A projector was chosen as it allows better visualization and improves collaboration among users.

  • Intermediate Interface. A mobile device is shown. It is only used as a communication device between tangibles interfaces and the main interface. Whenever the user approaches a tangible object, the mobile device emits a sound informing of the correct recognition of the object.

  • Primary or Tangible Interface. Some tangible interfaces have been designed based on physical objects. They are very easy to use and be assimilated by users. In order to facilitate the use of the system to therapists and parents, tangible menus have been chosen so that they can control the system remotely, change the game, refresh data, exit the game, etc. (Fig. 11.4).

    Fig. 11.4
    figure 4

    Menu which allow user to change game, go back, refresh and exit the system

    Full size image

11.4.4 Advantages and Disadvantages

The advantages of this distribution system are: The system can have a private interface (tangible interfaces, each user has their own) so this makes users more confident. Moreover, it provides a shared interface.

  • Tangible interaction is more natural for users. They only have to bring the objects closer to the mobile device. In this way we provide flexibility in the space, you can work in the same room and o remotely interact with the system.

  • It offers the possibility for multiple users to interact simultaneously, thus facilitating participation.

  • In a collaborative environment we can distribute and maintain collaborative interface users’ private spaces with their own device. Considering Streng study [14] and the importance of working with individual space, our conclusion is that users are more confident when interacting and working with the system.

  • The distribution of user interfaces in the environment allows us to simulate the way people usually work. In order to improve human-computer interaction in multi-device environments that support DUIs it is necessary to know and take into account how mental models and the cognitive system of users work.

  • Direct interaction with the objects provides a better understanding of the task. The tangible interfaces emphasize the connection between the body and the cognitive process, thus facilitating thinking through physical actions.

The disadvantage is the scalability of tangible interfaces. These are stationary and designed to engage five users. One of the future works would be to allow users to edit the games and tangible interfaces easily.

11.5 Conclusions and Future Work

A challenge in realizing the benefits of multi-device environments (MDEs) is developing interaction mechanisms which allow use the system easily. In this paper, we described the interactive triangle. It shows the most important points to consider in a MDE system, such as architecture, devices and Distributed and Tangibles User Interfaces as a way attractive and intuitive to interact with the system. In order to test the benefits of this type of systems MDE we developed an application called TraInAb (Training Intellectual Abilities). This is an interactive and collaborative game designed to stimulate people with cognitive disabilities. It integrates a new form of human-computer interaction. The user can interact with the system through everyday objects such as cards, toys, coins. This style of interaction is simple and intuitive; its purpose is to eliminate the technological barrier for people with cognitive disabilities helping them to improve their skills in a funny way.