Keywords

1 Introduction

Internet of Things (IoT) is enabling Smart Cites and supports data ingestion and digestion [1]. There is not a single universal IoT user interface that can support any possible use case or application scenario. For example, for the driver behind the wheel a voice and/or graphic interface is preferable, for the user in a public place a graphical or textual interface is more suitable. User interface, which provides simultaneously several means of communication: speech, gestures, text, etc., so that the user could choose from useful and convenient tools is called heterogeneous (multimodal) interface [2]. Heterogeneous interface for Smart City applications should also have mechanisms of automatic selection of an appropriate modality or combination of modalities.

In this paper we address the challenge of context-driven and situation-aware modality selection mechanism. We propose a method of Context-driven Heterogeneous Interface Selection for Smart City Applications. The proposed method is based on the concept that the intelligent system must be context-aware and situation-aware. Within the framework of the research, graphics, text, voice, audio, video modalities are conceded.

Validation and implementation of the proposed approach has been carried out as part of Solid Waste Management project, which in turn is a use case of bIoTope project [3]. Efficient waste management is an essential part of Smart City [4], which requires a complex multi-criteria approach [5]. The bIoTope project builds open IoT ecosystems on the basis of innovative IoT technologies and use cases. The project is funded under the EU Horizon 2020 Program. The system described in this paper is part of St. Petersburg pilot of bIoTope project on Smart Waste Management.

The rest of the paper is organized as follows. Section 2 presents Background and Related Work. Section 3 includes description of method of context-driven heterogeneous interface selection. Section 4 describes Implementation of proposed method. Section 5 describes Sample Scenarios and tests carried out at this stage. Section 6 concludes the paper.

2 Background and Related Work

2.1 Context and Situation

In this paper the context is understood as “any information that can be used to characterize situation of an entity” [6]. We define contextual information as ambient temperature, illumination, noise level, type and characteristics of the user’s device, etc. A context-aware system is a system, which “uses context to provide relevant information and/or services to the user, where relevancy depends on the user’s task” [6]. We consider a situation as context abstraction and generalization. The following situations are considered in the article: user is behind the wheel, user walks down the street, user is in a public place, etc.

2.2 Related Work

The article [7] describes a heterogeneous interface consisting of a graphical and a voice interface. The task of the interface is to simplify the work of doctors. This system is suitable only for medical specifics and uses only a mobile phone. The article [8] proposes the concept of an interactive system for people with disabilities, equipped with an adaptive context-sensitive user interface. The basic idea is to use a multi-level context processing. The main disadvantage is the absence of implication of the proposed idea. In [9], multi-level data processing is described. The algorithm is based on the use of contextual models and weights. The system can be easily adapted using three context models: user, device and environment. However, the proposed approach assumes that the developer must manually fill in the coefficients. Also, the system is suitable only for a small range of simple scenarios due to its architecture.

3 Method of Context-Driven Heterogeneous Interface Selection

Within the Solid Waste Management system, visual and multimedia modalities are necessary for users. Visual modalities include graphics (sending photo, graphical interface) and text; multimedia modalities include audio and video. Each modality has the following uses: input (receiving information from the user) and output (the user receives information from system). Heterogeneous interface can include a single modality or a combination of modalities (complex modality).

We developed a solid waste management (SWM) domain ontology, which is available online [10] for semantic manipulations over entities used in algorithms described further in this paper.

3.1 Appropriate Modality Selection Algorithm

To select a suitable modality or combination of modalities, a multi-step analysis of context is used. The main steps of the algorithm are shown in Fig. 1.

Fig. 1.
figure 1

Appropriate modality selection algorithm

Full size image

Algorithm contains 4 main parts. At the first stage, we need to personalize user interface. In our case, this means choosing a role. The following user roles were defined: citizen, fleet dispatcher of waste management company, driver, city administration, and janitor. Next step is detailing the situation. Situations determine the environment around the user, which can significantly affect the decision in the choice of interface. Within the framework of the Solid Waste Management use case, 6 main situations were identified:

  1. 1.

    user is in the building

  2. 2.

    user is at home

  3. 3.

    user is behind the wheel

  4. 4.

    user walks down the street

  5. 5.

    user is in a public place

  6. 6.

    user has his hands full.

For each situation, a set of context parameters defining this situation was proposed. To illustrate the algorithm, we use context matrixes. In total, we have three context matrices (T): the matrix of roles (R), the situation matrix (S) and the base matrix (B) that contains CR, CS and CB context rules respectively.

For example, the roles matrix includes rules for define user level like advanced or beginner: AdvancedIntelegence and NoobIntelegence; situation matrix includes rules NightEnviroment, NoiseEnviroment, LightnessEnviroment. A base matrix is a matrix that contains all context rules that do not contain a role matrix or a situation matrix. For example, SmarphoneType, CPUDevice.

$$ T_{m} = \left[ {\begin{array}{*{20}c} {w_{11}^{m} } & \cdots & {w_{1i}^{m} } \\ \vdots & \ddots & \vdots \\ {w_{c1}^{m} } & \cdots & {w_{ci}^{m} } \\ \end{array} } \right] $$

Each row of the context matrix (\( T_{m} \)) describes how context rule (c) influences modality (i) for selection decision:

$$ {\text{influence}} = \left\{ {\begin{array}{*{20}c} {w > 0,} & {preferred\,for\,this\,context\,rule} \\ {w < 0,} & {using\,is\,undesirable} \\ {w = 0,} & {does\,not\,affect} \\ \end{array} } \right. $$

Active context rule is a rule that is activated by context. For example, we could make a rule that works only if the weather is rainy.

In the summation phase, we must select and summarize active context rules (c) for every modality (i) from the context matrices. After that we will calculate the sum of all coefficients for every modality (\( MSUM^{i} \)).

$$ ISUM^{i} = \mathop \sum \limits_{c = 0}^{{C^{R} }} R_{i} + \mathop \sum \limits_{c = 0}^{{C^{S} }} S_{i} + \mathop \sum \limits_{c = 0}^{{C^{B} }} B_{i} $$

The third stage represents the sum of coefficients of modalities (\( ISUM^{i} \)) according to templates (t). A template is a predefined set of modalities (I) that could be useful together. For example, using text and graphics the best choice for PC in most types of information. Also, we have to separate the types in accordance with the direction (d) of interaction: input and output, because the interaction for reception and transmission can differ among themselves.

$$ TSUM_{d}^{t} = \mathop \sum \limits_{i = 0}^{{I_{d} }} ISUM^{i} $$

At the 4th stage, we must select the template that has the maximum sum (\( TSUM_{d}^{t} \)) and modalities (i) in this template (t) will be the best choice for interaction with user (\( W_{d} \)).

$$ W_{d} = MAX\left( {TSUM_{d} } \right) $$

Pseudocode for appropriate modality selection algorithm:

figure a

4 Implementation

Implementation of the proposed method was carried out on the basis of ITMO SWM project [11]. ITMO SWM project is part of St. Petersburg pilot of bIoTope project, which aims to develop The Smart Waste Management System (SWMS). SWMS includes: Central Processing, Web Portal, Mobile applications, communication with IoT (means for collecting and processing data from SGB sensors and sensors installed on waste trucks), the model for IoT data sharing, dynamic route optimization in real time, security and privacy policies, and dynamic route optimization in real time. Web Portal supports three modes of operation: dispatchers of waste collection companies, citizens, and city administration; Mobile applications support two user roles: janitor and driver of waste truck.

The source language for this development is Python. CMS Plone and Zope built-in transactional object database (ZODB) are used for developing web applications. The development framework of Mobile applications is Kivy.

We extended and qualitatively enhanced the functionality of the ITMO SWM described in [11].

Web applications have the following functionality:

  • City dashboard (visualization and presentation of current situation with the process of waste collection);

  • Various reports on the quality of the collection of solid waste (tools for analyzing the volume and quality of work performed by drivers);

  • Driving quality control;

  • Traffic jams and road works;

  • Complaints & Reviews (mechanism to leave or review a complaint about the quality of waste collection);

  • Automated scheduling of vehicles.

Mobile applications have the following functionality:

  • display the situation on the map, show the optimal route accordingly the cloud systems data;

  • send messages about inability to pick up a garbage bin or other problems encountered, including text, photo, video and voice messages;

  • display messages, including text, photo, video and voice messages.

Graphics modality (graphical interface) is represented as (1) City Dashboard, (2) various reports, (3) Traffic jams and road works visualization, (4) Complaints & Reviews. Graphics (sending photo), text and video modalities are implemented using the Messages Module of Mobile application. Audio modality (speech recognition and synthesis) is available for Mobile applications and Web Portal for all users. The implementation of audio modality is based on the algorithm described in [12].

Heterogeneous interface selection module is implemented in Python. The architecture of the developed web application is shown in Fig. 2. Sample scenarios and tests of this system are described in more details in the following section.

Fig. 2.
figure 2

Architecture

Full size image

5 Sample Scenarios and Testing

Sample Scenario 1. The driver of the waste truck drives at night and requests a route to the garbage bin using smartphone or tablet.

The admins interface is shown in Fig. 3.

Fig. 3.
figure 3

Admins interface

Full size image

The driver interface is shown in Fig. 4.

Fig. 4.
figure 4

Driver’s Mobile Application interface on devices Samsung Galaxy S7, Sony Xperia Z3 Tablet Compact и iPad Pro

Full size image

Sample Scenario 2. The citizen is in office (public place) and wants to complain about the quality of garbage collection using PC. The noise level is high.

The admins interface is shown in Fig. 5.

Fig. 5.
figure 5

Admins interface

Full size image

The citizens interface is shown in Fig. 6.

Fig. 6.
figure 6

Citizens interface

Full size image

6 Conclusion

We developed and demonstrated method of Context-driven Heterogeneous Interface Selection for Smart City Applications. It implements automatic selection of an appropriate modality or combination of modalities based on context-driven and situation-aware modality selection mechanism. Several sample scenarios of the functioning of the system are described and tested. It is shown that heterogeneous interfaces allow IoT information system provide better service to its users.

In future work, we will refine the developed method by including larger number of situations and context rules.