Keywords

1 Introduction

The service-oriented architecture (SOA) is considered to be a key concept in the Internet of Services (IoS), which enables service providers to deploy data, software and business processes as Web services or cloud services on the Internet [34]. Infrastructures based on SOA help service users create various real-world applications easily by invoking and composing available cloud services. A typical example of SOA usage is the Language Grid; it supports the sharing and creation of various types of language services on the Internet by encouraging service users to develop customized multi-language communication tools and intercultural collaboration environments [16]. On the other hand, the emergence of Internet of Things (IoT) in recent years has enabled things to be connected on the Internet. Things in the IoT environments are monitored and controlled by sensors and actuators [41]. The information collected by the IoT devices can be analyzed and used for making decisions as well as triggering cloud services instantiated on the IoS. IoS-based cloud services can also trigger sensors and actuators in IoT environments.

To enable the easy development and deployment of application systems that employ various cloud services on the Internet and things in the physical world, it is necessary to build an integrated infrastructure to support interaction between them. There are, however, several barriers to the integration of IoS and IoT; key issues are extreme heterogeneity, ultra-large scale, and the dynamic nature [17]. First, the extreme heterogeneity of cloud services and things renders service composition based on the traditional SOA rather inefficient due to the issue of interoperability. We need to deal with the interoperability of the cloud services in the IoS infrastructure and things in the IoT environment. Second, the ultra-large scale of cloud services and things negatively impact the performance of any integrated architecture of IoS and IoT. Third, it is necessary to realize new mechanisms for service composition that can deal with dynamic changes in the availability of cloud services and things, as well as the user’s situation. Moreover, the application systems based on IoS and IoT, which can be regarded as Internet-based sociotechnical systems, always involve multiple stakeholders and so are difficult to design.

Since the integration of IoS and IoT involves a complex social process of service composition with important issues of interaction, dynamics, scalability, heterogeneity and decision making, it can be considered from a perspective of a multiagent system, which is composed of a set of agents that perform complicated tasks by negotiation and cooperation [47,48,49]. The multiagent perspective has already been used to investigate and interpret many existing systems such as sensor networks [4], social networks [19] and crowdsourcing systems [18], which will also be useful to explore effective methodologies for understanding behaviors of various stakeholders and modeling comprehensive sociotechnical systems built based on the integration of IoS and IoT.

In this paper, we propose an integrated architecture of IoS and IoT by using the Language Grid as an example of IoS infrastructure. In the framework, we mainly focus on the components for service composition based on cloud services on IoS and things on IoT. Then we describe the proposed framework from a multiagent perspective while emphasizing two specific aspects: architecture integration and application design. We discuss the possibility of developing the integrated architecture of IoS and IoT as a multiagent-based architecture and applying multiagent methodologies for designing sociotechnical systems based on the integrated architecture. In this sense, we are not aiming to propose a new agent framework for IoS/IoT integration in this paper; rather we try to understand important issues and key concepts of the integration from the viewpoint of multiagent systems.

The rest of this paper is organized as follows: In Sect. 2, we provide the background of IoS and IoT, and then introduce the issues for integrating IoS and IoT in Sect. 3. Section 4 describes a multiagent perspective for the integration of IoS and IoT from two different aspects. Section 5 provides an example application of multilingual environment design. Finally, the conclusion is presented in the last section.

2 Background of IoS and IoT

2.1 Internet of Services (IoS)

The Internet of Services (IoS) can be explained in different ways depending on the different definitions of services. In the research area of service-oriented computing (SOC), services are usually defined as “self-describing, platform-agnostic computational elements that support rapid, low-cost composition of distributed applications [35].”

Service-oriented architecture (SOA) is a key concept to realize the IoS [37], where software, data or business processes are deployed as Web services by service providers and used by service clients or service users. In SOA, there are basic services or atomic services that provide single functions of software or operations of data, as well as composite services composed by multiple atomic services to realize complicated functions or operations. Web services are evaluated by Quality of Services (QoS), the metrics of which include several non-functional attributes including cost, response time, availability, throughput, reputation and so on [50]. Therefore, QoS-aware service selection and service composition are essential technologies for IoS realization, which have been widely studied in the research area of SOC.

IoS has already been discussed a lot from the multiagent perspective [12, 43]. Service-oriented systems have been said to realize many of the ideas generated in the research of multiagent systems. The multiagent-based SOC research challenges proposed involve pervasive service environments, society-inspired systems, and computational service mechanisms [13]. Moreover, interaction protocols in multiagent systems were applied for SOA in previous research and realized as a commitment-based SOA [42].

Since IoS involves various services and stakeholders, it can be regarded as a sociotechnical system where interaction between people and technology frequently occurs. We adopted the sociotechnical approach [8, 40] to develop the Language Grid, an IoS infrastructure for language services [16]. The Language Grid is a multilingual service infrastructure for supporting multi-language communication and intercultural collaboration activities, and enables easy registration and sharing of various language resources such as online dictionaries, bilingual corpora, morphological analyzers, and machine translators [14, 27]. The concept of the Language Grid is to address the language resource interoperability by defining and implementing standardized language service interfaces for service composition and customization. The Language Grid was used in our previous studies on QoS-aware service composition and recommendation [10, 26], combination of human services and Web services [23], and policy-aware service execution [46].

2.2 Internet of Things (IoT)

The Internet of Things (IoT) fuses things with the Internet to enable easy integration of the physical world into cyberspaces [2]. IoT is regarded as a key infrastructure in various application domains including the smart home, smart city, smart factory, healthcare, transportation, agriculture and so on. Typical enabling technologies in IoT include sensors and actuators, which are used to monitor and control things in the real world. The information produced by things in the IoT environments can be processed by cloud services and used by various stakeholders for decision-making in their application domains. Therefore, the technologies key to enabling current IoT-based applications include collecting, processing and analyzing information from things, and making decisions from the processed information. Since the information from IoT devices is regarded as big data in many situations, advanced machine learning technologies are often applied for analyzing such data. Moreover, high performance computing environments are necessary to handle IoT big data for ensuring real-time processing in various application domains.

Since IoT is an emerging idea, its architecture, enabling technologies, protocols and standards are being widely discussed from different perspectives. Identification, sensing, communication, computation, service and semantics are seen as key IoT elements, while big data analytics, cloud computing, high performance computing and fog computing are regarded as supporting technologies for IoT in a previous study [2]. IoT was also studied from an SOC perspective, where the IoT architecture was described as consisting of sensing layer, networking layer, service layer and interface layer [9]. However, IoT faces many challenging issues related to Quality of Services (QoS) including availability, reliability, mobility, performance, scalability, security, management, and trust. Interoperability is another important issue considering the heterogeneity of IoT device types and specifications.

IoT has also been studied from a multiagent perspective, where an IoT-enabled application is regarded as a social process involving multiple autonomous parties making it possible to be realized as decentralized multiagent systems [41]. Moreover, decentralization, asynchrony and decoupled enactment, governance of security, accountability and privacy were summarized as several important elements of IoT. Further, possible research directions in decentralized MAS has been proposed including programming models, interaction-oriented software engineering and enlightened governance. Since there are a large amount of devices connected in the environment, massively multiagent systems could be a promising design paradigm for IoT.

3 Issues for Integrating IoS and IoT

To build Internet-based sociotechnical systems like the multilingual environments, we are focused on the technologies needed to compose cloud services for IoS environments. With the development of IoT infrastructure and the availability of various sensors and actuators, it has become possible to provide high-quality services to users by satisfying their requirements in various situations. For example, sensors can be used to detect users’ situational information like user interests and degree of satisfaction with a multilingual interactive agent. Based on this situational information, customized atomic services and composite services in the IoS can be provided to users in different situations. Therefore, it is important to consider integrating IoS and IoT for sociotechnical systems. However, there are several issues that we need to consider when integrating IoS and IoT. Previous research in the SOC field has studied how the SOA paradigm may be revisited to address the challenges posed by IoT, i.e. extreme heterogeneity, ultra-large scale, and dynamic changes [17]. These IoT challenges greatly impact the integration of IoS and IoT.

First, the extreme heterogeneity of IoS and IoT makes it difficult to form composites of services and things. For example, the streaming data collected by sensors in the IoT and the data of function-based cloud services in the IoS have totally different granularities, making it difficult to combine them. In the traditional SOA, service users can easily access and invoke Web services through standardized protocols or lightweight message transfer frameworks. However, the diversity of things and interaction styles between service users and providers in the IoT environment makes it difficult to apply the SOA approach as is. Therefore, it is necessary to consider middleware-level components for service access in IoT [32, 36]. That is, we need to deal with the issues by addressing how to handle the interoperability of the services in the IoS infrastructure and things in the IoT environment.

Second, the ultra-large scale of services and things poses a huge challenge to the performance of the integrated IoS/IoT environment. Therefore, we need to deal with the issue for ensuring real-time execution of services. In a previous study, we proposed a framework of parallel service execution in the IoS infrastructure considering the policies of service providers [46]. We note that the concepts of fog computing [5] and edge computing [39] were proposed to deal with this issue in the IoT environments by handling data processing and event processing in different layers of servers. However, more factors need to be considered for improving the performance of the integrated IoS/IoT environments, including mechanisms for parallel execution of rules and handling of the massive complex event processing.

Third, the dynamics of the integrated IoS/IoT environments has two aspects: dynamic changes in available services/things, and dynamic changes in user situation. Therefore, the important issue here is that the integration framework must ensure that service composition and recommendation can accept dynamic changes in environment. Although we have conducted several studies on dynamic service selection [26] and service recommendation [10] with the Language Grid on the IoS platform, it is necessary to extend these studies into integrated IoS/IoT environments.

4 A Multiagent Perspective on Integrated IoS/IoT

4.1 Integration of IoS and IoT

For the simplicity of explaining the integration of IoS and IoT, we use the Language Grid as an example of IoS infrastructure. We developed and then operated the Language Grid as an IoS infrastructure to support intercultural collaboration for more than ten years. The Language Grid is built on general service grid server software, and consists of five parts: service manager, service supervisor, grid composer, service database, and composite service container [28]. Based on the federated service grid architecture which seamlessly connects multiple service grids [29], we also realized the federated operation of the Language Grid in several Asian countries including Japan, Thailand, Indonesia and China. As of March 2019, 226 language services are being shared on the federated Language Grid. Moreover, an extended framework for service design was proposed with the Language Grid by bridging the gap between stakeholders involved in language service infrastructures and those who develop and operate multi-language systems [25].

The key concept of IoS infrastructures like the Language Grid is to standardize the service interface for each category of resource (data, software, business process, etc.), and enable flexible service management and service composition. In IoT environments, standardization and composition of things are also regarded as important fundamentals [3]. For example, OpenIoT is one of the typical initiatives to realize standards for IoT [44], while Web of things is another proposal for composing embedded devices on IoT [11]. Moreover, a SOA-based IoT architecture has been proposed to realize flexible service composition with trust management for IoT environments [7].

There are two aspects that we should consider in the integration of IoS and IoT: development of the integrated architecture and design of real-world applications based on the integrated architecture. The remaining parts of this section will discuss these two aspects from a multiagent perspective.

4.2 The Multiagent-Based Architecture

The first aspect is how to develop the integrated architecture by addressing the complications hindering service composition and the interactions between IoS and IoT. Here we consider some typical patterns of service composition and interaction in the integrated IoS/IoT environments.

  1. 1.

    Information of things in the IoT is collected and aggregated by sensors, and it triggers the invocation of atomic services or composition of atomic services in the IoS platform. For example, appropriate language supporting services are invoked based on the information collected and analyzed from an eyetracker that identifies a non-native speaker’s difficulties during a multilingual conversation [6].

  2. 2.

    Actuators in the IoT trigger invocation of atomic services or composition of atomic services in the IoS platform. For example, the lightening of different colors of IoT LED or push of different buttons triggers the invocation of machine translation services with different input parameters (language combinations), which will be useful for multilingual support in international conferences [30].

  3. 3.

    Execution of atomic services or composition of atomic services in the IoS platform drives actuators or sensors on IoT. For example, whenever a composite translation service is executed by a certain user, a group of sensors like GPS and counter are driven to record the behavioral or situational data of the user together with the service invocation information, which can help improve the accuracy of situated service recommendation in the integrated IoS/IoT environments [10].

Fig. 1.
figure 1

Multiagent-based integration of IoS and IoT

Full size image

Since there are various types of complicated interaction, we can regard the integrated IoS/IoT environments as a multiagent-based architecture as shown in Fig. 1. The interaction between IoS and IoT can be realized by complex event processing agents and rule agents that manage various rules including sensor data aggregation rules, actuator driving rules, and service invocation rules. Complex event processing agents and rule agents are key parts of the integration components. Agents in the integration components interact with sensor data aggregation agents, actuator driving agents, and service invocation agents in the IoS and IoT sides to realize service composition.

The most important feature of the multiagent-based architecture is that it should be designed to address the major issues of IoS/IoT integration described in the previous section. Traditional characteristics of multiagent systems can help deal with these issues in different ways. To deal with the heterogeneity of cloud services and IoT devices, agents in both the IoS side and the IoT side need to deal with the interoperability by interface matching and learning mechanisms. For example, sensor data aggregation agents are required to discover an unregistered sensor that has the similar interface with a known type of IoT devices. To address the challenge of ultra-large scale of services and things in the environments, the rule agents and complex event processing agents need to provide optimization mechanisms to ensure real-time processing of rules and execution of services, which can probably be realized by solving constraint optimization problems in the domain of traditional multiagent systems. To address the dynamics of the integrated IoS/IoT environments, agent adaptations need to be applied to deal with various dynamic changes, e.g. the fluctuation of available services and devices in the environments.

4.3 Multiagent Methodologies for Service Design

We discussed the possibility of using the multiagent-based approach to deal with the integrated components for IoS/IoT. However, it is also important to design the application systems to be run on the integrated IoS/IoT environments, which is the second aspect of the integration. Service design process with integrated IoS/IoT environments always involves multiple stakeholders with different incentives, and therefore multiagent methodologies can be applied.

In previous work, we studied the service design process for the IoS infrastructure [24]. To design sociotechnical systems for the real world, the iterative service design approach is always applied; it consists of four phases: the observation phase is to understand user requirements and evaluation criteria for QoS; the modeling phase is to define the service process that can best satisfy user requirements by combining available cloud services based on QoS evaluations; the implementation phase is to realize service composition by embedding composited services into application systems; the analysis phase is to evaluate the designed service by analyzing the log data and interview results based on the defined QoS evaluation criteria [22]. This iterative service design process can also be applied to the service design for integrated IoS/IoT environments.

Based on numerous experiences in service design, we adopted the multiagent approach to propose field-oriented design methodologies [15]; this allows us to deal with the service design in the field, where complex issues often arise. The service design process always starts by understanding the problems in the field and proposing services for solving those problems at an early stage. Due to the interdependency between problems and their changes over time, it is more important to develop a continuous problem solving process than to solve just current problems. In this context, multiagent methodologies were proposed for experiments on service design with multiple stakeholders, including role playing games [45], participatory simulations [21] and gaming simulations [31].

5 An Example Application of Multilingual Environment Design

As our application, we use the example of designing a multilingual environment to illustrate the integration of IoS and IoT. Figure 2 shows a scenario of multilingual conversion between an interactive agent and a user. In the scenario, a German visitor is shopping at the Nishiki Market in Kyoto, a historic Japanese marketplace; the multilingual interactive agent provides necessary support. When the visitor asks something in German, the interactive agent first selects a composite translation service to translate it into Japanese. According to the difficulty and language of the input sentences of the visitor, the service invocation agent on IoS recommends a composite translation service, which usually consists of a speech recognition service, a machine translation service, and a dictionary service of domain knowledge. Meanwhile, the sensor data aggregation agent on IoT is interacting with the rule agent and complex event processing agent by sending the event information about what the visitor is looking at. The rule agent and complex event processing agent then trigger a composite dialogue generation service by interacting with the service invocation agent. In the example shown in Fig. 2, the service invocation agent recommends a cultural-oriented dialogue generation service that maps one concept in one culture (e.g. Suguki, traditional Kyoto pickles shown in the top-right part of Fig. 2) into the close concept in the other culture (e.g. Sauerkraut, which is a well-known dish in Germany) for explanation. Similar examples of culturally-situated agent were introduced in our previous work [1, 20].

The dialogue between the interactive agent and the visitor in above example seems simple, but its generation involves complicated interactions between various agents on the integrated IoS/IoT environments and service composition based on situation-aware recommendation mechanisms. To realize such high-quality multilingual environments with IoS and IoT, we must deal with two major issues: realizing key technologies of the integrated IoS/IoT architecture for service composition and designing multilingual interactive agents based on the integrated architecture.

Situated Service Composition. Service composition is one of the most important topics in the area of SOC, and this is also true in the integrated IoS/IoT environments. Traditionally, service composition is a technology that combines multiple atomic services in the IoS infrastructure to satisfy user requirements of Quality of Services (QoS). Service composition can be realized by automatically generating the service workflow through planning, constraint-based approach, QoS optimization, and user-centered approach. The key lies in how to select appropriate services to execute the service workflow since there are always numerous atomic service candidates for the same function. One of the major challenges in realizing this application is to combine various services on the integrated service platform and establishing interaction between agents across IoS and IoT. Unfortunately, existing service recommendation approaches do not fully consider the influence of situation information, such as time, location, and user relations thoroughly. Our solution is a situated QoS prediction for service recommendation that combines observed factor learning and latent factor learning. The detailed proposed mechanism is described in our previous work [10]. The situated service composition mechanism can be realized based on the multiagent integrated architecture described in Fig. 1.

Fig. 2.
figure 2

An application of multilingual environment design with IoS and IoT

Full size image

Multilingual Interactive Agents. Another issue in the example application is to realize multilingual interactive agents. The interactive agents play essential roles in many fields [38]. For example, there are increasing demands in the fields of tourism and healthcare to use interactive agents because of the human resource shortage due to the aging population. In such circumstances, interactive agents are required to handle dialogues with end users in various situations. Moreover, it becomes important for interactive agents to provide multilingual support to end users. However, it is always difficult to ensure the quality of dialogues generated by the agent, especially at an early stage. Previous studies widely used Wizard of Oz (WoZ) for prototyping and evaluating dialogue-based computer systems that have not yet been realized. In our service design process, we also apply WoZ to prototype and improve the interactive agent gradually. In our proposed framework, the recommender system provides a candidate list of dialogues to the Wizard based on service composition. The Wizard then selects or creates appropriate dialogues and provides them to the interactive agent. The detailed framework of interactive agent is described in our previous studies [33]. The implementation process of the interactive agents can be regarded as a participatory social system design process, which is a multiagent approach for service design.

6 Conclusion

This paper dealt with the issues of integrating Internet of Services (IoS) and Internet of Things (IoT) for realizing Internet-based sociotechnical systems. We described the integrated IoS/IoT architecture from a multiagent perspective. First, we described the multiagent-based architecture suitable for the integration by defining several interaction patterns between IoS and IoT. Then, we discussed the possibilities of using multiagent methodologies in the design of sociotechnical systems based on the integrated architecture. Further, we used an example of designing multilingual environments to illustrate the multiagent perspective.

Our future work is to implement the key concepts in the multiagent perspective discussed in this paper for integrating IoS and IoT in comprehensive and effective manner. Moreover, the discussed multiagent architectures and methodologies in this paper need to be extended to deal with the massive scale in the context of integrated IoS/IoT, which may result in related future possible research directions in massively multiagent systems.