Keywords

1 Introduction

The dedicated exploration and exploitation of knowledge has the potential to bring together people that are engaged in organizations and society with technology that acts on behalf of humans in an increasingly autonomous manner. Combining the strengths of both is crucial for the vision of a prosperous future in which the co-creation and reciprocal influence between people and technology enables advanced benefits. In recent years, many such benefits are wished for in different application domains. Some examples are (1) distributed, collaborative and automated design & manufacturing workflows realized by Cyber-Physical System, Internet of Things, Cloud Computing and Big Data Analytics in Industry 4.0 [1], (2) urban development turned to technology, innovation, and globalization in Smart Cities [2], (3) innovative creativity and productivity support for information exploration in new media [3], and ontology evolution in the Semantic Web [4]. One general issue in these examples is that the alignment of knowledge is required between what that is captured in the heads of people and what is encoded by technology.

The conceptual framework for tackling the issue is based on two ideas. The first is to extract innovative scenarios that humans come up with from the heads of people, to capture them in design thinking artifacts, and to decompose and represent the knowledge captured by design thinking artifacts in conceptual models. The second is to enhance technical systems into constituting elements of intelligent environments, to put intelligent environments into operation, and to abstract in conceptual models the data, information, and knowledge encoded by the operation of intelligent environments. From this results the challenge that knowledge contained in conceptual models differs in form, quality, and semantic richness, which is why a conceptual model-based transformation of knowledge is required. Thereby, manual engineering tasks should be substituted by (semi-) automated, intelligent ones as human engineering effort does not scale. By searching for patterns to describe the relation between design thinking, conceptual models, and intelligent environments on a meta-level, the problem is to find and to come up with systematic foundations that enable reuse and horizontal integration across different projects addressing the topic. To explore the pattern space, the PRINTEPS [5] and s*IoT [6] project are considered as instances that realize the patterns under scrutiny. Based on these instances, the method to find and to come up with systematic foundations is an agile composition of qualitative meta-synthesis [7, 8] and model-based design science research [9, 10]. The validation of the identified patterns is achieved by two evaluation cases, a SWOT analysis, and the transfer of tools between PRINTEPS and s*IoT.

After the introduction, the state of the art on design thinking and intelligent environments is provided based on s*IoT and PRINTEPS in Sect. 2. Section 3 explores how PRINTEPS and s*IoT instantiate common systematic foundations from the OMiLAB ecosystem. Validation cases for the resulting patterns that employ knowledge-based conceptual modelling are presented in Sect. 4. The discussion of how common systematic foundations are useful is conducted in Sect. 5 before concluding the study.

Fig. 1.
figure 1

Topic of the study.

Full size image

2 Bridging Design Thinking and Intelligent Environments

The alignment of knowledge between people and technology becomes increasingly necessary and complex for building digital products like innovative product-service systems. Figure 1 illustrates the related problem of combining domain knowledge in the heads of people and environments in which this knowledge is put into operation. To tackle the problem, human engineers often use ICT systems to relate both elements. However, this approach does not scale as the speed of change increases for domain knowledge and environments. Rather, different projects like PRINTEPS and S*IoT employ knowledge-based conceptual models to bridge the gap between design thinking and intelligent environments in a (semi-) automated, intelligent manner. PRINTEPS focuses on making intelligent environments accessible to human end-users, while s*IoT focuses on extending design thinking to represent and automate the knowledge that is captured in the heads of people. By integrating PRINTEPS and s*IoT based on patterns of common systematic foundations, it is possible to realize a bridge between design thinking and intelligent environments. This requires a knowledge-based conceptual modelling approach, as described in Sect. 3. Before this explanation, details are provided on how PRINTEPS aligns knowledge between people and technology by focusing on intelligent environments, while s*IoT aligns knowledge between people and technology by focusing on design thinking.

2.1 Design Thinking in s*IoT

A topic in s*IoT [6], which is an extended conceptual modelling approach, is to support design and use of innovative scenarios in a business context. This makes it necessary to access human knowledge about, e.g., a domain, human interaction, and context; and to turn human knowledge into a core element of digital products. Therefore, s*IoT employs design thinking in collaborative workshops with the goal of finding innovative business models. This requires creativity, insight, and understanding of customer needs. However, the result of design thinking workshops rarely is a fully fledged business model. Rather, often paper-based and semantically poor design thinking specific artifacts are created. The result is a semantic gap between design thinking specific artifacts and the business models in a specific company context. To bridge the semantic gap, s*IoT employs dedicated conceptual modelling tools like, e.g., the Scene2Model tool [11]. As a consequence, it becomes possible to link design thinking artifacts with business models in the context of an enterprise. This is the input for the s*IoT and other tool which employ knowledge representation schemes, AI, and technologies from the semantic web stack to connect business models and cyber-physical systems to enable the operationalization of innovative scenarios. In summary, s*IoT employs tools for building “smart” models to put the artifacts of design thinking into operation. These “smart” models are at the centre between two layers of interpretability: one for people concerned with innovating business scenarios, the other for cyber-physical systems that operate in complex real-world settings.

Fig. 2.
figure 2

Design thinking and intelligent environments in s*IoT and PRINTEPS.

Full size image

2.2 Intelligent Environments in PRINTEPS

A topic in PRINTEPS [5], which is a platform to develop practical intelligent applications, is to support the setting of environments in which different AI approaches enable robots to operate. These intelligent environments are characterized by increased complexity due to a feedback loop between all the systems that interact during operation. To enable smooth operation, knowledge is encoded in intelligent environments by the function, structure, and behaviour of these systems and their connection. Business insights and end-user requirements are a part of this knowledge. However, human end-users and business experts generally do not have the skills or tools to encode their knowledge in intelligent environments. Therefore, PRINTEPS abstracts the knowledge encoded in intelligent environments by providing intricate capabilities of intelligent environments. The relation between abstract, intricate capabilities and the operation of intelligent environments again takes the form of a feedback loop. The problem is that different computational paradigms, granularity of detail, and language of presentation are used on both ends of the feedback loop. To manage this complexity, PRINTEPS employs the robot operating system, multi-knowledge based models, and the (semi-)automated, intelligent transformation between them. As a result, PRINTEPS makes it possible to encoding knowledge about intelligent environments in workflow models. Nevertheless, human end-users and business experts still wish for more intuitive means of knowledge acquisition.

2.3 Bridging the Gap

Figure 2 shows the focus of PRINTEPS and s*IoT. On the one hand, the focus of PRINTEPS is on intelligent environments that provide capabilities to human end-users. As human end-users are generally not able to transform their knowledge directly into intelligent environments, conceptual models are employed as an intermediary. In particular, a multi-knowledge-based transformation is employed in PRINTEPS to align knowledge between end-users and intelligent environments. On the other hand, the focus of s*IoT is on design thinking to represent innovative scenarios. As the knowledge captured by design thinking artifacts generally cannot be directly used for operationalization in a business context, conceptual models are employed as an intermediary. In particular, an integrated conceptual modelling environment is employed in s*IoT to align knowledge between design thinking and cyber-physical systems. As PRINTEPS and s*IoT both employ knowledge-aware conceptual models to connect people and technology, a knowledge-based conceptual modelling approach is identified as an invariant between PRINTEPS and s*IoT.

3 Knowledge-Based Conceptual Modelling Approach

Additional benefits emerge by combining the strengths of PRINTEPS and s*IoT, as both focus on different aspects of aligning knowledge between people and technology when building digital products. To combine the two research projects, common systematic foundations surrounding a knowledge-based conceptual modelling approach have to be found. This section illustrates these systematic foundations and their integration with other patterns from the OMiLAB ecosystem.

Fig. 3.
figure 3

Design thinking and intelligent environments in s*IoT and PRINTEPS.

Full size image

3.1 Concept Synthesis in the OMiLAB Ecosystem

PRINTEPS and s*IoT are two projects that align knowledge between people and technology, as shown in Fig. 2. The combination of their different strengths can be achieved by synthesizing common systematic foundations on the meta-level [12]. These common systematic foundations can be related to PRINTEPS and s*IoT through instantiation, which facilitates cross-project reuse of results. By synthesizing common systematic foundations for PRINTEPS and s*IoT on the meta-level, a three layer pattern emerged through active design science research. Figure 3 shows the synthesized pattern, its integration with other patterns in the OMiLAB ecosystem, and its instantiation for PRINTEPS and s*IoT.

The pattern to combine PRINTEPS and s*IoT consists of three layers: scenario layer, conceptual modelling layer, and execution layer. On the scenario layer, knowledge about innovation is created by people engaged in different aspects of society. Knowledge is transformed between the scenario layer and the conceptual modelling layer by the decomposition of the former. On the conceptual modelling layer, knowledge is represented in different kinds of conceptual models. Model transformations are employed on the conceptual modelling layer to relate different conceptual models and the knowledge within them. Knowledge is also transformed between the conceptual modelling layer and the execution layer by the abstraction of the latter. On the execution layer, different systems put knowledge into operation. The name coined for this three layer pattern is digital product design lab, as its instantiation and consequent realization enables the creation of digital products that require an alignment of knowledge between people and technology. The pattern of the digital product design lab is integrated in the OMiLAB ecosystem. The OMiLAB ecosystem is an open platform, environment, and knowledge base to support the conceptualization and operationalization of conceptual modelling methods based on a general understanding of model values [13]. The OMiLAB ecosystem is managed by the OMiLAB NPO which is located in Berlin, Germany. Concepts of the OMiLAB ecosystem are domain-independent patterns like the modelling method seed concept [14] or the agile modelling method engineering life-cycle [15]. The pattern of the digital product design lab from the the OMiLAB ecosystem can be instantiated for PRINTEPS and s*IoT. This instantiation is beneficial as the pattern can be refined to enable the different foci from which the different strengths of the PRINTEPS and s*IoT projects emerge. Likewise, PRINTEPS refines the execution layer in a layer for intelligent environments while reducing the scope of the scenario layer layer to end-user aspects. Alternatively, s*IoT refines the scenario layer by incorporating design thinking while reducing the scope of the execution layer by merely considering proof-of-concept aspects.

Fig. 4.
figure 4

Realized instance of the OMiLAB ecosystem in the OMiLAB associated partner laboratory at Keio University.

Full size image

3.2 Digital Product Design Lab Realization for PRINTEPS

The PRINTEPS instance of the OMiLAB ecosystem, including the pattern for a digital product design lab is realized in the OMiLAB associated partner laboratory at Keio University. Figure 4 shows the installation that realizes different patterns of the OMiLAB ecosystem instance. Next to this physical realization, a virtual component is also necessary. The essential virtual component is PRINTEPS. PRINTEPS is a user-centric platform to develop integrated intelligent applications by combining four types of modules; namely, knowledge-based reasoning, spoken dialogue, image sensing, and motion management. PRINTEPS provides a user-friendly workflow editor based on Robot Operating System (ROS) and Service Oriented Architecture (SOA). It supports end users in AI applications design and development.

In detail, Fig. 4 shows the realization of two patterns in the OMiLAB associated partner laboratory at Keio University. Domain-independent invariants of the OMiLAB ecosystem are realized in the Workflow Modelling Space, while the PRINTEPS instance of the digital product design lab is realized in the Digital Product Space. Since PRINTEPS workflow editor is implemented as a web application, domain experts are able to create workflows using web browsers installed on any laptop PC, at any location. The left side of Fig. 4 shows an example of Workflow Modelling Space in a laboratory of Keio University. The workflow editor is applied in the Digital Product Space to build digital products based on the workflows that can be converted to executable Python source codes on ROS. Some of these products have already been realized, e.g., teaching assistant robots and robot teahouse (the right side of Fig. 4) which is described in more detail in Sect. 4. Decomposing services and processes into modules in the workflows while abstracting from proof-of-concept realizations based on a myriad of existing products (Pepper humanoid robot, HSR human support robot, Jaco2 arm type robot, and several sensors), these digital products align people and technology with knowledge.

3.3 Digital Product Design Lab Realization for s*IoT

The s*IoT instance of the OMiLAB ecosystem including the pattern for a digital product design lab is realized in the OMiLAB node at the University of Vienna. Figure 5 shows the installation that realizes different patterns of the OMiLAB ecosystem instance. Next to this physical realization, virtual components are also necessary. Two essential virtual components are ADOxx and OLIVE. ADOxx is an integrated conceptual modelling environment that stretches across the metamodel hierarchy. ADOxx can be used to engineer different modelling methods for building conceptual models that represent knowledge. Furthermore, ADOxx enables the integration of knowledge-driven AI to process the knowledge that is represented in conceptual models. OLIVE is a framework that extends the capabilities of ADOxx with, e.g., dashboards, data stores on the edge, and connectivity to the IoT. Using OLIVE, it is possible to integrate conceptual models and data-driven AI. The term “smart” models has been coined for the integration of conceptual models, knowledge-driven AI, and data-driven AI. All these elements are bound together in a service-driven architecture.

Fig. 5.
figure 5

Realized instance of the OMiLAB ecosystem in the OMiLAB node of the University of Vienna.

Full size image

In detail, Fig. 5 shows the realization of two patterns in the OMiLAB node at the University of Vienna. Domain-independent invariants of the OMiLAB ecosystem are realized in the Modelling Method Space and the s*IoT instance of the digital product design lab is realized in the Digital Product Space. The former is a manufacturing line for modelling methods inspired by the modelling method seed concept and agile modelling method engineering. The result of this manufacturing line are modelling methods and tools. These modelling methods and tools are applied in the Digital Product Space to build digital products based on “smart” models that align knowledge between people and technology. Some such digital products are already realized, e.g., a shopping assistant for disabled people, delivery on demand in Industry 4.0, and a smart city tourist guide which is described in more detail in Sect. 4. By decomposing design thinking artifacts in “smart” models while abstracting from proof-of-concept realizations based on, e.g., NAO humanoid robots, robotic arms, and flying drones, these digital products align knowledge between people and technology.

4 Case Studies

This chapter presents two demonstrative cases in which the realized PRINTEPS and s*IoT instances of the digital product design lab pattern are put to action. Thereby, a common understanding is facilitated based on the pattern for a digital product design lab from the OMiLAB ecosystem. This common understanding enables cross-project reuse, which is elaborated further in Sect. 5.

4.1 PRINTEPS Teahouse Case

A digital product space is realized in the OMiLAB associated partner laboratory at Keio University, corresponding to the PRINTEPS instances of the digital product design lab pattern. This realization is further refined according to Fig. 6 and put to action in the PRINTEPS robot teahouse case. The robot teahouse has been developed with the aim of realizing teahouses where human staff and machines work in synergy to provide services. Currently, the robot teahouse consists of six services: customer detection, greeting, guiding customers to a table, order taking, serving drinks, and cleaning up. To realize the robot teahouse, it is necessary to acquire expert knowledge from teahouse owners and express them as conceptual models using workflows, business rules, and ontologies. In the robot teahouse, the workflows are used to describe standard business procedures for robots with sensors. The business processes are used by robots to change the way they greet customers and take orders according to customers’ attributes such as age, gender, and accompanying person. The ontologies are used to describe agents, objects, and environment in the robot teahouse such as robots, menus, and table positions. Although, currently, we have acquired the expert knowledge by interviewing teahouse owners, this process should be supported by tools and methods such as the design thinking approach.

Fig. 6.
figure 6

Digital product design lab pattern for the PRINTEPS teahouse case.

Full size image

It is also necessary to integrate multiple robots (Pepper humanoid robot, HSR human support robot, Jaco2 arm type robot, etc.) and sensors (Microphones, Kinect, omni-directional camera, etc.) with knowledge-based reasoning, image sensing, spoken dialogue, and motion management. In the robot teahouse, knowledge-based reasoning techniques are used for greeting and guiding customers to a table. Image sensing techniques are used for customer and table detection. Spoken dialogue techniques are used for order taking. Motion management techniques are used for guiding customers to a table, serving drinks, and cleaning up. Since there is a big gap between the knowledge-based conceptual modelling layer and the intelligent environment layer, we have developed PRINTEPS including multi-knowledge-based editor and spoken dialogue system to facilitate the alignment of knowledge between people and technology.

4.2 s*IoT Smart City Tour Guide Case

A digital product space is realized in the OMiLAB node at the University of Vienna, corresponding to the s*IoT instances of the digital product design lab pattern. This realization is further refined according to Fig. 7 and put to action in the s*IoT smart city tour guide case. The general idea for this case is to support tourism in a smart city by exploring the potential of new business-aware service ideas, knowledge-driven and data-driven AI, and emerging robotic systems. This case can be tackled from different angles, e.g., by conceptual model transformation and the abstraction of capabilities from dronesFootnote 1 and by decomposing stakeholder requirements in smart city models to create intelligent tours using a NAO robotFootnote 2.

Fig. 7.
figure 7

Digital product design lab pattern for the s*IoT smart city tour guide case.

Full size image

The integration of knowledge-driven and data-driven AI with conceptual models is crucial for the different angles that tackle the s*IoT smart city tour guide case. One concrete example is a smart tour that requires input about what human participants know or want to know, the execution of the tour supported by emerging robotic systems, and knowledge-based conceptual models that facilitate the alignment of knowledge between people and technology. Thereby, knowledge-based conceptual models support knowledge acquisition using, e.g., speech and gesture recognition, the integration of open ontologies containing smart city knowledge and data, and semantic-aware services orchestration for e.g., landmark detection.

5 Discussion

Common systematic foundations between PRINTEPS and s*IoT, enable the integration of the two projects. This has been demonstrated by a concrete collaboration that made it necessary to bridge design thinking and intelligent environments. In design thinking workshops during this collaboration, the Scene2Model tool, which is a part of the s*IoT project, is used to elicit stakeholder requirements for PRINTEPS cases. This proved to be useful as knowledge acquisition is a current topic in the AI community. As a result, human knowledge about innovative scenarios could be represented in conceptual models that relate to workflow models of PRINTEPS and their decomposition in the operation of intelligent environments.

To further evaluate the knowledge-based conceptual modelling approach and its integration as a pattern in the OMiLAB ecosystem, a SWOT analysis is useful. The strength of the pattern is that it can be instantiated for different projects to enable reuse, transfer, and collaboration. Furthermore, the pattern enables the integration of knowledge-driven and data-driven AI from which the benefits of knowledge-based conceptual models emerge. The weakness of the pattern is that it might not be applicable to all kinds of projects, i.e., projects that are only interested in advancing theoretical foundations might not care about how theoretical foundations benefit from the design of innovative scenarios or about operation environments in which the theoretical foundations can be put to use. The opportunity for the pattern is that many research project tackle the alignment of knowledge between people and technology from different angles. Research, education, and application can benefit from the possible integration of these projects. The threat for the pattern is that openness and the willingness to share and collaborate are essential prerequisites. For closed projects that are not seeking relations with the bigger community only limited usefulness is provided.

6 Conclusion

This study examines the hypothesis that design and use of digital products requires scalable alignment of knowledge between what is in the heads of people and what is encoded by technology. As different projects already tackle the issue, the question is how to facilitate reuse, transfer, and combination of results. PRINTEPS and s*IoT are two such projects that employ conceptual models to focus on a subsection of knowledge alignment between people and technology. Respectively, they focus on the role that design thinking and intelligent environments play. To integrate both projects and potentially others as well, a pattern for systematic foundations on the meta-level is identified and integrated in the OMiLAB ecosystem. At the core of this pattern lies a knowledge-based conceptual modelling approach that employs data-driven and knowledge-driven AI. From the instantiation of this pattern for PRINTEPS and s*IoT emerges the potential to bridge design thinking and intelligent environments. This potential is further illustrated by evaluation cases, a SWOT analysis, and preliminary collaboration results. The generalization of the results from this study is currently in progress as new OMiLAB nodes are spawned based on the identified pattern within an EU project. Future steps are underway to intensify the collaboration between PRINTEPS and s*IoT to realize the full vision of bridging design thinking and intelligent environments. Concrete collaboration is planned on a front-end for PRINTEPS end-users based on design thinking and an intelligent environment for s*IoT based on ROS.