Keywords

1 Introduction

The current transformation of conventional products into hybrid, networked artifacts represents a qualitative leap that challenges the distinction between ‘digital’ and ‘analog’, ‘material’ and ‘immaterial’ [40, 41, 44]. The development, design and implementation of these hybrid artifacts is an interdisciplinary task, in which various professions – product-, graphic-, interaction-, and user experience design, engineering and computer science – are involved. These disciplines have different roots, their own thought leaders, constituent discourses, specialized theories and methods [37]. Due to its much longer history, design today has a broad foundation of coexisting and complementary theoretical approaches, methods, and a humanistic philosophical stance. In contrast to design, the field of HCI is not only younger, but much more determined by technical rationalism and pragmatic interests [3, 4]. Accordingly, the development of theoretical foundations was primarily directed towards guidelines, principles, and methods aimed at ensuring the usefulness and usability of computer programs and websites.

On the whole, a parallel but temporally staggered development of the theories and methods of design and HCI can be noted, where HCI has often caught up with what was already state of the art in the field of design. At the same time, there are also bodies of knowledge and methods that have so far been used almost exclusively in one of the fields, despite the fact that their application would also be beneficial for the other fields involved in product development.

Taking into consideration the current progress and dissemination of hybrid artifacts, artificial intelligence (AI), and machine learning, this preliminary study will argue for a further integration and advancement of existing theories, principles, and methods from design, HCI and user experience as well as a stronger involvement of designers and design researchers at an early stage of product development.

Since design has historically played a key role in socio-cultural innovations and the embedding of new technologies in the everyday life of people [5, 40, 48], the question arises as to how design, in interdisciplinary collaboration with computer science, cognitive psychology and environmental psychology, among others, can help shape the digital transformation of artifacts in a human-centered and sustainable way. In this context, the discipline not only has a responsibility, but also a special competence, since design has always been an integrative activity that brings together knowledge and competences from different fields – from the natural sciences, engineering, humanities and social sciences – for practical purposes [6, 15] and is able to act competently in this inter- and transdisciplinary field of tension.

1.1 State of the Art of Convergence of Design and HCI

In the last few decades numerous practitioners and researchers have worked to bring theories, processes, and methods of design and designerly ways of thinking and acting to computer science to humanize and improve the interaction between people and computers [14, 45]. Frequently a development was repeated in the field of HCI that had already taken place decades earlier in the field of product design.

One of the most important developments was a paradigm shift in both disciplines which can be briefly described as “from function to meaning” [7]. While modern design and its theoretical principles were mainly focused on requirements of practical product usage and industrial mass production, in the course of the 1970s semantic and symbolic product qualities and the related emotional bond moved to the fore in order to meet the hitherto neglected psychological needs of users [18]. About thirty years later, with the public accessibility of the Internet and dissemination of digital devises the nascent field of interface and interaction design experienced the same development stages: from function, usefulness, and usability to ‘user experience’ [34], ‘hedonic qualities’ [12, 22], and ‘emotional design’ [35]. Unaware of the already existing body of knowledge in product design, particularly with regard to design semantics and semiotics, the HCI-community started over again developing or adapting theoretical terms and models in order to guide interface- and interaction design practice. Although not exhaustive, a number of overlaps between theories, concepts and methods of design and HCI should be mentioned:

Workflow.

Oswald [37] recalled that the labor- and time-saving working processes and a reduction of the walking routes in the Frankfurt kitchen, developed by Margarete Schütte-Lihotzky in 1926, was an early predecessor of workflow and process design, which are currently embraced by usability engineering.

Semiotics/Theory of Product Language.

Oberquelle [36], who dealt with design issues in HCI, had early on pointed out the relevance of semiotics for user-centered HCI. Yet, in the existing HCI literature, semiotics has been referred to almost only in the context of computer-generated art and ‘aesthetic computing’ [32], the graphic design of websites [1, 11, 46, 52], or very generally to the communicative and interactive nature of digital objects [cf. 2, 9].

Semantic Differential.

The method of the Semantic Differential borrowed from psychology was introduced to product design or rather the so-called ‘theory of product language’ by Gros [19] to investigate product’s impression (look & feel) on beholders and target groups. At a later time Hassenzahl, Burmester, and Koller [23] applied this method for measuring the hedonic and pragmatic quality of websites and interactive applications as perceived by users.

Product Personality/Persona.

Gros [20] used the concept of persona or as he put it product personality within the theory of product language for the embodiment of product’s expression or – seen from another angle – characterization of product’s target group. Later Cooper [10] suggested that computer scientists could use personas to get a more realistic idea of the users of their software applications.

Affordance/Indication Sign.

The term affordance, which goes back to the psychologist Gibson [17], was introduced into the fields of design and HCI by Norman [33] and Krippendorff [29, 30]. The term refers to the fact that, depending on specific properties, an artifact invites to use it for certain purposes and in a certain way. However, the concept of the so-called indication sign [13], which has its roots in the philosophy of Langer [31], is a key concept in the theory of product language that comprises the scope of the aforementioned affordances and even goes beyond this; in addition, they communicate product’s properties such as performance, quality standard, technical status, etc.

Metaphors.

The invention of the desktop metaphor by Kay and its application to the Apple interface, including numerous other icons, contributed decisively to the easy readability and intuitive usability of the PC. Notwithstanding metaphors referring to historical forms, engineering or nature, were early on also applied to housing design of electronic devices to make the functions and operation of a ‘black box’ filled with technical components understandable and easily accessible [8]. At present, the metaphor design approach is used only sporadically for smart products, e.g., humanoid robots or Sony’s robotic dog ‘Aibo’.

Pattern Language.

In 1977, Alexander and colleagues developed the concept of a so-called pattern language by systematically showing exemplary solution patterns for more than 250 general problems, which are transferable to comparable applications. This approach took on a new relevance in interface design, where developers quickly arrive at good design solutions by resorting to predefined patterns [7, 11, 21].

2 Challenges in Development and Design of Hybrid Artifacts

The digital transformation of the world of things brings about a fundamental change in the relationship of people and objects, since smart objects develop a life of their own. At the same time, they currently still look much the same as their dumb predecessors. The design of virtual assistants like Google Nest or Google Home mini, to name a few, is still based on the outdated functionalist doctrine of the so-called Good Design, according to which less is more. As is often the case when new technologies are introduced, they are initially stuck with the old patterns. Thus, an in-depth analysis of ten voice assistants and social robots (including Amazon’s Alexa, Google Home, and Jibo) in terms of physical appearance, input and output modalities, feedback systems, and discoverability of functions concludes that the materialization of AI functions into tangible objects is still deficient. Even basic principles of interaction design have not been sufficiently considered [47]. Two of the most important challenges of smart assistants’ design are addressed below.

2.1 Control Dilemma

The control dilemma arises from the fact that on the one hand, smart assistants such as the smartphone are assigned certain tasks that they are supposed to manage independently to relieve users by reducing cognitive load [51]. But on the other hand, it is annoying when the assistants initiate undesirable actions or when their activities are not transparently traceable [28]. In both cases, users can feel unpleasantly surprised, ignored, or even patronized by the smartness of hybrid artifacts.

2.2 Invisibility Dilemma

The invisibility dilemma is at least equally problematic: When smartness is seamlessly integrated into Good Design-style voice assistants, supposedly familiar everyday objects such as consumer electronics, or the built environment this new quality of the artifacts is not apparent to people. Therefore, people have no opportunity to adjust their expectations and behavior accordingly. However, in the context of product categories, such as household appliances or automobiles, designers had repeatedly advocated that hiding advanced technology, new materials or ideas in an old design does not make sense. There is a widespread consensus that groundbreaking products should clearly show their innovative character, so that users can perceive the added value [27, 42]. There are strong reasons to demand this from smart products as well, especially if the devices encroach on users’ privacy and affect data security. An AI iconography, a set of visual signs or rather small icons that communicate various key factors of AI functions and operations to stakeholders, as suggested by Pilling et al. [38], is at least one possible semiotic approach to resolve this dilemma.

2.3 Interplay Between Hardware, Software and Interface

To this day, at least a few computer scientists consider the hardware of hybrid artifacts merely as a so-called ‘form factor’ [39] that lies outside of one’s own field of expertise and does not need to be taken into account. However, separate development and design of hardware, software, interface and interaction is problematic, especially for new products. The importance of a holistic approach can be exemplified by Apple’s products, in which all components always form a unique symbiosis: Starting with the intuitive operation of the graphical user interfaces and touchscreens, the design language of the housings, to the innovative handling of materials, construction and manufacturing techniques [2, 43].

Furthermore, it is supported by so-called digital materialism studies, which reject the dualism between nature and culture, mind and matter and seeks to mediate between natural science, the arts, and environmental ethics [41]. To this end, it is necessary, for example, to scrutinize and uncover concrete connections between for instance the materiality of the touchscreen, operating gestures, and the perception of the media content by people [50].

3 Next Step of Cooperation and Integration of Design and HCI

As has been shown issues around sensual perception, communication and interpretation of hybrid artifacts by users are crucial. Like any other product, hybrid artifacts are perceived as signs, and their use and operation are guided by users’ interpretation of signs inscribed in the artifact. In fact, various terms and concepts of product design semantics have already been redeveloped or borrowed by HCI to catch up. However, since these terms and concepts have not yet been systematically related to each other, and since a consistent semiotic perspective on the subject matter is also missing so far, this is suggested for further research.

Furthermore, a design semiotics approach can be used to deal with relevant and so far neglected issues, just to mention sustainability concerns. In her analysis of the interface design of technical devices, Weber [49] raised the question of why users of consumer electronics, for example, “receive visual feedback on volume or performance, while feedback on energy consumption is still lacking”. Since the increasing energy consumption associated with the digital transformation is highly relevant, but at the same time massively underestimated [24], this consumption should also be made perceptible – as well as the before mentioned equally invisible AI of hybrid artifacts.

However, whether users perceive hybrid artifacts as such, whether they develop a basic understanding of smartness, whether they can perform only certain predefined tasks, or whether they are also able to use them in accordance with situationally deviating individual needs and purposes is a crucial question – and at the same time a huge challenge for design in the broadest sense [16]. How hybrid artifacts are received by the users, which ‘technology emotions’ [25] are triggered and how an innovative technology is ultimately accepted, depends much on a holistic human-centered design approach. As Junginger [26] has pointed out, a human-centered design approach goes beyond user-centered design as it is currently understood and practiced in the field of HCI. While user-centered design only focuses on a person’s immediate experience with a particular product or service, the human-centered design approach takes a systemic perspective that strives to create and realize new opportunities for desirable relationships and interactions between people, products or services, and their living environments. This means that human-centered design addresses not only issues such as interactions between hybrid artifacts and users, but also includes artifacts’ consequences on society and the environment. Thus, design should not only be seen as a service provider when it comes to adapting technology to user needs and making applications usable and enjoyable. Design, understood as a holistic, human-centered approach, should already be involved in negotiating which technical functions and hybrid artefacts are really desirable and should be further developed. The challenge lies not only in the design of the hybrid artifacts, but in the design of the technology itself. Or as Geiger [16] put it in a nutshell: “Technology design is much more than the design of beautiful product housings and easy-to-use surfaces. Because it raises the question of power: Who rules over whom – man over technology or vice versa?”.