Keywords

FormalPara Editors’ Log: Chapter 05

A remarkable feature of Star Trek is the ease with which devices and interfaces are used by humans and aliens to interact with complex, pervasive, and often almost-invisible technical systems. In this chapter, psychologist and computer scientist Gerhard Leitner and Human Factors specialist John NA Brown explore the concepts of Human Computer Interaction (HCI) and User Experience Design (UX Design) that are found in Star Trek, and show us how they differ from the state-of-the-art interfaces that currently connect man and machine. (Eds.)

Star Trek: The Original Series, 01×21, “Tomorrow Is Yesterday” (1967)

  • Kirk: Captain’s log, supplemental. Engineering Officer Scott informs warp engines damaged, but can be made operational and re-energized.

  • Computer: Computed and recorded, dear.

  • Kirk: Computer, you will not address me in that manner. Compute.

  • Computer: Computed, dear.

  • Kirk: Mister Spock, I ordered this computer and its interlinking systems repaired.

  • Spock: I have investigated it, Captain. To correct the fault will require an overhaul of the entire computer system and a minimum of three weeks at a starbase.

  • Kirk: I wouldn’t mind so much if it didn’t get so affectionate.

  • Spock: It also has an unfortunate tendency to giggle.

Star Trek: The Animated Series, 02×03, “The Practical Joker” (1974)

  • Spock: Question. Why are we unable to communicate with crewmembers McCoy, Sulu and Uhura?

  • Computer: Answer. That is for me to know and for you to find out.

  • Kirk: Did I hear that right?

  • Spock: Affirmative. The dysfunction is more severe than I thought. Question. Are you deliberately holding our missing crewmembers prisoner?

  • Computer: I’ll never tell.

  • Kirk: Let me try. This is Captain James T Kirk speaking. You are programmed to obey any direct order I may give, correct?

  • Computer: Correct.

  • Kirk: Very well. I order you to release crewmembers McCoy, Sulu and Uhura immediately.

  • Computer: Say please.

  • Kirk: Well, I’ll be.

  • Spock: I suggest compliance, Captain.

  • Kirk: Pu-leese.

  • Computer: Say pretty please with sugar on.

Star Trek: The Next Generation, 02x16, “Q Who” (1989)

  • Sonya: Hot chocolate, please.

  • LaForge: We don’t ordinarily say please to food dispensers around here.

  • Sonya: Well, since it’s listed as intelligent circuitry, why not? After all, working with so much artificial intelligence can be dehumanising, right? So why not combat that tendency with a little simple courtesy. Thank you.

figure a

“It’s interaction, Jim, but not interaction as we know it.”

“There is more information available at our fingertips during a walk to the woods than in any computer system, yet people find a walk among trees relaxing and computers frustrating. Machines that fit the human environment instead of forcing humans to enter theirs will make using a computer as refreshing as taking a walk in the woods.” [1]

Can you imagine how long a Star Trek episode would have lasted if they had to use the kind of standard speech recognition engine that we have on our current computers or smartphones?

If you’re not sure what the big deal is, please try to issue a Star Trek-style voice command to Siri, Cortana, or Google’ voice assistant and see what happens. Even if these systems are usually quite stable, would you want to trust them to work all of the time? Could you trust them enough to calmly say “energize”, counting on the system to work the first time and teleport you to safety in the very last second before the planet you are currently on explodes?

Kirk could.

Accurate and reliable voice recognition is only one aspect which differs between the interaction in Star Trek and real life. Others will be discussed in detail within this article. We work in Smart Homes and draw our examples from the principle Star Trek locations – the Enterprise and other starships, and space stations such as Deep Space Nine. These are hybrid locations, not only representing the work place of the crews, but also their homes, with accommodations for guests from the four quadrants of the universe. It might seem that many interactive features of the franchise could be beneficial in a modern-day home, if that home were just “smart” enough.

“The helm is not responding, Captain. Permission to turn it off and on again?”

At the end of the 1930s, Popular Mechanics Magazine published an issue entitled “The Electric Home of the Future” [2]. Since then, every few years we hear of future innovations, with new labels. The current label is the Internet of Things (IoT) but, in principal, the basic concepts have not changed very much. The interior design of the imagined home from 1939 is reminiscent of the look and feel of the interiors from Star Trek: The Original Series (TOS). This might be due to the fact that Star Trek creator Gene Roddenberry was also developing his mental model of the future under the socio-cultural influences of roughly the same time-space coordinates. However, the functional range and ideas for the future home were and are still mundane when compared to those imagined by Roddenberry.

Much of this imagined technology is beyond the scope of this chapter. For example, let’s briefly consider the teleporters. As we write this, teleportation is still in its earliest stages of theoretical development, at least for molecular structures bigger than a photon. Once developed to a point of Star Trek-like efficiency and accuracy, it will reduce rescue and recovery missions to a mundane flick of a switch and solve the traffic problems on our entire planet (so long as we make sure that there is no fly in the teleporter... but that is a different story). The main focus of this chapter are those devices and features involved in the interaction between humans and their built environment, the classic domain of Human-Computer Interaction (HCI).

“I can nae get the power, Captain. She’s configuring her updates!”

One central concept of Star Trek environments which is quite different to current technology in the home is the idea of a single holistic computer that supports everyone in all tasks. The “Computer” is accessible from each device from every location (inside and outside the ship, in a shuttle and even on the surface of an alien planet) and provides everything that is appropriate, not only in terms of content and function, but also in terms of modality. For example: “Computer, please show me the status of the surface of planet X” issued on the bridge would provide visual information on the display in front of the person who issued the command. Asking the computer for information from within the captain’s private room would result in an audio message containing the required information.

When taking a look into current homes, be it fully smart, partly smart, or not smart at all (in terms of technology), the vast majority are characterized by a smorgasbord of electronic devices, hardware controls, software interfaces, and remote controls. Most of the time, the components or subsystems are not compatible with one another, do not cooperate, and cannot be used in combination. This is a big contrast to Star Trek’s computerized premises wherein the “Computer” presents itself to its human cohabitants as a holistic entity, which can be easily, accurately, and reliably accessed in a wide variety of natural ways.

The user does not have to worry about “the backend”; about what is going on behind the curtain. This is a beautiful way to imagine the interaction between human and machine, and it was reflected, 25 years after the launch of Star Trek, in the inspirational writings of Mark Weiser, who is quoted at the head of this chapter. For a few years in the late 20th century, Weiser was one of the people driving the development of new devices at Xerox’s Palo Alto Research Centre (Xerox PARC). He had a notion for a different way to look at the design of HCI, inspired in part by the philosophical and psychological work of his occasional co-author John Seely Brown and many others [3]. This notion was that technology that can be used well does not require the conscious attention of the user. Rather than being the centre of attention, the well-designed tool sits at what Brown called “the periphery” [4]. The article Weiser published 25 years before this writing, the first of a half-dozen seminal works in the field, described the state-of-the-art of HCI at the time and discussed the psychological and technological changes that would be required for an advance towards the HCI he imagined would exist in the next generation. To summarize his core idea, the computer should allow humans to stay focussed on their tasks, rather than forcing them to focus on the computer’s own processes. Towards this end, PARC was developing portable computers which were limited in power and functionality, ranging in size between 1 m “boards” and 2,5 cm “tabs”. The power of these devices was that they were networked with a single core computer, and could be shared between users in a common environment. As he put it, “Even the most powerful notebook computer, with access to a world-wide information network, still focuses attention on a single box. By analogy with writing, carrying a super laptop is like owning just one very important book.” [1]

“Siri, where are the nuclear vessels?” “I’m sorry, I do not understand: ‘Fear art. See new, clear weasels’.”

This concept of a central entity in a common environment makes it easier to understand and interact with the system, not only for the characters in Star Trek (who are, sorry to destroy the illusion, fictional and portrayed by actors), but more importantly – for the more important “users”, the audience. This can be considered one important principle for the success of the series. It seems that – although the technologies presented to the viewers are unknown – they seem at least to be plausible and at best to be natural.

In contrast to some interface designers and IT engineers responsible for the design of current devices and interfaces, the screenwriters and scientific advisors of Star Trek seemed to have had human capabilities in mind and seem to have focussed on how to address those capabilities and their associated limitations, as well.

Although the variety of devices in our homes is now comparable to the variety of fictional ideas presented in Star Trek, we are far from achieving their smooth interoperation. Desktop computers in the home office, laptops on the dining table, tablet PCs and smartphones to move around, all have separate and different user experiences through different devices, interfaces and networks. This miasma of differences is also related to another philosophy brilliantly conveyed in Star Trek, another philosophy that humanity has not yet achieved. Compatibility, interoperability, and unified interaction, look, and feel all have an economic value that is important to competitive markets and the market economy, and we have not overcome the idea that these should be driving forces in the design of software and hardware. In that way, it seems as though we are actually closer to the Ferengi, who put profit-based economy before all other considerations, than we are to Starfleet and the belief that people are more important than profit. However, there have been some developments which could be considered first steps towards the 23rd century.

“Captain, I cannot raise the incoming craft. It is using an incompatible interface and an out of date codex.”

Those of you who witnessed the beginning of Star Trek might also remember the beginning of the Internet. When the World Wide Web (WWW) first became available to a broader population, one had to deal with browser compatibility issues. Fortunately, the “best viewed with browser X” issue was only temporary. It has almost disappeared (at least as an explicit statement). Thanks to developments such as the multiplatform Opera browser, there is no longer a general need to use specific devices or brands in order to access the “web” via a specific type of wired or mobile network. Instead, users can rely on the infrastructure to ensure that content and functions are appropriately rendered and presented. Although there are still some issues in supported technology (Flash, Java-script), things seem to be moving in the right direction.

In fact, the aforementioned Opera browser relates to the computer of the 23rd century in another important way. It is an example of community-based, open-source development – content and function are not developed a priori for business purposes, but are intended to advance the community. However, although software issues may be solved in the future, this is currently not the case in regard to hardware (e.g. when observing the situation of charging and data transfer adaptors for smartphones).

“Dammit Jim, I’m a doctor not a search engine!”

Another example which is not as mature as the multiplatform WWW, but still demonstrates respective possibilities, is data storage and retrieval. We mentioned universal access to data from every device and in each location over the holistic entity of the computer. This would implicitly require sophisticated data management.

Despite that implicit demand, we do not remember a single situation where a Star Trek crew member spent any time worrying about where to store their content, or from where to retrieve it again. To be honest, we do not even remember anyone explicitly pressing a “save” button or issuing an equivalent voice command. Is it possible that some critical information was “lost in space” because the users forgot to explicitly save it? Of course not; Autosave – which is also becoming common in several applications and websites that we use – was assumed to be a natural aspect of HCI in Star Trek. The question is – why isn’t this function also considered natural on our computers? What is speaking against it?

The same applies to retrieving content. Even the temperamental and outspoken Dr. McCoy never had cause to say: “Damn, where did I save that file?”. Can you imagine the following scene? A planet fills the display on the bridge, and the captain calls for information only to have Mr. Spock answer: “I am sorry captain, but I cannot find the relevant file. Might you be able to give me a hint as to where it could have been stored?”. No, every piece of information is available, every time, everywhere and in a format that is appropriate for every context.

However, there is also potential for enhancement of functionality in this regard; specifically in the context of the home. We have seen first attempts to move in the direction of Star Trek technology. Cloud-based storage enabling the management of public, shared, individual, or private data is available via our digital devices. Personal and shared clouds let users store their information and digital content on spaces that are accessible from different devices, contexts and locations. Beside the big players in IT, there are also enterprises whose market model is the provision of such spaces. This eases data management because one can have a cloud drive where all the content is accessible from each location and each device. Theoretically, this is like Star Trek’s “Computer”, but in reality, software add-ons are available for many platforms, and some manufacturers automatically support cloud based services, but most of the time only if they are accessed by their own devices. Again, this is not optimal, but it is a step in the right direction.

“Sir, the ensigns are refusing to wear red shirts and threatening to dox anyone who sends them on an away mission.”

Security and privacy issues are strongly related to the question of storing and retrieving content. We must assume that certain parts of the databanks of the Star Trek “Computer” were restricted and classified. Otherwise episodes in which heroes or villains tried to steal secret information would have involved hackers rather than infiltrators wearing fake ears or antennae.

But how was the security issue resolved in Star Trek? In the illusion provided, this was another feature the “Computer” was responsible for. It seemed to be natural that data would be stored with links to whatever security and privacy information would be relevant to their future retrieval. Who owns the data and who has access to it was therefore never explicitly addressed. However, such meta data has to be handled with great care in the cloud-based services here in the twenty-first century. People are frequently warned about putting work-related content into data clouds or using shared document editing. In some companies it is even forbidden. Password-secured systems do not seem to have high enough security. In Star Trek, when it came to accessing confidential data, or to issuing commands which might have significant consequences – such as self-destruction of the ship – the “Computer” asked for access information to make sure that the person issuing the command is both entitled to do the intended thing, and aware of the consequences of their endeavour. This was done by combining several security features rather than relying on a single one. That is, it was not enough to have the right password. That code had to be recognized, but so did the voice and irises of the speaker, for example. This combination of input signals made sure that a dramatic command could not be issued by the wrong person or by accident.

This combination, seen in Star Trek as a reflection of the security protocols used in the military and in early satellite communication, inspired us as designers of the user experience. It was clear that communication ought to feel natural to the user, but did ease of use have to limit security and robustness? If not, then current concepts of both natural HCI and robust security would have to be changed.

“Scanning the planet’s surface for Snarks, Captain, but detecting only Boojums.”

We have seen many developments in this regard. Finger print scans are integrated into keyboards, smartphones, door lock systems, and more, and iris scans are on the verge of entering the mass market. While these individual advances in commonplace security technology are impressive, what is missing is an equal advance in the common understanding of how security can be improved. Combining devices increases security while decreasing ease of use; but what if the combination could be designed based on natural human signals? This question, discussed in more detail elsewhere [5], led to the development of a system that could “Synchronize Natural Actions and React Knowledgeably” (S.N.A.R.K.) and a simple iterative circuit was designed to this end. This way, individual signals (whether detected mistakenly from noise, or accurately, but out of context) could be ignored and false positives could be avoided. In the early days of communication between the earth’s surface and satellites in orbit, the signal was unreliable, with a lot of noise. The solution was to use triple modular redundancy, that is, the solution was to send each message via three different channels. Receipt of a single signal could always be ignored, and receipt of three signals of equal meaning could always be trusted. Two signals of equal meaning triggered a query. In this way, the designers ensured that the interaction was always deliberate.

Not having a system like this would have meant that nobody in Star Trek would have been able to say the word “computer” in a normal conversation without accidentally having the second half of their sentence taken as a command.

An Upgrade from the Technicians of Cygnet-14

It might be worthwhile to take a look at one of the commercial products currently trying to bring voice interaction into the twenty-first century. The Amazon Echo© proposes the Star Trek-like idea of having a single holistic background entity in the home. Unfortunately, the Echo does not use the same name at home that it uses in public. It does not even use a neutral name like “Computer”.

Instead, by default, it has to be called “Alexa”. Clearly this was intended to give the system a kind of personal touch. Echo, or Alexa to her flatmates, seems to be a technological step towards the twenty-third century, at least in terms of digital features. The user can access information from the internet, and can stream music and audio books. Alexa also has access to a calendar, and to a weather forecast, and she can tell jokes and help the user edit shopping lists. Unfortunately, Amazon has not provided a commercial food replicator, though Alexa can start the oven remotely, so that the user can be welcomed by a ready-to-eat roasted pork when returning from a battle at the shopping mall’s delta quadrant. In fact, Alexa is able to control a number of devices in the home via interoperability features with smart home systems, which is wonderful.

However, Amazon seems not to have noticed the issue of accidently triggering commands. As seen in Fig. 1, this problem is clearly illustrated in a tweet from a would-be user who wrote: “My girlfriend’s name is Alexa, so this would accidentally trigger like 30 times a day.”

Fig. 1
figure 1

A tweet in response to the obvious problem of false positives in Amazon’s attempt to introduce a voice-operated digital assistant for help around the home (Retrieved from http://www.ibtimes.com/pulse/27-names-call-your-amazon-echo-instead-alexa-1720482)

Full size image

Fortunately, and as we have discussed, the solution is out there. Amazon just needs to go on a S.N.A.R.K. hunt.

Boldly Going Where No One Has Gone Before, in Order to Make It Accessible to Everyone

To summarize, despite the many examples of advanced HCI that already exist in the home, we are still very far from the twenty-third century, and there is a long way to go before we reach a level of HCI comparable to that seen in Star Trek. That said, one of the next steps has already been taken. It is now possible to have reliable and secure voice-based interaction that seems natural and intuitive to the user, provided designers and developers are willing to take the time needed to build it.

What about the step after that? We conclude this chapter with an excerpt from a book by one of the authors. This passage illustrates an imagined next step between the S.N.A.R.K. circuit and the computer of the twenty-third century, a next step inspired to no small degree by the examples of HCI in Star Trek:

“Sometimes it is the case that I go to bed after writing an article with the plan to do the final reviewing there. When my wife is awake it is no problem to take the e-book reader and read the produced outcome (with additional lights or without). Searching for the document takes some steps, because I have to go through the hierarchical structure of the storage, find the correct folder and file. In the case my wife has already fallen asleep, the procedure is different, because the manipulation on the e-book reader, the needed light and my movements would wake her up. The most unobtrusive way of reading a document would be to listen to it. On the technical level, this would not be a problem, if I had not forgotten to manually transcode the file into an audio, e.g. in mp3 format. However, the handling of the program streaming the audio file is based on visual and tactile interaction. I have to search the same hierarchy mentioned before to select the file, and this would also wake up my wife. Currently all the preparation would have to be done beforehand. A really natural way would be to handle content and information in a way as it has been demonstrated in Star Trek. Depending on the context, the available device and infrastructure one could spontaneously decide to interact with the holistic system in the backend and content is provided in the appropriate modality. This could even be established on the basis of components that are already present in a state of the art home, for example motion sensors. If, when going to bed, I were to recognize that my wife is asleep, I would not switch on the lights. I would put on a headset and, in my softest voice, whisper” James, recent”. The system would immediately respond by an audio stream of my recently edited document, because it is aware of my context and that I am not using a visual device. With the command” recent” it would scan my recent activities (similar to functions that are already offered in diverse software programs, but across different devices) and find the recently-saved document, where ever it would be, either on my office computer (or on a networked attached storage). Recognizing that the document is in PDF, the system would simply transcode it and play it to me - all without disturbing anyone else.” [6]

With the greatest of respect, we return now to Mark Weiser’s seminal paper on interaction [1] and end by reformulating the famous citation we quote in our opening lines: “Machines that fit the human environment, instead of forcing humans to enter theirs, will make using the computer as delightful as it is on Star Trek”.