1 Introduction

Tabletop groupware systems combine real-world work surfaces with computational interaction, allowing people to collaborate over digital artifacts while still maintaining a co-located face-to-face working style. Tabletop groupware is now becoming more common with the increasing availability of large-scale input and display technologies, and tables have been proposed for a variety of situations, including meetings, design work, games, and leisure activities. Since real-world tables are already common sites for collaboration, tabletop systems have several natural advantages for group work: people can communicate naturally over the table, can see one another work, and can use many of the same coordination mechanisms that they use in the real world.

Despite these advantages, many current tabletop groupware systems still have usability problems. These problems arise in part because of the differences in the way that collaboration is organized around tables, differences that designers of traditional applications are often unaware of or are ill-equipped to support. It is evident from observing work around a table that this kind of collaboration is built on different principles, for example, people work at different sides of the surface, which means that there is no absolute sense of direction as there is in a desktop application; similarly, the strong cues about where people are means that there must be different ideas about shared and personal spaces on a table than in a more traditional groupware system. The degree to which tabletop groupware supports the necessary activities and mechanisms of tabletop collaboration can be considered as a rough definition of tabletop-groupware usability.

One of the ways that tabletop usability problems can be found and addressed is through evaluation techniques that are specific to table-based groupware. In this paper, we introduce a usability inspection technique that incorporates an understanding of the basic actions and elements of tabletop collaboration. The technique is based on an existing groupware-evaluation framework, Collaboration Usability Analysis [1], and so is called Table-CUA (T-CUA). CUA is a formative groupware evaluation technique for analyzing real-world group tasks, and views group activity as being divided up into two areas: taskwork—the actions needed to complete the task and teamwork—the actions needed to complete the task as a group (i.e., the work of working together). Where singleware usability evaluations assess support for taskwork, groupware usability evaluations must also assess support for teamwork.

Existing groupware evaluation techniques, including CUA, were developed with a focus on distributed collaboration [2–4], and so the main new contribution in T-CUA is in adapting the CUA method to the specifics of tabletop collaboration. In particular, CUA analyzes collaborative tasks using the mechanics of collaboration—a set of collaboration primitives that specify low-level actions that are needed to carry out a task in a shared manner, such as communicating with other members of the group, keeping track of what others are doing, negotiating access to shared tools or empty spaces in the workspace, and transferring objects and tools to others. T-CUA adapts the mechanics of collaboration to better fit the realities of tabletop activities, and brings in a number of new issues that are specific to tables: the location of users around the display, the orientation of objects on the display, management of shared and personal spaces, co-present interaction techniques, and transfer of objects between co-present individuals.

In this paper, we introduce the Table-CUA technique, and report on a study that we carried out to compare the new method with a simple expert-inspection technique. We had experienced evaluators assess an early paper-based prototype of a tabletop groupware system, using either Table-CUA or expert review. We wanted to find out whether Table-CUA helped evaluators structure their inspection around issues that were more relevant to tabletop usability, whether people using T-CUA would be able to find more teamwork problems, and whether T-CUA was more difficult to use than the simpler technique. Our results provide evidence that Table-CUA is a better way to evaluate tabletop prototypes than simple inspection. Evaluators using Table-CUA were more focused on teamwork issues, found more usability problems, and found several types of problems that were not seen by any of the expert-inspection evaluators. In addition, most participants felt that T-CUA was more effective, and was as easy to use as the other technique. The main reason for Table-CUA’s success in our study is not surprising; it helps evaluators remember issues that are important in tabletop collaboration, and provides them with a structure in which they can look for usability problems related to each of those issues. Therefore, one of the main contributions of Table-CUA is the idea that in order to adequately design and evaluate tabletop groupware systems, we must have an understanding of the components and mechanisms of this type of group work, and must build these into tools and techniques that can be used by tabletop system designers.

2 Formative usability evaluation for groupware

In user-centered software engineering, developers iterate through a process of design, implementation, and evaluation. To be practical, it should be possible to carry out each step in the cycle quickly, easily, and at reasonable cost. Much of this iterative development is focused on the early detection and repair of usability problems. For maximum impact, evaluation techniques need to work with early low-fidelity interface designs. This is the reason why discount evaluation methods have been so well accepted, as they help evaluators rapidly find usability problems in even very early system prototypes. Popular examples include consistency inspections and standards inspections [5], pluralistic walkthroughs [6, 7], cognitive walkthroughs [8, 9], and heuristic evaluations [10, 11]. Although discount methods have limitations such as dissociation from the real work settings and a lack of a real theoretical basis, they have proven to be valuable tools in software development [12]. Even without real users in the actual work setting, these methods have become successful because they provide evaluators with enough detail about the work and task context for them to find legitimate usability problems.

Although developing groupware interfaces is similar in many ways to developing interfaces for traditional single-user applications, standard discount evaluation techniques do not work well for groupware. The main problem is that discount evaluation methods are strongly oriented toward individual work: the contextual information they provide and the criteria they use for judging usability are focused on tasks individuals must carry out while working toward a goal, and social aspects of work such as communication and coordination are overlooked.

Recently, several discount methods have been proposed for assessing groupware usability [1–4, 13]. These methods view group activity as being divided into two areas: taskwork, the actions needed to complete the task, and teamwork, the actions needed to complete the task as a group. Where singleware usability evaluations assess support for taskwork, groupware usability evaluations must assess support for teamwork. This becomes our definition of groupware usability: the extent to which a groupware system allows teamwork to occur—effectively, efficiently, and satisfactorily—for a particular group and a particular group activity [1]. Groupware evaluation techniques now include basic inspection [4], a walkthrough technique based on cognitive walkthrough [2], and a variant of heuristic evaluation [3].

Most groupware evaluation techniques were partially developed from studies of distributed groupware systems [2–4], and many of the issues that are important in tabletop groupware systems (e.g., orientation, territoriality) are not explicitly addressed. Furthermore, studies have not yet been carried out to assess how effective existing techniques are in evaluating tabletop systems.

3 Collaboration usability analysis

In this paper, we adapt a groupware evaluation technique called Collaboration Usability Analysis (CUA) so that it can be used to evaluate tabletop groupware systems [1]. CUA focuses on the teamwork that goes on in group activities rather than the taskwork, and it is based on the notion that real-world activities can provide a sound basis for assessing how well teamwork support is provided in an interactive system.

The analysis framework for CUA has two main parts: the mechanics of collaboration and a set of typical actions (see Table 1). The mechanics of collaboration are a set of group work primitives [14] that represent the basic operations of teamwork—the small-scale actions and interactions that group members must carry out in order to get a task done in a collaborative fashion. They are the things that will be common to a shared task even under a variety of social and organizational factors, such as communicating with other members of the group, keeping track of what others are doing, negotiating access to shared tools or empty spaces in the workspace, and transferring objects and tools to others. The mechanics are a useful level of analysis for evaluation because they provide a fine-grained view of teamwork, allowing evaluators to consider collaboration support in groupware designs.

Table 1 The mechanics of collaboration (adapted from Pinelle et al. [1]
Full size table

The mechanics cover two general types of activity: communication and coordination. Communication is broken into two categories: explicit communication and information gathering, and coordination is broken into two categories: shared access and transfer. Table 1 shows the mechanics that are associated with each of these categories, and each mechanic is described in more detail elsewhere [1].

The mechanics of collaboration do not provide a complete picture of task execution. They describe commonly encountered collaborative primitives, but do not provide a concrete description of how collaboration takes place in a given context. Therefore, we include an additional specification in the analysis framework. Each mechanic can be carried out through a set of possible actions that describe how the mechanic is typically carried out in the real world. For example, if a person gestures to indicate an item on a whiteboard, common actions for accomplishing this may be drawing or pointing (both actions related to the gestural messages mechanic). Either of these actions is sufficient for accomplishing the intended goal; therefore, actions are usually presented as a list of reasonable alternatives (see Table 1).

CUA has shown promise as a technique for evaluating groupware since it describes real world group activities using a mechanical level of analysis, a level that allows inspection for usability problems [4, 14]. In the next sections, we describe how we adapted CUA for use with tabletop groupware systems. The main changes involve a reinterpretation of the mechanics to better fit the realities of tabletop collaboration, and a new set of actions that are derived from observations of tabletop work. Both of these changes are based on an understanding of the structural differences between tabletop collaboration and other forms of shared work; in the next section, we develop the foundation for these adaptations by looking more closely at these differences.

4 Differences in tabletop collaboration

When people engage in activities at tables, they are able to arrange their work in ways that are not usually seen in distributed work. It is easy to work together in a highly integrated fashion—face to face communication is possible and people can directly observe others’ activities, making it easy for them to coordinate their actions. However, tables place some constraints on how people can collaborate since they share a single working surface and have different perspectives of the table and of shared items.

4.1 Visibility of people, artifacts, and actions

Working in close proximity at a table allows people to directly observe others and to maintain an up-to-date awareness of their activities [15]. This makes it relatively easy for them to coordination their actions—each person can adjust their activities based on the actions of others, and can explicitly plan their activities when more intense coordination is needed. Similarly, working in close proximity makes it easier to initiate conversations since it is clear when others are available, and face-to-face communication enables them to resolve issues while minimizing the potential for ambiguities and social misunderstandings.

4.2 Orientation

In tabletop work, each person views the table surface from a different position, and shared items must often be repositioned so that individuals can view and interact with them. Kruger et al. [16] found that collaborators use orientation to comprehend information, to communicate, and to coordinate work. Orientation enables comprehension since repositioning an object enables individuals to easily read an objects or carry out a task using the object. Changes in objects’ orientations also communicate information to others—when an object is turned toward an individual, it is clear that he or she plans to use that object. Similarly, when an individual turns an object toward others, it is clear that any ensuing verbal and gestural communication will probably refer to the object.

4.3 Partitioning the workspace

In tabletop activities, individuals collaborate around a common work surface, and they informally partition the space into separate areas that support shared and individual work. Scott et al. [17] found that groups automatically divide the space on the tabletop into three territories: personal, group, and storage. The territories are not necessarily mutually exclusive, and they are partially based on users’ positions around the table. Personal territories are used to support individual work, and each individual’s personal territory is usually located directly in front of them. Group territories and storage territories are shared spaces; group territories are usually located outside the personal territories, often in the center of the table. Storage territories, which are used to store task resources, often overlap other territories and migrate around the table.

Kruger et al. [16] found that collaborators also partition the space on the table using the orientation of artifacts. Artifacts that are oriented toward an individual help to establish their personal workspace, and artifacts that have a neutral orientation help to establish the group workspace. They also found that orientation has a strong association with the ownership of artifacts, and when artifacts are in close proximity to a person and are turned toward them, others see them as being owned by that person. When artifacts are oriented toward others or have an arbitrary rotation, people tend to see them as being available.

4.4 Highly integrated coordination

The close proximity interactions that take place around tables can allow people to carry out activities in a highly integrated fashion, where people work closely together to achieve group goals. Several common tabletop activities can be carried out in a highly integrated fashion, such as brainstorming, group design tasks, and project planning. In previous work, we studied highly integrated activities on regular tables, and discussed several work patterns that are common in this style of collaborative work [15]. First, we found that people regularly reached into others’ personal spaces to take items. Second, we found that people moved around the table and changed their personal workspace so that they could coordinate their activities more closely with others. Third, we found that when people had to work in a limited work area (e.g., on a particular artifact or on a set of artifacts), they had difficulties managing access to the area due to space limitations and orientation problems.

5 T-CUA: adapting CUA for tabletop groupware

CUA has the potential to be useful in carrying out formative usability evaluations of tabletop groupware systems. The mechanics of collaboration can be applied to a wide range of collaborative situations, and provide a good starting point for considering how tabletop systems can be evaluated. However, one of the main criticisms of the mechanics is that they are not concrete and do not describe specific actions, making them difficult to operationalize when designing and evaluating groupware systems [18].

We developed Table-CUA (T-CUA), a formative usability technique for tabletop groupware, by modifying the actions component of the CUA framework so that it specifies common operations that are used for collaboration in tabletop work. This provides a concrete link between the mechanics of collaboration and the typical ways that activities are carried out on tables. We developed the actions from observational studies of group activities on regular tables [15–17, 19], and they are primarily based on the four characteristics discussed in the last section. The tabletop actions provide a way for evaluators to explore the possible collaborative situations that can arise when a system is deployed, and can help them identify instances where a groupware design does not provide adequate consideration for common patterns of interaction on tables. Table 2 presents the modified set of actions.

Table 2 T-CUA framework with actions for tabletop groupware applications
Full size table

The explicit communication mechanics specify how people engage in intentional communication. The T-CUA actions for this category are summarized below:

  • Spoken messages: People can communicate face-to-face, so social conversations and verbal communication about artifacts and activities are common.

  • Written messages: People can annotate objects and can share documents such as minutes, agendas, and plans.

  • Gestural messages: People can communicate by changing an object’s orientation to draw others’ interests or to assist with comprehension, and they can gesture to indicate interest or to ask others’ to pass an item or a tool.

  • Combinations of verbal and gestural messages: People can combine both verbal and gestural messages.

Information gathering mechanics specify how people stay aware of the activities of other group members.

  • Basic awareness: People need to stay aware of others’ positions and actions on the table, and of the objects they are using in their activities.

  • Activity information from objects: People can obtain information about others’ activities through the orientation of an object or by its proximity to others.

  • Activity information from people’s bodies: People can obtain information about others’ activities when they interact with objects, interact with other people, or change their position at the table.

The shared access mechanics specify how people coordinate access to objects and space.

  • Obtain resource: People can obtain objects by taking them from several areas of the table: distant locations, group territories, or others’ personal territories; they can also ask others to hand them an object.

  • Reserve resource: People can reserve resources for future use by moving an object into their personal space, by rotating an object so that it faces them, by putting a hand on an object, by holding and object, or by moving next to an area on the tabletop.

  • Protect work: People can protect their work from others’ interference by moving next to an area, by marking an object to indicate ownership, by working with an object, or by keeping objects in their personal spaces.

The transfer mechanics specify how individuals transfer objects to another person.

  • Handoff: People can synchronously hand an item to others by putting it directly into their hand, by putting it in front of them, or by rotating and translating the object to them.

  • Deposit: People can asynchronously leave objects for others by moving them into group territories or into another person’s personal territory, by rotating an object so that it faces another person or has an ambiguous orientation, or by moving away from a protected area.

6 Evaluation of T-CUA

We evaluated T-CUA in a study where 12 participants used the framework to carry out a usability inspection of low-fidelity paper prototypes of tabletop groupware applications. We were interested in learning whether T-CUA would allow evaluators to identify usability problems related to teamwork support, and whether those problems were qualitatively different than those found with expert review. Furthermore, we wanted to assess whether evaluators could easily incorporate concepts from the T-CUA framework into usability evaluations.

We compared T-CUA with informal expert review, a common approach for carrying out usability evaluations [12]. In expert reviews, evaluators rely on their knowledge of usability principles to help them identify usability problems. Expert review provides evaluators with a flexible approach that allows them to incorporate their knowledge of teamwork and tabletop collaboration into the evaluation process. We use this technique in our study by training people on basic collaborative issues seen in tabletop systems (orientation and territoriality), and by encouraging evaluators to consider taskwork issues when carrying out the evaluation.

6.1 Participants

Twelve participants, ten male and two female, were recruited from a local university. Participants ranged in age from 21 to 45 years (mean 29.25 years). All had some past experience with conducting usability evaluations: three had professional experience with conducting evaluations, and nine had experience with conducting evaluations in either research or classroom projects. Eleven had some experience with using groupware, and ten had experience with using digital table systems.

6.2 Procedure

Each participant evaluated two low fidelity prototypes. Half of them used T-CUA to evaluate the first prototype, and half used expert review. Participants evaluated the second prototype using the other evaluation scheme; therefore, all participants saw both techniques (although in different orders) (see Table 3). All participants were given scenario descriptions that helped to frame the goals of the prototype design. Scenarios described the activity that the system was designed to support, the typical users of the system, the outcome users hoped to achieve by undertaking the activity, and the role the prototype was designed to play in the activity (see Table 4).

Table 3 Evaluation sequence by participant group
Full size table
Table 4 Scenario description from the house design prototype
Full size table

At the beginning of the session, the participant was given training on carrying out an expert review and on using scenario descriptions during usability evaluations. They were also given a brief tutorial on tabletop groupware systems and on the role that orientation and territoriality play in tabletop collaboration. All participants were also given training on T-CUA. Group 1 received this training after evaluating prototype 1, and group 2 received the training prior to evaluating prototype 1. The training included a brief overview of the mechanics of collaboration and a description of the tabletop actions that are shown in Table 2. Training sessions lasted for approximately 15 min.

6.2.1 Expert review

Participants were given the scenario description for the prototype, an instruction sheet detailing the steps to follow in the evaluation, and a set of problem sheets. They were asked to review the scenario, and they were given a brief orientation to the prototype, where the design goals and main features were explained to them. They were told to identify areas where the prototype did not provide adequate support for the collaborative activities that were described in the scenario. They were asked to draw on their experiences working with groups to help identify other problems with the design that might interfere with collaboration between groups of users. They were instructed to record all problems on the problem sheets. Participants were given approximately 45 min to complete the evaluation.

6.2.2 T-CUA

At the start of the T-CUA evaluation, participants were given a scenario description, an instruction sheet, a handout containing the T-CUA framework shown in Table 2, and problem sheets. Similar to the expert review evaluation, they were asked to identify mismatched support for the scenarios, and to rely on their experiences working in groups to help them identify problems. However, they were also told to use the T-CUA framework to help identify usability problems. They were asked to review each mechanic and its associated actions, and to identify instances where the prototype did not provide adequate support. They were told to write all problems that they identified on the problem sheets, and if a problem was identified using a mechanic, to record the name of the mechanic next to the problem description. Participants were given approximately 45 min to complete the evaluation.

At the end of the study, participants completed a questionnaire that asked them to describe the strengths and weaknesses of each technique. They were also asked whether they could identify problems more effectively with one technique than the other, and they were asked which technique was easiest to use.

6.3 Prototype 1: design the layout of a house

Participants were given a 28 × 22 in poster board that contained a low-fidelity paper prototype of a tabletop groupware system that allowed users to design floor plans (see Fig. 1). The prototype showed the user-interface components in the system and examples of how the system could be used to create a simple floor plan. Descriptions and drawings were included to show how users could interact with the workspace, artifacts, and controls. Table 4 shows the scenario description that was given to participants when they evaluated the prototype, and Fig. 1 shows the low fidelity prototype.

Fig. 1
figure 1

Prototype 1: tabletop system for designing the layout of a house

Full size image

In the prototype, each user has a separate mouse and keyboard. The system does not differentiate between personal and group territories, and all of the space on the table can be used to create the diagram. All users must share two toolbars. The “Shapes Toolbar” allows users to drag and drop predefined shapes into the workspace, and it includes representations for common diagram components: doors, walls, stairs, and windows. Each shape on the toolbar has two identical representations, but one is rotated 90° from the other, and users can choose either representation when adding a new shape. The “Control Toolbar” allows people to change system settings or to change the type of interaction they can perform with the system. They can use the toolbar to change input modes so that they can draw or write in the workspace. They can also delete existing objects, and move objects (but there is no support for rotation), and the color slider allows people to specify the color that will be used by the draw tool.

6.4 Prototype 2: plan a software project

Participants were given a poster board that contained a low-fidelity paper prototype of a tabletop groupware system that allowed users to create a PERT chart to show the task sequence and dependencies in a software project. In the prototype, each user has a stylus that they can use to interact with the system. The table does not differentiate between personal and group territories, and all of the space on the table can be used to create the PERT chart. Each user has a separate toolbar that they can use to add nodes and connectors to the diagram, and to change the type of content that they can add using the stylus. Toolbar settings allow users to draw and write on the workspace, and to delete, move, resize, and rotate existing content.

6.5 Study design and analysis

Participants were divided into two groups: one used T-CUA and then expert review, and the other used expert review and then T-CUA. We balanced participants between the groups according to their professional and academic experience with carrying out usability evaluations. While this could not be done precisely, we attempted to guarantee that we had an equal number of expert and novice evaluators in each group.

We made an a priori decision to focus our analysis primarily on results from prototype 1. In our discussion of study results in the next sections, we include prototype 2 data in some analyses for the sake of completeness, but we feel that the data from prototype 1 provides a more valid measure of participants’ performance since fatigue and training effects were not an issue.

We conducted a basic analysis on the results from both prototypes and an in-depth analysis on the results from prototype 1. In the basic analysis, we reviewed the problem lists for each participant. Problems that were repeated in their evaluation of a prototype were removed. We also removed problems that were not really problems, which consisted of two classes: (1) those that reflected a misunderstanding of the design that was specified on the prototype, and (2) those that did not describe a specific usability problem. We coded each of the remaining problems according to whether they referred to teamwork issues or taskwork issues. We analyzed problems from prototype 1 in more detail, and classified them according to the feature they addressed (e.g., delete function, color control) or according to the issues that they addressed (e.g., protection, transfer).

We did not address problem severity in our analysis. We felt that our focus on teamwork made it is difficult to judge severity since group collaboration patterns are highly variable, making it challenging to determine how extensively a problem will interfere with people’s interactions with the system. Also, issues such as group size and composition can play a significant role in how the system is used and in how pervasive some usability problems will be within a given session. Furthermore, since the prototypes did not provide actual interactive functionality, it was more difficult to explore the system and to envision how extensively design decisions would impact the overall usability.

7 Findings

All evaluators finished their tasks within the allotted time, and did not have any major difficulties with two evaluation techniques. All evaluators found at least some usability problems in the prototypes: evaluators found an average of 10.5 problems per prototype, although there was some variation across evaluators.

We organize our results into two main parts: first, we ask a set of questions that consider the effectiveness of the two evaluation techniques, and look at issues such as number and type of problems found; second, we ask a set of questions about participants’ experiences using the two techniques, and discuss subjective feedback that they provided about the methods.

7.1 Performance differences

7.1.1 How many problems were found with each technique?

Participants identified more problems using T-CUA than they did with expert review. Overall, evaluators identified an average of 11.33 problems across both prototypes using T-CUA, and 9.75 problems using expert review. In prototype 1, an average of 13.2 problems were identified by each evaluator using T-CUA, and 10.17 were identified using expert review (see Fig. 2). In prototype 2, an average of 9.5 problems were identified by each evaluator using T-CUA, and 9.3 were identified using expert review. We carried out an ANOVA test with the data from prototype 1 to determine if the differences between T-CUA and expert review were reliable. For the overall number of problems, there was not a significant main effect of evaluation technique (F1,10 = 1.08, p = 0.323).

Fig. 2
figure 2

Average number of problems found by evaluators for both prototypes

Full size image

7.1.2 Which technique found more teamwork problems?

All participants were asked to focus on teamwork problems, regardless of the technique they were using. However, participants identified more teamwork problems using T-CUA than they did with expert review. Overall, 83% of the problems identified using T-CUA referred to teamwork issues, and 55% of the expert review problems referred to teamwork issues (see Fig. 2). We carried out an ANOVA test with the data from prototype 1 to determine if the differences between the numbers of teamwork problems were reliable. For the overall number of teamwork problems, there was a significant main effect of evaluation technique (F1,10 = 10.71, p < 0.001).

We believe the main reason that people using expert review found fewer teamwork problems is simply that it is difficult to remember and keep in mind the group-related usability issues that could come up in the use of a system. Very few visible features of the system (or of the scenario description) relate to teamwork, and the evaluation therefore required that evaluators imagine all the ways that the system would be used by groups. This is difficult to do (at least, perhaps, without extensive experience observing groups in similar situations), and as a result, we believe our expert-review evaluators simply ran out of ideas relating to teamwork. In contrast, T-CUA provides a series of reminders about types of interaction, which gives evaluators a much larger set of starting points for their inspection. Although evaluators still have to imagine group situations, T-CUA provides them with a set of guidelines (e.g., consider whether and when groups will use handoff) that can make the process easier.

7.1.3 What were the differences between problems found with each technique?

We analyzed problem descriptions from prototype 1 in detail to determine the qualitative differences between the problems that were reported using each technique. We began by categorizing problems according to the issues that they address. The categories were developed from the themes that we found in the problem descriptions—many of them were classified according to the mechanical actions that they describe (e.g., communication, information gathering, etc.), and others were classified according to the specific system feature that they address, such as the color control or the resize control. The problem categories are organized according to whether they describe teamwork or taskwork problems and are shown in Fig. 3.

Fig. 3
figure 3

Total number of problems, by category and technique, for prototype 1

Full size image

Taskwork problems primarily refer to existing features in the prototype. The first three problem categories, resize, color control, and delete, include problems that focus on controls that are included in the interface. The fourth category, need new controls, includes problems that deal with the need for additional controls to allow users to interact with the interface. The fifth category, managing distance, includes problems related to the difficulties people may have with viewing and interacting with objects and controls that are not physically close to them on the display. All other taskwork problems are included in the miscellaneous category.

Teamwork problems focus on insufficient support for common collaborative actions, which is not surprising given that T-CUA dominated the range and types of teamwork problems that were found. Most of the categories correspond with the mechanics of collaboration, including communication, information gathering, obtain resource, reserve resource, protect work, and transfer. Two other categories were also added, orientation and control conflicts. The orientation category includes issues related to the orientation of taskbars, artifacts, and workspaces, and it was included as a separate category since some orientation issues were pervasive and could interact with several different mechanics. The final category, control conflicts, refers to instances where system features could allow people to interfere with others’ activities. For example, a global color control could allow people to change the color that all people used when drawing. This category was added since control conflict problems were not specifically related to any of the mechanics.

Tables 5 and 6 list the problems that were identified for prototype 1. Table 5 shows that a large number of taskwork problems were identified using expert review. While T-CUA did identify some taskwork problems, many of them overlap with those identified by expert review, suggesting that they may have been obvious and easy to identify. Many of the taskwork problems deal with features that are identified explicitly in the prototype, and do not deal with areas of support that are missing in the design (with the exception of the new controls category), suggesting that both techniques may be limited in their abilities to identify less obvious taskwork problems in tabletop groupware systems.

Table 5 Taskwork problems from prototype 1
Full size table
Table 6 Teamwork problems from prototype 1
Full size table

Table 6 shows that a large number of teamwork problems were identified using T-CUA. Furthermore, most of the collaborative problems that were identified were uniquely identified by T-CUA, and many of these could not be directly inferred by inspecting the prototype without having an understanding of tabletop collaboration issues. For example, the need for personal regions where people can carry out individual work is not directly obvious by critiquing user interface features in the prototype, but the T-CUA framework emphasizes personal territories, making it easier for evaluators to identify their absence in the design. Participants using expert review were consistently able to identify two classes of collaboration problems: orientation and control conflicts. Orientation issues were obvious to participants, regardless of the technique they used. More control conflict problems were identified using expert review than with T-CUA, and this in part seems to be the result of the expert review evaluators spending more time finding problems using the scenario. The first two control conflict problems came directly from the scenario, and the second two seem to come from feature inspection.

7.1.4 Within T-CUA, how were the mechanics used?

Participants were asked to identify the mechanic that they used when they wrote problem reports during the T-CUA evaluation sessions. Figure 4 shows the total number of times each mechanic was recorded in problem descriptions from T-CUA evaluations for prototypes 1 and 2 combined. All mechanics were used during the evaluation sessions, but four were used more frequently: gestural messages, basic awareness, obtain resource, and protect work.

Fig. 4
figure 4

Total number of problems found using the mechanics during T-CUA evaluations of prototypes 1 and 2

Full size image

Some mechanics may be underrepresented since they are similar to other mechanics. Also, the sequencing of the mechanics in the T-CUA framework may play a role in the frequency with which some mechanics were cited. For example, basic awareness is similar to activity information from objects and from bodies, and it comes before the other two on the T-CUA handout. Therefore, people can record many information gathering problems under basic awareness. Similarly, handoff and deposit are both similar, and handoff, which comes first in the sequence, is cited more frequently than deposit.

Since we do not have severity ratings for our problems, it is impossible to determine the usefulness of each mechanic based on the number of times it was used. It is possible that the problems that were found using the mechanics that were used infrequently are important, and will have a strong influence on the overall usability of the system. Future work is needed to investigate this issue in more detail.

7.2 Subjective differences

7.2.1 Which technique did participants feel was the most effective?

In the questionnaires, most participants indicated that they felt T-CUA was more effective at identifying problems than expert review. Eight participants indicated that they felt T-CUA was most effective, one participant preferred expert review, one participant felt there was no difference between the techniques, and two participants indicated that each techniques was effective, but with different strengths.

Some of the people that chose T-CUA provided additional comments in the questionnaire. Five people indicated that T-CUA helped them to consider collaborative issues that they might otherwise overlook. For example, one participant wrote, “CUA forces you to review the different categories and issues one by one,” and another wrote, “The framework gave different perspectives that would have taken a bit more time for me to come up with on my own.”

Two people indicated that both techniques were useful, but that each has different strengths. One person wrote that, “Expert review was good for common sense, CUA was good for common cases.” The other person wrote that expert review “did not restrict the findings, so it might be faster to find problems (at least initially) using it. However, CUA would probably lead to more problems being uncovered given a longer evaluation period.”

7.2.2 Which technique did participants feel was the easiest to use?

Most of the participants indicated that they felt that T-CUA was easier to use than expert review. Seven people felt that T-CUA was easiest, three chose expert review, and two felt that there was no difference between the techniques. The three that felt that expert review was easier to use pointed out that it was less structured and less demanding than T-CUA. One person indicated that they preferred expert review because “identifying problems were not restricted by a set of guidelines.” Another wrote that, “Expert review is, in general, less demanding. The structure imposed by CUA is a little tiring sometimes.”

In contrast, those that chose T-CUA indicated that the structured nature of the approach made it easier to use than expert review. One person wrote that, “it focused the evaluation a lot more,” another added that “the list in CUA makes it easier to consider specific usability issues,” and a third person wrote that T-CUA was easier because “it gives you hints where to center your attention.” Finally, one person wrote that “CUA was easier and probably more thorough. It provides a fairly comprehensive lot of things to check for, and there was less uncertainty that something was being overlooked. I don’t think there was anything that expert review found that CUA wouldn’t.”

8 Discussion

8.1 Summary

Our results show that T-CUA can help evaluators to identify usability problems related to teamwork support in tabletop groupware prototypes. The main reason for Table-CUA’s success in our study is not surprising—it helps evaluators remember issues that are important in tabletop collaboration, and provides them with a structure in which they can look for usability problems relating to each of those issues. In subjective results, this was cited as one of the main strengths of the technique—that it helped evaluators to think about collaborative issues, and to identify problems that they might otherwise overlook. Therefore, one of the main contributions of Table-CUA is the idea that in order to adequately design and evaluate tabletop groupware systems, we must have an understanding of the components and mechanisms of this type of group work, and must build these into the tools and techniques that are used by tabletop system designers.

One possible criticism of T-CUA is that the structure of the analysis framework may force evaluators to view collaboration in a fixed, predefined way that can limit their ability to identify some collaboration problems and cause them to overemphasize others. However, just as with other usability inspection techniques, it is important for evaluators to interpret the assessment criteria in the context of the usage scenario.

In our study, we urged participants to carefully consider whether actions and mechanics should be supported in the system, and to avoid simply writing down a problem for every action and mechanic. Some of the problem descriptions from prototype 1 show the T-CUA found problems that are less obvious and that show considerable insight (see Table 6). For example, one evaluator pointed out that the absence of personal territories and the lack of support for object rotation could make it difficult for people to determine the availability of objects. This suggests that, if applied properly, the T-CUA framework can act as more than a checklist—it can help people to consider how the design will support common collaborative actions so that they can identify problems that are not obvious in an unstructured inspection.

8.2 Generalization

We developed CUA as a formative usability evaluation technique, and expect T-CUA to be used mainly in the evaluation of early prototypes. The main limitation of usability inspection techniques is that they are removed from the context of use. In groupware design, this means that it can be difficult for evaluators to consider social, political, and organizational issues in a given setting. However, usability inspections can play an important role in the iterative design process when a design is in early stages and has limited or no functionality, making it difficult to conduct more realistic testing.

T-CUA is likely to be easy to apply for people that have training in usability evaluation and a basic understanding of tabletop collaboration. In our study, most participants were able to identify several problems, regardless of variations in their experience levels. While most of our participants had some experience with tabletop systems, this experience was usually very limited, and most did not have a deep understanding of tabletop groupware design issues. We provided them with a total of 15 min of training on orientation and territoriality on tables, and on the T-CUA framework, and all participants were then able to use the evaluation without any noticeable difficulties. Therefore, the required level of specialized training needed to use T-CUA is low, and the technique should be accessible to usability professionals, regardless of their past experiences with tabletop systems.

In spite of our focus on early, formative evaluation, we feel that the T-CUA framework can also play an important role in focusing usability evaluations at later stages of development. The needs of designers may vary over the course of development, and the T-CUA framework can potentially be adapted so that it can be used in different ways by evaluators. For example, we previously described how the CUA framework can be used to carry out organized walkthroughs of prototypes, where evaluators simulate collaborative tasks to identify usability problems [2]. Also, Baker et al. [3] used the mechanics of collaboration to develop a heuristic evaluation technique for groupware. We feel that T-CUA can be used to develop similar techniques, so that evaluators can better consider tabletop collaboration issues at all stages of development. In the future, we plan to investigate these extensions in more detail.

8.3 Boundaries

The findings of the study show that while T-CUA was useful in identifying teamwork problems, it did not identify as many taskwork problems as expert review. Furthermore, the taskwork problems identified using T-CUA were more limited in scope, and did not cover as many problem categories. This is not surprising since T-CUA evaluations were primarily focused on uncovering teamwork issues using the framework. However, these findings suggest that T-CUA is not sufficient for covering the range of usability problems that can arise on tables, and that usability evaluation plans for tabletop groupware also need to include techniques that can address taskwork issues. One possibility is to include both expert review and T-CUA as part of the evaluation strategy so that coverage is provided for both teamwork and taskwork problems. There are also a range of other single-user evaluation techniques that may be useful in addressing taskwork issues on tables, such as task analysis [20, 21], cognitive walkthroughs [8, 9], and heuristic evaluation [10, 11]. However, the usefulness of these techniques in evaluating tabletop systems is still unclear, and it is possible that custom taskwork evaluation techniques may be needed for tables.

The T-CUA framework was based on studies of regular collaboration on real tables. However, tabletop groupware systems create opportunities for people to interfere with people in ways that would not normally occur in unsupported activities. For example, design decisions can make it easier for people to act in ways that would normally be restricted by social protocol on regular tables. These design decisions can make conflict more likely, and can cause teamwork usability problems. However, since T-CUA is based on studies of unsupported work, control conflicts are not explicitly addressed. In our study, people identified more control conflicts using expert review than with T-CUA, and this was the only teamwork category where we saw this difference.

For T-CUA to be effective at covering the range of collaborative problems that can be present in tabletop groupware, it needs to be modified to address control conflicts. However, these conflicts do not easily fit within the existing organizational structure of the mechanics of collaboration. Conflicts are related to the coordination section of the framework, but a conflict is not a mechanical action. We propose addressing this by adding a new category that exists outside of the framework shown in Table 2. We show the proposed addendum in Table 7—it does not include a mechanic, but rather a category name, and it contains a set of possible conflicts. We derived the list from the problem descriptions from prototype 1. The possible conflicts can be used in the same way that actions are used during a T-CUA evaluation. Evaluators can inspect the prototype in an effort to identify potential control conflicts using the list shown in Table 7.

Table 7 Control conflict addendum to the T-CUA framework
Full size table

9 Conclusions

In this paper, we introduce Table-CUA (T-CUA), a usability inspection technique that incorporates an understanding of the basic actions and elements in tabletop collaboration. T-CUA provides a set of low-level, concrete actions that describe how collaborative activities are carried out on tables, and it was developed to help evaluators to identify instances where designs do not provide adequate support for the collaborative actions that users are likely to carry out using the system. We carried out a study to compare T-CUA with a simple expert-inspection technique. Our results provide evidence that T-CUA is a better way to evaluate collaboration support in tabletop systems than simple inspection. Evaluators using T-CUA were more focused on teamwork issues, found more usability problems, and found several types of problems that were not seen by any of the expert-inspection evaluators. In addition, most participants felt that T-CUA was more effective than and as easy to use as the other technique.

In the future, we plan to carry out further studies on the evaluation of tabletop groupware systems. We plan to investigate how T-CUA can be expanded to deal with variations in table size and group size. Further work is also needed to determine the average number of evaluators that are needed to uncover the main usability problems that exist in a given prototype. We feel that T-CUA can play an important role in identifying teamwork problems, but we are also interested in investigating how it can be integrated into a more comprehensive evaluation strategy that addresses taskwork as well. Furthermore, we are interested in investigating possible future extensions to T-CUA so that control conflicts can be identified more effectively. We also plan to investigate the role that T-CUA framework can play in developing other types of teamwork evaluations for tabletop groupware systems. In particular, we feel that the framework can be used as a basis for developing heuristic evaluation techniques and walkthrough-based evaluation techniques, and that this can help to provide a range of options for evaluating tabletop groupware.