1 Introduction

The computerization of a business may drastically change user activities for the better or for the worse. A general recommendation from research in usability and user-centered design is to involve users in the production of the system design in order to make the computer systems as usable as possible (Constantine and Lockwood 1999; Cooper 1999; Dix et al. 1997; Greenbaum and Kyng 1991; Helander et al. 1997; Newman and Lamming 1995; Preece et al. 1995; Shneiderman 1998). There are various recommended guidelines and methods for performing such analysis, but generally these methods focus on how the systems development organization should involve a user perspective or adhere to usability guidelines. However, a problem often encountered is how to integrate usability in a general system-development process (see especially Carlshamre and Rantzer 2000; Grudin 1991; Gulliksen et al. 2003). When integration fails, the result is often that usability aspects are marginalized and not prioritized. The arguments for positioning usability and user-centered design in the developmental organization are that many requirements emerge during the design process and that there are mutual constraints created by user requirements and systems architecture. Still, besides the problems of integration, research is clear that user involvement as well as top management support is the most important success factors for systems development (Ward and Peppard 2003; Standish Group 1995). We are therefore interested in how procurer organizations reason about, as well as organize for, usability and its integration for tailored systems design.

Research has focused more on in-house development and product development and less on contract development. This may explain why procurement and procurer perspectives on usability have received less attention. However, contract development is a distinct developmental context. Users and the developers work in separate organizations, which may make user-involvement less salient or prioritized, at least for the supplier organization (Grudin 1991). Furthermore, contract development is constrained by the initial contract, which puts more pressure on the procurement organization to define and communicate system and development requirements in advance. The problem, of course, is that a developer organization will not do more than what is agreed upon, since to do so would constitute a breach of contract. Thus, in contract development, the procurer organization’s understanding and requirement of usability are of importance. We need to learn more about how procurer organizations understand, require, and go about doing usability. If usability and user-centered design has had difficulties in finding their way into general systems-development practice, the procurer’s role in defining usability in contract development is even more poorly understood.

There are relatively few studies of procurement in general and very few studies specifically focusing on usability procurement, which was acknowledged as early as 1994 by Buie. These researchers focused on the specific procurement procedure of government contracting, which undermines the possibility of negotiating development incitements between a request for tender and actual contract. The lack of opportunity to negotiate between a request for tender and an actual contract points to the problem that the procurer organization has to actually require usability and user involvement from the development organization or conduct independent user analyses before actual contracting, which in any case requires knowledge about usability and usability methods. Usability therefore depends on the procurer organization’s ability to see the benefit of doing or requiring usability at an early stage of the development. In such acquisition procedure, the question of usability is moved back and higher up in the decision-making management structure, and thus financial accountants are made responsible for making resources for usability available. Scown (1998) found that accountants at procurement organizations had too little knowledge of usability, which limited usability perspectives in the request for tender and calculations in return of investments. Svensson (2002) studied the power relationship between the developmental department and users in an in-house development project. The analysis concluded that the procurer steer group failed to take into consideration the usability questions that were actually given to them and thus gave the developmental organization the power to decide various usability issues, which in turn did not prioritize usability. In the trade-off between making a system of high usability adapted for the user group and using a new technology platform, usability was downplayed.

In one case study, Artman (2002) investigated how a procurer project leader and a contract project leader differed in their perspectives on usability. The procurer project leader had a process perspective, which meant that she emphasized how users were involved, while the contractor project leader focused on the product features (see also Näslund 1997; Følstad et al. 2004). This in turn had consequences for the actual development process, in that the contractor did not plan for user involvement while the procurer project leader undertook many of the user-activities herself with, in her own evaluation, quite meager results despite her good intentions. In the end, the procurer project leader actually designed the user interfaces herself and demanded that the contractor implement them. Thus, the actual usability competence of the procurer project leader defined the system’s usability. Interestingly, both the procurer project leader and the contractor project leader emphasized a business perspective when reasoning about usability although each understood good business in a different way. This kind of discursive discrepancy is seldom highlighted in usability literature; however, it is important to do so as it underscores the difference between a technology-centered notion of usability and a more consumer-oriented notion. The former perspective emphasizes that the technology is easy to use while the latter emphasizes that users will be motivated to use the system. These differing objectives introduce even more trade-offs when deciding to procure a systems development project (see also Markensten and Artman 2004).

Holmlid (2004) reviews the participatory design tradition in the light of procurement and argues that the pro-activity of the procurer organization is an undervalued asset within the participatory design tradition. Holmlid exemplifies how usability professionals at Ericsson have moved from project organization to marketing, arguing that this move made more room for doing usability. One conclusion from this study is that usability professionals must consider issues other than the developmental. Markensten and Artman (2004) studied how contractors using user-centered design methods and prototyping can conduct a pre-study that actually reflects and informs the business processes and allows the procurer organization to better understand the user’s use situation and thus the importance of conducting usability activities. One important conclusion in this case study was that management must take the time to reflect on what the users contribute to overall system performance in the light of their requirements and study new ways of segmenting the user population. Markensten (2003) did a case study of how an organization arranged for the procurement of a content management system. The study concluded that the procurer had problems defining usability requirements as a consequence of using idealized models of the users’ work practices as well as insufficient tools for eliciting requirements. Others have focused the costumer–developer relationship (Keil and Carmel 1995) or the way in which the procurer organization decided how to merge different technology environments (Forsgren 1996).

Based on these studies, it is our understanding that considering a procurer management perspective is very important in exploring issues of usability. A procurer management with a shallow and narrow view of usability can easily reduce usability to interface design in the requirement specifications while a mature and contextualized understanding of usability recognizes that the usability approach is as complex as a general information systems (IS) approach with many interrelated processes and strategic decisions (Ward and Peppard 2003; Earl 1993). However, IS approaches often overlook mundane user tasks in favor of general business processes. Based on some studies outlined above, Holmlid and Artman (2003) have therefore suggested a tentative model for usability procurement which points out the need for various usability roles (and perspectives), including business, design, and evaluation analysis in different phases of procurement and development. From a more traditional usability point of view, we suggest that usability and user-centered design competence can and should be employed by the procuring organization while making a concrete, interactive prototype before the developmental organization is contracted (Markensten and Artman 2004; Artman and Markensten 2004). This prototype can then be used both as a common ground for communicating about different requirements with the decision makers at an early stage. A prototype accompanied with user profiles is easier to understand than abstract and often written arguments for usability, for a prototype will not only demonstrate the design of technical aspects of the system, but will also indicate the organization of the business. Furthermore, a prototype in itself contains requirements by design and can be submitted to potential supplier or development organizations. It therefore enables the procuring organization to articulate requirements that supercede the prefabricated solutions of development organizations. Nevertheless, a procurer organization must also understand that a stand-alone prototype may have to be altered and refined during actual implementation as constraints of technology may demand such changes.

The concept of usability has now come to such maturity and recognition that organizations recognize the need for doing usability work, but this does not always mean that the company contracts someone to do it. How much a procurer organization can do by itself is a general research question. Our research project focuses on how, ever so implicitly, procurer organizations plan, communicate, monitor, and evaluate usability work as well as what benefits they gain so doing. Our first step is to understand how procurer organizations value, understand and reason about usability. This means that we have looked for usability activities even when they are not labeled as such. Instead of trying to change development practice from within, we think the usability research community should address the procurer organization’s management by acknowledging what they do today and possibly help them to refine those activities to achieve better usability procurement.

This article investigates how usability and user involvement have been managed by the procurement organization during the procurement process for a contract development of a taxi-dispatch system. The study was initiated by the second author who was at the time working at the taxi-company and had sensed the discrepancy between the employees’ expectations of the new dispatch system and their actual response when it first came into business. We examine how different actors within a procurer organization in the taxi business understood, required, and negotiated usability activities during the early stages of the procurement. The aim is to form a valid understanding of procurer organization’s perspective on usability as well as to acknowledge usability in procurement.

Before presenting the procurer organization’s view, we must describe some general aspects of the taxi business.

2 Taxi business

Although the taxi companies around the world are in the same business, they differ a lot in tradition, needs, and routines. In North America and many south European countries, customers simply grab a taxi on the street or wait for one at a taxi stand. In these countries, hardly anyone books a taxi in advance. In northern European countries such as Sweden, Denmark, Norway, and Finland, it is common for the customers to call a taxi company to order a taxi for immediate or future needs. In Sweden, for example, approximately 50% of the job offers are dispatched to the drivers via the companies’ operator centrals. This puts much higher pressure on the taxi companies’ dispatch systems for them to be able to handle large quantities of orders.

A typical Swedish scenario of booking a taxi is relatively straightforward and simple. When a customer phones a taxi company to order a taxi, an operator answers the call and asks the customer for necessary information such as the pick-up address, name, and telephone number. A taxi driver then receives the booking information, accepts it, sets off to the pick-up address, and completes the job by taking the customer to the desired destination. Nevertheless, problems may easily occur for a number of reasons: for example, there might not be any available taxis in the area, the taxi driver might have difficulties finding the right address, or get delayed due to traffic jams or a flat tire. The customer might choose another taxi instead of waiting for the ordered one or change his/her mind about traveling so that there is no one waiting at the pickup address, etc. In all these cases, the operator central is often the coordinating unit where customers and taxi drivers phone to receive help with their specific problem. In such cases, it is most likely that the caller receives help from an operator who did not initially handle the order and therefore solving the problem requires cooperation between the telephone operators. This situation is not unlike the division of labor and cooperation at an emergency central where a lot of information must be coordinated between several, different operators (see Artman and Waern 1999).

2.1 Case background

The taxi company we are examining is based in Stockholm and covers the city and its outskirts, which has a population of over a million citizens. The company has divided up the serviced area into 200 smaller zones in order to efficiently handle the large area. The company is one of the largest taxi companies in Stockholm and is collectively owned by approximately 1,000 independent taxi haulers. Together they employ 4,000 taxi drivers and 150 telephone operators. The drivers are mainly men in their late thirties to early sixties, with varying professional backgrounds. Each vehicle has a mobile computer (MC) installed that functions as the driver’s main communication link with the operator center. The MC is often used for transmitting short messages regarding estimated arrival time at the pick up address. These messages then work as notifications for the staff at the operator central.

The taxi company’s operator central is located in the center of Stockholm and receives telephone bookings day and night. Up to 40 operators are at all times occupied with answering and mediating bookings and other requests from customers. The taxi company also provides other services for their customers such as wake up calls in conjunction with pre-bookings. In general, the central is busy during rush and late hours as well as weekend nights. Apart from the operators, the central also includes group managers and a special unit, the supervision group, which consists of about 50 specially trained operators. The supervision group mainly functions as a communication link between the taxi drivers and the operators, but the team has other assignments as well.

They supervise all bookings, communicate with the drivers on the radio, handle faxed bookings, and monitor pre-bookings from the airports. The team also constantly supervises the availability of taxis in the 200 zones that the company has divided Stockholm into. As soon as one zone has a decrease in available taxis, the supervisors inform the operators by using pop-up messages. The taxi-drivers get the same information through their MC. Thus, when they take bookings, the operators then know that they have to notify customers about possible delays or even deny access to taxis in those areas if they cannot commit themselves to the costumers’ wishes.

The taxi company first computerized their business in the early 1980s. In 1990, they bought their current dispatch system. At that time, the procurement and implementation was unsuccessful because the company implemented the system overnight, a method that they refer to as the “big bang.” This resulted in a period with many malfunctions. The procurement was initially motivated by a need for a new radio-communication infrastructure and redesign of the MC used in the vehicles. With a new dispatch system, the company would be able to ensure better supervision of vehicles in order to minimize the lack of available taxis in different areas. Visions of the system included the possibility of personalizing the services provided to regular customers.

3 Method

During the fall 2003, we interviewed ten participants in the procurement organization team. The respondents included the IT-Manager, the project leader, and users (telephone operators, supervisors, and taxi drivers). The IT-manager helped us get in contact with most of the respondents; some participants were contacted by the second author.

The interviews were guided by a method called theory-led thematic analysis (Hayes 1997), which combines qualitative analysis with many of the features of traditional, quantitative research. Hypotheses based on former models and research guided and prioritized the interview questions, but the respondents were relatively unconstrained in answering. Then, in a second analytical phase, the data was organized in themes which were quantitatively assessed and used as a basis to form new hypotheses. Each theme or category was then used as the base for understanding the question at hand. This means that some statements may be found within several themes.

We formed the following hypothesis based on actual research about usability procurement as well as the second author’s former experience as an operator at the taxi central: when the company set out to procure its second system, usability and user-centered design had not been required to any particular extent and the procurement mainly focused on technical aspects.

The interview schedule was semi-structured and included topics rather than individual questions. Topics concerned the procurement process, how usability factors were required and how they were conducted, what general tradeoffs the project had performed, and what kind of knowledge the respondents had gained from this procurement project. They were conducted in a conference room at the taxi company, and each interview took about 1 h. All the interviews were audio recorded and later transcribed.

A written summary of our first interpretation of each interview was sent to the respondent involved. After the respondents had either approved the transcript or made some corrections to our first interpretations, we corrected the interpretations and commented the interviews accordingly. The quotations from the interviews presented in the next sections are used as instantiations of representative expressions. In this specific article, the presentation of results is based more on the actual chronological process than on the hypotheses.

4 The procurement of the dispatch system

Because of the taxi company’s experience of the last “Big Bang” implementation, they were cautious during this procurement. They wanted to have full control over the entire procurement process and the option of making alterations during the process. This is the suggested practice when developing large and complex computer systems (Gulliksen and Göransson 2002). The procurement process started in 1999 as a consequence of the need for a new radio infrastructure. The new taxi dispatch system was then introduced in stages at the operator central from January 2003, which means that the telephone operators were forced to work in two different systems during that whole year. The system was first put into full use in January 2004, although it was withdrawn 3 days after initial implementation as it was not operating fast enough. The problem was investigated, and the system was put into operation during the later part of 2004.

4.1 The procurement phases

As mentioned above, the procurement was initially motivated by a need for a new radio-communication infrastructure and new MCs for the vehicles. The dispatch system was considered at a later stage. Figure 1 tries to present the staging of the whole project.

Fig. 1
figure 1

The staging of the procurement project. The double arrow shows the approximate timeframe of the project we describe

Full size image

The implementation was meant to be seamless, so it was done in phases without any interruptions in business. The whole of the new technology had to adhere to the old system solutions. The old dispatch system created a bottleneck that new implementations had to be compatible with. From what we have seen in our other studies, it is quite common that systems development is initiated more by infrastructure than considerations of user tasks (Artman 2002; Markensten 2003), which makes usability less prioritized. However, with a stage-by-stage and seamless implementation, the procurer has the opportunity to actually reflect on the system’s impact on the social organization of work activities and refine the requirements. This is a hard trade-off to handle.

4.2 Selecting the supplier

The procurement process started in 1999 with a pre-study designed to locate and organize the needs of the company, a modern radio infrastructure and a more up-to-date MC system. The company scanned the market of suppliers of dispatch systems. The procurement steer group, consisting of managers and experienced users, went to visit the different suppliers to study their specific solutions. At this point, the team did not have any requirement specifications; instead, they used a general description of specific needs, formulated by the company board. The description contained a list that focused on the supplier as well as the system. According to the list, the supplier was to be well known, dedicated to the taxi business, and be able to seamlessly deliver a modular, user- and support-friendly system. The current system was at all times taken as a bench-mark. Finally, it also included a wish list of specific functions, for example, electronic maps and a pathfinder system for the taxi drivers to help them to find the best route.

It quickly became evident that only 10 out of 30 suppliers were appropriate. Some team members found it difficult to find an appropriate supplier due to high demands on and expectations from the new system. One respondent was quite surprised to find that so many of the dispatch systems did not support their work procedures and that the systems they offered were slower than the company’s current system.

Evaluating the systems provided by the suppliers was a really difficult task because there are not many companies around the world that do things the same way we do. We have very high demands on a system. The company has many vehicles and many employees who will use the system. Besides that, we always want to speed up the work, the system has to work fast. [...] I realized at an early stage that our current system works extremely fast compared to the ones we saw.

This shows that issues that conform to usability, such as performance and work organization, were implicitly accounted for but not explicitly required. The evaluation work resulted in a short list of three or four conceivable suppliers. Early in 2000, the taxi company selected a Canadian supplier who could deliver a modern system and technology in the primary interest areas of radio infrastructure and mobile technology. According to one of the respondents, the supplier also had strong incentives for approaching the European market, which made him more eager to provide the taxi company with the kind of system they required. The supplier agreed to develop a new MC in cooperation with the taxi company. At this stage, a new dispatch system was not included in the procurement, even though the supplier could provide them with one. Instead, the two parties included it as an option in their framework agreement:

In the beginning of the process, we decided to procure a new mobile terminal, a new radio infrastructure, and so on. The dispatch system, on the other hand, was never discussed at that point. Instead, the dispatch system was included as an option in the framework agreement for future procurement.

The reason for not including the dispatch system from the beginning was that the taxi company was still relatively satisfied with their current dispatch system as it was quick and established within the work organization. This is another reason that conforms to usability aspects and work organization. The two parties formulated a mutual framework agreement, which, due to the demand for a seamless implementation, was divided into eight minor agreements. Each contract contained additional information such as responsibilities, goals, time schedules, and costs for each phase. A primary concern during implementation was that the infrastructure would be compatible with the current dispatch system. This staging of the process and implementation strategy shows the maturity of the company in organizing a large-scale system implementation.

At this point, usability—or, rather, the term they used, “user-friendly” equipment—had been mentioned, but was not highly prioritized. When the term user-friendliness was used it more or less was described as adherence to graphical user interfaces or Windows standards. Thus even though the reasoning above must be seen as to adhere to usability issues at least from a professional view, the procurer organization did not associate them at this point in time. Differences between Canada and Sweden concerning the organization of the taxi business were not considered at this point as the contract mainly concerned infrastructure and not business strategy or company-related procedures. The supplier organization’s methods for approaching usability, or activity analysis, were not discussed during this phase.

4.3 Reconsidering the dispatch system

Soon after signing the first agreement, the taxi company decided that they would also procure a dispatch system. The reason for this reconsideration was that the new infrastructure made possible new functions that the old dispatch system was unable to support. To make changes in the old dispatch system was not an option as it was based on old technology which would have been too costly to reprogram and which still would not have been able to handle new, important features. Therefore, they purchased a Product Commercial Specification (PCS) from the selected supplier, which contained detailed information about the system and its functionality. However, the taxi company considered the system insufficient. A reference group within the procurer organization was made responsible for adding and changing functions and requirements into the PCS. The reference group consisted of different employees from the operator central who were familiar with the functionality of the current dispatch system. The initial purpose of the reference group was to secure the requirements formulated by the company board and to ensure that the new system contained the same functions as the old system. With this procedure, the procurer organization secured some aspects of the minimal requirements that the system should support. The reference group also added new functions that would benefit the operators. One respondent gave an example of an added function: the way the operators and the supervision group handle bookings that request attributes such as a limousine, an estate car, or a car safety seat. The new dispatch system should be able to automatically perform certain steps in these bookings that had to be handled manually by the operators in the old system. The operators’ knowledge of possible problems and problematic situations as well as the work context was valuable at this stage. Functionally, the supplier’s dispatch system was not suitable for the taxi company:

We have added a lot of functionality, but we wanted a more standardized system and not something we developed ourselves, although that is what we ended up doing.

The new functionality and work organization they required was not standard for the taxi business. The procurer had to re-specify the system to such a degree that it became a tailored system.

We did not buy an off-the-shelf product, we specified it together with [the supplier] and then decided, on a high functionality level that was to be included.

They worked together with the supplier in order to make sure that a modified system could handle all the new functions and attributes that they needed.

We have contributed quite a lot to the process by giving them many new ideas, and I think this has improved the value of their product.

The supplier’s original dispatch system could not handle attributes or services such as train–taxis and services for the disabled, which seems to be specific for the Swedish taxi business. The taxi company required that the system also should filter pre-bookings according to attributes so that they were only offered to vehicles and drivers with the requested attributes. These features of the system seem to be quite specific to the Swedish taxi business. As the supplier company was Canadian, they were not included and, furthermore, there was little understanding of their importance. This indicates the importance of an active procurer who precisely articulates aspects which are important for its specific business processes.

The main reasons for choosing the dispatch system were that it was modular, had stable performance, and would better fulfill functions in the new infrastructure. At this stage, the requirements were not focused on usability or workflows but rather on functions. This practice of using the supplier’s PCS as a platform for discussing the organization’s specific requirements and functions was very clever. The discussions became based on something which the supplier understood and was familiar with and which therefore created a common ground for including and expanding new functions. In this sense, both the supplier and the procurement group had strong positions for further development. The agreement for the final version of the PCS was signed in 2001.

4.4 Prototyping

After the PCS was revised, the procurer organization started work on revising the actual interfaces of the MC as well as the dispatch system. The design activities were performed in workshops and were conducted by two reference groups, one with taxi drivers working with the MC and one with employees from the operator central who did the PCS work. The reason for involving end users in this phase was to make sure that the system adhered to the necessary functions and had the appropriate design. Both reference groups worked, independently of each other, in workshops. The workshops were not formal or structured according to any specific workshop methodology, but were rather informal meetings with the aim of redesigning each interface. The groups worked in an iterative process by developing basic prototypes of the user interfaces. The prototypes were then sent to the supplier who updated the interface according to requests made from the two groups.

4.4.1 Prototyping for the new MC

As it turned out, the new MC was designed two times, first immediately after the first contract and then after the decision to procure the dispatch system. The specific aspects that were decided by the drivers during the first prototyping session were, for example, the physical design of the device and security aspects such as avoiding mirror reflections. The taxi drivers also asked for an interface with larger and fewer buttons and a more straightforward navigation. According to one of the participants, these aspects were necessary to ensure the safety of the drivers and the passengers.

The driver’s main task is to drive the car, not to focus on a MC. To support this, it is very important to use as few function keys as possible in the user interface. In that way, the drivers are able to monitor the MC with a kind of split vision.

According to one of the respondents, the content, concepts, and information structure used in the old MC were kept. A photo of the old terminal was used as a template for reconstructing all the functions in the graphical user interface and constituted the design work. The argument for designing the MC in this way was that the device would be much more user-friendly if the same structure and functionality were kept—that is, “user-friendliness” was defined by how familiar the users would be with the look of the new system. In effect, such practice restricts the options for redesign by attempting to improve the old way of doing things. However, the IT manager argued that keeping the familiar design would also save a lot of money for the taxi company because they would not have to provide the drivers with any additional education. The members of the reference group collectively decided how the device should be designed and which functions it should contain. To retain the feeling of pressing “real” buttons, the group incorporated confirmation windows in the navigation structure. When the driver presses “ok” the confirmation pop up disappears so that the driver knows that she/he has performed the action. This is a good way to achieve closure in human–computer dialogues; however, in the context of an attention-demanding activity such as driving a car, these forms of interaction might be disturbing. Another way to achieve closure and confirm actions in this context could be auditory signals, that is, the computer could make sounds to tell the user whether or not a request was implemented.

Despite the intention of designing the new MC so that it resembled the old one, the first solution was physically different from the old MC that had a small, narrow display with numbers in black and white. The new MC software was designed with a larger touch screen interface that displayed information in color.

Thus, much of the usability reasoning amounted to adhering to the familiar design in terms of interaction and navigation, rather than improving and reorganizing the software to better suit users and their tasks.

Because the taxi company procured a new dispatch system later on in the procurement process, the user interface of the MC had to be modified and redesigned a second time as new functions were needed. The dispatch system included a user interface for a mobile unit, but the reference group did not find this suitable as it was based on text-based buttons. With the incentive of adding new functions, redesign of interaction and layout were considered. The process of redesigning the interface was conducted in the same way as the previous interface. The main difference was that the group started off by using a prototype representing the supplier’s version of the user interface for a MC (see Fig. 2).

Fig. 2
figure 2

The MC as designed by the supplier

Full size image

The final design solution resulted in an interface quite different from the previous one. The interface of the new MC had a main menu consisting of icons and submenus in text. The respondents described the icons as “obvious.” For example, the reference group included a letter icon for the message section and a globe for access to different zones on an electronic map. This means that all the old and illogical abbreviations used in the previous interface were removed (see Fig. 3 for the final design). With the new MC, the drivers were also given full access to a graphical overview of the numbers of vehicles currently in service via the computerized map. The respondents pointed to this as an example of useful, visual support for inexperienced drivers. A red light highlighted zones with a lack of vehicles. In that way, the drivers would know where they should go to receive immediate bookings. This allowed for a kind of cooperative, indirect dialogue with the co-workers at the central. More direct dialogues would be still managed by cellular phones or by written messages.

Fig. 3
figure 3

The MC redesigned by the taxi-drivers (implemented by the supplier)

Full size image

4.4.2 Prototyping the dispatch interface

The reference group that worked with redesigning the dispatch system initiated many changes. Several participants explained that the user interface in the original supplier version of the system was very hard to use because one had to jiggle between several screens and because it did not have all the appropriate functions. Thus, the reference group had strong incentives for redesigning it. The original supplier version had emphasized direct manipulation, and it contained lots of information, icons, and buttons. The operator central demanded high-speed performance, which is not possible if the system provides the users with too much information or requires them to use too many pointing devices. Figure 4 shows the minimalistic design of the old dispatch system, which the operators found appropriate. The original supplier’s user interface was completely revised and specific functions which the taxi company needed were included. The greatest changes are accounted for below.

Fig. 4
figure 4

The design of the old dispatch system

Full size image

The design of the new interface for the dispatch system was conducted by paper prototypes. According to some of the representatives from the operator central, the group started off with a prototype representing the user interface of the original supplier version of the system. The prototype was redesigned by moving and restructuring the content. For example, the group choose to cluster important information together (such as pick-up place, costumer, and destination) and display it at the top of the user interface (see Fig. 5). This was very different from the former interface, which had a simpler structure with fewer, but more restricted, input areas.

Fig. 5
figure 5

The new interface of the new dispatch system, designed by the operators and implemented by the supplier

Full size image

Another example is that they choose to include some functions in the operator interface which previously had been used only by the supervision group (see Fig. 5 right-hand side). The reference group decided that the operators could just as well handle tasks such as repeated bookings that the supervision group had handled before.

I’d really wish the operators and the supervision group could perform all their tasks in one window, but unfortunately this is impossible. Instead, we took two of the windows used by the supervision group and integrated them into the interface used by the operators. This is because we believe that the operators can handle repeated bookings, for example, just as well as the supervisors.

Thus, by redesigning the interface, the group also took the initiative for rearranging the division of work. The prototyping added functions even if the design’s main purpose was reduced to articulate work flow and allocate task for various actors at the central. Thus, prototyping added to analyzing the PCS in an even more concrete way.

The design activity continued to be iterative until the group arrived at a final design solution. One respondent described this way of working as good because it gave everybody involved a chance to express his or her opinion.

It was really good to work with prototypes because it gave everybody a chance to say how they wanted the interface to display the information. The participants could, for example, suggest that certain information should be displayed on the right side or that a button should be placed somewhere else. I think this way of working really improved the process.

Due to these extensive content and structural changes which were required during the developmental process, the final interface differed considerably from the original supplier version. The most important change was that most of the information was collected on one screen rather than on different screens that the user had to jump inbetween. Another important function was that each taxi vehicle could be tagged with different attributes which helped the operators to filter and select an appropriate taxi. One of the representatives from the operator central explained that since the new user interface still contained a lot of information, the group tried to sort the information under different subsections, divided by tabs. This meant that the operators no longer had to switch to various subsystem views to perform certain tasks which they had to do in the old dispatch system. The original supplier version was also mainly designed for direct manipulation by mouse, but the reference group designed it to also work with commands and short cuts. Another detail changed for the operators was the necessity of pressing three keys to dispatch an order. Since this action is time consuming, the group required early on in the process that this should be changed to one key. According to one respondent, this requirement made the supplier quite confused because its designers could not see the purpose of that kind of modification. Given the difference in how many dispatches the operators make each day, this aspect may have greater importance for the Swedish market were bookings are more frequent than in other countries.

4.5 Technical tests

After the design phase was completed, both the MC and the dispatch system had to undergo technical tests as well as evaluations before they could be fully implemented. They let test drivers perform specific tasks such as an initial technical test and evaluation of the MC. The drivers were then asked to give feedback on the MC. According to a representative for the drivers, the test drivers were chosen on the basis of their personal interest in design issues. The procurer organization also performed technical acceptance tests of the dispatch system. When the developmental process reached the implementation phase, the system had to be modified even further even though this requirement had already been specified in detail when making the PCS. The reason for this need was, according to one respondent, that the reference group had misinterpreted, or rather taken for granted, the capacity of the standardized functions (for example, attributing vehicles and bookings). Such capacity aspects are hard to simulate in prototypes, and it is understandable that the operators took their experienced capacity for granted.

The dispatch system was tested continuously with operators. They did not use any formal test or evaluation protocols.

One of the respondents describes that the operators disliked the low performance and capacity of the system but had few opinions about the functions and the interface. A possible explanation for this, given by the respondent, is that the new graphical interface is much easier to work with because of the new structure with different sections divided by tabs.

As mentioned above, it was important for the taxi company that the implementation be seamless and incremental. In January 2003, the system was implemented in the operator central. This meant that the operators up until that time had to work in two parallel systems, the old dispatch system and the new one. Some, but far from all of the bookings could at the time be performed in the new dispatch system. This is because only a few of the vehicles had been transferred to the new dispatch system and some functions, such as services for the disabled, had not yet been implemented in the new system. Due to this, the operators had to jump between two very different systems, with different performance rates. This, of course, made many operators frustrated; some even refused to work with the new system. As mentioned at the beginning of this section, the new dispatch system was initially put into business in 2004, but was withdrawn shortly thereafter because it did not stand up to the performance needed.

4.6 Post-implementation evaluation

The respondents were asked if end users had to any extent evaluated the dispatch system. Some of the respondents, most of whom were managers, believed that evaluations of the dispatch system as well as the MC had been done. They exemplified this by mentioning that more experienced operators had measured the performance of the system through bookings in both systems and then comparing the performance times. The same respondent mentioned that between 30 and 50 test drivers used the new system, which had generated a considerable amount of feedback. The procurement process group also held weekly meetings where all the involved parties met and discussed different issues. The taxi company also arranged open meetings for the drivers once a week. According to a couple of respondents, these meetings were not that frequently attended by the drivers, which may be an indication that the implementation and the design process was successful. Another plausible reason for their poor attendance may be that the time required for attending these meetings competed with the time for driving and thus earning money. Other participants said that they did not consider these meetings as a proper evaluation, but rather as a way of receiving continuous feedback. Thus, there was no formal evaluation of the system, let alone any usability test. One respondent says that the taxi company should not have implemented the dispatch system at such an early stage. Taken together, this trial-and-error procedure made another respondent feel that the drivers were used as guinea pigs.

I feel that we, the procurer, have been used as a guinea pig for the supplier and that was not the initial purpose. At the same time, we do not have any other option because there is no product available that fully suits our needs.

At this stage, the various responses and reactions from the end users were organized and put together for later processing. Because of the delay in project delivery, the procurer organization now had to get the dispatch system into business as soon as possible and therefore there was no time for “ cosmetic changes.” Two of the representatives from the operator central said that they were fully aware that this way of prioritizing could cause problems for the operators. What is for the reference group or the supplier just a minor malfunction may be an extremely annoying disturbance for the operators who have to deal with it every day.

You think that smaller details or malfunctions don’t matter, but they become a major thing to the operators because they deal with them every day. If we take care of it, they will be quite satisfied. Even the supplier doesn’t see the necessity in taking care of these small details, but they do matter.

This shows a great awareness among users that usability matters, at least in hindsight.

4.7 How the concept of usability evolves

During the first phase of the procurement (selecting a supplier and evaluating the dispatch systems), appearance and stability were valued higher than performance. When the respondents were asked to describe what the term “usability” signifies, they all gave different definitions. Two respondents of high rank considered systems designed according to Windows standards as very user-friendly due to the user’s familiarity with graphical interfaces:

It is user-friendly because it functions in accordance with Windows standards.

One respondent maintained that the importance of a usable system was included in the requirement document used by the group during the supplier evaluation phase. The document included usability in different subsections but, according to the respondent, this was done without any explicit references to the research literature in the area and without any explicit measurable requirements. Instead, the group based this classification on previous knowledge and experiences within the company. During the later phases, usability was refined as pertaining more to performance than to appearance.

Other respondents quickly arrived at the concept of performance when defining usability. A usable system is “one that responds fast and without any unnecessary delays,” this respondent says. The reasoning goes as follows: if the system performs its activities slowly, this results not only in poor feedback for the end users, but also higher costs for the company due to the resulting reduction in bookings. A usable system is therefore considered as one that is an efficient tool for the end users when they perform their daily activities.

To judge efficiency, first impressions do not suffice. This is widely acknowledged in human–computer interaction (HCI) literature (see, for example, Preece et al. 1995; Shneiderman 1998): a system is at all times dependent upon the user, various information dependencies in context, and the interface. This is exemplified by one of the respondents who saw the double-sided nature of usability as both interface design and response times.

A system can at first glance appear to be very user-friendly, but if every interaction takes ten seconds to perform, you loose interest.

When it comes to display-design issues, the respondents emphasize the importance of considering the arrangement of frequently used buttons. Otherwise, the users might perform frequent mistakes. One respondent gave as an example the key used for cancellations in the dispatch system: if this key is placed next to another frequently used key, the interface arrangement increases the risk of performing false cancellations. According to the same respondent, this issue was not something that the supplier had thought of in the initial version of the interface. As stated by another participant, it is also extremely important to focus on the navigation structure so that the users do not have to perform unnecessary key pressing. Instead, all functions “should be three keystrokes away.”

The respondents also define a usable system as a familiar one. This can, for example, be accomplished by using a coherent design for various subsystems and for various user groups, i.e. both the operators and the supervision teams. The impression of familiarity can also be accomplished, one participant claimed, by making sure that the system employs a consistent use of language. If the system has to be translated, it is very important to make sure that the translation is accurate. Mistranslations can have fatal consequences, especially if the mistranslated word means something completely different in the specific business. One of the respondents, a representative for the operator central, mentions one such mistake that was made in this particular system:

When you look at some of the pop-up windows, you understand that they are mistranslated from the English word “Cancel.” In our business, the term cancel means something completely different to cancel a booking for example.

Another important reflection made by the respondents is that the term “usability” involves many aspects that, taken together, support single users as well as the company. The respondents describe the importance of focusing on the larger business picture and not on isolated design details. Only then can a usable system be achieved. One such aspect is the cooperation between different coworkers within the company.

According to a couple of the respondents, it is very common in other companies to separate the tasks between various teams. In such a division of labor, operators have rarely, or hardly ever, any contact with the staff that is responsible for supervising the traffic. It is even common to separate the teams physically by placing them on different floors. This strong division of labor was also reflected in the design of the original supplier system, which was designed for individual use. Dispatch systems are often meant to automate much of the communication between the operators and the drivers by means of sending simple messages between the cars and the central. In the taxi company, operators and the supervision team communicate with each other frequently and work together as one group. In order to illustrate this, one of the respondents presented a very common scenario, as when an operator receives a call from a customer who wishes to know why the ordered taxi has not yet shown up. To solve this kind of problem, the operator often has to communicate directly with people on the supervision team or with the coordinator. According to one respondent, this kind of communication pattern was very hard to grasp for the supplier, which failed to see the purpose of having operators communicating with supervisors. At the same time, the respondents mentioned that the typical, unproblematic workflow within the company is, in a way, organizationally divided. The operators rarely handle any calls from the drivers because their task is to handle as many bookings as possible, and the supplier’s original dispatch system was designed according to typical, unproblematic scenarios involving isolated and omnipotent operators. Such aspects of both typical and atypical procedures were only implicitly included in the process through the reference group’s use of problematic scenarios. The practice of involving users in order to get such first-hand information is very important even though a thorough analysis of the work context, the task allocation, and communicative means between operators would have articulated the work activities even further.

The respondent gives a slightly dejected view of how ignorant the supplier could be of the specifics of the Swedish taxi business on one hand, and of the cooperation within the operator central on the other hand. A supplier who has little or no understanding of how the Swedish taxi business is organized will also have a hard time understanding how this particular taxi company is influenced by the high emphasis on telephone bookings and a large geographical area to cover.

5 Discussion and implication

This interview study has illuminated one procurer team’s usability issues while procuring and implementing a computer system for a taxi business. Despite the first impression that usability and cooperative aspects of the operators’ working environment were not given specific consideration, the project requirements did eventually include usability, or at least user involvement, issues. Interestingly, the taxi company conducted these user-centered activities by itself and did not out-source these issues to the supplier. From our perspective, it is good that the procurer organization took such an active role in forming their new work environment as it is the employees who have knowledge of the requirements from an activity perspective. However, it is problematic that the procurer organization did not have any former experience of and competence in user-centered design methodologies. In a sense, the processes of involving users’ helps enforce democratic anchoring within the organization, but design is such an important aspect that it should not be handed over to a supplier without any experienced and competent guidance (Følstad et al. 2004). We also saw that the respondents’ understanding and awareness of usability issues go beyond a simplistic interface design perspective as it includes division of labor and work organization. This is, as we understand it, a very important aspect as the benefit of a system lies more in its appropriateness for work activities than in usability heuristics that many developer organizations believe satisfy usability considerations (and which often must satisfy usability considerations because of the lack of knowledge when designing the first system).

The taxi company decided at an early phase that it would not repeat the mistake it made the first time it procured a new dispatch system. This time, the process would be characterized by full control and the possibility of making alterations. In this sense, the taxi company showed a high level of maturity. In respect to usability, the company was very active in ensuring usability. The specification work involved end users, but many of the participants had only a limited degree of prior experience of this kind of work and therefore may not have had the competence required for recognizing which aspects apply to usability and cooperation issues and which do not. Nevertheless, the case shows that a procurer organization can do quite a lot on its own with fairly simple means such as using prototypes with an existing PCS. However, as Markensten (2003) points out in his study of a content management procurement, if the procurement group values usability as such but does not have all the appropriate tools for assessing it in a more formal sense, the lack of knowledge can result in what Markensten calls “idealized models of work.” Such idealized models—or, in this case, implicit models—may hamper the designing of new and more efficient work practices. For smooth implementation, one probably need to involve someone who has a professional knowledge of design, evaluation, usability, and cooperative work and not just rely on users or the supplier, since one cannot expect users to be fully competent in defining usability design. This is approximately the solution Markensten and Artman (2004) arrive at.

When the various participants were asked to define the concept of usability, they described it from different perspectives. According to a manager, a user-friendly system equals Windows standards. On the other hand, the term was also associated with less direct manipulation, which is not associated with systems that follow Windows standards. Other respondents used performance metrics when defining usability. The initial requirements document included the need for usability, but did not define how this was to be achieved. This problem is closely related to the various usability perspectives that Artman (2002) has found between the procurer and the contractor. It seems that usability has two more or less relevant and valid definitions in business, one focusing on the interface and the other focusing more general performance and even the design process. In both cases, we find that usability as a concept evolves during the development process from an almost casual and general feature to something which is more closely related to what the user needs to do. The difference between these concepts highlights the importance of communication between higher management and users who are involved in the procurement because users are often asked to decide on usability questions (Svensson 2002). As usability is important for the smooth and reliable operation of any complex computer system, usability must be discussed and defined at an early stage. This, of course, is problematic for two reasons. First, a procurer organization assumes that the system will be usable. Second, without any specific, formal knowledge of usability, it is hard to discuss the concept without referring to a concrete situation. In this study, this problem was handled by letting various users design the interface layout in informal workshops. Many requirements, such as performance metrics, learning time, and percentage of errors cannot be defined by visual layout alone. Nevertheless, visual layout is a way of presenting a concrete situation which users can refer to while discussing various requirements. We think that procurers should design interface layouts, but take some time to discuss the requirements in the organization in relation to different metrics (Markensten and Artman 2004). We also think that such a process would be facilitated by the methodology that is presented in general HCI-literature, even though an important research issue is to appropriate these methods to procurer organizations.

It is important to keep in mind that, from the beginning, the taxi company was mainly interested in procuring systems in the areas of radio infrastructure and mobile equipment. The need for a new dispatch system therefore sprang out of later needs caused by technological advancement. In other words, their primary goal was not to procure a more usable dispatch system. This is not an uncommon constraint when procuring and designing systems, but it should not be allowed as an excuse for not making usable systems as such an attitude can downplay the system’s efficiency and thus part of its investment is lost (see Artman 2002).

One initial requirement from the company board within the taxi company was that the supplier should be dedicated to the taxi business. The chosen supplier had sold their system to several taxi companies and was considerably well known in North America. Despite this, it demonstrated a lack of understanding of the workflow and the communication pattern that characterize the work at the operator central. This may be explained by the fact that the two companies were located in two different countries where the taxi business is organized differently. This is an indication of how problematic it can be to develop a usable system if you do not know the business constraints (Artman 2002; Markensten and Artman 2004).

We can see both the advantages and disadvantages of being a pioneer in procuring new technology and especially of contracting a supplier company that does not have a strong market imprint in one’s organization and business. The advantage is that the desire to accomplish a good systems design is shared by the procurer and the supplier. If the project fails, both fail; if it is a success, the supplier company can use the knowledge to attract new customers. This makes the general problem of finding new directions and digressing from the general plan feasible and less costly for the procurer. The disadvantages, at least from a time perspective, are that a lot of analysis and trial-and-error work are required until the system fits the organization. From the point of view of usability and methods of cooperation, it is questionable whether this disadvantage is in fact a disadvantage, but remember that the procurer management often understands the problem from a monetary angle (Scown 1998). For most systems of relative to high complexity, analysis and multiple evaluations are necessary for creating a supportive and usable system. This is, as we have seen in this and other cases, something which many procurers are reluctant to pay for (Artman 2002; Markensten 2003; Scown 1998). However, even if usability is achieved for the first procurer, subsequent procurers will encounter other problems in adapting the system to their specific needs because it is designed for a specific organization and any redesign will be costly and demanding. This might provide an answer to the question of why we are seeing so many projects failing in usability. The complex picture is burdened by the procurer’s initial visions and ideas: the requirements he makes initially constrain further development (Carlshamre 2001).

6 Conclusion

Through our corpus of five case studies (Artman 2002; Holmlid 2004; Holmlid and Artman 2003; Markensten 2003; Markensten and Artman 2004; Svensson 2002), we are beginning to understand the procurer organization’s perspective on usability. Contrary to our and many usability professionals’ first impression, procurer organizations seem to value usability. However, the concept is poorly understood in terms of requiring, producing, and monitoring systems development and is even sometimes confused with a democratic anchoring process (Følstad et al. 2004). Many organizations try to conduct user-centered activities (i.e. focus groups, workshops, interviews, work flow analysis) independently when they begin to envision the need for the system, but then have problems in transforming the results into plans and directives. Thus, when procuring an IT-systems development, they feel that they have already done many of the activities suggested by usability professionals. The solution is not only to demand usability, as the usability the developer organization can offer is seldom of such detailed understanding that it supports a specific work organization, but also to be active in the process and work intentionally towards a high degree of usability. The problem seems to be that procurer organizations generally do not have an adequate knowledge of usability design and user involvement. While they are well aware that many technical aspects are beyond their competence, or, at least, beyond the limits of their time schedule, they find usability to be intuitive and therefore fail to understand the importance of discussing usable design. This strengthens our understanding that the HCI-community must address procurer organizations more forcefully and develop methods that they can adopt. We must take the procurer perspective seriously.

We have argued that the common practice of involving usability professionals in the developmental phase results in a delay in initiating user-centered design activities. In order to involve such activities early on, we need to redefine what we mean by “early.” We claim that there is today often a gap between the procuring organizations’ business goals of initiating a procurement project and their objective in defining what is to be procured. In response to this problem, we claim that user-centered design must be involved even earlier than is the general norm today in order to find a path to usability (see Markensten 2005 for a model of procurement).

We must recognize that the user-centered design process does not necessarily have to be viewed as a systems development process, but can also be perceived as a grounded, business-development process or, more to the point, an approach to systems acquisition that links business objectives and systems requirements while strengthening the procurers’ ability to order a usable and appropriate system. This shift in perspective is not easy, but it is, in our experience, very rewarding. It can help to involve usability earlier in a more appreciative context in which the results are actually used. In the long perspective, we believe that this shift also would allow HCI education to be more focused on organizational and management issues and theories.