Keywords

Introduction

Worldwide map portals are very often used for identifying the geographical location of a target destination or searching for the best route to an unfamiliar place. There are a great number of web maps on the Internet which provide a map itself and many map tools, however much they differ in their level of user friendliness. Although web map portals have been analysed in numerous studies with respect to user interface and map visualization, no standard of web map design has been published so far.

This paper builds on and contributes to the work of Nivala et al. (2008), who examined four different mapping sites. Several usability tests were conducted on users, and usability problems were recognized and further analysed. However, these tests were carried out 5 years ago. Is it feasible to assume that all of these problems have already been solved?

This study provides additional insight into problems related to map visualization and designing the user interface of worldwide map portals. First, some problems of usability testing are identified. A usability test covering three worldwide map portals (Google Maps, Bing Maps and MapQuest) is then described. Finally, results of the testing and guidelines for designing highly usable map visualization are presented.

Previous Studies

Progressive development of the Internet in the last two decades led to the need for testing of web site usability. Therefore, there have been many studies dealing with usability evaluation. Nielsen (1993), who defined the elements of usability as efficiency, learnability, memorability, error rate and satisfaction, is considered a father of usability testing. Subsequently, many usability evaluations of web sites, software or information systems have been made, e. g. van Waes (2000) and Battleson et al. (2001). Travis (1999) published a Userfocus web portal, which deals with various usability issues in general and provides guidelines for designing technologies and usability testing. A usability international standard ISO 9241-210 (2010) was also adopted.

An important part of general usability issues is the evaluation of map portals. There are many studies dealing with usability testing of web maps from different points of view. Schobesberger (2009) presented various types of usability evaluation of web mapping portals. Interview, user observation, user surveys, remote evaluation and eye-tracking were defined as the main techniques. You et al. (2007) studied issues related to zoom and pan functions. The author examined five web map portals by evaluating simulated user interface and designed an interface with ideal zoom and pan functions.

Hub et al. (2011) and Komarova et al. (2007) have presented heuristic usability evaluations of Geoweb, which consist of defining potential problems, i.e. heuristics, and examining user interface and judging its harmony through known heuristics. He et al. (2012) and Komarkova et al. (2010) used the think-aloud method for testing of national map portals and a user test, and found that users were influenced by Google Maps when working with the national map portals. Komarkova et al. (2009) carried out a written questionnaire and practical user tasks on GIS applications of Czech regional authorities. The most significant finding was that users prefer to have the option of switching on/off all layers on all tested map portals.

Skarlatidou and Haklay (2006) analysed the usability of some world-wide map portals with a user test and questionnaires. The time needed to perform the task was measured and the success rate was calculated. Skarlatidou and Haklay came to the conclusion that Google Maps is the most usable map portal and Map Quest is the least usable one. Correspondingly, Nivala et al. (2008) presented a usability evaluation of world-wide map portals similar to the analysis presented in this paper. Four world-wide mapping sites were evaluated employing user tests and expert evaluations. Altogether, 403 usability problems were identified. The problems were divided into four categories: user interface, map, search operations, and help and guidance. Based on the evaluation of web map portals, some design guideline suggestions were discussed and presented as a result of this study.

Usability studies focusing on web map portals also generate inspiration for further development or adoption of new tools and visualization methods, e.g., more user-centred options of cartographic visualization mentioned in Konečný et al. (2011), Stachoň et al. (2013), etc.

Another approach to examining usability is using eye-tracking to follow the user’s eye. An example of an eye-tracking study focusing on the comparison of 2D and 3D cartographic visualization can be found in Popelka and Brychtová (2013).

Methods

It has been 5 years since Nivala et al. (2008) presented a usability evaluation of world-wide map portals. The present study deals with usability testing of the same map portals and tries to identify whether the problems described in 2008 have been solved or persist. Moreover, the potential deficiencies of up-to-date portals are indicated. Nivala examined Google Maps, MSN Maps & Directions, MapQuest, and Multimap. MSN Maps & Directions and Multimap have since been joined to Bing Maps, so the well-known map sites compared in this work are Google Maps (hereafter referred to as GM, http://maps.google.com/), MapQuest (MQ, http://www.mapquest.com/), and Bing Maps (BM, http://www.bing.com/maps/).

It was considered whether to include some alternative or local map portals in the study. The study was eventually limited to the above-mentioned map portals due to particular map portal functions, language dependencies that might limit future study, etc.

Visual Interpretation

The map portals were initially examined in terms of the problems analysed by Nivala et al. (2008), who defined four categories: user interface, map, search operations, and help and guidance. Since the map portals have seen certain improvements in the meantime, some of these problems have been solved while others remain unsolved.

The problems concerning user interface may be regarded as solved. Nevertheless, the main problem reported concerning user interface was the overloading of web map homepages with advertisements and additional information. Moreover, in some cases there was no link to the map on the homepage. Another problem was the fact that links opened in the same browser window as the map, and there was no quick way back to the homepage. The last major problem in this category was the form and placement of the function buttons.

Most of the defined problems were related to the map field. The principal problem was map visualization—maps looked like paper maps instead of web maps, and their visualization was messy and confusing. The other problems presented in 2008 were the absence of a print function or route direction tool and the fact that the scale bar showed only miles. Panning was sometimes found to be problematic and slow, or there was no possibility to add markers into the map. Most of these problems have since been resolved, but other problems in this category still persist: e.g., there is no legend; data is seen as inconsistent—some objects in a given category are shown on the map while others are not; there is no option to modify the map by checking boxes to show or hide different layers, and there is no option to highlight certain categories of objects, e.g., hotels or theatres. There are also problems with zooming and panning. For example, some symbols or text appear and disappear randomly with different scales, the step between map scales is too large, and the visualization between the scales is too varied. Some colours are considered unintuitive. Another problem is labelling—the placement of the text, its legibility, etc. The final group of problems in this category is the misinterpretation of some symbols.

The third category covered search operations. There were two search possibilities: a free search, which allows a user to use search criteria more liberally, and a search done using different boxes for a strictly defined input, e.g., a country or address. Some of these problems have been resolved, and others have not. The resolved problems include those related to default settings—if the user did not define the country of his location, the site would only give results from the USA. Another group of problems concerns route or address searching—there was no way to search for a route by clicking on the map itself, no possibility for multi-stop route searches and no good methods for route modification. A final problem was centring the map according to the result irrespective of its location. The unresolved problems include those concerning visualization of the search result—streets are indicated by a point instead of a line. There is no possibility of conducting several separate searches. Moreover, there is no option for enabling a direct return to the search result, and in some cases users do not know whether the search result is relevant or not.

The final category is help and guidance. A major problem was the absence of help, but this has already been settled. Some users identified problems with the comprehensibility of the help provided.

Following Nivala’s study, the map portals were compared with each other by visual exploration. Elements of functionality were described for each portal, and its potential vulnerabilities were recognized.

Null-Hypotheses and Pre-test

Based on an analysis of the problems defined by Nivala et al. (2008) and an exploration of up-to-date map portals, null-hypotheses for further testing were formulated. The null-hypotheses were encouraged by a simple pre-test, which was performed on 20 geography students. First, weaknesses of each portal were determined and the students were subsequently asked a simple question. If they agreed about a potential problem, it was proposed that the vulnerability be evaluated in a user test. For example, the colouring of bodies of water on the MQ map was considered unsuitable, since the pools in the parks were difficult to distinguish. The students were then asked how many pools they see in the park on GM, MQ, and BM.

The following null-hypotheses were formulated as a result of the pre-test and the exploration of the web map portals: GM will appear as the best map portal because users are used to working with it and the orientation is easy. The map visualization and symbology are also very familiar to the greatest number of users, and therefore they will consider its symbology to be the best. GM’s labels will also be considered the best, since the labels on MQ are too small and difficult to read, and labels on BM are disproportionate and they overlay additional information. Moreover, users will prefer the possibility of turning layers on and off and regulating the amount of information contained in the map. On MQ, it will be difficult to distinguish bodies of water in the parks.

User Test

The user test in the MUTEP (MUltivariate TEsting Program) was drawn up according to the null hypotheses and the pre-test results as described above. MUTEP was developed by the Department of Geography at Masaryk University for the purposes of objective experimental exploration and evaluation of cartographical products. This application uses an innovative cognitive testing method that combines quantitative and qualitative information; the software enables us to test various inputs, from isolated cartographic symbols to a set of complex interactive maps. Moreover, it allows us to explore various types of operations or cognitive processes. MUTEP enables group administration, automatic evaluation of results and export to a form that is suitable for statistical analyses (Šašinka and Morong 2012).

MUTEP was installed on a server; preparation and administration of the test battery were therefore realized online. The test was conducted under homogeneous conditions via a web browser and took between 15 and 20 min. The test consisted of an introductory portion and two portions of testing tasks. The introductory portion was necessary in order to obtain the data for the subsequent statistical evaluation.

After filling in a personal data questionnaire, the users were asked to answer several questions about the analysed web map portals, e.g. which portals they use and for what purposes (see Fig. 1).

Fig. 1
figure 1

Illustration from the user test—task 4 (How many pools are located in Güell Park?)

Full size image

In the second portion of the test, users were asked to choose one of four buttons according to the assigned task. Only one option could be chosen. The first task in this portion tested the subjective preferences of the users: the preferred map on two scales (see Fig. 2) and the most suitable symbol for a cinema and a school. These categories were chosen because they were present in all three web map portals, yet they were visualized with different symbols. The same type of task was used for determining the number of bodies of water in a park on MQ to confirm or contradict the results of the pre-test (Fig. 3, task 1–3). The final task in this set of tasks tested whether the users misinterpret the violet polygons in a park on BM as bodies of water (they actually represent the interior structure of public buildings) (see Figs. 1 and 3, task 4). The legibility of map labels for large cities on BM was examined in the same way—the users chose the name of the largest city on the map by clicking on an appropriate button (Fig. 3, tasks 5 and 6).

Fig. 2
figure 2

Different visualizations used on (1) MQ, (2) the old version of GM (until the first half of 2013), (3) the new version of GM, (4) BM

Full size image
Fig. 3
figure 3

(a) Correctness rate for the test tasks: 1–3 bodies of water on MQ, 4–public buildings on BM, 5, 6–labels on BM, 7–9 post offices on BM, GM, MQ, 10–12 pharmacies on BM, GM, MQ, 13–15 parks on BM, GM, MQ, 16–18 intersection of river and railway on MQ, BM, GM; (b) Preferences of tested portals regarding visualization; (c) symbology preferences

Full size image

The task in the third portion of the test involved marking an appropriate point by clicking on the map field. Only one point on the map could be marked. The purpose of this portion was to evaluate the distinctiveness of symbols and users’ efficiency when orienting themselves on a map. The respondents were asked to find the nearest post office to a given street intersection (Fig. 3, tasks 7–9), the nearest pharmacy to a given hotel (Fig. 3, tasks 10–12) and parks (Fig. 3, tasks 13–15) on all three examined map portals. The final task was to mark the intersection of a watercourse and a railway (Fig. 3, tasks 16–18). This set of tasks used similar map segments to ensure the comparability of all three evaluated map portals. Selected types of tasks had to be answered repeatedly to eliminate the influence of the order of the map portals on the test. It usually took more time to perform a task using the first map portal, compared to the time needed for the second and third evaluated portal, due to user familiarity with the task. Correct performance of a task and the time required to accomplish it were measured for the selected tasks.

Results

The main purpose of this study was to identify current usability problems on the following map portals: Google Maps, Bing Maps, and Map Quest. In this section, we compare the potential usability problems arising from our research with the results of Nivala et al. (2008). The results are divided into two parts and correspond to the main research methods: qualitative and quantitative.

Participants in User Test

The user test described in the previous section was taken by 127 respondents (67/60 male/female). The data gathered from the personal data questionnaire shows that the majority of the participants were students of Geography at Masaryk University (average age 22.9 years). Most of the participants use a computer every day, so the user group was quite homogenous.

Qualitative Data/Research

Most of the problems identified relate to the maps themselves. The map is the most important part of a map portal, since it contains all the information that is provided. Moreover, the usability errors concerning the user interface, search operations and help and guidance seem to have been solved since Nivala et al. (2008) conducted her study. It is possible to divide the remaining problems into two groups: map visualization and map tools.

Map visualization is a rather subjective issue; nevertheless, differences can be recognized among the evaluated portals. Several problems described by Nivala et al. (2008) still persist and have already been reported in this paper in the section about methodology. Nevertheless, some other problems have been identified.

Figure 3 shows the differences in map visualization of the tested web map portals of a certain scale. Map visualization can influence users’ ability to read information on a map. It is important to realize that a map is not only a presentation of spatial data, but it is especially a means of communication between the cartographer and the map user, as described by MacEachren (1979).

Qualitative information analysis focused on evaluating the most preferred visualization of the presented maps and on preferences for selected map symbols. Preferences are shown in Fig. 3b and reveal that most users prefer GM’s map visualization, especially in its newest version. Tested symbology concerned map signs for a school and for a cinema. Figure 3c demonstrates that users prefer BM’s symbology. There was no difference in preferences between male and female participants.

Users were also asked to reveal which map portals they use in everyday life. Options included the tested map portals as well as Mapy.cz, the favoured map portal in the Czech Republic. Mapy.cz was not a tested portal because we chose to concentrate on international map portals which are used world-wide. An interesting result is that almost all respondents use GM and Mapy.cz, but only 8.7 % of them work with BM and only 2.3 % use MQ. The popularity of GM may be explained by the amount of the data provided for the Czech Republic (or Europe), since the other map portals focus more on the USA. Another question was aimed at discovering the reasons respondents use map portals. The majority of respondents answered that they use map portals for searching addresses and routes, planning trips and looking for points of interest. An interesting result is that a large number of users (24.8 %) use topographic maps but not orthophotomaps.

Another subjective aspect related directly to Fig. 2 may be the map skeleton. The figures illustrate that the main orientation objects are rivers, parks and main roads on GM and BM, and the road network on MQ. The base map changes automatically when zooming on BM, which can cause changes in map orientation and thus confuse users.

The map content varies on each map and it is changing very quickly. During work on this study, the map content changed several times. For example, the point symbol categories on GM were reduced.

Quantitative Data/Research

Quantitative data provides more objective information, and so we focused on this data in the following parts of the test. First it was found out that the size of labels differs among the portals. It may be observed that the labels on MQ are rather small in comparison to labels on GM and BM. Moreover, labels on BM are disproportionate, so they overlap and obscure the content of the map itself. On GM, additional information about objects is provided using mouseover. The analysis of label size shows that labels in GM are the most usable ones. Another problem with BM is the layer order, which does not follow cartographic rules and makes the map less usable.

The chart in Fig. 3a shows the number of correct answers. Tasks focusing on map labels (tasks 5 and 6) proved that only 44.9 % of respondents were able to answer the question about city labelling correctly on BM (task 5), which confirmed the hypothesis that disproportionate labels may confuse the user. Task 6 has a high correctness rate, but the time required to perform the task was longer.

A potential problem could be found in the fact that map visualization varies geographically on GM. Buildings are presented on MQ and GM, while BM shows only a street network. GM provides shadowed 3D buildings, which cover a significant part of the map and might be considered disruptive. Moreover, on BM it is possible to display the interior structure of some public buildings (e.g. shopping malls, museums); buildings with an interior structure are violet, which is quite similar to the blue colour indicating bodies of water. This is confirmed by the fact that 67.8 % of respondents believed the violet colour denotes a body of water instead of a building, as shown in Fig. 3, task 4.

In tasks 1–3 in Fig. 3, respondents were asked to count bodies of water in parks on MQ. The bodies of water on MQ were ultimately recognized correctly by most users (over 80 % on average), but most said counting them was very difficult and the time needed to perform the task was too long.

Tasks 7–12 sought to analyse which type of searching logic on the map portals is more intuitive for users. MQ is the only portal with the possibility to switch layers on/off, and finding an object requires that the user first turn on a given layer. By comparison, on BM and GM all symbols are present on the map all the time. This implies different searching logics. Users were asked to find the post office nearest to a particular street intersection (tasks 7–9) and to find the pharmacy nearest to a particular hotel (tasks 10–12). The chart shows that the correctness rate was quite similar for all portals (around 90 %) with the exception of finding the pharmacy on MQ, which was 78 %.

Tasks 13–18 required users to find a particular location on all tested portals, such as a park or the intersection of a watercourse and a railway. As shown in Fig. 3, the results were similar for all portals. The correctness rate was around 95 %.

One interesting result was the fact that women required more time to perform the tasks than men did. The biggest difference was when locating the post office nearest to a given street intersection (Fig. 3, tasks 7–9), in which the average time for men was around 40 s and around 50 s for women.

Discussion

The test was designed to examine the above-mentioned persistent problems in map portal visualization. The secondary focus was on differences in map visualization among the portals tested. It can be said that there are some differences among the examined map portals, but they are not significant ones. GM seems to be the most usable map portal, since users needed the least time to perform the tasks when compared to BM and MQ. This fact can be explained by the popularity of GM in the Czech Republic, and respondents of a different nationality might produce different results. The results of a questionnaire established that the majority of respondents know and use only GM. By contrast, MQ may be considered the least intuitive map portal because the majority of respondents required the longest time to perform the tasks. MQ differs from the two other portals the most in the areas of map content and searching logic, and MQ is also the only map portal which offers the option of switching layers on/off. This may appear to be an advantage for users, since the map is not overloaded with map symbols. However, the results of the user test may prove that user orientation on a map is highly dependent on practice with the described scheme.

Further analysis focused on the differences between men and women in the user test. In general, women needed more time to perform most of the tasks. This finding needs to be further verified in extended studies. Another promising direction of future research lies in the area of cartographical visualization analyses of spatial data originating from spatial data infrastructures, such as INSPIRE (INfrastructure for Spatial InfoRmation in Europe) as described, for example, by Řezník (2013). At the same time, issues concerning the popularity and diversity of various map portals in different countries can be examined.

Conclusion

This paper presents a usability evaluation of the most popular world-wide map portals: Google Maps, Bing Maps, and Map Quest. The main aim of this study was to identify potential vulnerabilities in usability of these map portals and compare the results with a previous usability study by Nivala et al. (2008). Since it has been 5 years since Nivala conducted her tests, we asked which problems have already been solved, and we identified persistent problems and possible new weaknesses in map portals. The main methods used in this research were a combination of qualitative and quantitative tasks performed in an electronic user test conducted in a MUTEP environment.

We found that the greatest number of problems mentioned by Nivala et al. (2008) that still persist concern cartographic visualization. Problems regarding user interface, help and guidance and search operations have already been solved.

Table 1 summarizes the results obtained in the user test, establishing the main strengths and weaknesses in map visualization of each portal tested.

Table 1 Strengths and weaknesses of map visualization of GM, BM, and MQ
Full size table

According to Table 1, BM has the most weaknesses of all tested portals. Nevertheless, most users liked the symbology used by BM. GM may be considered to be the best map portal, since most users preferred its general map visualization and its labels seemed most effective. It may be assumed that MQ is the least usable portal, since it uses searching logic that is different from that of the other two portals and users found it to be quite confusing. On the other hand, MQ is the only portal with the option of switching layers on and off.

Visual interpretation helped to identify null-hypotheses that could be proven by the user test. The user test showed that the majority of null-hypotheses were true, with the exception of the hypothesis that considered the GM symbology to be the best one.

The user test results demonstrated the popularity of GM in the Czech environment. On one hand, there are no significant differences between the examined map portals. On the other hand, differences between men and women were identified, which cartographers might respond to by e.g. developing the more user-centred options of cartographic visualization mentioned above. Gender divergences might be further verified by e.g. using eye-tracking tools.

There is great potential for further extension of this study, especially by involving alternative or local map portals or by examining gender differences. It would also be possible to cover various new map portal technologies, such as Google Street View, 45° images or mobile applications.