Keywords

1 Introduction

University websites serve a multifaceted function, acting as a central information hub for students, providing access to everything from course materials to campus events and resources. With the increasing reliance on digital services, students with disabilities have a greater desire to access and engage with online information and services [14], i.e., they require assistive technologies adapted to their specific needs. Therefore, it is essential for universities to provide universally designed websites and thus meet the needs of “all people to the greatest extent possible, without the need for adaptation or specialized design” [23]. In the context of higher education, web accessibility aims to ensure inclusive learning by enabling all current and prospective students to access, perceive, navigate, and interact with university websites effectively, efficiently, and satisfactorily. To support academic success for all students, universities must guarantee that necessary information, announcements, course content, and online services are accessible with ease. By prioritizing web accessibility, universities can promote inclusive lifelong learning and create a positive learning environment, especially for those at risk of being excluded [12, 16, 18].

The World Wide Web Consortium (W3C) is recognized as the leading advocate for web accessibility and recommends accessibility guidelines for websites and equal access [28]. The guidelines include essential features that must be incorporated and are supported by technical specifications and educational materials to promote the development of accessible websites. However, despite regulatory frameworks for public and private organizations implemented for years in many countries [29], unmet accessibility guidelines - and thereby inaccessible features on university websites continue to hinder access, hence segregating and stigmatizing users. For instance, Laamanen et al. [17] explored the accessibility of all Finnish higher education institutions’ homepages. The study revealed that the evaluated websites did not meet current accessibility requirements, there were in fact significant variations among the institutions regarding accessibility compliance. A more comprehensive study conducted by Acosta-Vargas et al. [1] on higher education institutions in Latin America found that the evaluated websites violated many checkpoints in the accessibility guidelines, especially the one for alternative text. Similarly, Alim [3] evaluated the web accessibility of the top research-intensive universities in the United Kingdom and found that the most common violations were related to insufficient text alternatives for non-textual content, contrast problems, and the ability of webpages to work with upcoming technologies and tools.

Ismailova and Inal [12] tested the web accessibility of top universities in Kyrgyzstan, Kazakhstan, Azerbaijan, and Turkey and discovered that most websites failed to meet the guidelines. The distribution of errors revealed that they mostly stemmed from failing to adhere to success criteria pertaining to non-text elements and modifying text sizes. In 2007, 2008, and 2009, Espadinha et al. [7] tested the accessibility of all Portuguese public university websites’ landing pages aiming to detect the difference in accessibility compliance. Although the universities showed significant progress during the assessment period, the overall accessibility results were not at an acceptable level. In fact, only 12.5% of the universities supported services to students with a wide variety of characteristics. In another longitudinal study, Thompson et al. [22] assessed the accessibility of college and university websites in the United States over a five-year period. The findings revealed a noticeable improvement in accessibility according to certain measures. However, other measures demonstrated a deterioration in accessibility, such as full keyboard support.

The Norwegian government has shown a significant commitment to improving the quality of university websites by implementing regulatory frameworks for web accessibility. In 2013, Norway introduced legislation to force public and private sectors to meet accessibility guidelines [29]; and as of February 2023, all organizations must have an accessibility statement in place on their websites [25]. Nevertheless, previous research has consistently shown that professionals in general have low awareness of the field [8], and, moreover, the websites have significant accessibility issues [10, 19, 21]. This goes against national regulations and must be fixed to offer fully accessible online services to everyone in society. Since universities have the primary responsibility to provide information for all prospective and current students - no matter their skills or abilities, it is crucial to explore to what extent they have designed their websites to ensure accessibility, usability, and inclusion. Thus, in this study, ten Norwegian university websites were evaluated by performing manual and tool-based evaluations.

2 Methods

By conducting manual and automated tests, ten Norwegian universities - owned by the Ministry of Education and Research - were assessed regarding their compliance with WCAG 2.1 level AA [30]. The list of universities was obtained from the website of the Directorate for Higher Education and Skills [20], and all tests were conducted in February 2023. The names of the universities are the Norwegian University of Science and Technology (NTNU), Norwegian University of Life Sciences (NMBU), University of Bergen (UiB), University of Oslo (UiO), University of Stavanger (UiS), University of Tromsø - The Arctic University of Norway (UiT), University of Agder (UiA), Nord University (Nord), University of South-Eastern Norway (USN), Oslo Metropolitan University (OsloMet).

The regulations say that public websites must provide complete keyboard support and be compatible with assistive technologies, such as screen readers, to ensure easy access and navigation for individuals who are blind, visually impaired, or otherwise rely on a keyboard. In this regard, a manual evaluation was conducted using a screen reader to identify accessibility violations in detail - combined with a commonly used online evaluation tool called WAVE [31]. By performing both manual and tool-based assessments, we aimed to identify as many accessibility violations as possible based on the WCAG guidelines.

2.1 Scope of the Evaluation

To ensure a practical evaluation framework, representative samples of pages were selected from each university website rather than testing all pages. This is common practice, followed by many researchers when evaluating website accessibility. Therefore, ten representative homepages from each university website were evaluated, including the homepages for the university, library, faculty, department, staff, admission for a bachelor’s program, admission for a master’s program, admission for a Ph.D. program, exchange agreement, and student life, respectively. Homepages are most often the visitors’ first point of contact before navigating to other pages; thus, they must comply with the accessibility guidelines. Likewise, inaccessible homepages, whether partially or completely, can cause difficulties when trying to access other website pages. Furthermore, accessibility violations are often repeated when two pages have the same layout and structure [I0], i.e., selected homepages from each university have their unique layout, structure, menu design, and content flow, thereby representing many subpages on the respective university website. Please note that these representative pages were deemed essential by current and prospective students seeking information about a university.

2.2 Measures

Online tools have been employed to evaluate website accessibility for years and provide useful feedback on the identified accessibility violations and help developers in rectifying the detected issues. WAVE (Web Accessibility Evaluation) has been extensively used in several accessibility studies [10, 15, 33] and was utilized to evaluate accessibility compliance in this study. The tool generates proposals to fix errors and alerts based on WCAG compliance and offers comprehensive accessibility evaluation information such as errors, contrast errors, alerts, features, structural elements, and ARIA (accessible rich internet applications). To conduct this research, we established the evaluation criteria to identify the number of errors on evaluated pages that compromise WCAG 2.1 conformance level AA.

However, previous research demonstrates that less than half of the available accessibility errors are detected by online tools [13, 27]. Therefore, elucidating the outcomes of accessibility evaluation derived from online tools requires vigilant scrutiny. For example, online evaluation tools can detect if the alternative text is linked to an image but cannot judge its accuracy without interpreting the image’s content. Consequently, manual evaluation is always needed to provide higher accuracy and coverage of accessibility issues [13, 27]. It is, moreover, the most accurate and reliable way of identifying accessibility issues on a website [26]. In this study, we used the Google Chrome Screen Reader to test each selected homepage individually. Afterward, we attempted to access each selected homepage from their respective websites.

3 Results

3.1 Manual Evaluation (with a Screen Reader)

We classified the various issues detected in the evaluation into different groups as per the categorization framework defined by WebAim [32]. These groups included focus indicators, navigation order, items that should not receive keyboard focus, inaccessible custom widgets, and lengthy navigation. The tests indicated that the provision of alternative text for non-textual content on the websites must be more consistent. Images that lack descriptive information, especially when text is already present within the image, should have alternative text to comply with regulations. Also, some web pages provided comprehensive descriptions of their images, while others did not include alternative text for non-textual content - or offered only perfunctory descriptions. There were tendencies for some websites to use animations; however, these elements were either not focusable or did not contain alternative text. Needless to say that providing accurate and comprehensive alternative text consistently for images, videos, and other non-textual content is crucial for users who rely on screen readers.

The websites also exhibited problems with keyboard access, making it difficult to follow the order of the content. On some pages, the keyboard focus shifted abruptly between different sections of the page, causing confusion and making it hard to determine the precise location. To ensure that website users can access all information on a page, regardless of their mode of access, websites must be designed to accommodate keyboard-based navigation. Additionally, some websites demonstrated illogical and unintuitive default keyboard navigation orders, particularly evident in the tabbing order - deviating from the standard visual flow from top to bottom and left to right. In some instances, some crucial areas were skipped, while insignificant portions were read aloud. Furthermore, some websites featured inaccessible custom widgets, such as buttons, hamburger menus, and chat bubbles.

A notable accessibility challenge in web design involves the limited functionality of screen readers, which may fail to accurately capture and convey certain text elements. Consequently, users who fully rely on screen readers may require assistance in interpreting the page content and may thus face incomplete or inaccurate information. The evaluation showed instances where screen readers selectively ignored the critical text and focused solely on links present on the page. Given the significance of these challenges, websites must be optimized to ensure that all content is accessible. Failure to do so can result in substantial barriers to accessing and comprehending important information, hence excluding and discriminating against screen readers.

The study also identified challenges in the navigation process on many of the websites. i.e., the abundance of interactive elements requires users to tab through numerous components, a process which exacerbates the challenge of accessing the desired information. Although the ‘skip to the main content’ feature enables easy keyboard navigation, it could have been more effective on some websites. In sum, this confuses the user’s location on the page, disrupts their sense of orientation, forces them to spend additional time figuring out their position, and highlights the need for such features.

3.2 Tool-Based Evaluation

The tool-based evaluation generally complemented the manual review by addressing similar issues. The results showed that none of the university websites were error-free (see Table 1). As a matter of fact, all websites displayed a certain number of errors in web accessibility and were furthermore not consistent with accessibility guidelines. In total, the University of Stavanger had the highest number of checkpoint violations (n = 1661), followed by the Norwegian University of Life Sciences (n = 465), Nord University (n = 111), the University of Adger (n = 97), the University of South-Eastern Norway (n = 75), and the University of Bergen (n = 72). Conversely, the Norwegian University of Science and Technology (n = 13), the University of Oslo (n = 18), Oslo Metropolitan University (n = 24), and the University of Tromsø - The Arctic University of Norway (n = 25) contained the least number of violations.

Table 1. Frequently repeated errors across the universities
Full size table

The most frequent accessibility issue detected (n = 1464) was ‘very low contrast’, which relates to 1.4.3 Contrast (minimum) at the conformance Level AA. Insufficient contrast between image and text color negatively impacts user experience, particularly for individuals with limited vision or color deficiency. Therefore, background and foreground color combinations should be suitable for all users. The University of Stavanger (n = 796) and the Norwegian University of Life Sciences (n = 408) had the highest number of issues with very low contrast, while the Norwegian University of Science and Technology and Oslo Metropolitan University only had three contrast errors each.

The ‘linked image missing alternative text’ error was the second-highest error incidence, relevant to 1.1.1 Non-text Content (Level A) and 2.4.4 Link Purpose (Level A). This error occurs when images and links lack alternative text and images within a link - that do not contain alternative text, results in an empty link. Another common error was the absence of alternative text, indicating that the universities did not provide descriptive text for non-text content. Without this, non-text content will be unavailable to screen readers. Hence, it is necessary to add the ‘alt’ attribute to each non-text content along with an adequate and equivalent description describing the content, as well as presenting the function of the link.

Other errors with a high incidence were ‘empty link’ (n = 76), ‘missing alternative text’ (n = 68), ‘empty button’ (n = 61), ‘missing form label’ (n = 36), and ‘empty heading’ (n = 31). Relevant checkpoints include 1.1.1 Non-text Content (Level A), 1.3.1 Info and Relationships (Level A), and 2.4.4 Link Purpose (Level A). Only one university included the following errors; ‘image map area missing alternative text’ (n = 15), relevant to 1.1.1 Non-text Content and 2.4.4 Link Purpose, and ‘language missing or invalid’ (n = 7), relevant to 3.1.1 Language of Page. These errors can negatively impact the ability of people to comprehend the content and navigate the website efficiently. Sufficient descriptions must accompany all design elements to prevent the loss of control and confusion for screen readers. Rectifying these issues can be achieved by incorporating a descriptive title for the form element and providing text within the link.

Table 2. Accessibility errors identified on evaluated homepages
Full size table

Moreover, the findings indicate that the university homepages exhibited the highest number of accessibility errors (n = 676), followed by the library (n = 557), department (n = 348), and faculty (n = 305) homepages (Table 2). Among the universities, the highest number was detected on the University of Stavanger homepage (n = 561), followed by the Norwegian University of Life Sciences (n = 66). Conversely, the homepages of the University of Tromsø - The Arctic University of Norway (n = 1) and Oslo Metropolitan University (n = 2) had the least number of errors. The student life (n = 67) and staff (n = 76) homepages were found to have the least accessibility issues. In terms of admission homepages, the bachelor’s, master’s, and Ph.D. program admission pages contained 117, 99, and 113 accessibility errors, respectively. These results suggest that university homepages are particularly prone to accessibility errors, and efforts should be made as soon as possible to enhance the accessibility of these pages - thus ensuring a more inclusive and user-friendly online experience for all users.

4 Discussion

This study explored the current state of accessibility of ten Norwegian university websites by performing both manual and tool-based evaluations to determine their level of compliance with WCAG 2.1 Level AA. The results showed that all websites failed to meet accessibility compliance, i.e., they lacked the required features to adhere to the recommended level of accessibility conformity and had a considerable number of accessibility errors. Consequently, this implies that the impact on user navigation, particularly for those with visual impairment and blindness, is adverse.

The most common errors were related to providing a text equivalent for non-text content, as outlined in WCAG 2.1 checkpoint 1.1.1 Non-text content. The main violations identified in the tool-based evaluation included missing alternative text, linked images missing alternative text, and image map areas missing alternative text - findings that are consistent with previous research [1, 2, 12, 17, 18]. Moreover, the manual evaluation revealed that the websites show inconsistency in offering alternative text for non-textual content to further enhance the use of screen readers. Some pages on the same website provided detailed descriptions for their images, while others did not - or provided only brief descriptions that may not accurately convey the image’s essence and content. Previous research supports this finding and concludes that people with visual impairments cannot comprehend the content properly unless an adequate and equivalent description of non-text content is provided [10].

Concerning design elements, the tool-based evaluation identified empty links, empty buttons, empty headings, and missing form labels as the primary accessibility violations, which are all necessary to provide clear and relevant information on a website. Alhadreti [2] and Calvo et al. [4] reported that the lack of adequate description for design elements and inappropriate use of headings is the most commonly violated accessibility error, respectively. Both issues can leave screen readers feeling disoriented, severely impacting their engagement and performance. The tool-based evaluation also revealed that all websites had low contrast between the background color and foreground text color, which causes problems for users with low vision and color deficiency. This finding aligns with those of Laamanen et al. [17], who identified inadequate color contrast of the text and background on higher education institutions’ web pages, with 58% of the total errors relating to low color contrast. Likewise, the research of Alim [3] reported that almost three-quarters of the university websites contained contrast errors - with an average of nine errors.

Regarding keyboard access, the manual evaluation identified numerous issues that negatively impact screen readers’ comprehension of the website’s structure. On some pages, the content order was extra challenging to follow; and, in some cases, the keyboard focus suddenly changed between different sections leading to confusion and difficulty pinpointing the exact location. Besides, some websites had unintuitive keyboard navigation orders, which should be structured from top to bottom and left to right. These findings are consistent with Parajuli and Eika’s [19] study, where navigation was identified as the most frustrating issue screen readers encounter when accessing websites. Furthermore, some websites lacked functional focus indicators - notably with sufficient color contrast - enabling users to navigate through interactive elements such as links, buttons, images, and input fields on the page.

Last but not least, testing the websites using a screen reader showed that the navigation process often was excessively lengthy, which creates a significant obstacle for screen readers. Adding a ‘skip to main content’ feature is an effective way to improve the navigation of websites. However, some of the websites were found to have implemented this feature ineffectively. This improper usage may lead to additional user confusion and, moreover, users feeling a lack of control over their browsing experience.

Taken together, these findings clearly show that an absence of conformity with the accessibility guidelines results in web accessibility barriers that may impact an individual’s daily life to various extents. The violations prevent users, particularly those with visual impairments, blindness, and difficulties in controlling mouse movements due to various physical and motor conditions, from easily accessing and navigating any text, image, audio, video, or program on a web page. Considering the fact that all people have the same rights when it comes to accessing online information and services provided by public institutions, e.g., universities and university colleges, the urgent need for equal access to information for all cannot be ignored to the same degree as today. Inal and Ismailova [9] suggest that web accessibility is an indication of human development, measuring various countries’ levels of social and economic growth. Moreover, seeing that the EU’s Action plan on human rights and democracy was implemented in 2020 [6], prioritizing human rights and democracy ought to be the primary goal for every European country by now.

Despite the fact that legal obligation has been implemented in many countries - Norway included - to ensure the accessibility of web content, and furthermore that country-specific regulations have been instrumental in promoting awareness among practitioners concerning implementing accessibility practices in the web development process [2, 8], a significant number of websites still remain inaccessible. Needless to say that this is a serious situation that hampers diversity, equity, and inclusion in digital society worldwide [e.g., 3, 10, 17]. Moreover, it indicates that enacting laws and regulations at the national level does not necessarily guarantee accessibility, and additionally, that changing practitioners’ mindsets, etc., might take more time and effort than first expected. That said, previous studies show that raising awareness among practitioners regarding the importance of accessibility leads to the incorporation of accessibility attributes into digital systems [11, 17], and so we argue that a positive change is possible.

5 Conclusion and Recommendations for Further Work

This study aimed to assess the accessibility of ten university websites in Norway through a combination of manual and tool-based evaluations conducted in accordance with WCAG compliance. Overall, none of the university websites passed the test and the most common violations were lack of alternative text and very low contrast. As discussed above, the results from this study provide clear evidence that not adhering to accessibility guidelines creates obstacles to web accessibility - with a variety of negative consequences for the users. Thus, we recommend improvements in the following key areas:

  • Provide text equivalents for non-text content

  • Address design element issues

  • Optimize keyboard access

  • Streamline navigation process

  • Raise awareness and provide education amongst all stakeholders

  • Embrace inclusive education - therefore, accepting diversity amongst students

In line with the UN’s focus on “leaving no one behind” [24] and the European higher education policies, that strongly promote inclusive education to provide equal opportunities for diverse students [5], the universities examined in this study should consider increasing both understanding for - and awareness of the accessibility requirements to further enhance the accessibility, usability, and inclusion of their websites. Seeing that most of the detected errors in this study were associated with a lack of awareness and a general understanding of the accessibility prerequisites of people regardless of skills or abilities, we find it imperative to keep educating and training all stakeholders - including content editors, designers, developers, and managers - in the ecosystem to ensure an even more accessible web, thus promoting access and improved outcomes for all students in higher education in Norway. Also, we call upon the research society to conduct more studies focusing on user evaluation, including potential users with various disabilities, as this would help to understand the bigger picture and why it is important.