Keywords

1 Introduction

The growing change in the development and use of information and communication technologies (ICT) and the need to create information services from a broad perspective, has led to a large part of these services being provided through Web portals [27]. In the educational field, the institutions of Higher Education University publish on their websites the most relevant information and services they provide [1].

Access to communications by persons with disabilities is a right, which is covered by the UN convention, in order to eliminate the discrimination to which they are subject [1]. Therefore, in order to allow more people to access the Web, it is important to develop features that allow accessibility [1].

In Peru, the legislation on accessibility on the web consists of:

  • Law No. 28530 - Law on the promotion of Internet access for people with disabilities and the adaptation of the physical space in public Internet booths [5].

  • Ministerial Resolution 126-2009-PCM [1] of the Presidency of the Council of Ministers, which approved the guidelines for accessibility to web pages and mobile telephony applications for public institutions of the National Computing System. The purpose of the document is that the public institutions of the Peruvian State apply the Web Content Accessibility Guidelines 1.0 (WCAG 1.0) [27].

The Ministerial Resolution is still deficient in terms of accessibility, since it is also necessary to adopt WCAG 2.0. It is also important to note that it is only directed to the web pages of public institutions [26].

Knowing this problem, this evaluation work seeks to make a current and uniform diagnosis of a set of Peruvian Public Universities, which will allow to know the web accessibility problems they present and provide appropriate recommendations [18].

For this purpose a set of web pages of Peruvian public universities were selected considering the recommendations of the Web Accessibility Initiative (WAI) of the World Wide Web Consortium (W3C) and its methodology Conformity Assessment Web Accessibility version 2.0 [13, 14], which is considered an international standard adopted by many countries and is already an ISO called ISO/IEC (ISO/IEC 40500: 2012) [36].

1.1 Web Accessibility

Web accessibility can be defined as a universal access to the Web, regardless of hardware, software, language, culture, geographical location or the physical or mental abilities of users [30].

The goal of web accessibility is to ensure that the information or services delivered through web sites are available and can be used by the widest possible audience [22].

1.2 WCAG Web Accessibility Guidelines

The WCAG guidelines are aimed at those who generate content for the Web and consist of specific recommendations whose focus is on making the content presented in an accessible form. WAI elaborated annexed guides that exploit each point and detail the steps to follow to implement them [21].

WCAG version 1.0

WCAG version 1.0 was an important advance in making the Internet more accessible for people with disabilities. Completed in 1999, WCAG 1.0 provides 14 guidelines and 65 verification points that can be used to determine if the accessibility of a web page is met, through 3 priorities and 3 levels of compliance or adequacy levels [38, 31]:

  • Priority 1: Must be fulfilled. It is a basic requirement. It refers to those requirements of the verification points that a Web page must comply with in order not to hinder or prevent access to a large group of users.

  • Priority 2: It should be fulfilled. Remove certain barriers. That is, those elements that “must” be grouped to eliminate important barriers in access by different groups are grouped.

  • Priority 3: It could be fulfilled. It would improve accessibility for certain groups. It refers to certain characteristics that the Website must comply with in order to reduce the difficulties that some groups of users may encounter when accessing it. These are recommendations that affect a smaller number of potential users, but that improve the accessibility and usability of the page in general.

In accordance with these priorities, there are 3 levels of compliance in the accessibility assessment:

  • Level A. All the checks of priority 1 have been met.

  • Level AA. All the checks of priority 1 and 2 have been met.

  • Level AAA. All the checks of priority 1, 2 and 3 have been satisfied.

WCAG version 2.0

WCAG 2.0 improves the initial standard by applying to more advanced technologies, being more flexible, detailing in a more precise way which are the checks to be carried out and incorporating better documentation [22].

Unlike the WCAG 1.0 developed exclusively for web technologies that existed in 1999 (HTML, CSS and JavaScript), the WCAG 2.0 was developed in a technologically neutral way. However, sufficient techniques were also developed that provide guidance and examples to meet the guidelines using specific technologies.

he WCAG 2.0 guidelines are composed of 4 principles, 12 guidelines and 61 criteria, each of which will have different levels of compliance (A, AA, AAA), in addition to a set of sufficient techniques and advisory techniques [35, 39]:

  • Principle 1: Perceptible: The information and the components of the user interface must be presented in the way they can be perceived. It consists of 4 guidelines and 22 compliance criteria.

  • Principle 2: Operable: The components of the user interface and navigation must be operable. It consists of 4 guidelines and 20 compliance criteria.

  • Principle 3: Understandable: Information and user interface management must be understandable. It consists of 3 guidelines and 17 compliance criteria.

  • Principle 4: Robust: The content must be robust enough to rely on its interpretation by a wide variety of users. It consists of 1 guideline and 2 compliance criteria.

1.3 Tools for Automatic Review of Accessibility

Tools for automatic review of accessibility are programs or online services that check the level of accessibility of a web site using a variety of tools. Using these automatic testing tools helps minimize time and seems less complicated [25]. Some of these automatic tools are described on the W3C website [37].

The use of these accessibility tools will depend on the size and scope of the websites. Some tools are easier to use, others allow to evaluate a page at a time, some provide only a quick scan and evaluation, while others offer a greater focus on the details and are able to review a website with multiple pages [32].

It is a fact that automated tests save time and labor, but manual tests (performed by humans) give greater precision [40]. According to Saleem’s review [25] there are 3 categories of results when evaluating websites. First, the real positives are real problems discovered by these tools. Secondly, false positives are errors reported by tools, but when human judgment is used, errors do not occur. Finally, false negatives are defined as errors that cannot be detected by tools [29].

It is for this reason that studies indicate that when performing an accessibility evaluation, a combination of manual and automatic evaluations should be used as much as possible [25].

2 WCAG 2.0 Web Accessibility Evaluation on University Websites

In the study by Zaphiris et al. [20] an accessibility assessment work was carried out on the websites of 7 universities in Cyprus (3 public and 4 private), for which an evaluation was carried out that combined manual and automatic tests. In addition, questionnaires were carried out to determine whether the university authorities knew about the provisions and regulations for web accessibility within their study centers.

In the study by Hilera et al. [12] describes an evaluation of accessibility of the contents of the Web portals of some of the most important foreign and Spanish universities according to three university rankings, checking compliance with WCAG 2.0.

On the other hand, Kane et al. [16] conducted an accessibility study using a multi-method analysis on the homepages of the top 100 universities at the international level, where each site was analyzed by reviewing compliance with WCAG 2.0 accessibility standards and using automatic tools and manual tests.

In the paper Pendergast et al. [24], an examination is made of the requirements of accessibility laws, the formation of the accessibility initiative and the consequent WCAG 2.0. The accessibility testing tools for web sites and web content are discussed and then used to measure the level of compliance of several universities in the United States, finding that in all the websites of the universities had multiple accessibility errors.

In both Navarrete and Lujan Mora [22] perform an evaluation work through which is verified through software tools, the level of compliance with the guidelines on web accessibility and the observance of the syntax of markup languages and style sheets, in the web portals of the main universities of Ecuador. From the results of the analysis a series of guidelines is established to improve the design of the web portals of the universities. The study finds its motivation in the regulations on web accessibility in Ecuador that is part of the declaration of the National Plan for Good Living, 2013–2017.

On the other hand, Laitano [18] presents a study where he makes a first diagnosis of web accessibility carried out in 2012 on a sample of pages of the Argentine public university space. The evaluation verifies the compliance of the WCAG 2.0, contemplating the methodological recommendations of the W3C. The results suggest that the web accessibility barriers found are mostly serious (level A). The most frequent are related to the syntax of the markup language, with the presentation of the content, with the non-textual content and with the visual readability of the text. It also shows that certain groups of people could be particularly favored by the solution of these barriers.

In as much Cordova Solís [6] makes a comparative study of web accessibility in portals of Peruvian universities that offer distance education at the undergraduate level, following the fulfillment of the priorities of the web accessibility contemplated in the Peruvian legislation based on the directives of the W3C. To this end, evaluations are carried out using accessibility assessment tools (HERA and TAW), showing a high degree of non-compliance with the priorities, which constitute information barriers and access to people with a disability who wish to study in a distance mode.

In the work of Ismail and Kuppusamy [15], an exploratory study was conducted on the accessibility of the web pages of 320 universities in India. For this evaluation, automatic tools were used, and a classification was made considering the compliance guidelines of WCAG 2.0. After this evaluation, a comparative classification was made in 3 layers depending on the level of accessibility. The results of the analysis present proposals for the improvement of the websites.

In the work of Kesswani and Kumar [17] an accessibility assessment is carried out to prestigious universities in India using automatic tools, as well as evaluating the websites of prestigious universities in Germany, China and Russia. The results indicate that most of these study centers follow less than 50% of the WCAG 2.0 accessibility guidelines.

While in the work of Acosta-Vargas et al. [23] a web accessibility assessment is carried out using the content accessibility guidelines of WCAG 2.0 and for this they chose 20 universities that are within the Webometrics ranking. These universities are from North America, Latin America, Asia, Africa and Oceania. For this, automatic tools were used with which they obtained different results.

In conclusion it can be said that the largest number of studies on web accessibility evaluation with the WCAG 2.0 guidelines have been carried out in European universities and in North American universities.

With regard to studies in Latin American universities, there is a growing interest in improving the web accessibility of websites both for online and face-to-face education, taking into account that many of these countries have regulations that indicate the use of guidelines of web accessibility based on WCAG 2.0 and even on WCAG 1.0, which are mostly ignored due to lack of knowledge, making the websites of their universities have a very low level of accessibility.

It is also observed that a large part of the studies is carried out with automatic tools, because many of them are quite reliable and provide information that can be evaluated and analyzed quickly without the need for manual tests.

On the other hand, there is very little work on evaluation of web accessibility in Peruvian universities, only found a work on universities that provide distance education, these universities being only private.

3 Methodology

We analyzed a set of web pages of Peruvian public universities that are within the Webometrics Ranking of 2016 considering the WCAG 2.0 [39] and the Methodology of Evaluation of Web Accessibility Conformance (WCAG-EM) [11].

Taking these recommendations into account and what is indicated in the accessibility evaluation works on the websites of the Universities of Lujan-Mora et al. [1, 22], and Laitano [18], the following steps are proposed:

  1. 1.

    Selection of Universities for analysis.

  2. 2.

    Definition of the web pages of the selected universities.

  3. 3.

    Definition of the level of compliance.

  4. 4.

    Selection of the tools for the analysis.

  5. 5.

    Execution of the evaluation.

  6. 6.

    Analyze the results.

3.1 Selection of Universities for Analysis

Based on the work of Hilera et al. [12], the main objective of this work is to perform an accessibility evaluation of web pages of a selection of Peruvian universities that are in an international academic university ranking, in this case the Ranking of Universities of Webometrics, which has a great academic reputation. This ranking, called “Webometrics Ranking of World Universities”, is a portal of the Cybernetics Laboratory of the Spanish CSIC [8], which considers the productivity and effect of academic products placed on the Internet.

Before selecting universities, it was revised in Peru there are 51 public universities recognized by the National Superintendence of Higher Education University (SUNEDU) [28], of which 11 are from Lima and 40 other departments. Of this group only 30 public universities are in the Webometrics Portal Ranking 2016 [7]. From this group of 30 public universities it was decided to choose the first 14 of the Ranking as a sample and shown in Table 1.

Table 1. 14 public universities selected for the study
Full size table

3.2 Web Pages of the Selected Universities

A preliminary step to the evaluation of accessibility is the selection of the web pages of each university, whose accessibility will be analyzed [12]. That is why different works use different amounts of pages per university, among them we have: Laitano [18], Lujan-Mora et al. [3] and Mohamad and Ahmi [4] who used only the main page (home), in the works of Hilera et al. [12], Lujan-Mora and Acosta [2] were evaluated 3 pages and Navarrete and Luján Mora [22] evaluated 6 pages.

For the present work we have selected 2 sufficiently representative pages that are common for each of the 14 public universities selected in the sample, according to the following criteria:

  1. 1.

    The main page of the website (home), being the most important page in terms of accessibility, because if the main page is not accessible, users may have problems to reach other pages of the site.

  2. 2.

    Academic Offer, Careers. For this case, only one informative web page of an undergraduate program should be taken.

In this way when analyzing 2 web pages of each of the universities you can obtain more balanced results than if we only analyze the main page, and there is a high probability that between the 2 include a larger number of problematic components compared to accessibility.

It is true that a more exhaustive study of all the web pages of the university would lead to more precise results, but it would be very expensive, and it is probable that most pages of a site follow the same pattern.

3.3 Level of Compliance

After having defined the set of web pages that should be evaluated for the 14 selected public universities, the WCAG 2.0 accessibility assessment indicators are defined, taking into account that WCAGs have a high degree of acceptance at international level, and that they constitute the fundamental indicator in the works of Laitano [18], Lujan-Mora et al. [2, 3, 22], Mohamad and Ahmi [4] and Hilera et al. [12].

Even though the Peruvian regulations establish version 1.0 of the WCAG [9], for the WCAG 2.0 version it has been chosen because of the advantage of being technologically neutral and on the presumption that Peruvian legislation will soon adopt the new standard [18].

There are 3 compliance criteria (A, AA, AAA) for the WCAG guidelines so the conformance levels A and AA were applied, since that is what the WCAG-EM conformity assessment methodology [10] advises and recommends for a good evaluation. The AAA level is desirable, so it is not considered. The use of this level of compliance is supported by the works of Laitano [18], Mohamad and Ahmi [4] and Lujan-Mora et al. [2, 3, 22].

3.4 Tools for the Analysis

There are numerous web accessibility evaluation tools, of which 92 are recommended by the W3C through a list [40], which fulfill several functions:

  • Tools for automatic evaluation using various types of standards such as WCAG 2.0, WCAG 1.0, Sect. 508, etc.

  • Support tools for manual evaluation that will be used by experts in accessibility evaluation, using WCAG 2.0, WCAG 1.0, Sect. 508, etc.

  • Tools for HTML, CSS (or grammar) markup validation.

  • Tools for color and contrast evaluation.

  • Tools for epilepsy detection.

  • Tools for the readability evaluation, etc.

That is why, following the suggestions of the WAI [32] evaluations were made using automatic tools. They can only check compliance criteria that support automation.

Because be evaluated accessibility of websites universities, it has made a summary of accessibility evaluation tools that are listed in the Table 2, considering web evaluation work at universities and government entities:

Table 2. Tools for accessibility evaluation
Full size table

3.5 Evaluation

In this stage it was verified that each web page complies with the requirements of conformity and with the adequacy level AA of the WCAG 2.0. using the selected automatic evaluation tools, which will check those compliance criteria that support automation, in this case the tools: Achequer, TAW and Examinator.

In addition, grammar validation tests were carried out, which included a review of the proper use of the valid HTML and CSS code. For this, two W3C services were used: Markup Validation Services that verifies the validity of the marking of web documents in HTML, XHTML, etc. [34] and CSS Validator Service that checks the style of the pages of a web page for the specifications of the CSS [33].

3.6 Results

In this stage, the specific details of the evaluation of each web page of the selected universities were registered, in the following way:

  • The summary results of the evaluations using automatic tools, in templates where the type of web page is specified (in this case the main page and the academic information page), the tool with which it was evaluated and the result of this evaluation according to the criteria and guidelines of the WCAG 2.0, with their respective scores. With the analysis of these scores you can have a preliminary result of the level of compliance level A and AA.

  • The results of grammar assessments (validation of web documents in HTML, XHTML, CSS), by means of automatic tools.

  • Tables and graphs of the problems found by compliance criteria A and AA, in units and percentages, were prepared.

4 Accessibility Evaluation

After defining the tools for evaluating the web pages of the selected public universities, the evaluation was carried out.

4.1 Grammar Validation Evaluation

The results obtained from the validation of HTML and CSS for the home of the selected universities are shown in Table 3 and in Table 4. the range of errors used. With respect to the evaluation of the undergraduate program page, the results obtained by the grammar assessment tools are shown in Table 5.

Table 3. HTML y CSS
Full size table
Table 4. Range of errors by color
Full size table
Table 5. HTML y CSS
Full size table

4.2 Web Accessibility Evaluation Through Automatic Review

To carry out this evaluation, 3 automatic evaluation tools were used: AChecker, eXaminator and Taw 2.0.

AChecker

The first evaluation was carried out with AChecker using WCAG 2.0 guidelines up to level AA. AChecker classifies accessibility problems into three broad categories: Know Problem, which are problems that have been identified as accurate barriers; Likely problems, which are problems that have been identified as probable barriers and require human judgment; and Potential problems that also require human judgment [19].

For this evaluation, only Known Problems have been taken into account, since according to AChecker are the problems that should be repaired immediately [19].

The problems encountered using AChecker of type A (celestial bar) and AA (green bar) when evaluating the home page are shown in Fig. 1 and in Fig. 2 when evaluating undergraduate program page.

Fig. 1.
figure 1

Evaluation of level A and AA of the home with AChecker (Color figure online)

Full size image
Fig. 2.
figure 2

Level A and AA assessment of the undergraduate program page (Color figure online)

Full size image

eXaminator

The second evaluation was done with the eXaminator tool, which is a free service to check the accessibility of a web page and has the following characteristics:

  • Evaluates the application of accessibility guidelines in the HTML and CSS content of a page.

  • Use WCAG 2.0 as a reference.

  • Rate each test on a scale of 1 to 10 depending on the number of errors and hits detected.

  • Provides help with links to WCAG 2.0 documents.

  • Weight the notes according to the importance and reliability of each test.

  • Summarize the results in a general score.

From this overall rating of 1 to 10 that the tool performs, it can be said that the highest value indicates that the page has the best level of accessibility and the lowest value indicates that the page has the lowest level of accessibility. For this, the following scoring ranges have been defined: High (7-10), Medium (6.9-4.0), Low (0-3.9).

It should be noted that for evaluation of the home pages of the 14 public universities, the website of Universidad Nacional de Trujillo could not be evaluated by the tool, therefore 13 of the 14 selected universities were evaluated.

Below in Fig. 3 you can see the results of the evaluation on the Home page (green bar) and the undergraduate program page (red bar).

Fig. 3.
figure 3

Evaluation using eXaminator (Color figure online)

Full size image

TAW

The third evaluation was made with the Taw 2.0 tool, which is an online tool that analyzes the page, based on WCAG 2.0 and generates a report.

The resulting page is a summary document, which shows the total of the Problems (corrections are necessary), the Warnings (must be reviewed manually) and the Unverified Points (which require a complete manual analysis) and organizes them for each principle (Perceptible, Operable, Understandable and Robust).

For the present work only, the Problems have been considered. The Warnings and Unverified Points have been discarded due to the need for expert manual evaluation.

In the case of the evaluation of the Home page of Universidad Nacional San Cristobal de Huamanga, the TAW tool could not evaluate it, so 13 universities of the 14 were evaluated (Fig. 4). For the case of the evaluation of the undergraduate program page, only 12 universities of the 14 were evaluated (Fig. 5).

Fig. 4.
figure 4

Accessibility evaluation of level A and AA with Taw 2.0 (Home page)

Full size image
Fig. 5.
figure 5

Accessibility evaluation of level A and AA with Taw 2.0 (Undergraduate program page)

Full size image

5 Results

5.1 Accessibility Levels

It should be noted that the results of automated assessment tools are not comparable because they use different evaluation criteria, however they can give a similar view of the level of accessibility of the pages.

To show the general results of the evaluation of automatic tools an equivalence shown in Table 6 has been designed.

Table 6. Ranges by error and score
Full size table

Table 7 summarizes the results of the 3 automatic tools aChecker, eXaminator and Taw 2.0 at AA level of the Home page of the universities and Table 8 of the undergraduate program page.

Table 7. Accesssibility results (Home page)
Full size table
Table 8. Accessibility results (undergraduate program page)
Full size table

In general, none of the university pages achieves the AA compliance level of WCAG 2.0 and therefore would not meet the minimum level recommended by the WCAG-EM Conformity Assessment Methodology to indicate that a Website is accessible.

5.2 Most Common Accessibility Problems

Using the tools TAW 2.0 and AChecker, it has been possible to classify the most common accessibility problems, based on non-compliance with WCAG 2.0 criteria at levels A and AA for the 14 selected universities. It must be borne in mind that each tool does not necessarily evaluate the same criteria, so some differences will be found.

5.2.1 Processing (Level A)

The error in the compliance criterion 4.1.1 is presented due to the duplication of type ID values. Duplication can be problematic for user applications that depend on this attribute to correctly present relationships between different parts of the content. Also presented by misuse of labels attribute start and end markers, which would prevent assistive technology to interpret the page.

5.2.2 Purpose of the Links (Level A)

The error in criterion 2.4.4 is shown because the Title attribute does not provide additional information to clarify in more detail the purpose of the link. To do this, it must be verified that the Title attribute, together with the link text, described the purpose of the link.

If the supplementary information provided by the title attribute is something that the user must know before following the link, as a warning, then it must be provided in the link text and not in the title attribute.

5.2.3 In-Formation and Relationships (Level A) and 1.4.4 - Change of Text Size (Level AA)

It is recommended to use CSS to control the visual presentation of the text.

5.2.4 Non-Text Content (Level A)

The failure of criterion 1.1.1 arises due to the omission of the alt attribute in elements img, area, and input of type “image”. Alternative texts are the main means to make information accessible, since they can be interpreted through any sensory modality (visual, auditory or tactile) that best meets the needs of the user.

5.2.5 Contrast (Level AA)

The error in criterion 1.4.3 is due to the specification of colors of the front without specifying background colors or vice versa. It is recommended that the foreground and background color be defined in the same CSS rule. In this way, users with loss of vision or cognitive ability, with language and learning problems, are given access to a web page.

5.2.6 Keyboard (Level A)

It is recommended, to comply with criterion 2.1.1, to enable all the functionalities for the use of the keyboard, in this way users who use the keyboard and the wide variety of technical aids that emulate the pressure of the keys will be able to access them.

5.2.7 Language of the Page (Level A)

It is recommended, to comply with criterion 3.1.1, that the content developers provide on the page the information to correctly present the texts and other linguistic contents, with the purpose that applications such as screen readers can load the pronunciation rules, or the Media players can display subtitles correctly.

5.2.8 Labels or Instructions (Level A)

It is recommended, to comply with criterion 3.3.2, to design user interfaces that provide simple instructions for entering information, such as label associated with visually connected input.

6 Conclusions and Future Works

The WCAG-EM methodology has been applied to analyze the degree of compliance with guidelines 2.0 because it is a known international standard. Two pages of each of the 14 public universities were evaluated: the home page (or home) and a page of undergraduate program, in order to have more balanced results and because in the two pages have found more of problematic components.

The grammatical level evaluation of the 14 pages with the tools W3C Markup Validation Service and W3C CSS Validation Service, indicated that all pages show errors at the level of HTML and CSS, and even universities present a high number of errors on homepage, which would indicate that the code should be reviewed.

The result of the evaluation with the tools Taw 2.0 and AChecker indicates that all the evaluated pages of the public universities, present breaches in the criteria of the WCAG 2.0, both at level A and AA, which is mostly serious. The most frequent are related to the syntax of the markup language, with the presentation of the content, with the non-textual content, with the purpose of the links and the readability of the text. This makes it impossible to guarantee the perception of the contents by most people and the reliable interpretation of them by a wide range of user assistance applications.

Regarding the evaluation carried out by the eXaminator tool, the average evaluation of home pages and undergraduate programs are 4.8 and 5.2 of accessibility level respectively, which would indicate that there is no substantial difference in level of accessibility between the sites. And indicate that both pages have a fairly low level of accessibility.

As it has been possible to demonstrate through this work, the Peruvian public universities do not reach the minimum level of compliance required by WCAG 2.0. This positions Peru in a situation like that of other countries in Latin America. As shown by a study in Chile, where it was shown that none of the 58 university portals reached the minimum level of WCGA 2.0, similarly in Ecuador a study showed that none of the pages are accessible.

Regarding the lines of future work, the following are identified:

Extend the study to the total of Peruvian public universities, and even involve private universities, with the purpose of knowing more widely the level of accessibility of Peruvian universities. Also consider increasing the study of pages by university.

Involve users of universities in the evaluation, as suggested by WCAG-EM, to better understand the use of the evaluated web pages.