Accessibility barriers for users of screen readers in the Moodle learning content management system

Abstract

In recent decades, the use of the Internet has spread rapidly into diverse social spheres including that of education. Currently, most educational centers make use of e-learning environments created through authoring tool applications like learning content management systems (LCMSs). However, most of these applications currently present accessibility barriers that make the creation of accessible e-learning environments difficult for teachers and administrators. In this paper, the accessibility of the Moodle authoring tool, one of the most frequently used LCMSs worldwide, is evaluated. More specifically, the evaluation is carried out from the perspective of two visually impaired users accessing content through screen readers, as well as a heuristic evaluation considering the World Wide Web Consortium’s Authoring Tool Accessibility Guidelines. The evaluation results demonstrate that Moodle presents barriers for screen reader users, limiting their ability to access the tool. One example of accessibility problems for visually impaired users is the frequent inability to publish learning contents without assistance. In light of these results, the paper offers recommendations that can be followed to reduce or eliminate these accessibility barriers.

1 Introduction

In educational institutions around the world, the importance and presence of e-learning programs and approaches is growing at a rapid pace. To create and manage e-learning environments, platforms, and course Web sites, authoring tools including learning content management systems (LCMSs) are used. Responding to this growing use, the number of tools available has also increased.

Despite these overall advances, studies of different educational and course Web sites have nevertheless detected accessibility-limiting barriers for certain users [1, 2]. Moreover, accessibility-related problems have also been identified in numerous LCMSs currently used (see Sect. 2.4). For instance, difficulties have been detected in certain LCMSs where authors’ prior programming skills are assumed or required [1]. Visually impaired users, in particular, are constantly confronted with barriers when surfing the Internet that often take the form of (1) identically named, redundant or confusingly similar links [3], (2) images used without accompanying, alternative text, and (3) colors or other visual, non-linguistic elements—used to convey information [4].

Regardless of the specific causes, accessibility barriers in an educational environment can seriously limit disabled teachers’ and administrators’ ability to effectively use the platforms, be it in managing a course or uploading course resources. The matter takes on greater significance when one considers that, by limiting disabled users’ ability to independently incorporate e-learning trends and their home institutions’ educational environments in their courses, such individuals are put at a professional disadvantage that may make it more difficult for them to obtain and maintain a job.

The Moodle¹ LCMS was selected for evaluation in this study in light of two principal considerations, namely the fewer accessibility-related barriers identified in prior studies relative to other LCMSs, as well as the former’s status as the most commonly used and recommended LCMS around the world [5]. This paper presents user and heuristic evaluations of Moodle’s accessibility for screen reader users attempting to create and manage a course on the platform [6]. For the former evaluation, two different users analyzed accessibility with respect to predetermined tasks often performed by administrators and teachers. In the latter evaluation, Moodle’s conformance with the World Wide Web Consortium’s (W3C)² Authoring Tool Accessibility Guidelines (ATAG) 2.0 is tested by an expert.

2 Background and related work

In the following subsections, topics on accessibility, the way in which visually impaired users interact with computers while surfing the Internet, as well as studies evaluating the accessibility of different LCMSs are presented and discussed.

2.1 The importance of accessibility in working environments

According to figures and estimates by the World Health Organization (WHO),³ more than one billion people—approximately 15 % of the world population—currently live with a disability [7]. Basing on the same source, these individuals generally have fewer opportunities to study or work than the members of the non-disabled population. Among the many causes of this employment opportunity gap are infrastructures that, to varying degrees, limit the participation and access of disabled individuals. In educational institutions, such barriers can appear in the very buildings where learning takes place [8]. Even where these infrastructures can be accessed by individuals with disabilities, they often are not accessed in the same way as by non-disabled individuals, resulting in the need for greater flexibility in other aspects of the disabled person’s professional life (e.g., scheduling modifications) [7].

By circumventing the problems posed by traditional infrastructures and offering the possibility to work from a computer and a reliable Internet connection, information technology (IT) offers, at least in theory, one way to close the opportunity gap between disabled and non-disabled individuals [9]. Nevertheless, in practice, barriers to accessibility have been detected even in the very IT tools originally intended to help close this gap [10]. Thus, if IT is to be considered part of the solution rather than part of the accessibility problem, barriers in IT software and content must be identified, studied, and eliminated.

In an attempt to ensure IT system accessibility for disabled users, a number of laws are in force in diverse countries and regions. With regard to Internet accessibility, some of the most important legislation currently includes eEurope in the European Union [11], Section 508 of the Rehabilitation Act in the United States [12], and the Equal Opportunities, Non-Discrimination and Universal Accessibility Act in Spain [13]. Laws of note focusing, more specifically, on the accessibility of IT systems in educational environments include the Disability Discrimination Act in the United Kingdom [14], Section 504 of the previously mentioned U.S. Rehabilitation Act [13], and the Organic Law on Education (LOE) in Spain [15].

2.2 Visually impaired individuals and the Internet

According to a survey conducted by the WHO on disabilities, 314 million people were found to be visually impaired, 45 million of whom being completely blind [16]. Thus, the lack of consideration by Website developers of the needs of visually impaired users would clearly result in the blanket denial of Internet access to an extremely large number of individuals around the world.

Both in Spain—country in which the authors of the present study are based—and in other countries, organizations like Fundación ONCE ⁴ (Spain) and the Royal National Institute of Blind People⁵ (UK) provide resources and support for the accessible use of ITs by visually impaired individuals. With respect to Internet use, numerous different assistive technologies (ATs) are available and respond to different user needs and contexts of use. Two such ATs used frequently by visually impaired individuals are screen readers and screen magnifiers. According to an earlier study [17], the two screen readers preferred most by visually impaired users are the JAWS⁶ commercial software and the open-source NonVisual Desktop Access (NVDA)⁷ software, with the use of the latter growing with respect to the former. In another interesting comparison of ITs, the same study concluded that Internet Explorer (IE)⁸ continues to be the Web browser of choice among visually impaired individuals.

Everyday, users with disabilities—the visually impaired being no exception—confront numerous difficulties when surfing the Internet. Indeed, users of screen readers not only encounter such barriers in Web sites, but also in the ATs on which they depend for their daily work. In two studies [18, 19], visually impaired individuals using screen readers were found to take three times as long as their non-disabled counterparts with no screen reader for the completion of a given task. According to another study [18], screen reader users commonly face difficulties including (1) the lack of labels associated with controls like inputText or ComboBoxes, (2) overly complex Web page structures and layouts, (3) the lack of alternative texts for images, (4) the impracticality of certain navigation techniques, (5) the inadequacy of color contrasts used, and (6) the incorrect size of elements present. Whether a specific barrier to accessibility is found in the Web site itself or in the ATs used, the end result is often the same: the limited access of the screen reader user to an application and its contents.

In learning environments, visually impaired users often cannot access course Web sites or their learning content, while visually impaired teachers and administrators cannot upload these learning materials or effectively manage their courses on their own.

2.3 Standards and guidelines for accessible authoring tools

In order to facilitate the creation of accessible authoring tools, guidelines have been created for developers. Perhaps the most important of such guidelines are those published by the W3C which, in addition to the ATAG [21], has also produced the Web Content Accessibility Guidelines (WCAG) [20].

Especially in LCMSs, it is important to note that not only should the tool itself be accessible for all users, but so too should the documents and resources created by teachers and presented by the LCMSs. Thus, for screen reader users, alternative text must be created to accompany images such that the users may understand the meaning and importance of the image. For the hearing impaired, to give another example, videos uploaded by the teacher should include appropriate subtitles [22].

2.4 Evaluation of accessibility in educational authoring tools

A number of studies exist which evaluate LCMSs according to accessibility guidelines. In most of these studies, e-learning tools like Moodle, dotLRN,⁹ and Blackboard¹⁰ have been found to contain serious barriers to accessibility with respect to WCAG 1.0 [23]. In a more recent study [5], a deep evaluation of the accessibility of the Moodle, ATutor,¹¹ and Sakai¹² LCMSs with respect to four parameters, WCAG 1.0 and ATAG 2.0 were undertaken, the results of which uncover accessibility barriers in each of the three LCMSs. Nevertheless, the same study also showed Moodle to present the fewest barriers of the three. As mentioned before, this fact constitutes one of the two main factors contributing to the selection of Moodle for evaluation in the present study. From the specific perspective of visually impaired users, two different problems detected in LCMSs [4] are (1) the inability to fully access Web content or functions through the keyboard and (2) the lack of accompanying text for visual Web information that is detectable and readable by screen readers. In another study [24], accessibility problems for visually impaired individuals in Moodle were identified in the diverse images used to convey information and a general lack of headings used in the application. As a result, the study concluded that Moodle did not conform to ATAG 2.0 or WCAG 2.0. Finally, in the evaluation of diverse authoring tools including Moodle and AContent,¹³ a different paper [25] found the latter to be the only authoring tool conforming ATAG 2.0.

2.5 Improving Moodle accessibility

Projects have been launched to offer solutions for the accessibility problems identified in learning environments. Two such projects are the Accessible Multimedia Service Learning Project¹⁴—aiming to improve the accessible creation of learning courses—and the ALERT project¹⁵—providing guidelines for the improvement of accessibility in online learning for the achievement of learning goals by students.

Focusing on accessible adaptations of LCMSs, and especially Moodle, the EU4ALL project¹⁶ aims to improve accessibility in higher education and facilitate lifelong learning. Another project of note is the Open University’s creation of the accessible adaptation of Moodle, OpenLearn,¹⁷ to achieve conformance with W3C standards.

In addition to such projects, plug-ins have also been created for course administrators to improve Moodle accessibility. Two such plug-ins include Blocks: Accessibility,¹⁸ allowing users to customize Moodle according to their visual needs, and SimpleSpeak,¹⁹ providing text-to-speech synthesis.

Finally, studies can be found in the literature aiming to directly improve Moodle’s accessibility. In one study [25, 26], a platform is created adapting Moodle for people with cognitive disabilities. In another study focusing on individuals with visual impairments [27], a tool is proposed for the creation of accessible content in Moodle 2.0, which renders information in different ways.

3 Discussion

This paper presents an evaluation of the accessibility of Moodle, understood as an authoring tool used by teachers and administrators to create and manage online learning courses.

To the knowledge of the authors of the present paper, prior studies evaluating Moodle have done so from the perspective of students and have generally neglected executable functions by teachers and administrators for course management and creation. Moreover, the vast majority of the previous studies evaluate LCMSs like Moodle from the perspective of WCAG, whereas only one study could be identified that evaluated LCMSs according to ATAG. This latter study, however, is not as exhaustive as the present study insofar as it does not specify the guidelines not followed and does not clearly analyze each task executed.

As presented above, while a number of tools and projects exist that aim to improve the accessibility of Moodle, the improvements are not universally applicable and some require prior technical knowledge on the part of the user for their implementation. As many people cannot be expected to possess the information, knowledge, or expertise necessary to use these adaptations or plug-ins, accessibility barriers in Moodle should be eliminated directly within the LCMS.

Having considered the current status of Moodle, the literature, and existing approaches, this paper aims to provide a deep evaluation of the accessibility of one of the latest and most commonly used versions of Moodle for teachers and administrators using screen readers and with respect to ATAG 2.0.

4 Evaluation design

The following subsections present the design of the accessibility evaluation, beginning with the general evaluation objective and followed by the evaluation process.

4.1 Evaluation objective

This study evaluates Moodle 1.9²⁰—currently the most frequently used version of the application [28] (see Fig. 1)—from the perspective of screen reader users like teachers and administrators for course creation and management.

Fig. 1

Registered Moodle users by version (as of September 2012)

The study evaluates accessibility from two perspectives:

• 1. Accessibility of the Moodle authoring tool, analyzing problems that arise for users when executing a task (e.g., a teacher who tries to create a course) and

• 2. Accessibility of the Web pages generated by the Moodle authoring tool following task execution.

It is important to note here that the degree of accessibility of author-created elements through the Moodle default editor—including images and lists—is not studied here. This is due to the fact that, as shown in previous studies, the Web site default editor itself is not accessible [29] and, therefore, cannot be evaluated by a blind study participant.

4.2 Evaluation environment

As Spanish was the mother tongue of users and evaluators, the present study focused on the Spanish language version of Moodle.

The evaluation was carried out using the Windows XP Operating System (OS)—currently one of the most frequently used OSs on the market²¹ (see Fig. 2)—and the IE 8 Web browser.

Fig. 2

Operating system usage statistics

4.3 Evaluation method

In order to demonstrate the accessibility, or lack thereof, of a tool, a complete evaluation must take both expert evaluations and user experiences into account [30]. In an attempt to integrate these two perspectives, the present study follows the evaluation method detailed in the following subsections.

Moreover, in order to perform these two types of evaluations, a series of preliminary steps from the W3C evaluation methodology [31] and other studies [32] was taken, namely:

• 1. Selection of tasks to be evaluated. The tasks selected for this study were those whose execution was identified as necessary for the creation and management of a course in Moodle, as well as the management of students in that course. Figure 3 offers a scheme of the modules and tasks evaluated (see “Appendix” for a comprehensive list).

  Fig. 3

  

  Scheme of tasks evaluated

• 2. Definition of role of the individual executing the tasks. In Moodle, individuals may interact with the LCMS in three different ways, namely as a student, teacher, or administrator. The present study evaluates Moodle from the perspective of the administrator, the role from which all possible system tasks may be executed, including those a teacher can execute.

4.4 Heuristic evaluation method

Any heuristic evaluation may be conducted automatically, semi-automatically, and manually by experts [30]. As mentioned earlier, the present study aims to evaluate Moodle 1.9 with respect to ATAG 2.0. Since, to the knowledge of the authors of this article, no automatic tool currently exists for evaluation according to these particular guidelines, a deep manual evaluation was performed by an accessibility expert with the help of semi-automatic tools [33].

4.4.1 Objective

The main objective of an expert evaluation is to determine whether a tool being evaluated is in conformance with a given set of W3C accessibility guidelines. As Moodle is an authoring tool, the present study analyzed its conformance with ATAG 2.0, guidelines elaborated to aid developers in the creation of accessible IT systems and Web content. It is important to emphasize that ATAG 2.0 is still just a working draft. However, as these guidelines are nevertheless based on the WCAG 2.0 Recommendation, the authors considered ATAG 2.0 to be sufficiently robust to justify its use in the evaluation.

4.4.2 Participants

The evaluation was carried out by an expert with 2 years of experience in the evaluation and creation of accessible Web sites.

4.4.3 Environment

The environment selected for the expert evaluation is described in the Sect. 3.2.

4.4.4 Description

As explained above, a manual expert evaluation was carried out in this study. In the evaluation, the accessibility of each task, as described in the evaluation design task, was assessed according to ATAG 2.0 and taking into account its different priority levels, from A to AAA. In ATAG 2.0, guidelines are divided into two parts, with Part A being related to the creation of an accessible interface and Part B related to the creation of accessible content [34]. Additionally, each part of ATAG 2.0 is further classified into layers of guidance, that is, principles, guidelines and success criteria (SC). A principle is divided into guidelines providing basic aims to follow in the development of accessible tools. The guidelines are then divided into SC which are testable and can be classified by conformance levels A, AA, or AAA, where AAA is the highest and A the lowest.

4.5 User evaluation method

In a user evaluation, individuals both with and without disabilities are involved [35]. The selection of the specific disabilities, participants, tasks to be completed, task completion times, and the evaluation environment is elaborated below in the following subsections [36].

4.5.1 Objective

The evaluation conducted in this study attempted to determine whether a visually impaired user with the help of screen reader ATs is able to complete a set of predefined tasks on Moodle without encountering accessibility barriers. Study participants used two different screen readers, and all differences between the two with regard to the accessibility of Moodle were recorded.

4.5.2 Participants

The evaluation was performed by two participants. One is a journalist and frequent IT user—including tools like Moodle—who has been blind since birth. The other user, an accessibility and IT expert with no visual impairment, simulated blindness for the evaluation by switching off the computer screen.

These two evaluation participants were selected so as to represent the wide range of visually impaired individuals that require screen readers to access Internet applications. The former participant represented the group of visually impaired individuals who have been blind from birth and, therefore, have been able to develop other skills to compensate for their disability. These individuals are generally more accustomed to using ATs in their daily lives. The latter user evaluation participant represented the group of visually impaired individuals whose disability resulted from an illness or accident later in life. Such users often have not developed the same alternative internal skills or long-term familiarity with ATs as individuals from the former group.

4.5.3 Environment

To complete the predetermined tasks in Moodle, each participant used the NVDA 2010.2²² and JAWS 10²³ screen readers—currently the most commonly used screen readers available [17]—and produced independent evaluations for each. The former screen reader is open-source and compatible with Web browsers like Mozilla Firefox and IE, while the latter is a commercial software that, in the version studied here, works only with IE. Both screen readers can read aloud in Spanish, the native language of both users.

4.5.4 Description

In the user evaluation, each participant received a list of predefined tasks to be executed on Moodle. The tasks evaluated were those necessary for course creation and management. Figure 3 shows the main tasks evaluated in the user study (see “Appendix” for comprehensive list). In the evaluation, each user independently answered a questionnaire without the supervision of a technician or an accessibility expert, explaining whether each predefined task could successfully be completed and describing any accessibility barriers encountered along the way.

5 Evaluation results

The following two subsections present the results obtained in the expert and user evaluations, respectively, with Moodle’s conformance with ATAG 2.0 being analyzed in the former and the errors and accessibility barriers encountered by users being recorded in the latter.

5.1 Heuristic evaluation

The results of this subsection were obtained from the manual expert evaluation of Moodle 1.9 with respect to ATAG 2.0. The analysis of the findings shows that Moodle does not conform to the A priority level in either Part A or B of the guidelines.

The results obtained in the expert evaluation are presented in two distinct manners: the number of errors per SC and priority level, as well as the number of errors per principle and priority level. Table 1 shows the number of errors per SC and per priority level of each principle. The first column presents the principles, while the second, third, and fourth columns show each unfulfilled SC, with the number of times each was not fulfilled sorted according to priority level.

Table 1 ATAG 2.0 errors per principle and priority level

Additionally, Figs. 4 and 5 compare the SC in the guidelines for the three priority levels of conformance (A, AA, and AAA) in Parts A and B, respectively. The figures show the number of errors detected per guideline and, at the same time, sort these errors by priority level.

Fig. 4

Number of accessibility barriers found in Moodle for each ATAG 2.0 Part A guideline and priority level

Fig. 5

Number of accessibility barriers found in Moodle for each ATAG 2.0 Part B guideline and priority level

A complete list of features and accessibility barriers present in each task can be found at the Web site http://labda.inf.uc3m.es/MAE.²⁴ What follows below is an explanation of the specific guidelines according to which the learning tool fails in the evaluation.

With respect to Part A of ATAG 2.0 specifying that the authoring tool user interface must be accessible, evaluation results demonstrate that Moodle does not satisfy certain SC of an A priority level. Therefore, it can be concluded that Moodle fails to conform to Part A of ATAG 2.0.

The first principle of ATAG 2.0, Principle A.1, specifies that a tool interface must be developed according to accessibility guidelines. This principle is divided into two guidelines, A.1.1, which is related to Web-based functionality, and A.1.2, which is related to non-Web-based functionality. Concerning Principle A.1.1, Moodle does not follow WCAG 2.0, at least at the A level of conformity for its Web content, since not all of its tasks are accessible, as demonstrated in a previous study [29]. This constitutes, of course, an error of great importance. Furthermore, Moodle’s non-Web-based functionality is not accessible, as specified by the A.1.2 guideline. For instance, the use of the Moodle HTML default editor, What You See Is What You Get (WYSIWYG), is not accessible and Moodle does not inform users of the existence of accessible interfaces in other WYSIWYG editors.

According to Principle A.2, editing views should be perceivable. However, this principle is not fulfilled by Moodle either, since the tool does not inform users about changes produced in text modified by the author (A.2.2.2 SC). As the HTML default editor view does not conform to WCAG 2.0, the presentation properties for text font, text style, and so on cannot be determined programmatically.

Furthermore, Moodle does not follow Principle A.3 specifying that editing views should be operable. According to the A.3.1 guideline, Moodle’s features should be accessible by keyboard. However, keyboard access to tool features is not possible and shortcuts are sometimes overlapped by OS shortcuts (A.3.1.1 and A.3.1.4 SC). This was found mainly in some non-Web-based Moodle content. Another problem observed is the inability of the user to change the shortcuts provided by the tool in order to avoid these aforementioned keyboard conflicts or to add new shortcuts (A.3.1.5 SC). Additionally, the user cannot press a keyboard command to see all tool shortcuts (A.3.1.6 SC). According to guideline A.3.2, the authoring tool should provide authors sufficient time to act. Moodle, however, does not allow this in some cases, since it refreshes content without prior notice (A.3.2.2 SC) and does not permit auto-saving of modified content (A.3.2.4 SC). With regard to guideline A.3.4, an authoring tool should provide ways to navigate and edit via content structure. Nevertheless, the default HTML editor shows that the author markup element cannot access content modified in the HTML default editor or in the newly generated modules by structure (A.3.4.1 SC) or programmatic relationships (A.3.4.2 SC). Considering guideline A.3.5, as Moodle does not provide the possibility of a text search within the content, the guideline is not fulfilled in the majority of tasks. While guideline A.3.7 is not fulfilled either, this does not constitute a frequent error. When there are previews, these are not previewed in an In-Market User Agent, they do not fulfill UAAG [37] (A.3.7.1 SC), and users cannot specify the user agent (A.3.7.2 SC).

Finally, Principle A.4 specifying that editing views should be understandable is also not fulfilled by Moodle. With regard to the problems with guideline A.4.1, evaluation results show that Moodle does not help the author reverse actions (A.4.1.1 SC) or provide the possibility to reverse actions sequentially (A.4.1.3 SC). Moodle does not conform either to guideline A.4.2, specifying that the user interface should be documented. While Moodle provides extensive documentation on the Internet, the documentation is often (1) incomplete for certain tasks, (2) not properly linked, or (3) not correctly identified (A.4.2.2 SC). Besides, this documentation does not specify which features have been included to satisfy Part A of ATAG 2.0 (A.4.2.1 SC).

Concerning ATAG 2.0 Part B related to the generation of accessible content, as the following paragraphs will show, there are certain A level priority criteria not satisfied by Moodle. Thus, just as the tool does not completely conform to Part A, neither can it be said to completely conform to Part B.

According to Principle B.1 of ATAG 2.0, the production of accessible content should be enabled. In Moodle, however, the content generated is not always accessible and no internal mechanism exists to check its accessibility. As a result, Moodle does not conform to guideline B.1.1 or its SC.

Principle B.2 of ATAG 2.0 states that authoring tools should help authors create accessible content. Nevertheless, this help is not completely provided in Moodle. Since Moodle allows authors to produce inaccessible content without providing mechanisms to inform the author of the problem (e.g., the author can create a table for layout that Moodle does not prevent), guideline B.2.1 is not followed. Moreover, Moodle does not follow the SCs from guideline B.2.2. For instance, the author can include a video without Moodle specifying that alternative content should be uploaded. Additionally, Moodle does not inform authors when they do not specify alternative text. Concerning guideline B.2.3, Moodle does not include any steps to assist in the creation of alternative content for any non-text content inserted in the authoring tool. That said, Moodle does offer a mechanism to provide alternative text in copied images in the tool without any pre-existing alternative text. However, this feature is not completely accessible inasmuch as the alternative text provided is the name of the image rather than a descriptive text (B.2.3.3 SC).

Regarding Principle B.3 indicating that authors must be supported in improving the accessibility of existing content, Moodle does not check the accessibility of content (B.3.1.1 SC), as evidenced by the fact that users can create tables for layout.

Considering Principle B.4, authoring tools must promote and integrate their accessibility features. However, as explained earlier, the author can create inaccessible content without any specific documentation (B.4.1.3 SC) or elements to note the errors (B.4.1.5 SC). Moreover, guideline B.4.2 specifies that the tool should include suggestions for the author on the accessible development of content.

Finally, the expert evaluation determined that not only are these accessibility barriers present in Moodle occasionally, but were encountered frequently throughout the majority of the tasks evaluated. Therefore, it may be concluded from the expert evaluation that Moodle presents serious shortcomings for accessibility with respect to ATAG 2.0.

5.2 User evaluation

As discussed in an earlier section (see Sect. 3.3), the user evaluation in this study was performed by a blind user and a non-impaired accessibility expert who simulated blindness. As the present section will demonstrate, different accessibility barriers were encountered in Moodle by the evaluation participants as they used the tool with screen readers. As a result, it can be concluded that Moodle is not accessible for visually impaired individuals using screen readers. These findings and conclusion are explained in greater detail in the following paragraphs.

Comparing the evaluation results obtained by the blind user and the non-impaired user simulating blindness, while the latter user initially experienced greater difficulties interacting with Moodle—consistent with the difficulty of learning how to use a screen reader—this initial lack of experience with screen readers did not ultimately prove to be a barrier for the non-impaired user. Indeed, after having learned the screen reader control commands, the non-impaired user was able to complete the same tasks as the blind user. Consequently, the results obtained by the two users in the evaluation were quite similar. For instance, as both were accustomed (or had become accustomed in the case of the non-impaired user) to navigating through Moodle using headings, both users experienced greater difficulties when performing task for which headings did not appear. Additionally, in cases where a control was not accompanied by an associated label, both users had difficulty guessing the correct information specified in the control. Thus, in reviewing the complete results recorded by both users, it can be concluded that both experienced similar difficulties in the different tasks to be completed.

Looking at the evaluation results obtained with the two different screen readers, the difficulties experienced by users for a given task tended to be similar with each screen reader. While the users noted problems related to the usability of the screen readers including the readability of certain tables, the use of certain screen reader shortcuts or the use of text like “Choose or upload file…” in some buttons, these usability barriers fall outside of the scope of the present study and, nevertheless, did not affect the relevant results obtained.

The diverse problems users faced while interacting with Moodle have been grouped and labeled as follows (see “Appendix” for a comprehensive list):

• 1. E1. Control has no label associated with a descriptive text;

• 2. E2. Page refreshes without prior warning;

• 3. E3. User is redirected to another page without prior warning;

• 4. E4. Web page appearance is not uniform;

• 5. E5. Table is used for layout;

• 6. E6. Information is communicated only through images of text;

• 7. E7. Task completion procedures are difficult to understand or follow;

• 8. E8. English-language text appears despite the prior selection of Spanish;

• 9. E9. No button appears for the cancelation of an operation;

• 10. E10. Screen reader cannot read table well due to a table design problem;

• 11. E11. Headings are used inappropriately;

• 12. E12. Table has many rows, making it difficult to discern overall table structure;

• 13. E13. HTML default editor is not accessible;

• 14. E14. Text description is incorrect;

• 15. E15. No message appears to avoid or correct an error;

• 16. E16. Text is not read correctly by screen reader.

Considering the results obtained, it is important to emphasize that the users were able to complete the majority of the tasks required. However, the users still experienced important accessibility difficulties that made the use of Moodle more difficult.

Figure 6 shows the number of times an instance of one of the above error categories was encountered by a user in the evaluation. The figure clearly demonstrates that the most frequent error encountered was related to the non-uniform appearance of the Web site (E4). This lack of uniformity—characterized by (1) disappearing blocks of information, (2) variable structures, and (3) the occasional appearance on the Web page of nothing more than a question and a button—proved most problematic and prevented the users from knowing at particular moments if they were still on the same Web site. Additionally, the use of input controls without an accompanying label (E1) proved to be a critical source of difficulty, since users were unable to determine the data they were supposed to introduce. Another frequently appearing problem was the incorrect use of headings for the creation of good Web page structure (E11). This is useful for users of screen readers since, by allowing them to create a Web site structure in their minds, they can complete given tasks with greater ease.

Fig. 6

Instances of errors by type in the user evaluation with the NVDA and JAWS screen readers

Finally, it is important to note that certain errors—(1) use of useless descriptions (E14), (2) display of excessive information (E12), and (3) presence of inserted data that is not evaluated (E15)—carry extremely important consequences despite their having been detected just once in the accessibility evaluation. Finally, ten instances were recorded in which English appeared despite the prior selection by users of Spanish as the Moodle language (E8). This, of course, presents a serious barrier to accessibility for any user (visually impaired or not) who does not understand English or the meaning of the specific English texts that appeared.

6 Recommendations

Based on the evaluation results presented and consistent with the overall objectives of the present paper, a set of recommendations has been elaborated in order to improve the accessibility of the Moodle interface and content created therein.

With regard to the Moodle interface, it is important that it can be created in an accessible way, so authors may use it without encountering barriers to accessibility. The following recommendations are aimed at removing such barriers.

• 1. Use accessible non - Web - based content. The accessibility of all external elements should be checked before their use in Moodle. This is due to the fact that they can compromise the overall accessibility of the tool.

• 2. Avoid common accessibility errors in Web-based functionalities including:

  • (a) The use of tables for layout inasmuch as they must be used to show structured information.

  • (b) The use of different page structures and functionality structures throughout the Web site.

  • (c) The failure to divide information into small and easily understandable pieces.

• 3. Provide mechanisms to allow the author avoid mistakes or solve them if they appear.

  • (a) Provide buttons or functionalities for the cancelation of each task or, at the very least, of important tasks.

  • (b) Provide mechanisms to undo the final actions executed, like a button to undo the final task.

  • (c) Provide functionalities to periodically save the actions executed to prevent information loss.

  • (d) Provide ways to save user preferences regarding auto-saving, colors, font size, etc.

• 4. Content search. Given its importance for screen reader users, allow the author to search for content by name, category, or structure. For instance, the user should be able to search headings, lists, and tables.

Concerning the creation of accessible Moodle content, the tool should both help the author create accessible content and guide him/her in that creation. To achieve this overarching goal, the recommendations below should be followed.

• 1. Guide the author in the generation of accessible content, since he/she may not be an accessibility expert.

  • (a) For the generation of content, it is safer for the user and fewer errors will be committed if data are entered into fields of a form, rather than if created by the author alone.

  • (b) Elements and advice should be provided allowing the user to create alternative content with ease.

• 2. Check the accessibility of the user generated content, since even when the author is an accessibility expert, errors can nevertheless be committed.

  • (a) Provide mechanisms to check the accessibility of content generated and allow the user to specify whether the content should conform at the A, AA, or AAA levels.

  • (b) After checking for accessibility, inform the user of any errors/barriers found and advise him/her on how they may be eliminated.

  • (c) Provide mechanisms for the automatic correction of accessibility errors/barriers detected.

• 3. Provide accessibility documentation, since the author may not know something about a particular barrier.

  • (a) While it is true that Moodle provides Web documentation, this documentation should nevertheless be properly linked to each task. Thus, the user may understand the task and know how it may be efficiently and accessibly completed.

  • (b) The documentation should be complete. This means that all images should be provided and no part of the documentation should be inaccessible due to its being under construction.

  • (c) If Moodle cannot implement certain elements to create accessible content, the author should be informed.

In conclusion, the authors believe that following these recommendations serve to greatly improve the accessibility of the Moodle interface and Moodle content for visually impaired users of screen readers.

7 Conclusions

This paper has presented an evaluation of Moodle accessibility for screen reader users. Whereas other previous studies had focused on student experiences, this paper has analyzed Moodle’s accessibility as an authoring tool used by administrators and teachers. Having analyzed the results of the expert and user evaluations, it can be concluded that Moodle is not accessible for screen reader users and does not conform with ATAG 2.0, at least at the A priority level in Parts A and B of the guidelines.

Taking into account the recommendations for Moodle presented above would allow users—taking into account a great functional diversity—to interact with the software without encountering barriers to accessibility in the software itself or the content generated. Following these recommendations, therefore, would go a long way to reduce the opportunities gap between disabled and non-disabled individuals in the use of this popular LCMS.

8 Limitations of study and areas for future research

The evaluation of Moodle presented here was carried out exclusively from the perspective of visually impaired individuals using screen readers. No other type of user disabilities was taken into account. While this, strictly speaking, does not constitute a limitation for the study, in terms of universal design Moodle should nevertheless be accessible for all user groups and not only for users of screen readers. In response to this reality, the authors are currently researching Moodle accessibility for individuals with other types of impairments such as deafness and reduced mobility. Moreover, while the Moodle version selected for analysis is currently the most used of all available versions, it is nevertheless not the most recent version of Moodle. It is important to emphasize, therefore, that some of the barriers identified in the earlier and most popular version may have been eliminated in the newer version. On the other hand, barriers may be present in the newer version that were not encountered in the one evaluated here.

Therefore, possible future extensions of this study could include evaluations of other types of ATs, browsers, and disabilities, as well as newer versions of Moodle.

Appendix

Table 2 presents the main errors encountered in the user evaluation or, in other words, by users interacting with Moodle via a screen reader. Cells from column 1 of the table identify the profile(s) of users able to complete a given task and may include (1) A for administrators, (2) T for teachers, and (3) S for students. Columns 2 and 3 present the functionality groups and the names of each task executed. Moreover, column 4 shows the different errors encountered for each task. These errors are identified according to their numeric classification presented in the Sect. 4.2. Finally, column 5 specifies whether the task could be completed with possible responses being (1) Yes, (2) Yes*, and (3) No, where Yes indicates that the task could be completed without accessibility barriers, Yes* indicates that the task could be completed with the presence of barriers to accessibility, and No indicates a task could not be completed due to the accessibility barriers.

Table 2 Difficulties found for tasks executed in the Moodle user evaluation