1 Introduction

Numerous studies over the years have documented accessibility-related barriers in various categories of websites. Generally, these barriers impact Blind users the hardest, as the barriers tend to be most problematic for screen reader users. However, the studies often focus on determining the presence of accessibility barriers, rather than determining the specific impact of those barriers. What has not yet been quantified is the impact that these accessibility barriers have, in terms of the increased amount of time that it takes Blind users to accomplish tasks.

In the U.S., government agencies at the federal level are legally required to make their websites (and other technologies) accessible for people with disabilities, under Section 508 of the Rehabilitation Act. State and local government must make their websites accessible under Title II of the Americans with Disabilities Act (Lazar et al. 2015). Any private organization that is classified as one of the 12 categories of public accommodations under Title III of the Americans with Disabilities Act must also make its website accessible. On 15 January 2019, the U.S. 9th Circuit Court of Appeals reversed a lower court decision, reaffirming yet again that the Americans with Disabilities Act addresses websites of public accommodations, even in the absence of specific technical guidance or regulations on what interface accessibility standards to use (Lf legal 2019). Also, in October 2019, the U.S. Supreme Court refused to review the Robles v. Domino’s Pizza case at that time (known as a “denial of a petition for a writ of certiorari”) letting the 9th circuit’s decision stand (Bloomberg 2019).

In the U.K., the Equality Act 2010 prohibits exclusion from the use of services because of a disability, and this includes online services (GEO 2015). Under this legislation, providers must make “reasonable adjustments” to the services that they offer online. To strengthen the Equality Act 2010, a new regulation was enacted called the Public Sector Bodies (Websites and Mobile Applications) (No. 2) Accessibility Regulations 2018. This new regulation requires new public sector websites and mobile applications to be accessible, but it also required existing public sector websites to be made accessible by September 2020 (2021 for mobile applications). This includes compliance with WCAG 2.1 AA and the publication of an accessibility statement (National Archives 2018). Since 2014 in the EU, the European standard for digital accessibility is a guideline known as EN 301 549 (EFTA 2014). The premise of this standard for the public sector is that all information and communication technology (including websites) needs to be accessible to everyone, which includes people with disabilities. Similar to the UK regulations, EN 301 549 has been updated to point to WCAG 2.1 for its guidelines.

The primary accessibility guidelines for webpages and applications are maintained by the World Wide Web Consortium (W3C), which is an international consortium that produces standards for web content. The guidelines relevant to websites are known as the Web Content Accessibility Guidelines, or WCAG. The current version of WCAG (as of June 2018) is WCAG 2.1, and it outlays 13 core principles of web content being perceivable, operable, understandable and robust (W3C 2018). Within those standards are a subset of testable “Success Criteria” which determine level of accessibility for those broader guidelines. There are three levels for criteria: A, AA, and AAA, with AAA being the highest. Most governmental laws and policies strive for a minimum of AA. For example, there is a principle that requires things to be “Operable” on a website, but within that principle, there are guidelines such as “Keyboard Accessible” and “Enough Time.” Within a guideline such as “Keyboard Accessible” there would be success criterion for “Keyboard” (2.1.1) that states that all the content functionality must work from a keyboard (i.e., not just through a touch screen or mouse input).

There is a strong need for empirical data on the time impact on Blind users regarding accessibility barriers. This empirical data is necessary for understanding how much time is lost by Blind users, when they face specific barriers. This data is needed for both: (1) economic models used in regulations and (2) for future accessibility guidelines.

As governments around the world increase or clarify the requirements related to web accessibility for public accommodations, they often require some level of cost-benefit analysis for the proposed regulation (for instance, in the U.S., this is known as a regulatory impact analysis). As a part of these cost-benefit analyses, it’s important to understand not only the costs of making technology accessible, but also the costs to Blind users (and people with other disabilities) of not making the technologies accessible.

This empirical data on the time lost due to accessibility barriers is also necessary for understanding that all of the success criteria within accessibility guidelines are not equal—compliance with some success criteria has a minor impact on users, while compliance with other success criteria has a major impact (and the impact may differ on what type of disability you have). So, quantifying time data on the disparity between high and low accessibility could potentially impact both future accessibility guidelines, as well as regulations. Furthermore, many public and private organizations fail to view web accessibility from a perspective that accounts for lost time and the resulting impact on worker productivity. When technologies aren’t accessible for employees with disabilities, that time lost is a cost to the company. Significant time differences due to inaccessibility could negatively impact the ability of individuals to conduct tasks—for example, searching for public information, conducting financial transactions, and applying for employment. Government, researchers, developers, and disability advocates would benefit from some quantifiable estimates relating to the time disparities for Blind users between accessible and inaccessible web-based interfaces.

2 Background

In a 2007 study (Lazar et al. 2007), it was discovered that frustrating situations involving the use of the Web led to Blind users losing, on average, 30.4% of time. However, the amount of time lost exclusively due to accessibility barriers was not quantified. It is important to acknowledge that the definition of a “frustrating situation” can be misunderstood as a situation arising solely from web inaccessibility independent from other factors. While some of the problems that led to frustration and time lost were a result of inaccessibility encountered on web pages, there were frustrating and time-consuming problems related to poor performance of the assistive technology (AT) being used by the Blind users. There were also web pages users encountered that were fairly accessible but were not usable, so more time was needed to complete basic tasks. These problems included “(a) page layout causing confusing screen reader feedback; (b) conflict between screen reader and application; (c) poorly designed/unlabeled forms; (d) no alt text for pictures; and (e) 3-way tie between misleading links, inaccessible PDF, and a screen reader crash” (Lazar et al. 2007). In 2017, the U.S. Access Board Final Regulatory Impact Analysis for the updated version of Sect. 508 cited the 2007 study conducted by Lazar et al. as support for the notion that web accessibility causes time lost for Blind users. The text from the regulatory impact analysis is as follows:

“Lazar et al. (2007) find that blind participants reported losing, on average, 30.4% of time spent on the computer due to the frustration with situations from inaccessible/unusable web pages and AT malfunctions. Other studies reached similar conclusions”.

(U.S. Access Board 2017)

Although it is true that those situations contributed to frustration and time lost, it would not be accurate to say that the frustration and time lost experienced by users was solely due to the inaccessibility encountered on the Web. Rather, the frustration arose from some situations of inaccessibility, but also screen reader crashes, screen reader incompatibility, poor formatting, and other factors related to AT and not solely the Web. The 2017 U.S. Access Board Final Regulatory Impact Analysis also mentions a 2004 study conducted by the Disability Rights Commission – “The Web: Access & Inclusion for Disabled People.” The Disability Rights Commission conducted this research to formally investigate various shortcomings involving web inaccessibility and policy that should enforce accessibility. A segment of the investigation included a controlled study that observed task and time performance of Blind users and sighted users on websites with low accessibility and high accessibility. The purpose of this portion of the investigation was to identify barriers to accessibility for people with disabilities and then help develop solutions for them. The recorded results indicated that even on high accessibility websites, the Blind users needed more time to complete a task than sighted users. However, it was also concluded that “both blind users and non-impaired users took far longer on low accessibility sites than on high accessibility sites, and that this effect was not much more pronounced for disabled users…” (Disability Rights Commission 2004). While the studies conducted by Lazar et al. and the Disability Rights Commission both yield results involving time lost by Blind users on the web, neither study had the primary focus of determining how web accessibility barriers exclusively impact the time it takes Blind users to complete a task.

The idea that Blind users spend more time on the Web than other users due to accessibility barriers is not a novel one. In 2010 research by Babu et al. (2010) barriers faced by users in the context of accessibility and usability were studied. A portion of the study involved observing Blind users responding to short answer questions when navigating to an input field. They found that “the user’s productivity is adversely affected due to the time and effort spent identifying the input field” because the input fields were not entirely compliant with “the understandability factor” of WCAG. While the study did find a situation in which an accessibility barrier caused Blind users to spend extra time on a task, the time was not quantified, and the study also considered usability barriers not directly caused by WCAG noncompliance. Additionally, in a 2007 study by Bigham et al. (2007) the behaviors of Blind users and their time spent browsing the Web were recorded. Bigham concluded that the Blind users “took more time to access all pages than their sighted counterparts.” However, Bigham’s analysis did not quantify the time spent on specific tasks with accessibility barriers due exclusively to WCAG violations.

Although the aforementioned studies did not quantify time spent or lost by Blind users in tasks, a different study conducted by Watanabe in 2009 quantified the task completion time of Blind and sighted users on controlled websites. One of the websites included properly labelled headings, and the other did not. After analyzing the data, it was concluded that Blind users had a decreased task completion time when headings were properly labelled. Watanabe explains that “for task 1, some of the blind subjects spent about 20 to 40 s to navigate through the structured site, while they typically spent from 60 to 70 s to navigate through the unstructured site.” Even though the task completion time was decreased for Blind users, Wantanabe states “blind subjects tend to take about twice as long to complete a task for both structured and unstructured sites.”

The conclusions from Watanabe’s 2009 research are relevant, but they do not entirely encapsulate all time lost due to accessibility barriers. WCAG Success Criterion 2.4.6 states that in order for a website to be accessible to level AA, “Headings and labels describe topic or purpose” (W3C 2018). However, it was noted in the study that the controlled websites were entirely accessible when the heading variable was not considered. While Watanabe’s study revealed time data that quantified the time spent by Blind users on controlled, static websites with one specific accessibility barrier, it does not provide a comprehensive understanding of how much time is lost by Blind users due exclusively to WCAG violations in a less controlled, dynamic setting that is much more representative of the websites Blind users access regularly.

3 Research Methods

When doing research involving users with disabilities, there are multiple methodological issues to consider, such as the sample size (typically smaller than for more general populations), material format (e.g. audio, large print, or braille?), and distribution of participants geographically (if there aren’t enough users with a specific disability nearby, whether you should do remote research) (Lazar et al. 2017). One of the most important methodological considerations is whether you should utilize a user’s own technical environment, with their own assistive technologies and settings, or use a consistent technology environment, which is identical and used for all participants.

3.1 Potential Research Methodologies

For users with disabilities, when you utilize a user’s own technology, it documents the optimal performance of the users, because the user’s setup has already been designed around the needs of that specific user (Lazar et al. 2017). All of the settings (e.g., screen reader rate of speech, chosen voice) are maximized to what a user needs. At the same time, it means that the technical environment differs from user to user, and therefore may not be appropriate using some types of experimental design where you must control for the technology across all users. However, when the research design focuses on studying user behavior or performance, you don’t need to control for technology used, because you “want” to measure users at their best performance level, with other factors being controlled for (Lazar et al. 2017). For the current study, where we want to measure user performance when users encounter various barriers, it is appropriate to have users utilizing their own technologies, so that the data collected will be conservative and appropriate.

The core methodological approach of the current study is to have users attempt identical tasks, one on a high accessibility website, and one on a low accessibility site, and compare the time differences. Note that we describe it differently from the common terms “accessible” and “inaccessible.” If a site is completely inaccessible, tasks cannot be completed by users with disabilities due to accessibility barriers. Yet just saying, “it’s inaccessible” does not give any useful data, since the time to complete a task would either be zero or infinity (depending on how you frame the inability to complete a task, but that’s more of a philosophical argument). So it’s necessary to compare high accessibility sites (with no or minimal barriers) to low accessibility sites, where there are accessibility barriers but the tasks can still be completed. Since the goal is to collect data that can be utilized in policy, we must be sensitive to the fact that we need to create tasks that users can complete. We can’t simply create tasks that result in users giving up on every task (even though we are aware that some users will give up on some tasks), because that doesn’t get us to the data that are needed for regulatory impact analysis, even if that might be acceptable for a typical usability test.

There is another question to be considered regarding control of the test environment. Ideally, we want to have users interacting with existing, live websites, because that would be most representative of the actual amount of time lost due to low accessibility websites. However, it’s unknown if there are simply too many uncontrollable factors in live websites. For instance, while sites can be evaluated for low accessibility and high accessibility, it’s unknown if there are too many uncontrollable variables, and whether creating simple sites with simple interactions (and without extraneous factors), would be better.

Therefore, this research had two potential methodological approaches, both of which will be evaluated through the use of pilot studies, to determine which is the most effective. The first approach that we completed was to use existing, “live” websites, which after evaluation are deemed to be either low accessibility or high accessibility. The other approach is to use very basic website structures, created from scratch for the study, which remove the various confounding factors from the existing sites (using an approach similar to Hochheiser and Lazar in 2010).

3.2 Pilot Study for Approach One: Using Existing, “Live” Sites

The websites for the pilot study were selected as sites that would fall under the purview of the 12 categories of public accommodations per the U.S. Title III of the Americans with Disabilities Act. It was decided that category 1 (places of lodging), category 2 (establishments serving food and drink), category 6 (service establishments, including health care providers), and category 10 (places of education) would be used as a sample of the categories for site selection. Then, within the four categories, a significant number of websites were inspected for WCAG compliance and violations to determine both high and low accessibility sites for each. As noted previously, the low accessibility sites had the additional challenge of trying to ensure that the level of inaccessibility would not prevent users from being able to complete tasks.

The pilot testing included a total of five participants, four females and one male, ranging in age from 22 to 64. Each participant had at least five years of experience using a screen reader and the average for screen reader experience is 19.8 years. Four out of the five participants use JAWS as their primary screen reader, while the remaining participant uses VoiceOver as their primary screen reader (as noted earlier, the participants all used their own technologies maximized with their own settings). Participant experience with the Web/Internet ranged from 10 years to 30 years, with the average being 19 years of experience on the Web/Internet. Each of the five participants has some college experience. One participant is currently a sophomore undergraduate student, two participants are college graduates, one participant has a Master’s degree, and one participant has both a Master’s degree and a PhD.

The task list was chosen after careful consideration of the accessibility barriers present on each website, and also with regards to what would be considered a representative user task. Each task type also had to be equally replicated across all of the websites in a category. So, if one grocery website had a data retrieval task, then all of the grocery store websites needed to support equal, but not identical, data retrieval tasks.

After completing a pilot study of approach one, we determined that approach one is not an effective method for obtaining the quantified time data for high versus low accessibility. After conducting the pilot studies and analyzing the results, we determined that there are often confounding factors on existing, “live” websites, and the necessary approach of using each individual user’s device and assistive technology on the live sites further complicates clear time data. This approach cannot provide a distinct “high accessibility” versus “low accessibility” framework to use for quantitative analysis.

For example, one of the users thought that a website did not have an accessible “Find a Store” search, even though the button was accessible with regards to WCAG. The screen reader would say “Find a Store button”, but if the user navigated to the element next to it, the screen reader would say “Search button”. The sequence of those buttons created confusion for the user because they thought the “Search” button initiated the search for “Find a Store” button. What really happened was that the user was taken to a page that searches the entire website upon clicking the “Search” button, and they could not find the store address information they were looking for.

A second website example was a store that had an occasional pop-up dialog box regarding a current promotion. This may or may not be displayed to users during the collection of data, and when that pop-up was displayed, it could present a case where the task on the website could not be completed at all (even though this would otherwise be a task that should be able to generate time data).

4 Future Steps: Pilot Study for Approach Two

The next stage of this project will involve the preparation of two separate website structures that we will create solely for the project. One site will include high accessibility design practices, and the second site will include common accessibility barriers that are found on websites. We will then conduct a pilot study with Blind users to determine whether this methodology provides more concrete time data than the previously attempted methodological approach. If the pilot study shows promise in terms of collecting data, we will then begin a more expansive study with a larger number of users.

The eligibility of participants for the pilot study for approach two will be determined the same way as for approach one. The format of the pilot study for this approach will have a similar structure to the original pilot tests that have been completed. Using the websites that we created, participants will be instructed to complete representative user tasks in a high accessibility and a low accessibility setting. The time it takes the participant to complete each user task will be recorded, and then all of the time data will be analyzed to quantify how much time was lost due to the presence of accessibility barriers.