Keywords

1 Introduction

The integration of ASD and UX has been a major research interest of both ASD and UX communities since the late 2000s [6]. Despite an abundance of related literature [20], their integration remains challenging, due to frictions between the approaches [35]. First, each community has diverging needs. ASD aims to satisfy customers by frequently delivering working software [6, 17] and avoiding failed projects by responding to changing customer requirements [12]. UX aims to satisfy user needs and requirements while limiting late design changes to reduce development time and costs or preventing user errors to reduce technical support requests [3]. Second, ASD and UX principles seem opposed: ASD welcomes changing requirements, whereas UX does not [6, 12, 41, 49]. Third, the focus on developers and code production in ASD versus the focus on users in UX shows the further discrepancy between the two approaches [20]. These incompatibilities prevent or slow down the successful integration of ASD and UX.

To overcome these barriers, several models for integrating ASD and UX have emerged. A 2022 systematic literature review [20] reports on 18 primary models for ASD UX integration. Each comprises a series of generic principles for ASD UX integration related to lifecycle or primary processes, from upfront UX design a sprint ahead of agile [44] to parallel and synchronised tracks [6]. Nevertheless, these models do not integrate enough UX. First, the requirements specification remains product-oriented, as none of these models provide usable guidance for integrating user needs into the requirements. This lack of UX is surprising, since user requirements are necessary to deliver products that people actually want [12]. Second, none provide UX designers with a formal decision-making role, e.g., involving UX designers in iteration backlog or planning activities [8], although it is essential to prioritise system features according to UX so as to meet user needs [12]. Further, we identify a lack of UX literacy, characterised by misunderstanding of UX, as another barrier to UX integration. Symptoms of lack of UX literacy in ASD or software development, in general, include mistaking UX for aesthetics or visual design [15], mistaking users with customers or domain-experts [2, 12], belief that performing UX requires no UX expertise [3], belief that UX can be performed informally [5, 8], contentious attitudes towards users [12], and lack of understanding of UX return on investment (ROI) [3]. Lack of UX literacy is typical in low UX maturity contexts [40] and may lead to ostracism of UX experts, e.g. by excluding them from decision-making processes [23].

Despite these barriers, it now seems a given in the ASD community that users need a good user experience to adopt systems, and therefore that developers need UX to deliver competitive systems  [6, 12, 20]. But then, how to explain the aforementioned lack of UX literacy, especially in agile-UX settings? Furthermore, how do ASD stakeholders working in an agile-UX setting perceive UX? How do their perceptions of UX translate into their UX practices? How do their actual and perceived UX practices compare?

To answer these questions, we conducted a case study in two organisations working in an agile-UX setting, referred to in the following as Org. A and Org. B. We used questionnaire, interview and observation to explore how ASD stakeholders perceive UX in lower UX maturity contexts during early phases of ASD UX integration, and the consequences of their perception of UX on UX practices. This research was motivated by our experiences within industrial projects encountering the aforementioned barriers while integrating UX into formal software development model: 1\(^{st}\) author as UX consultant in Org. A (Nov 2021 to Nov 2022), 2\(^{nd}\) author as UX researcher in Org. B (Nov 2018 to present), 3\(^{rd}\) as UX researcher (2012–2017) and as UX advisor in both Org. A and B (Nov 2018 to present). Our missions in Org. A and B did not include the study of UX practices, but focused on the integration of UX into software development models. These missions gave us hands-on experience with the barriers, so we took this opportunity to explore the gap between UX literacy and UX practices at different maturity levels. To the best of our knowledge, the literature has not addressed this gap, although problems in ASD UX integration were identified [10, 12, 25,26,27,28, 44, 45]. The remainder of this article is organised as follows: after a background section, we present the methodology, the results and their discussion, before concluding the paper.

2 Background

2.1 UX and UX Strategy

UX is defined as “a person’s perceptions and responses that result from the use or anticipated use of a product, system or service” [22]. It is an umbrella term for the hedonic and pragmatic aspects of how users interact with the product or service in different physical or temporal contexts [19]. UX is grounded in user-centred design (UCD), a process that places the users, their needs, and their tasks at the centre of development focus. UCD has four phases: specification of the context of use, specification of the user requirements, production of design solutions, and evaluation of design solutions [22].

UX strategy aims to align business goals and UX activities while improving the development and UX of products [4, 16]. Implementing UX strategy achieves economy of scale while reducing maintenance and development costs, need for technical support of users, documentation, and training, job turnover, user errors and mistrust [3, 40, 48]. Despite the obvious benefits of UX, its integration into software development is paved with obstacles [26, 46] stemming from an insufficient understanding of UCD [2, 7], poor comprehension of UX and UX expertise [4, 7, 8], contentious attitudes towards users [12], and lack of awareness of UX ROI [3, 7]. Project failure due to poor UX may present decision-makers an opportunity to push UX adoption and overcome barriers [4].

2.2 UX Maturity/Capability Models (UXCMMs)

UX maturity refers to the ability of an organisation to consistently implement UX processes, while UX capability refers to the ability to achieve the required goals of UX processes [13, 34]. To address obstacles preventing the adoption of UX, UX capability/maturity models (UXCMMs) [7, 14, 42, 48] have emerged allowing organisations to assess their UX capabilities and improve their UX maturity. The literature highlights the importance of UXCMMs during both project planning and project execution [18]. Attributes of UX maturity include, but are not limited to: integration of UX practices in the development cycle, human-centred leadership or organisational human-centredness [14]; a success stories database, education and training, budget and dedicated staffing [42]; focus on users, process management, infrastructure and resources [48].

One of the prominent models for measuring human-centredness in organisations is Earthy’s usability maturity model (UMM) [14]. The model has five levels: X (Unrecognised), A (Recognised), B (Considered), C (Implemented), D (Integrated), and E (Institutionalised). At level X, organisations have no UX practices and are unaware of UX ROI. At level A, stakeholders recognise the need to improve software development practices due to poor UX with their products. Practices that could inform user requirements are performed inconsistently. At level B, ASD stakeholders are aware of the importance of quality of use, engage in awareness raising and training to improve UX literacy, and account for user requirements during development. At level C, the organisation implements UX processes and techniques appropriate for each new project. At Level D, the software development model integrates UX to ensure high quality in all relevant products. Adequate resources are allocated for UX activities and staff members can use UX artifacts. At Level E, organisations are driven by UX, and leverage it to increase the value of internal and external products. UX issues are given equal treatment to other system issues, and human-centered skills are held to the same standard as engineering skills. Each level is described using a set of attributes that are rated on a 4-point scale (none (N), partially (P), largely (L), and fully (F)). To transition to a higher maturity level, an organisation must first fully or largely achieve the attributes of the current level.

In this paper, we argue that high UX maturity cannot be achieved without UX literacy and we break down UX literacy into four attributes: understanding of UCD processes, understanding of UX, attitude towards users, and awareness of UX ROI. We consider these attributes prerequisite for successfully performing UX processes [14, 48], involving users [14], and integrating UX with other processes [14, 42]. The earlier users are involved in development, the higher the UX maturity [18].

2.3 ASD and UX Integration

Although superficially compatible, integration of ASD and UX challenges practitioners and organisations attempting it [10, 11, 45]. Challenges include: power struggles between developers and designers to maintain involvement in projects, lack of a common vision of the product, high workloads for too few UX designers [25], low usability and user needs prioritisation, lack of time for upfront activities, and lack or poor communication between designers and developers [1, 24]. These challenges may occur in all organisations, regardless of size [7]. Moreover, “a communication gap between UX and non-UX practitioners” represents a major challenge for ASD practitioners when integrating UX practices into the organisational software development model [28].

However, up-front UX work such as user research and UI design, also referred to as sprint zero, might help ASD teams to build a shared UX vision and increase the speed of later sprints [44]. Further, Brhel et al. [6] advocate for a shift from up-front design to up-front analysis so as to deliver the right product, i.e. one with a high degree of innovation, usable and useful, beyond the scope of ASD. The authors also recommend that UX and ASD activities should be iterative and incremental, organised in parallel tracks, and continuously involve users.

3 Methodology

3.1 Study Goals and Overview

To answer questions raised in Sect. 1, we set three goals: (1) assess changes in the UX literacy and perception of UX of ASD stakeholders working in an agile-UX setting; (2) observe their UX practices; (3) compare actual UX practices and UX literacy, and identify problems during ASD UX integration. To achieve these goals, we used a mixed-method approach involving survey, observation, and interview (Fig. 1). The survey assessed UX literacy (goal 1), the observation captured UX practices (goal 2), and the interview gathered insights about their beliefs regarding UX, opportunities for UX, and barriers to UX (goal 3).

We adopted a case study approach involving the ASD stakeholders from Org. A and Org. B, two organisations working in an agile-UX setting. We studied both organisations for over a year, starting at the end of 2021, collecting data from a single project in Org. A and from multiple projects in Org. B. We compared UX literacy data collected by survey and interview to UX practice data collected by participant observation. We administered the questionnaire and conducted interviews in two rounds, December 2021 to June 2022 (R1) and July 2022 to December 2022 (R2). Following the agile-UX lifecycle presented in [29], UX and developers worked along parallel, interwoven tracks, and we conducted user tests at the end of each iteration. We used the UX process reference model presented in [30] to select UX methods, based on teams’ immediate objectives rather than on their UX maturity.

Fig. 1.
figure 1

Methodology: overview.

Full size image

3.2 Organisations A and B

Org. A is a medium-sized company desiring to create software to support their employees through automation. They decided to integrate UX activities into the development of this software, having recently experienced a project failure due to a lack of UX considerations, significantly reducing their organisational efficiencies. Org. A hired an external ASD team to develop the new software, and the 1\(^{st}\) and 3\(^{rd}\) authors as UX practitioner and advisor respectively. Both Org. A and the external ASD team were integrating UX activities into ASD processes for the first time. This is indicative of low UX maturity, Recognised (level A) on Earthy’s model [14], as Org. A’s management is beginning to understand the UX ROI, and recognise a need to improve the UX of its systems.

Org. B is a large company and major automotive parts supplier. This study covers multiple projects of a department in Org. B that underwent an agile transformation in 2012 and primarily develops software for automotive solutions. Authors 2 and 3 were enrolled in 2018 to formally perform UX activities, moving from ad-hoc and scattered UX to budgeted and more structured UX, with the goal of promoting UX across the organisation. This indicates an intermediate UX maturity, between Considered (level B) and Implemented (level C) on Earthy’s model [14]. Implemented UX processes show good results; however, skilled UX staff are not yet involved in all stages of development or when required and some ASD stakeholders are unaware of UX as an attribute of the system.

3.3 Participants and Stakeholder Groups

As UX practitioners, we planned and executed UX activities, advised, and worked directly with ASD teams. This allowed us to informally discuss with ASD stakeholders, observe the conduct of UX activities, take field notes, and recruit participants for this study. In both Org. A and B, we recruited participants from three ASD stakeholder groups for analysis and comparison purposes: developers, managers, and senior managers. Developers’ primary task is software development, from which we excluded designers. Managers oversee developers, designers, and UX staff. Senior managers are top-level managers responsible for strategic decisions at project and company level.

The survey involved 30 participants, 13 from Org. A and 17 from Org. B (Table 1). Eleven participants from Org. A and 13 from Org. B participated in both rounds, totalling 48 responses. Eight participants were interviewed, four per organisation. No Org. A developer was available for interview in R2 and no Org. B senior manager was available in R1. The same Org. A senior manager, and Org. B manager were interviewed in both rounds. Participants read and signed a consent form, on paper before the interview, electronically before the survey. The subcontracting agreement executed with Org. A and the memorandum of understanding entered into with Org. B authorised collection of observational data.

Table 1. Summary profile of participants sorted by ASD stakeholder group (Dev: developers; Mgr: managers; SMgr: senior managers), organisation and round.
Full size table

3.4 Methods

We created a 6-module questionnaire (Table 2), inspired by [32]. The first four modules measure ASD stakeholders’ UX literacy: understanding of user-centred design processes (UCD); understanding of UX concepts, roles and definitions (UUX); attitude towards users, their needs and requirements (ATU); awareness of UX ROI and the benefits of integrating UX into development (ROI). The last two modules, opportunities for UX integration (OPP) and barriers to UX integration (BAR), collect data on problem recognition and integration, two attributes of Earthy’s UMM [14]. Problem recognition is “the extent to which members of the organisation understand that there is a problem with the quality in use of the systems produced”. Integration is “the extent to which human-centred processes are integrated with other processes”. We used these two attributes to collect more accurate data related to maturity levels A to D, having estimated the UX maturity level of Org. A as level A, and level B or C for Org. B.

Except for UUX, which contains six, each module contains four statements, half being reverse-worded to reduce agreement bias [47]. Participants rate their agreement with each statement between 1 (strongly disagree) and 5 (strongly agree) on a Likert scale. Answer accuracy tends to decline over time, so we maximise answer quality through exclusive use of close-ended questions, which requires less participant motivation and skill, and progressively decrease question complexity throughout [47]. The questionnaire takes 5 to 10 min to complete.

We used participant observation through the 1\(^{st}\) and 2\(^{nd}\) authors, who were directly involved in projects in Org. A and B, respectively. We kept notes on events occurring in the field (e.g., meetings, decisions, and discussions) between ASD stakeholders. We used these notes as sources of observational data.

To elaborate upon the survey, we conducted semi-structured interviews based on UX attributes and observational data. We covered nine topics, each providing insights into one or more of the attributes, and ASD stakeholders’ current understandings and beliefs regarding UX practices. Table 3 displays the topics, their attributes, and with whom they were discussed. We interviewed and recorded participants in person or via online video conference.

Table 2. Questionnaire statements by UX attribute, rated on a 5-point Likert scale from strongly disagree to strongly agree (standard) or from strongly agree to strongly disagree (reverse).
Full size table

To analyse survey data, we calculated mean scores per UX attribute and stakeholder group and looked for outliers. We sought statements with notable low means across all respondents, and for each organisation per round, to use as cues to investigate qualitative data. Further, we transcribed interviews verbatim, and collected quotations linked to UX attributes (Table 3). In addition, since interview and survey questions were structured per UX attribute, we were able to cross-analyse interview and survey results, linking qualitative and quantitative data. Finally, we connected our findings to relevant observational data.

Table 3. Semi-structured interview: guiding questions. UCD stands for understanding of user-centred design, UUX for understanding of UX, ATU for attitude towards users, ROI for awareness of UX ROI, BAR for barriers, and OPP for opportunities. Dev stands for developers, Mgr for managers, and SMgr for senior managers.
Full size table

4 Results

4.1 Survey

As shown in Table 4, ATU, BAR, and OPP notably decreased between rounds in Org. A; conversely, UCD, ATU, ROI, and OPP notably increased between rounds in Org. B. Table 4 also presents the average score of UX attributes. Org. A developers and managers exhibited similar mean levels of UX literacy. Excepting OPP, senior managers scored higher in all UX attributes. Org. B developers had higher means than managers. In R2 senior managers showed notably higher UCD and ROI than developers and managers. Table 5 shows differences in scoring for participants from both organisations who took part in both rounds. The ATU, BAR, and OPP scores decreased in Org. A, while ROI and UUX scores increased in Org. B.

Table 4. Questionnaire: mean scores per UX attribute and ASD stakeholder group. Round average in second column for all ASD stakeholders. Orange/Green: noteworthy decrease/increase (± 0.2 points) in scores between rounds. AVG: UX attributes average.
Full size table

Eight individual questionnaire statements had mean values below a neutral score (3) across all ASD stakeholder groups. In the ATU attribute, ASD stakeholders disagreed with the statement “Users are able to express what they want” (M = 2.83, SD = 0.72 in Org. A; M = 2.81, SD = 0.96 in Org. B) and agreed with “User expectations are difficult to manage” (M = 2.46, SD = 0.79 in Org. A; M = 2.56, SD = 0.92 in Org. B.). In the ROI attribute, Org. A generally agreed that “UX activities increase development costs and time” (M = 2.67, SD = 1.08). In contrast, Org. B. scored better (M = 3.22, SD = 1.05). Results for this statement are a negative outlier within the attribute. Each statement from the OPP attribute received negative scores.

Table 5. Variation between rounds for both round participants.
Full size table

4.2 Observation

Table 6 summarises discrepancies between planned and executed UX activities. In Org. A, at the start of the project, managers attended a 2-hour UX training session. The same training was planned for developers, but never occurred. The first key touch point, at the end of the UX analysis, allowed stakeholders to develop a shared understanding of user needs and requirements, technical restraints, and opportunities. Lacking prior UX experience, ASD stakeholders were reluctant to adopt users’ points of view. Although expedited, ASD stakeholders started development while UX research was ongoing. Planned UX activities were abandoned. Overlooking UX research results and UCD, ASD stakeholders discussed and ‘validated’ data models and prototypes with users in ASD-led activities. UX research results went unheeded. Results from successful UX activities with managers did not reach developers. The low-fidelity prototype was evaluated while being altered. No time to re-evaluate it was allocated, leaving key UX issues unresolved. During coding, software modification became resource intensive and had to be delayed post-launch. Final user tests were conducted with few users on an unstable software version, hence suboptimal usability at launch.

Org. B more successfully executed planned activities, possibly due to commitment to, and experience in, UX-driven projects. During R1, Org. B was engaged in a long-term project. The team adopted ASD while following UCD processes, wherein the final design solution resulted from six iterations of UX evaluations with six to eight users. At touchpoints, ASD and UX stakeholders analysed UX evaluation data and deliberated design changes. However, implementing the two parallel interwoven ASD UX tracks was challenging. In a subsequent project, Org. B struggled to convince the customer and business developers to start UX activities (e.g., user interviews, personas). The development team focused on delivering a functional prototype to satisfy the customer, who thought UX could be performed later. UX was not integrated in all projects and UX staff frequently had to jump between projects and finish many tasks on short deadlines.

Table 6. Discrepancies between planned and actual execution of UX activities, with status of objectives. Obj. = objective A = achieved; PA = partially achieved; F = failed.
Full size table

4.3 Interview

UCD. Org. A’s developer believed developers need not be involved in UX activities, “not [caring] about [them]”. UX activities were criticised by the and a senior manager due to resource-intensity and “untimeliness”, believing development “had to” precede UX research completion. They wanted UX research to start a sprint ahead of development so as to fully leverage it in the future. Since UX evaluation started during software programming, fixing UX issues would lead to time and budgetary overrun, and R2’s manager lamented resultantly insufficient improvements. The senior manager recognised UX awareness alone is insufficient to meet user needs and requirements, and observation of users performing tasks is necessary. Org. B understood the goals and outcomes of UCD’s four phases, and follows the analysis-design-evaluation cycle. The senior manager believed UCD reduces late design changes and decreases the risk of developing suboptimal products. Nevertheless, much convincing is required to run UX-driven projects, as Org. B’s culture tends to be engineering-oriented.

UUX. R1’s Org. A developer’s believed users are “just humans”, and thus inherently uniform, disregarding user background and characteristics in development, perceiving users’ specific needs and requirements as unjustified. R2’s manager and the senior manager did not share this view, recognising UX as key to solving workflow problems and improving user efficiency and general hedonic aspects. Org. B’s ASD stakeholders see the value of UX in identifying and solving real user problems. They believed UX is necessary for any project unless the timeframe is prohibitive or the project is purely technical. They recognised the roles and importance of UX-trained staff, believing in a transversal UX approach, wherein discussion of UX activities is a basis for stakeholder meetings. However, only some projects were driven by users’ needs and requirements.

ATU. Org. A’s developer regarded UX and user involvement as relevant only to the low-level details of software otherwise conceptualised by managers. ASD stakeholders struggle to fully take into account users’ feedback, believing users are biased by their current cognitive work models. The senior manager, believing users should be at the core, was particularly dismayed by developers’ “lack of empathy”, as “the end user would never be capable of using what they deliver”. Having observed UX evaluations, Org. B’s developers supported UCD, and understood how UX activities help reduce bias, stating “It’s normal to take [user] motivations, choices, and desires into account”. R1’s manager believed “even if [users] cannot explain [their preferences]”, their feedback aids in design, and multiple prototypes and iterative evaluations are necessary.

ROI. Org. A’s senior manager, though unsurprised by UX findings, understood the need to formalise them. The managers and the senior manager recognised UX reduces late development revisions and increases product value, yet prematurely proceeded with development, believing the then incomplete UX analysis’ findings would be irrelevant. By R2, having delayed release, the senior manager regretted this decision, recognising the software would require post-launch rectification to improve UX and user efficiency. The senior manager was disappointed by the insufficient leveraging of UX findings throughout development. Having conducted several UX-driven projects, Org. B’s ASD stakeholders regarded UX as necessary, requiring sufficient time, staff, and budget. The senior manager and manager recognised user data as valuable to development decision-making and convincing customers to revise products. R2’s manager believed UX activities uncover UX issues key to development. Org. B’s ASD stakeholders believed UX costs are offset by returns from product attractiveness, improved user satisfaction and performance, shorter delivery time, and reduced late design changes.

OPP. Org. A’s R1 manager believed UX revealed the “pain” of users’ work among other otherwise overlooked issues, while the R2 manager appreciated the bias reduction derived from UX. Both managers believed UX clarifies user needs, enabling the creation of UX goals and adaptation of user stories. In R2, the senior manager opined, “User experience is not a luxury, [...] it’s a necessity” but the difficulty lies in balancing user and technical requirements. User involvement is key to development, since without it, the project would be a “probable failure”. Org. B has past success with UX integration, providing a basis for ASD stakeholders to further expand UX activities and strengthen ASD UX integration. They often claimed issues lie in balancing the work of UX and development staff, resulting in rushed sprints and constant re-prioritisation of the sprint backlog.

BAR. In R1, Org. A ASD stakeholders believed UX analysis results were in line with already planned solutions. They also believed UX reduces requirements for software revision in late development, and that the product’s value increased. At the end of R2, the managers and senior manager concluded that despite desiring to adopt UCD, timeline and budgetary constraints prevented UX integration. Org. B. faced barriers when expanding UX to new projects; ASD stakeholders often had to be persuaded to include UX activities. The senior manager regarded many ASD stakeholders as not understanding UX and explained that “we will save time later” by conducting UX activities instead of rushing into development.

5 Discussion

5.1 Interpretation

Table 7 shows the UX maturity assessment of both organisations using Earthy’s UMM [14]. Each author assessed their organisation using observational data and personal experiences gained from fieldwork. To maintain consistency, the 3\(^{rd}\) author, having knowledge of practices in both organisations, provided additional insights to reach consensus. We rated attributes of levels B (Considered) and C (Implemented), as lower levels are achieved, while higher ones are not.

At level B, two attributes are analysed: B1 assesses ASD stakeholders’ awareness of quality in use (i.e., efficiency, effectiveness, and user satisfaction), while B2 evaluates user focus by considering end users’ needs and requirements throughout the development process. At level C, three attributes are analysed: the user involvement attribute (C1) represents the extent of user data elicited from representative users; the human factor (HF) technology attribute (C2), the extent UCD methods and techniques are used; the HF skills attribute (C3), the extent to which HF skills are used in human-centred processes.

Table 7. Level B and C of Usability Maturity Model [14] management practices and assessment of both organisations. N: not achieved, P: partially achieved, L: largely achieved, F: fully achieved.
Full size table

Org. A. Org. A attained level B UX maturity, partially achieving the practices of this level. ASD stakeholders were only partially aware of quality in use, as only one external consultant had sufficient UX training (B1.1) and only the managers had any UX training, which was brief (B1.2). ASD stakeholders often mistook UX for UI and underestimated the scope and impact of UX activities (B1.3). Further, ASD stakeholders struggled to prioritise user needs and requirements: the value of UX evaluations was understood, though UX research was disregarded (B2.1). User background and contexts of use were frequently disregarded, resulting in software developed without users in mind. Neglecting end-users is particularly concerning for UX, which prioritises their experiential response during system interactions [35]. However, by the end of R2, the managers and senior manager understood users have specific needs and requirements (B2.2). Further, Org. A only partially achieved three practices from level C: user tests were conducted on medium- and high-fidelity prototypes with end users; however, the allocated time was inadequate, as were the number of iterations and participants (C1.2); the number and the extent of UX evaluations were insufficient, and occurred after development started (C1.4); and, since facilities and tools were unsuitable, outsourcing was necessary (C2.2). These findings are indicative of low UX maturity [7], as is the absence of UX in decision-making [24] and limited UX resources [35].

Org. B. Org. B attained level C UX maturity, as they largely achieved practices from level B, but failed to achieve all from level C. Staff executing UCD processes were largely aware of quality in use as a system attribute, and those performing processes relating to user-facing elements accounted for the human beings who will use it. Throughout development, appropriate UX methods using representative users identified user needs and requirements: in all development phases user feedback was collected, analysed, and integrated to improve the product, though not in all projects (C1.1); users interacted with phase-appropriate prototypes (C1.2); user characteristics defined the evaluation measures of quality in use for prototypes (C1.3); and UX evaluations were performed until quality in use was satisfactory (C1.4). UCD methods and techniques were selected during development: user needs defined project objectives, UX methods to elicit user needs and evaluate prototypes were appropriate to each phase (C2.1); tools and facilities invested in (C2.2); however, UX staff resources were insufficient (C2.3). HF skills were partially used in human-centred processes: required competencies identified (C3.1) and some UX skills partially developed (C3.2); but limited UX staff meant UX was absent in some phases (C3.3). Lack of UX resources limits achievable outcomes and creates bottlenecks [36]. ASD UX integration requires coordination through mutual adjustment and frequent communication [43], necessitating efficient collaboration. Successful designer-developer collaboration depends on seven factors [24] and is key to raising UX maturity. Despite Org. B’s designers and developers working closely, communicating often, and from an early stage, UX processes are far from being stable and embedded into organisational culture, as UX involvement varied with project and customer, echoing [43].

UX literacy vs. UX practices. The gap between UX literacy and UX practices is most salient in Org. A and their moderate UX literacy prevents UX-driven development. Although UX consultants increased its capacity to do UX (e.g., correctly applying UX methods), Org. A still lacked the capacity to use UX (e.g., using UX knowledge during development processes), the difference between do and use being introduced in [37]. ASD stakeholders need expertise and communication to avoid relying on assumptions and to properly understand user needs in UX activities [33]. Within Org. A, ingrained belief that solutions based on personal opinion are equal to or better than UCD trumped UX literacy. This belief resulted in neglecting UX research findings and initiating development prior to completing key UX activities, thereby causing delays in software launch and raising costs: lack of UX resources stifles UX’s potential, bottlenecking development [36]. The decrease in ATU derives from a belief that simply asking users for their needs and requirements suffices, and changes in users’ opinions reflect users’ unreliability. A combination of aversion to UCD, lack of understanding of UX artifacts and reliance on pseudo-UX activities led (senior) managers to often deviate from strategic UX planning and insufficiently grasp user characteristics, needs, and requirements. UX evaluations were conducted too late and insufficient time was allocated to address UX issues. These obstacles are in line with findings in [36]. The increase in time and development cost led to a decrease in OPP and alerted ASD stakeholders to the necessity of conducting UX activities.

Org. B has higher UX literacy and is becoming UX-driven. The manager and senior managers had the highest UX ROI awareness, while developers supported UX integration. Even though commitment from top management is crucial to improve UX maturity [18], some remaining barriers prevent Org. B from adopting UCD. Only innovation projects were UX-driven, showing room for further expansion of UX practices and culture. Org. B’s in-house UX team, with four years experience and wide knowledge of UX, produced relatively high UCD, UUX, and ROI scores. Org. B seeks a competitive advantage through integration of UX, recognising the value of solving real user problems.

Prospects. Both Org. A and B are striving to improve their UX maturity level. Org. A aims to involve more users, perform additional user tests, prioritise UX by temporarily halting development until UX research is complete, and leverage UX to more effectively manage their IT budget. Org. B needs to expand their UX workforce, enhance the UX literacy of ASD stakeholders, and systematically start projects with UX research. We cannot provide standardised recommendations to facilitate the integration of ASD and UX, since according to [18] such recommendations must depend on the context of the organisation (e.g., UX maturity), project (e.g., related business goals), and team (e.g., UX literacy).

5.2 Limitations and Strengths

The limitations of this paper mainly regard internal validity. First, we cannot use Cronbach’s alpha to statistically validate the questionnaire, since participants with low UX literacy produce inconsistent scores within questionnaire attributes. Second, some ASD stakeholders left between rounds, preventing us from tracking the evolution of some participants’ perceptions and attitudes toward UX throughout the study. We offset this by incorporating new participants and interviewing members of all ASD stakeholder groups at least once. Third, our relationships with participants may have influenced their responses, as may have the introduced UX methods. Specifically, we worked closely with some participants for over a year, integrating UX activities into projects, suggesting methods, and guiding implementation during software development. Nevertheless, we reduced this potential bias with the following counter-measures. On the one hand, we controlled the instrumentation threat to internal validity [9] by using the same interview guide in both organisations (Table 3). On the other hand, we committed to our role as UX practitioners throughout our missions. Lastly, our consulting work presented an opportunity to undertake this study in the field but did not change our approach to UX consulting or our behaviour as practitioners.

The primary strength of this work lies in the mixed-method approach, which minimises result subjectivity, as the observational data gives nuance to self-reported survey and interview data. When possible, we triangulated data sources collected with all three methods, cross-checked the findings, and 1\(^{st}\) and 2\(^{nd}\) authors independently analysed and discussed their interpretations for coherency [21]. E.g., quantitative survey results guided our search for qualitative evidence in observation data to elaborate why ASD stakeholders held certain beliefs, helping us to explain the gap between their UX practices and literacy. Our longitudinal approach strengthens result validity beyond a mere snapshot: we repeatedly collected data for over a year, contextualised the data, and identified patterns in UX practices. Finally, we carefully selected participants from different stakeholder groups (i.e. developers, managers, and senior managers) with different roles and perspectives on UX, so as to target key positions in software development.

6 Conclusion and Future Work

We performed a case study of two organisations presenting the characteristics of lower levels of UX maturity, working in agile-UX settings. From this case study, we showed ASD UX integration is better achieved when ASD stakeholders exhibit higher level of UX literacy, which enables organisations to improve their UX maturity. However, their UX practices under-perform compared to their UX literacy, as there is a gap between what they understand, what they think they do, and what they actually do. Through a mixed-method approach, we identified a gap between ASD stakeholders’ knowledge of UX and their UX practices. Further, we linked our findings to UX maturity levels using Earthy’s model. This case study shows the variation and differences in this gap between UX literacy and UX practices in two organisations.

Org. A exhibits a low UX maturity level; Org. B exhibits a medium level. In Org. A, the largest issues stem from low UX literacy, which hinders UX practices implementation during product design and development, resulting in deviation from strategically planned UX activities. This leads to friction between ASD stakeholders and UX practitioners, and to an end product with poor UX. Failure to conduct proper UX activities delayed product launch, enabling ASD stakeholders to recognise the criticality of UX to product success. In Org. B, initial barriers to UX integration were overcome before the study started. Our findings reveal that beliefs and practices may shift depending on the course of action and struggles that occur within an organisation. Prior success stories help foster positive attitudes towards UX. Managers were aware of the value of UX and developers were unopposed to user involvement in development.

In future work, this study should be followed by longitudinal studies containing various touch points, multiple participants and organisations, across all levels of UX maturity. These studies would enable collection of further empirical evidence on the UX literacy and UX practices gap and its nature across all UX maturity levels. To provide recommendations for integration of UX in agile work practices, we should focus on providing guidelines for reducing the gap between UX literacy and UX practices.