Keywords

1 Introduction

Business organizations constantly search for new ways of gaining advantage over their competitors [2, 13]. This concerns the entire business value chain ranging all the way from procurement to production and sales. While in the past businesses often aimed at boosting production without considering an exact demand prediction [17], today’s ongoing propagation of Enterprise Resource Planning (ERP), Production Planning (PP), Business Intelligence (BI) and other, similar analytical systems, allows for more efficient ways of steering business decisions [18]. All too often, however, these systems fail to meet the high expectations assigned to their implementation [14]. Moreover, a great number of implementations are late, over budget or simply not successful [1]. Based on a real case, our work thus aims at better understanding the implementation process of such a tool. We explore the pre-implementation as well as the post-implementation stage and highlight several challenges that had to be tackled and lessons that had to be learned. To guide to our research, we focused on the identification of key requirements (pre-implementation stage) and success factors (post-implementation stage), and how to best explicate these aspects through different technology acceptance models.

While we do realize that similar studies conducted in different settings may produce deviating results, we strongly believe that many of the insights reported in this paper are conferrable to other cases and thus that our analysis is of significant relevance to the information systems community.

2 Study Setting and Research Question

Our team was asked to accompany the implementation of a Text Analysis System (TAS) based on SAP HANA; the goal of the proposed system being the support of the company’s quality management department in automating and improving their current complaint handling process. Pooling, combining, and linking vast amounts of data to produce relevant insights may be considered a key success factor in such a complaint handling process. Generic applications supporting this task are, however, barely available and so the company considered building a proprietary solution. Understanding the perspective of the staff members whose work is directly linked to the existing data-flow and whose daily routines may thus be affected by a changing tool landscape, was considered a significant cornerstone supporting the implementation process – one whose influences may be even greater than those expected from various technical decisions. Consequently, the analysis we present in this paper treats the technical implementation as a secondary aspect and rather puts its focus on the people who are affected. To guide this exploration process, we followed previous studies of technology implementation and acceptance (cf. [3, 8, 19]), being particularly inspired by the Technology Acceptance Model (TAM) [6], the Hospitality Metaphor (HM) [4], and the Unified Theory of Acceptance and Use of Technology (UTAUT) [21]. The goal was to understand the changes such a new tool would introduce into daily routines and to highlight both positive and negative aspects of acceptance. Based on the assumption that the correct implementation of the system would lead to a relatively quick and straight forward adoption, we believed that changes should be easily identified [8]. Only those aspects that require additional learning would need more in-depth follow up analyses for them to be better understood [9]. Consequently, our work may be defined as an initial before-after evaluation exploring the following research question: What are apparent challenges concerning the implementation of an industrial text analysis system and how do existing acceptance models such as TAM, HM and UTAUT compare in identifying them?

3 Task Setting and Problem Space

The technical task was to implement a text analysis tool based on SAP HANA. A dataset including customer complaints (i.e. customers who identified a defect with their product after its purchase) as well as complaints filed by affiliated firms (i.e. authorized shops and other resellers) served as a starting point. From there the goal was to create a dashboard application capable of providing structured information on the current complaint situation all the way down to the product level. It should highlight words, feelings and common problems associated with distinct products. Different types of defects should be given different codes so that in the future a more holistic problem analysis would be supported. While one may argue that the market would offer better tools than SAP HANA to deal with this type of task (particularly tools specialized in text and sentiment analysis such as SAS’s Analytics and Oracle’s Social Cloud) the company chose SAP for three reasons. First, they had been using SAP for many years, and hitherto were never disappointed by its performance (note: we as authors neither support nor reject this reason, we simply report on what had been presented to us). Second, a follow up project should consider adopting the resulting system for company internal data analysis, and third, the company did not feel comfortable interacting with other, 3rd party solutions providers; i.e. their trust had been with SAP for many years. Consequently, a first prototype was implemented using the ‘Voice of Customer’ (VOC) feature. The pre-defined text analysis dictionaries offered by this feature were extended by a custom dictionary which allowed for linking the alphanumeric code of a product with its name, country code and responsible business sales unit. Data to test the prototype came from internal sources. A typical example for how such a data stream would be triggered starts with a customer who identifies a defect with a bought product and files a complaint at the point of sales. The complaint is then manually added to the system adhering to the following structure: ComplaintID, ComplaineeInfo, ComplaintDescription. The SAP HANA VOC feature processes the complaint and consequently updates the dashboard application, which staff members can then use to display and analyze all relevant issues. While this workflow seems relatively straight forward it had to deal with two types of challenges i.e. (1) the technical challenge of text mining, and (2) the rather social challenge of accepting this new technical tool. In the following we particularly focus on the latter of these two challenges.

4 Social Challenge: Technology Acceptance

In search for some guidance on understanding the social challenge of accepting new technologies we have decided to use three of the existing theories reported by Venkatesh et al. [19] and explore how well their constructs align with the interview-based research methodology we used to understand the challenges and success factors of implementing the company’s TAS. The Technology Acceptance Model (TAM) was chosen because of its meticulousness [10] and its strong empirical support [16]. The Hospitality Metaphor (HM), on the other hand, should provide a powerful theoretical framework for investigating technology adoption in a messy, realistic and highly emotive environment [5]. And, finally, the Unified Theory of Acceptance and Use of Technology (UTAUT) should integrate different models to offer a more holistic understanding of technology acceptance. The following sections describe these three theories in some more detail.

4.1 Technology Acceptance Model (TAM)

The Technology Acceptance Model (TAM) consists of four core constructs, i.e. the external variables Perceived Ease of Use (PEOU) and Perceived Usefulness (PU) explain the Attitude Toward Using (ATU), which consequently defines the Behavioural Intention to Use (BIU) a system [6, 10]. Per previous work (cf. [6, 10, 19]) PU is used both as an independent and a dependent variable, as it derives from PEOU and, in turn, predicts both ATU and BIU. TAM’s rational is based on the assumption that technology acceptance is mediated by a user’s attitudes and beliefs, where beliefs are understood as the degree of instrumentality that is tied to an action while attitude is considered to be purely affective [7]. Consequently, one may see beliefs as being related to a person’s subjective judgment of whether performing a given behavior results in a specific consequence, whereas a person’s subjective attitude (either positive or negative) affects the performance of said behavior [10]. It can thus be argued that, according to TAM, the way an individual accepts a technology depends on his/her attitude towards its usage, which furthermore is defined by its perceived usefulness and ease of use.

4.2 Hospitality Metaphor (HM)

Contrasting the clear variable constructs employed by TAM, the Hospitality Metaphor (HM) is seen as a rather theoretical lens that helps understand innovation processes and technology adoption [4]. In fact, some researchers claim that HM’s emphasis on emotional aspects of users’ everyday dealing and struggling with technology makes the metaphor a perfect tool for exploring acceptance [5, 12]. HM is based on the concept that a technology may be perceived as an alien, which is exemplifying and embodying its alien affordances and culture, and that a successful implementation can only be achieved if the ‘host’ organization can extend complaisance and is able to assimilate, absorb and fit the alien’s culture where it offers leverage and benefits (e.g. in new working procedures) [4]. Thus, HM is focusing on how and whether people use technology, give it meaning and make sense of its innovation, which in this case may be interpreted as a ‘guest requiring hospitality’ [12]. According to Coleman, once and organization adopts this perspective, it is able to bridge the cultural boundaries between guests and host. This implies that guests can behave as if in their own environment and therefore hosts may need to relinquish control over the environment. In other words, hosting requires duties and efforts, i.e. it usually requires the modification of everyday routines and practices [5].

4.3 Unified Theory of Acceptance and Use of Technology (UTAUT)

The final perspective we consider relevant for our analysis is manifested by the Unified Theory of Acceptance and Use of Technology (UTAUT). The theory was established through reviewing and subsequently consolidating many of the previous acceptance models, providing a more holistic approach to exploring acceptance by including different perspectives. Consequently, UTAUT is believed to be more stable and robust than its predecessors. According to [19], the theory is based on four key elements; i.e. performance expectancy, effort expectancy, social influence and facilitating conditions. The first three directly determine the Behavioural Intention (BI), while the last element is a direct determinant of the usage behavior. In addition, Venkatesh and colleagues found that, gender, experience, age and voluntariness of use are details posited to moderate the impact of the four key elements. Other constructs that have shown an influence include the attitude towards using technology, self-efficacy, and anxiety. All in all, UTAUT has been one of the most employed acceptance research models of the last decade [15].

5 Study Methodology

In order to explore challenges, important requirements, and relevant success factors of implementing an industrial TAS we conducted two rounds of focus group interviews; one before and one after the implementation of the system. Our interview sample was restricted to staff members who were directly affected by the introduction of the software, including representatives of the implementation team, the current BI team as well as the QM team, amounting to a total of five people (Note: talking to five people meant that we would talk to most employees who were directly affected by the introduction of the TAS). Interviews were fully transcribed and coded, using a deductive coding scheme as guidance (extracted from [8]) and inductively produced codes to fill in the gaps. The focus of the pre-implementation interviews (1st round) was on identifying user requirements for the tool. The second round of interviews (i.e. after the tool implementation) focused on the challenges and success factors of the implementation process as well as on not or only barely fulfilled user expectations. Next, so as to link findings and existing theories, we compared our results with previously published studies on success factors for implementing business intelligence systems [20] and its challenges rooted in organizational change [8]. Finally, we wanted to understand to what extend existing models on technology acceptance (i.e. TAM, HM, and UTAUT) can explain our findings. Given that these models mostly employ quantitative survey data we sent a questionnaire to our participants one month after the tool went live. Although our small sample size does not suffice to draw any conclusion whatsoever on model related issues, it does help categorize the type of feedback TAM, HM and UTAUT provide with respect to the implementation of such a system.

6 Discussion of Results

Following we report on the results of the above describe research agenda. To provide some guidance, the discussion is divided into three different sections. The first section summarizes the three key requirements regarding the system implementation, which we could identify based on the pre-implementation focus group interviews. The second section focuses on the critical success factors we were able to extract from the post-implementation interviews, and the last section reflects on how our data fits the above described technology acceptance models (i.e. TAM, HM, and UTAUT).

6.1 Three Key Requirements

The most important factor pointed out during the focus group interviews concerned the level of automation the TAS should provide. All the interviewees stated that it was fundamental for them to keep control over the process and to not lose their decision-making authority, e.g. “We don’t ask you to build something that is overtaking our position here in the company. We still want to have control over the complaints. We want help from the system, that’s it. […] a human must remain in control at all time”, stated a user in the pre-implementation interview. Keeping the users informed at all times, i.e. keeping them in the loop during the development and implementation of the system, the company tried to fulfil this requirement as much as possible.

A second important aspect regarding the expectations concerned the amount of effort a user should put into learning the TAS. Users asked to create something easy to learn, given the limited time they would have for training. Consequently, we asked them after the implementation about their learning effort and despite the employed training method (i.e. self-learning, learning through trials and guidelines) they reported that it took them only a few days to use the system without any problems. Here, the fact that the implementation happened on a platform they were already accustomed with (i.e. SAP HANA), probably helped smoothen the adoption process.

Finally, the third major issue highlighted by the pre-implementation focus groups concerned the system’s core functionality. Here one of the participants stated: “What we really want is to avoid is a mistake happening twice. […] I want to have a dashboard showing how many times we had that complaint, all the products related to that complaint, all the complaints of that product and all the complaints of that customer. We want to have the chance to investigate complaints in a new and more efficient way […] from here we can then start creating a database of possible solutions”. The company aimed at tackling this request to the point that it became its key use case guiding system implementation.

6.2 Three Critical Success Factors

Guided by Yeoh and Koronios’s [20] Critical Success Factors (CSF) model the following reflects upon three critical aspects affecting the TAS implementation. First, focusing on the organizational dimension, users stated that it was important to have someone pushing for the system’s implementation and usage. Most of the technical people involved in the actual implementation were busy developing and consequently had no time to advocate and promote the new system. Thus, it was up to the team manager to show commitment and involve key staff members. This sort of commitment was perceived as an important key factor, since the resulting TAS was to be used by different types of staff members (respective end users) and so they had to be involved and pushed to participate in various implementation procedures (i.e. use case descriptions, test runs, feedback loops, etc.).

Second, exploring the process perspective, the existence of a well-balanced team is often considered a critical success factor, which was also highlighted by our participants. To that end it was important for the company to have people from different fields on board, so that users would not only deal with the IT department. The different backgrounds also helped to learn about realistic TAS use cases and affiliated tasks. A second aspect of this perspective relates to the iterative development and communication strategy. That is, clearly defined communication channels supported the implementation’s focus, and helped save both time and money.

Finally, a user-oriented change management process seemed critical [20]. Here it was clear that the TAS was going to change daily routines and workloads. Consequently, the company aimed at involving end users as much as possible in order to shape the TAS according to their expectations, needs and desires. On top of that, users also expected that the TAS would change the workload and procedures prevalent in other teams, e.g. a user firmly stated: “I don’t know exactly how this will influence the company in detail but, apart from our team, there will be a serious effect on the Sales Management, on the Quality Inspection and definitely on the Process Management team related to the Quality Department”.

6.3 Three Acceptance Models

Trying to relate the results of our focus group interviews and their subsequent survey study to Davis’ TAM we looked at the data from a more structured point of view; i.e. we used a short questionnaire to collect model specific data.

All our participants considered themselves as experts, and they stated that the TAS had already become an essential tool for their daily routines (although they had only been using the system for one month). According to the TAM we may argue that the acceptance of a certain technology depends on the Behavioural Intention to Use (BIU) which, in turn, is strictly linked to the user’s Attitude Towards its Usage (ATU). ATU, as we can see in the model, depends on its Perceived Usefulness (PU) and its Perceived Ease of Use (PEOU). As we have seen before, our participants considered the TAS implementation a success as they rated the final system to be useful and able to improve the quality of their work. Except for some doubts with respect to its level of independence and the consequent reliability of produced results, participants were confident that the system had become an essential component of their everyday work procedures. This optimism and trust toward the TAS, and the perceived usability provided through a rather user-centered development approach, seem to have been the main drivers of adoption, although other external variables may have been positive influencers as well. Personal differences or organizational interventions, for example, might have modified or altered a user’s PU and PEOU. Thus, while our results support the influence of PU and PEOU on ATU and BIU, they also highlight that some other aspects, which lie outside these core TAM constructs, should be considered.

To explore the system implementation from an HM perspective we investigated whether the TAS was perceived to evoke a threat to the users’ everyday working procedures. Ciborra stated in his research that a successful implementation can only be achieved if the ‘host’ organization is able to extend complaisance and assimilates, absorbs and fits the alien’s culture where it offers leverage and benefits [3]. Both pre- and post-implementation interviews showed that none of our participants perceived the TAS to be an alien that can harm their work. The users felt comfortable with the system from the very beginning so, that once it was in place they even started extending its functions rather than just reading its guidelines and executing pre-defined commands. However, the ability to be in control of the ‘guest’ seemed to be a particularly important factor for this positive adoption, as during the pre-implementation interviews we were often confronted with distinct fears of participants related to the potential power the system would eventually have. This insight is very much in line with the HM theory as it shows that, even if the ‘guest’ is welcome, the ‘host’ wants to keep control over the situation. In the post-implementation interviews, participants stated that they were perfectly aligned with the TAS simply because the system itself does not act or behave without previous authorization. Rather, it shows data in a different way, which enhances the quality of their work without endangering their position. To this end, an important aspect of HM is viewed in the concept of ‘hostility’, which may lead to two different types of behavior; i.e. resistance with respect to new technologies (when a host does not allow a guest to enter the environment and thus, even knowing its characteristics, is not interested in sharing its habitat), and hostile rejection (when the host perceives the guest as an enemy and therefore will treat it as something that can harm and endanger its current situation). Here, our study results particularly point to the easiness with which the TAS has been accepted as a part of an existing working environment. None of the staff members we talked to felt endangered because of the TAS and therefore none of them showed hostility towards this new system. In summary, we thus found that by treating the situation more like a complex organism rather than a static set of influencing constructs, HM was able to offer a sound description of the given implementation challenges.

Finally, based on the existence of both qualitative focus group data and a small number of completed questionnaires, it was possible to also explore the suitability of the UTAUT model as a potential means to explain the given implementation case. Here, the data showed that participants expected strong performance from the TAS. That is, they clearly hoped from the system to significantly improve their prevalent job performance. Thus, similar to what we were able to see when looking at the data through the TAM perspective, also the UTAUT lens identified the given trust in the expected outcome as the most important determinant for the system’s acceptance. On the other hand, the expected effort required to assure the system’s success was rated rather low. That is, in the pre-implementation focus group interviews, participants already stated that they wanted from the company a system which would not require too much effort in learning. From the very beginning of the implementation process this demand was expected to be met and eventually positively confirmed through the post-implementation interviews. Thus, it seems that the user-centered development process, which continuously kept end users informed, also helped in keeping up this instalment of trust towards the usability of the final product. The questionnaire data confirmed this assumption, showing that participants had a positive feeling regarding the system’s ease of use and did not experience any intricacies operating it. As for other social influences, the small sample size kept us from identifying any important external variables which would exhibit a strong effect on the given implementation scenario, although we are very aware that with systems that affect the working procedures of more people such social determinants would most definitely play an important role. However, in the given case the social environment rather acted as a facilitator. This is, from the very beginning participants felt positive about the project’s outcome, which made them consider the complete IT department, the distinct implementation team as well as the higher management, who originally initiated the implementation of the system, a facilitating condition assuring their expectations and consequently their acceptance. According to Venkatesh et al. [19] this facilitating condition can be split into three different components. First we speak of perceived behavioral control (encompassing self-efficacy and technological/resource facilitating conditions), which was achieved by keeping the end users in control of the system, giving them the resources to learn and use the system (e.g. written guidelines) and, finally supporting its complete adoption by utilizing an underlying system platform which already serves as their daily starting point at work (i.e. SAP HANA). Next, we find the core facilitating conditions themselves (i.e. environmental factors which support a successful implementation process), which in our case were represented by the close contact to the implementation team throughout the entire development process.

Finally, the last facilitating condition may be found in the reign of compatibility. That is, given that the TAS was built on top of the already employed SAP HANA platform, a rather seamless integration with other systems was easily achieved. This aspect was also clearly perceived and notably mentioned by our participants. In summary, we thus may argue that an overall positive intention towards the system (i.e. high outcome expectations and low effort expectations) paired with the existence of additional facilitating conditions led to a successful implementation. Examining the situation from this perspective we may thus argue that the UTAUT model very well explains the given implementation process.

7 Concluding Remarks and Future Work

The goal of the work reported in this paper was to analytically describe key requirements and success factors of implementing a new text analysis tool in an industrial setting. An additional goal was to apply the core concepts described by three different technology acceptance models (i.e. TAM, HM, and UTAUT) and evaluate their fit for the given use case. The tool, i.e. an industrial text analysis system, has been successfully implemented and we believe that the above analysis helps identify the core requirements and success factors that contributed to this achievement. As for the acceptance models, we have seen that they exhibited similarities but also showed different strengths in explaining our data. The TAM model may be considered a robust and powerful instrument for studying and predicting the users’ attitude towards using the system. It was easy to identify factors that contributed to the model’s two main determinants Perceived Usefulness (PU) and Perceived Ease of Use (PEOU) and consequently to show their influence on Attitude Toward Use (ATU). A completely different angle was provided by the HM, which predominantly encompasses emotional aspects. The model treated the new system as a ‘guest’ that aims at entering an existing environment. Here we could see that our users were ready to acknowledge the system but wanted to maintain control over it, highlighting a certain fear of novelty. They argued that, while being generally in accordance with the doings of the system, they require time to build up the necessary trust towards its actions. This human characteristic is often perceived as an insurmountable aspect of technology implementation processes. Finally, the UTAUT model was able to describe the users’ expectations assigned to the new tool. Similar to what was already shown by TAM we found high performance expectancies combined with low effort expectancies. In addition, we were able to identify facilitating conditions, such as the roles of involved parties, as important factors influencing the overall acceptance and consequent adoption of the system.

Regarding future work, we plan to conduct similar studies within other companies in order to verify our results. These studies should also add an additional departmental perspective to the data, as we believe that there may be differences between different company departments, particularly when it comes to employees’ age and technology affinity (two parameters which were rather constant in our study setting). Finally, we want to find adequate (potentially better) ways of combining different technology acceptance models and analysis perspectives so as to eventually provide a blue print for a more holistic analysis method capable of describing the implementation and adoption of industrial software systems.