1 Introduction

One route to understanding business processes is to experience Enterprise Resource Planning systems in the classroom. The aim of this research is to measure students’ understanding of business processes after experiencing ERP and to understand why students learn the way that they do. The educational study of business processes education is important because business organizations are increasingly emphasizing them. From global competition to the speed of today’s transactions, organizations are changing the way they think from a functional perspective to one of a cross-functional nature. The cross-functional view requires sharing information, information systems and integrated processes (Amrani et al. 2006). For an organization to be more efficient, it needs to improve its way of doing business, which often equates to improving the business processes (Harmon 2007). Enterprise Resource Planning systems (ERP) are very large software programs that control every aspect of a company from sales to accounting to supply chain to human resources. With ERP systems, business processes become standardized, information flows across functional lines (Barua et al. 2007) enabling companies to become more efficient and competitive. Enterprise Resource Planning systems software has been used to experientially teach integrated business processes over the last 10 or so years (Wagner et al. 2000; Wang 2011; Rienzo and Han 2010). The importance of understanding business processes is evident in the literature and in practice. Higher education should be confident that the students are actually learning business processes. However, assessing that learning of business processes is challenging.

An extensive review of past studies attempting to measure business process learning with ERP systems resulted in surprisingly little conclusive research on the effectiveness of teaching ERP systems in a university setting (Monk and Lycett 2011). Understanding whether a student gains in knowledge of business processes after learning ERP therefore remains a challenge. While most of the past research reviewed stresses positive feelings for using ERP, there is little statistically significant quantitative data to validate the effectiveness; much of the data are based on studies of self-efficacy. Students should feel positive about their learning environment, but their own perceptions do not measure their true understanding of business processes. In summary, a more concrete measurement of knowledge is required for conclusive evidence. In this research, it is argued that business simulation could be a more appropriate measure of business process understanding that stretches to the higher levels of Bloom’s cognitive taxonomy (Bloom et al. 1956).

Computer simulation of business is common in higher education and experiential learning theory accelerated the use of simulation games in business education (Keys and Wolfe 1990). Kolb’s experiential learning theory is based on his philosophy that “learning is the process whereby knowledge is created through the transformation of experience” (Kolb 1984, p. 28). Experiential learning theory proposes that a learner can understand more with experience. Kolb’s experiential learning cycle can be simplified as shown in Fig. 1. Within the learning process experience, the learner is reflecting, understanding, and acting. In the reflect phase, the learner is taking in all the information, identifying problems, and assessing the situation. From there, the learner moves to understand from the previous phase, builds up a strategy, and looks for a solution. Finally, the learner acts on that strategy and solution formulated in the previous phase. Those three steps together form the concrete learning experience.

Fig. 1
figure 1

Simplified Kolbian experiential learning cycle

Full size image

An example of experiential learning is the use of simulation games at universities to teach ERP concepts. Leger claims faculty have found that students struggle with the concepts that ERP systems introduce—integrated business processes—and yet also noted that today’s business student is very computer-literate (Leger 2006). The ERP educational simulation developed at HEC Montreal involves teams competing to run a cereal manufacturing business. Working with different time periods, the teams decide on production and sales and use an actual ERP system (SAP in this instance) to determine production runs. Results and statistics are generated in the form of reports and Excel files to give feedback to the teams so they can make adjustments. In this gaming classroom, students move from a silo-oriented approach to running a company, to a process-oriented approach.

Computer business simulation games can do a good job of mimicking managers’ decision making processes in a business situation. And these simulation games could also be used as an assessment to measure how well a student understands the various parts of a business and how they interact. Students should be asked to apply their knowledge as they would in a real life situation for assessment purposes (Weiss 1998); they should be able to solve new problems in unknown circumstances which will assess at a deep level (Biggs 1999). In this research it is argued that an experiential simulation game can be used as assessment for determining business process understanding in any class with or without experiencing ERP.

This paper is organized as follows: The introduction stresses the importance of business process education and the challenges to assessing that pedagogy. Computer simulation is offered as a superior assessment tool. The Method of Study section describes the research study, including a discussion of the philosophy of critical realism. This is followed by a section on quantitative and qualitative results of that study. The paper then concludes with the discussion of the results and educational implications.

2 Method of study

The research philosophy pursued in this study follows critical realism, which attempts to understand the mechanisms beneath causal relationships (Danermark et al. 2002). As an alternative to the traditional philosophies such as positivism or interpretivism, critical realism uses elements of both to explain the impact of “sociotechnical phenomena” (Wynn and Williams 2012, p. 787). Under this philosophy, empirical results alone are inadequate to determine theory. To complete a critical realism research study, mechanisms are hypothesized to explain the quantitative and qualitative data (Danermark et al. 2002; Bygstad and Munkvold 2011). Within the context of the data, there are forces for (causal powers) and against (liabilities) the workings of these mechanisms by which the results occur; essentially the ways in which things actually act. Powers and liabilities each demonstrate potential mechanisms that may or may not happen in a given research situation (Sayer 1992). Bygstad and Munkvold summarize the goal of critical realism research as, not to come up with generalizations, but to uncover the mechanisms that cause the outcomes. Those mechanisms may not be evident from simple observations (Bygstad, Munkvold 2011).

Critical realism research, being diverse by nature and holistic in its view, demands multiple tactics (Bhaskar and Hartwig 2010). Mingers (2001) and Orlikowski and Baroudi (1991) present a set of cogent arguments in favour of using multiple research methods in a single research project.

  • The world is complex, Mingers argues, and so one research method cannot explain all its complexities (2001).

  • If only one research method is used, then the researcher may only be looking at one aspect of the project, and missing out on other facets, showing limited results (Orlikowski and Baroudi 1991).

  • The triangulation of methods used in research will make it a stronger research project because the different methods can explain different aspects of the project and fill in any holes (Mingers 2001; Orlikowski and Baroudi 1991).

  • The field of information systems has expanded to that of a social science therefore research within the field should employ a variety of research methods (Mingers 2001).

  • Research projects run through various phases and each phase can use a different research approach (Mingers 2001).

In the research here, data collection began with a quantitative approach when the students from three separate undergraduate classes (1 with ERP and 2 without) played a business simulation game and were assessed by the outcomes of the game. This phase seeks to measure what the students have learned about business processes with or without ERP pedagogy. ERP Pedagogy in this experiment consisted of learning about ERP systems through books, readings, and lectures and using a live SAP system to perform basic business cycles such as order to cash. The second phase of the research is qualitative, where the students participated in focus groups and then in one-on-one interviews. The resulting data answers why the students played the game as they did, understanding the influence of ERP knowledge on using business processes to run a virtual company. Beginning with the proposition that students will learn about business processes when experiencing ERP systems in the classroom, such as using a live SAP system to perform business transactions, this multi-phase research tests and refines the proposition. To understand the students’ thought processes while they played the simulation game and to understand why they did what they did in relation to business processes, this qualitative approach is appropriate for the second phase. By interviewing students, this triangulated approach fills in the gaps left open and results in a refined and well-supported theory. Past research was lacking in that mostly quantitative data was collected, not allowing for a deep(er) understanding of how students learn. It is argued that critical realism is a more comprehensive research strategy that will produce definitive answers and recommendations for future work.

The three undergraduate classes tested were similar in level: Business Information Systems (with ERP), Accounting Information Systems (without ERP), and Programming Business Applications (without ERP). Before entering any of these classes, students must first successfully complete an introductory class, Business Computing: Tools and Concepts. It can therefore be assumed that each type of class begins with a similarly-prepared group of students within the subject of information systems. In addition, any one of these classes can count towards an undergraduate degree in an equivalent fashion. In other words, if a student takes one of these 3 classes, they do not have to take either of the other two to graduate. However, the Accounting students take their Accounting Information Systems class a year later than the students who take Business Information Systems (with ERP) and Programming Business Applications (without ERP). This maturity and additional knowledge gained from an extra year of university may have an effect on this study.

Students in each class were asked to play a business simulation game either before any education began or during the first week of their classes. The outcome of the game, net profit, was recorded for each student’s game result. At the end of the semester, students were again asked to play the business simulation game, with net income outcomes recorded. This quantitative data was analysed statistically by t-tests between classes of the same university.

Focus groups and email interviews were then arranged and conducted with students to briefly discuss strategies during the game. From these preliminary interviews, the qualitative data was analysed and students were selected for further in-depth interviews. These selected students were interviewed to discuss the business simulation game and other topics surrounding business processes. Using content analysis, all interviews were transcribed and coded with the software Nvivo. Categorical codes, previously identified in the literature, were used and new codes discovered when analysing the interview transcriptions. A mechanism for learning was developed for each school. Further delving into the data revealed specific objects or properties that have the power to trigger the learning mechanism or to inhibit the working of the learning mechanism. Those powers and liabilities were analysed, discussed, and compared between the different classes. Implications for education were then concluded from the results.

3 Results

3.1 Quantitative data analysis

A t-test analysis was performed on the net profit (simulation game outcome) difference for the 3 different classes. The difference of the net profit score of the class with ERP was compared to those without ERP. Results of the statistical data analysis follow.

3.1.1 Results of classes with ERP versus classes without ERP

T-tests were run three times: Once for the comparison of Business Information Systems (with ERP) to that of Accounting Information Systems (without ERP), once for the comparison of Business Information Systems (with ERP) to that of Programming Business Applications (without ERP), and once for the comparison of Accounting Information Systems (without ERP) to that of Programming Business Applications (without ERP). Prior to the t-tests, the data was checked for normality.

The t-test compares each set of two groups, by testing the null hypothesis that both means are equal. For comparisons of the class with ERP to those without ERP, the p-value is 0.213 and 0.494 respectively, which are both above the 0.05 significance level. Therefore, the analysis failed to reject the null hypothesis and it can be assumed that there is no difference between the outcomes of the classes without ERP compared to those with ERP. In other words, the difference in Net Profit for Business Information Systems (With ERP) is the same as the difference in Net Profit for Accounting Information Systems (Without ERP) or Programming Business Applications (Without ERP) for all the students in the classes tested. In addition, the Accounting Information Systems (Without ERP) was compared to the Programming Business Applications (Without ERP) via a t-test with the results again insignificant at p-value of 0.562.

Along with the game play, students were asked to record their grade point average, known as GPA. GPA is the most-used indicator of university performance, used for graduate school acceptances and post-graduate employment (Richardson et al 2012) and was thought to be a good way to compare the different sets of classes to ensure the students were similar. The first step was to check the normality of the GPA data. For Business Information Systems (with ERP) compared with Accounting Information Systems (without ERP) the p-value is 0.000 which indicates that the GPAs of the two classes are different (Accounting Information Systems’ GPA was also significantly different from that of the Programming Business Applications’ GPA). Since the Accounting Information Systems students’ data is not normal, a Mann–Whitney test was additionally run on the data. The results, showing the p value at 0.0002, confirm the difference between the two classes’ GPAs. Surprisingly, the GPA difference between the two classes does not appear to make a difference in the outcomes of the delta between the simulation game plays, but every student from the high GPA class (Accounting Information Systems) had a positive net profit the first time they played the game (bar one student). This is an indication that there are differences between the groups, which can only be discovered by analysing the qualitative data. For the other class comparison, Business Information Systems (with ERP) compared with Programming Business Applications (without ERP), the p-value is 0.303 which indicates that the GPAs of the two classes are not different.

For those students who were interviewed in-depth, the data looks different when figuring which students increased their score during the second time they played the simulation game. All students who experienced ERP increased their score after the ERP class, as shown in the following Fig. 2.

Fig. 2
figure 2

Interviewed students who experienced ERP

Full size image

In addition, if the students who learned with ERP (Business Information Systems) are compared with the Accounting Information Systems students who did not learn with ERP, the difference in means for the students who participated in the in-depth interviews is statistically significant. These results are displayed in Table 1.

Table 1 BUSINESS INFORMATION SYSTEMS (with ERP) versus ACCOUNTING INFORMATION SYSTEMS (without ERP) for those who were interviewed
Full size table
Table 2 Frequency count for in-depth interviews
Full size table
Table 3 Sample necessary structure statements
Full size table

The p-value is 0.039 which is below the 0.05 confidence level, so the null hypothesis can be rejected and it is shown that there is a statistical significant difference in the means between Business Information Systems and Accounting Information Systems for those who participated in the in-depth interviews. However, the sample is too small for any definitive claims, but the results may give clues for the qualitative analysis that the two groups have some differences. For those students interviewed in Programming Business Applications (without ERP), the p-value was above the 0.05 confidence level so the null hypothesis could not be rejected for that group compared with Business Information Systems.

Fig. 3
figure 3

The SELECT Model

Full size image

The demographic data from all three classes was reviewed to see if students were basing their simulation game play or interviews on past work experience. Some of the students had summer work experience that would affect their results. (Note: any jobs had by these students were part-time or short-term jobs such as waiters, cashiers, financial analysts, clothing store employees, phone store employees, or accounting firm summer internships.)

We now turn to the qualitative analysis to explore differences between the classes.

3.2 Qualitative data analysis—focus groups and individual interviews

The second phase of this research is qualitative, beginning with focus group discussions and individual interviews. These were conducted with 53 students after the final playing of the simulation game, with 24 of them from Business Information Systems (with ERP), 16 from Accounting Information Systems (without ERP) and 13 from Programming Business Applications (without ERP). Some of these interviews were conducted via email because students had left campus for the holiday break and were unavailable in person. All discussions and interviews were imported into the software Nvivo and then transcribed within Nvivo.

The initial coding phase then commenced. The categorical objects and their properties that were used for this research project are initially prescribed and originate from the theoretical framework developed from the literature as suggested by Danermark (Danermark et al 2002), developed from key facets of business processes. Students who understand business processes should understand this list.

They are as follows:

  1. 1.

    Business Processes

    1. a.

      Connections

    2. b.

      Integration

    3. c.

      Communication

    4. d.

      Value

    5. e.

      Collaboration

    6. f.

      Involvement

    7. g.

      Terminology

  2. 2.

    Enterprise systems—ERP

    1. a.

      ERP integration

    2. b.

      Info view

    3. c.

      Efficiency

    4. d.

      Performance control

    5. e.

      Communication

    6. f.

      Empathy

    7. g.

      Management control

  3. 3.

    Experiential learning

  4. 4.

    Business Knowledge

    1. a.

      Financial

    2. b.

      Marketing

    3. c.

      Pricing

    4. d.

      Manufacturing

    5. e.

      Quality

    6. f.

      Choice

  5. 5.

    Broad view

  6. 6.

    Complexity

Once the focus group and individual interview data was coded with the aforementioned objects, new objects emerged from the analysis and therefore were added to the categorization of the data. The new objects are as (1) Level of Interest, (2) Competition, (3) Strategy, (4) Environment, (5) User Interface, (6) Timing, and (7) Product and Market Knowledge.

To fully understand the impact of the data, a frequency count was undertaken to find out which categorical objects and their properties, both predetermined and new, are most important.

The data are not rich enough to draw any conclusions, or to develop the learning mechanism, but were used to identify students that would be ideal candidates for the in-depth interviews. Some observations may be drawn from this data:

  • Business knowledge is summarized from specific discussion of the simulation game. Every property of business knowledge was discussed by at least one class’s focus groups and/or individual interview. It is obvious that students are articulating their understanding of business when demonstrating their decision-making skills during the simulation game play.

  • Very few, if any, objects were discussed surrounding ERP systems. Less than 10 % of students in any class brought up any properties within ERP. This is not surprising for students in Accounting Information Systems and Programming Business Applications since they didn’t study with the software during the semester. But it is odd that the group using ERP didn’t mention it much in the focus groups or individual interviews.

  • In the area of business process objects, 30–50 % of all groups touched on the property that a business process involves terminology. In other properties in this area, Business Information Systems students and Accounting Information Systems students were more likely to discuss business processes than were the students in Programming Business Applications.

  • There were a number of new objects discovered in the focus groups and individual interviews. Those that are most prevalent were listed above. All groups discussed competition in the business simulation game. At the end of the game, the player is awarded a company car. Depending on the level of net income, the award possibilities are a sports car, a family sedan, or an old three-wheeled car. Students got excited about the reward of a car and discussed it after they played the game. This excitement translated to discussion of the competition during the focus groups and individual interviews.

  • At least 30 % of the students in the Accounting Information Systems class mentioned the environment and how the economy wasn’t part of the game. They also mentioned other external factors that were missing from the business simulation. Perhaps students in that curriculum are more attuned to economic factors affecting financial outcomes. Also note that these students have had an extra year of business education. Students in Business Information Systems spoke of the timing of the game as an issue, either good or bad.

Since the data was deemed too shallow, other means were used to identify candidates for the next round of interviews. The criteria used for selecting the students to participate in these interviews were those students who exhibited the following in the focus group and individual brief interviews: (1) User interface problems; (b) drew on information learned in curriculum; (c) complained about aspects of the game; (d) focused on varied information; (e) expressed interest in helping further research; (f) were voluble; (g) drew parallels between game and real life; (h) were bored by game. These criteria were identified because they suggested that the students thought deeply about the simulation game, or had problems with the game, and could articulate those deep thoughts in an interview format.

3.3 Qualitative data analysis—in-depth interviews

In-depth interviews were then conducted with 19 students. Seven of those students experienced ERP systems within their classes the past semester (MIS-ERP Business Information Systems). Twelve of the students did not learn ERP (MIS Class—Programming Business Applications and ACC-MIS Accounting Information Systems).

3.3.1 Conceptual analysis

All interviews were transcribed and selected statements were coded using the categorical objects and their properties predetermined in the literature review. The objects were tallied up in a frequency count and those objects that met the threshold of being contained in overall at least 50 % of all interviews (the next lower frequency was less than 20 % overall) were considered objects of a necessary structure with ERP Learning only for the class of students who had used ERP in their coursework, as highlighted in Table 4. The “Stopping Rule” (Anderson et al 2006, p. 107) for developing mechanisms from identified categorical objects suggests this cut-off method as acceptable. Those particular objects were identified as being necessary because they must be present for the business process learning to take place that is, causing certain outcomes to happen. Necessary objects are dependent on another (Sayer 1992). For example, business process understanding is dependent on business knowledge and in some cases ERP learning, in this research setting, as defined by Sayer as the “structure of a system of interest” (Sayer 1992, p. 62).

Table 4 Sample contingent aspects statements
Full size table

Because the research cannot only be restricted to what exists in the literature, additional objects were identified and added for new objects, some becoming contingent aspects, meaning that they may produce a different outcome depending on the situation. Known as the contingent causality (Smith 2008), these objects result in mechanisms that interact and produce outcomes varying by context (Smith 2006; Bygstad and Munkvold 2011). These are new objects emanating from the in-depth interviews, not carried over from the focus group and individual initial interviews. Contingent aspects are those that influence a mechanism positively or negatively. They can exist independently (Sayer 1992) and define the boundaries of the mechanism. In the data they were identified as being prevalent in at least 25 % of the interviews. The frequency count for each object is displayed in Table 2.

From that frequency count, the SELECT model is developed. The model, at its simplest, is as follows in Fig. 3.

The model is labelled as SELECT, an acronym for Source of Experiential Learning Educational Causal Theory. The model contends that in order for students to have success in experiential learning, they must begin with business knowledge, reinforced by experiencing ERP systems in order to learn business processes. There are positive and negative pressures exerted on that learning. Some of their classes’ impact classroom learning in a positive way, in that some students were excited to discuss what they had learned and how they applied it to the business simulation game. Market Understanding can aid or detract from university learning and override concepts taught in the classroom as evidenced by some students not analysing the products they chose to sell from marketing or cost basis, but just picking what they liked or what was popular with their friends. In a similar fashion, some students relied on their intuition to play the game successfully, but some had poor results and spoke of “common sense” and “second nature”, showing their neglecting important game information and just playing on gut feel.

As in the focus groups and individual initial interviews, during the in-depth interviews students freely and easily spoke about all properties under the object business knowledge. It is therefore reasonable to assume that all students begin with a basis of business knowledge, as indicated as the starting concept in the necessary structure of the SELECT model. This is evidenced by every single student interviewed discussing business knowledge, as shown in the frequency count of Table 2, in both the ERP class and the non-ERP classes. Students spoke about their first time in playing the game, indicating business knowledge to start with, as indicated by this quote: “The first time I made the mistake by trying to stay in front of everyone as far as the amount of technology in the product, the quality and the price. And the problem with that was that the other firms I guess in the game would start lowering the prices and kind of put me as an outlier, so I would be caught in there and would have to struggle to get back in the middle and then they will shift again and by the time I eventually caught up to them I was too in debt.”

For ERP Learning, only the class that learned about ERP actually discussed frequently the properties within that category. Interestingly, 100 % of all students who experienced ERP systems were able to relay their thoughts on business processes while playing the game, often claiming that their class material was ingrained. This indicates the Kolbian experiential learning theory at work. For those that did not study ERP, their knowledge of business processes came more naturally and they focused more on the financial and accounting processes, which makes sense given their majors and perhaps their summer work experiences.

The necessary structure is supported by these sample statements by the student interviews in Table 3.

3.3.2 Qualitative results—causal analysis

The contingent aspects in the SELECT model positively or negatively affect business process learning. For example, students who had learned with ERP systems cited a number of different classes as being helpful in forming this knowledge: Operations Management, Marketing, Accounting and Finance, Economics, and basic core business classes. That group with ERP discussed specific topics within those classes such as marketing a product, determining the number of factories, and generally feeling armed with “enough foundation so you weren’t guessing in the game.” The students who had not learned with ERP systems focused also on those classes, with more emphasis on the financial and accounting areas, with topics highlighted such as cost/benefit ratio, cash flow analysis, and income statements. This is unsurprising, since many of the students in that cohort were Accounting or Finance majors. Both groups also cited the market understanding or the current market as a source of business decision-making. It may be argued that undergraduate students in the US are great consumers, so it would be only natural for them to think of today’s current market during the business simulation game.

Interestingly, 100 % of students in the class Programming Business Applications, discussed their classes as influencing how they made decisions in the business simulation game. The content of this class includes database design and usage, as in the other two classes. It also includes visual basic programming, which normally attracts a more technical type of student.

An obvious difference between the two groups manifested itself in the contingent aspect of intuition specifically in the area of how students initially played the business simulation game. The non-ERP groups were confident of their ability to play the game the first time. They admitted to using similar strategies for playing the game both times and exuded confidence in their abilities. For example one student, who had a positive net profit for both game plays, claimed “People that didn’t have business background that I had wouldn’t be able to do it as well as I did. They might have been able to do well, but may not be able to do as well as I did throughout the process.” The ERP group was much more tentative the first time they played the game, and they used words such as “overwhelmed” and “multitasking too much” displaying their lack of confidence or understanding. It is argued that at times this tentativeness is a power for learning, since the transformation from being overwhelmed the first time to being successful the subsequent time was dramatic. Every student who learned with ERP systems improved the second time they played the business simulation game. One experiential ERP student who felt they were out of control during the first time they played the game, discussed concepts such as marketing and classes such as Operations Management that helped them during the second game play so it wasn’t simply a game of guessing. For those who did not learn with ERP, half of that group actually did worse the second time they played the game. The confidence exuded by this cohort, or their experience with an extra year of study or Accounting summer work experience, may have overshadowed their absorbing full information to make better decisions the second time they played the game.

The contingent aspects are supported by these sample in-depth interview responses in Table 4.

The next step in the qualitative data analysis is to determine why and how a particular mechanism is being fired (or not). In essence, this step is retroducing the theory, refining it and explaining how the contingent aspects trigger or suppress the causal powers of certain properties. Within each property, the interviews were combed for statements that might identify a causal power or liability that would trigger the mechanism that was the level above. These powers and liabilities are listed in the following Table 5, with a plus sign (+) indicating a power and a minus sign (-) indicating a liability. The two different types of classes were separated into groups of “With ERP” and “Without ERP”, with sample quotes from the in-depth interviews supporting these concepts in Table 5.

Table 5 Powers and liabilities
Full size table

Under the concept of Business Knowledge, all the students who learned with ERP systems discussed choice or product influences; that is, how outside influences affected their choice of product during the business simulation game. All of these students did better the second time playing the business simulation game, so for them market influences enhanced their learning. Only 40 % of the non-ERP students mentioned product influences and most of them did poorly on their second try at the business simulation game. For this group, choice of product leading to the contingent aspect of market understanding was suppression on the necessary structure of the mechanism, thereby inhibiting learning. This is taken as a liability since it affects how the student thinks through business processes and may detract from their using university or work-gained knowledge. At first glance, it might be said that using outside influences such as current events or popularity of products would be considered a power for decision-making. However, the data indicated that in this case, it would be a liability in that students ignored data presented to them in the game and solely were influenced by preconceived information. It is thought that today’s market is a powerful force for the undergraduates and that force may dominate their decision-making.

Both groups definitely used their pedagogical foundation as a basis for their business knowledge of financial, marketing, and manufacturing. For all three classes, more than 70 % of respondents spoke of it, as exemplified by this one student’s comments “I think in the beginning of the semester I hadn’t learnt much in the classes yet, towards the end I had gotten through most of the course so I was taking marketing at the same time so I learnt how to market a product and all the things you need to take into account in doing that; And then I was taking operations management so I kind of knew how things were made and how you have to market your products in the different factories I guess like how many factories you need to do so much production.” Students had a tendency to discuss past courses in mostly general business subjects. It was refreshing to hear that the students had remembered what they learned and applied the principles to playing the business simulation game, demonstrating a positive influence on learning.

Unsurprisingly, the ERP learning categories were discussed almost exclusively by the students in the classes that had experienced ERP systems. For that group efficiency, which was expressed as equalizing supply and demand, was a common theme in their interviews and more than 70 % of the students mentioned it. Statements like [I] “focused a lot on production and that was my main thing because I wanted to meet the demands of my customers” and “To make sure that was meeting the demand…keep your demand and production equal” demonstrate that those students were thinking about how the functional areas of a business fit together, in effect, a power for learning business processes. In addition, more than 50 % of the students in that group spoke around the idea that info view or information flow was vital in decision making. One interviewee remarked “Using the email and like the graphs that were on the system where you could flow between different interfaces and just see how you were doing that month,” demonstrating information view as a trigger for learning. Every one of these students improved their score the second time they played the business simulation game, indicating some change in their ability to make business decisions. However, none of these students spoke directly of the course that used ERP experientially, nor did any even mention Enterprise Resource Planning software.

The focus of this research is to understand how students learn business processes. Under that concept, all properties highlighted had the power to stimulate the SELECT mechanism. Specifically, students focused on connections within the organization in the business simulation game. For example, a typical student said “Accounting helps a lot because I have to know the over-time and the factory. Should I expand or not and things like the income statement that report. That helps a lot and also the per unit cost and things like that also.” Only the majority of the students that learned with ERP systems discussed integration with the usefulness of flowcharts and how functional areas within a company are connected, as evidenced by this quote “Definitely just knowing how marketing flows and the production. You have to have sales. If you do not have sales, production doesn’t really matter and knowing the importance of all different parts you need to be able to meet demand.” These ERP students who had spoken of learning with the system all did better on the business simulation game after their coursework had ended. Again, this result points to a change in their ability to understand and use business processes. Overall, it’s interesting to note that at least 50 % of students in all three different classes discussed business processes. This prevalence of discussion, for both experiential ERP students and those not experiencing ERP, leads to the idea that teaching business processes can be accomplished in different ways. More than half of the non-ERP students study accounting or finance. From the in-depth interviews with these students, it is thought that the accounting curriculum inherently teaches business processes, most likely through various accounting cycles. A graphical summary of the powers and liabilities superimposed on the necessary mechanism is displayed in the following Fig. 4.

Fig. 4
figure 4

Necessary mechanism with powers and liabilities

Full size image

4 Discussion and educational implications

Past research attempting to measure the difference between classes experientially using ERP systems and those not experiencing the systems has been somewhat inconclusive and lacking in rigor. In measuring those differences and probing the basic question—‘What components enable a student to learn business processes’—this research attempts to unlock the relationship between the courses and determine which relations are necessary and which are contingent in a critical realism view for student learning (Sayer 1992; Easton 2010). In doing so, the learning mechanism is uncovered and properties that promote or inhibit that mechanism are realized. The quantitative results from the undergraduate courses as a whole at a large university show no statistical difference in means between understandings of business processes, as measured by playing a business simulation game, between classes that have experienced ERP systems as opposed to classes that have not used ERP. These results are similar to the pattern of past research in that they are inconclusive, which initially led the researchers to dig deeper to understand how students learn. Following critical realism, a comprehensive research strategy with a thorough qualitative analysis gives clues to business process learning.

All students who experienced ERP systems performed better on the business simulation game the second time they played it, after their class was completed. For the students in classes that didn’t study with ERP, their results the second game play were mixed. Half of that group of students did worse the second time they played the game. In addition, GPAs were significantly higher in the one non-ERP class. Consequently, it may be inferred that there is something in the ERP curriculum that allowed these lower GPA students to absorb more from their class to enable them to improve on the business simulation game. This work has sought to answer this question using a critical realist philosophy. By performing a second phase of the research, understanding of the mechanisms and causality behind students’ learning in the two distinct classes was achieved. Under this view of critical realism, it is inadequate to determine any theory from empirical observations solely. The qualitative research portion sheds light on the differences between the two classes and validates the modified Kolbian experiential learning theory of understand, act, and reflect, described as follows:

  1. 1.

    Fundamentally, business process learning begins with a basic understanding of business in general, as shown in the start of the SELECT learning model. Students from all groups clearly demonstrated their understanding of business in the focus groups, individual interviews, and in the in-depth interviews. This is considered the Understand phase of the theory, where problem-solving is learned.

  2. 2.

    Students that were interviewed in-depth frequently mentioned that today’s market influenced their decision making process. This conclusion is impactful for educators of business undergraduate students. If our students are relying on their knowledge of current markets to make decisions, we need to be cognizant of that and work to eradicate any misconceptions and ensure their knowledge of current markets is correct. In this phase, students are acting to execute the goals of the organization, the business simulation game company.

  3. 3.

    Similarly, some students used only their intuition to make business decisions in the simulation game. Acumen is a trait valued in business. However, it should be impressed upon students to incorporate appropriate information before making decisions. Again, this occurs in the Act phase when the students are running the game.

  4. 4.

    Students interviewed overwhelmingly mentioned their coursework as a source for their knowledge of business. In addition, coursework was mentioned again and again from all groups within the interviews when discussing business process usage during the game, validating the educators’ efforts in imparting and instilling not only facts, but knowledge that is able to be applied. These students’ interviews are indicating that learning is taking place on the higher level cognitive scale of Bloom’s taxonomy at the application level, not simply rote memorization. This completes the loop of the modified experiential learning theory. In this the reflect phase, the students are thinking and questioning their decisions in light of the knowledge they’ve gained in coursework.

  5. 5.

    Once students have a basic knowledge of business they can experience ERP systems to solidify their understanding of business processes. For the group of students who did learn with these large computer programs, the concepts of equalizing supply and demand in an organization and the idea that information is essential for decision making were firmly entrenched in their minds when discussing business processes. Students in this group also thought more often about the power of information and how parts of an organization were connected together. From a pedagogical point of view, these ideas articulated in the interviews and used to play the business simulation game show that students are digesting their experiences with ERP systems and applying that knowledge in business decision-making situations. This conclusion validates the experiential learning theory: Experiencing is crucial to learning. Within the confines of the experiment here, students who experienced ERP systems in their curriculum will be able to use that experiential knowledge for future success in the business world. With regard to the modified Kolbian experiential learning theory, students who experienced ERP systems reflected on the connectivity of business and on how information was key to decision making. These reflections indicate an understanding of business processes. The non-ERP group, being mostly finance and accounting majors, reflected greatly on the financial aspects of the game.

  6. 6.

    Unsurprisingly, students in the group without experiential ERP focused on financial statements when discussing the subject of business processes. Since the majority of those students are accounting or finance majors, it is only natural that they would focus on the financial aspect. This raises an interesting implication for future educators. Another route to learning business processes may be through the traditional accounting classes. From the quantitative data, the large classes of students statistically did just as well as the ERP students in mastering the business simulation game. However, out of this group of accounting and finance majors, more than half of those that participated in the in-depth interviews did poorly on the second time they played the business simulation game. In contrast, in their qualitative interviews, they were able to discuss business processes and how it impacted their decision making in the game as were the ERP students. Timing of classwork may be an issue here. Most of this group of students brought up ideas from past year’s accounting classes; only 2 mentioned finance classes they had just finished. Students who took those classes in prior semesters are already beginning with a basic knowledge of business processes to play the game the first time, and the Accounting students have had an additional year of university. Another explanation is that this group of students, most of which exhibit a high level of confidence, used that confidence to play the game well the first time. It is thought that these students came into playing the game the first time with business process knowledge, perhaps from classes or summer Accounting work. Their second game play results were simply randomly high or low. This is validated by the quantitative data. For the ERP students who were interviewed, all first game play results were negative, bar two. For the Accounting students, all first game play results were positive, bar one, confirming their entry into the game playing with preacquired business process knowledge.

  7. 7.

    Yet another surprising result is that this same group of students appear to be very confident about their business process knowledge and relied on their business sense or even common sense to articulate their thoughts on business processes. As mentioned in the quantitative data analysis, the GPAs of the non-ERP group from the Accounting classes were higher than those of the ERP group. Perhaps the higher GPA students are more confident of their abilities and feel that their business sense and common sense is sufficient for decision making, acquired from an extra year of university or from a summer Accounting internship. At the university where the research took place, the accounting program is the most rigorous within the college and it attracts the higher achieving students. With this rigor and achievement, confidence exudes. On the flip side, the group of students who learned with ERP was quite tentative and overwhelmed when they played the business simulation game the first time with all improving the second game play, as they gained confidence in business process decision-making. Again, this is most likely due to lower average GPA translating into a lack of self-confidence, but a greater capacity for change.

This research resulted in the following educational Implications:

Firstly, students need to have a basic understanding of business before attempting to learn business processes from experiencing ERP systems. This point was validated by the qualitative data analysis of the in-depth interviews after students played the business simulation game; basic business knowledge enables students to learn from ERP systems. This point makes sense: how can a student understand the complexities of business processes in ERP without understanding the basics of a business? The outcome of this conclusion for educators is to make sure that students understand those business basics before moving to complex software with business processes. The other part of that equation is that educators need to be clear in what those basics are.

Through a rigorous analysis of the qualitative data, this research study shows that one route to learning business processes is through experiential ERP. By affirming ERP learning, the modified Kolbian experiential learning theory is also validated. The emphasis by organizations on business processes is compelling. Students need to understand and make decisions based on business processes. ERP systems have been used for over 10 years as one way to teach this important topic. Universities should continue to do so, confident in the result that learning with a live ERP system indeed facilitates business process learning. In addition, it was evident from the in-depth interviews that overwhelming majority of students relied on their classes to help them make business decisions. This is a gratifying result and one that should be emphasized to flaunt educational success.

There are forces that can work for or against students’ using their knowledge gained in classes to make business decisions. For the undergraduates, some students have a tendency to rely on their intuition to make business decisions. While it is commendable to have a “business sense”, it is highly unlikely that inexperienced undergraduates can make business decisions based on their innate common sense alone. In addition, some students appear to use what they know about today’s popular market to make business decisions, i.e. the environmental impact. These students use their information gleaned from products and popular culture at times to the exclusion of other information provided. These two sources of information have a significant educational implication: any curriculum taught should encourage use of full information and not simply falling back on intuition or market knowledge. Educators should also ensure that any intuition or market knowledge used is accurate and not misunderstood.