Keywords

1 Introduction

This study is an ongoing multi-university project into student learning using business simulations. As of this writing, 15 faculty at 13 US universities ask their students to participate in a post-simulation survey to evaluate their learning experience using various strategic management and project management simulation games. The data collection is ongoing, and it is providing extensive information about the learning that occurs while students participate in business simulations.

Related to this research, at the ABSEL 2018 Conference, Teach [1] discussed the difficulty of measuring learning; Chasteen, Teach and Szot [12] discussed student impressions of difficulty vs realism; Teach and Szot [3] described the survey and presented preliminary findings; and Nugent [4] explored the impact of reflective observation questions after each round of a simulation. The effect of debriefing was further explored using this data at ISAGA 2018, Teach and Szot, [5].

This paper discusses relationships found between self-assessed participant learning skills and self-reported opinions about a set of often consider incidental issues when planning to use a business simulation as an experiential learning activity in a classroom.

2 The Survey

Measuring the learning claimed by business simulation participants and their perceptions of the experience is an on-going element of this research methodology.

The survey uses an 8-point Likert-like scale to measure self-reported learning from “I did not improve this skill in this area at all” to “my skill in this area was greatly increased” and a 6-point Likert-like scale from “disagree strongly” to “agree strongly” to measure student beliefs and opinions about the simulation. Teach and Szot, [3]. Table 1 lists the 16 learning skills evaluated and Table 2 lists the questions involving demographics, behaviors, beliefs, and opinions about “the little things”.

Table 1. The learning skills set. The ability to…
Full size table

The data for this research were obtained from an extensive web-based survey that included a feature that allowed the student to leave the survey and return later to continue. We measured the time-on-task to complete this survey comparing the time and date when each student started and completed the survey. Most completed it in one sitting. The median time taken to complete the survey was 9.9 min and 80% completed the task within 15 min. Nevertheless, because of the “stop and continue later” feature, a few students took more than 24 h.

In addition to the 15 behavior, belief, and opinion questions (variables A through O in Table 2). These 15 behavior, belief, and opinion questions were mostly derived from the authors experiences in using business simulation in university classrooms since 1962, and there are very few research papers published that detail the effects of these behaviors, beliefs or opinions. The survey also collected demographic and ranking data and used open-ended questions to capture the name of the simulation used in the student’s class and the country where they graduated from high school. Questions P, Q and R. represent demographic characteristics and the final position of the participating students’ teams. There are references listed for the items in which the authors found relevant published works.

Table 2. The questions involving demographics, behaviors, beliefs, and opinions; the “little things”.
Full size table

For this study, variables A through O were rescaled from the 6-point Likert-like scale to a 2-point (agree-disagree) scale due to insufficient data to support analysis using the 6-point scale.

3 Methodology

The survey was developed using Qualtrics survey generator and hosted on a Qualtrics web server. Supporting faculty provided the anonymous participation link to their students following completion of the simulation activity in their course and encouraged students to participate in the survey. Data was exported from Qualtrics and imported into SPSS for the analysis.

The survey was approved by the Georgia Tech IRB, which required each student to be aware that he or she had the option of not participating in the survey or stopping the survey process whenever the participant decided to quit. Those who decided not to participate at the start did not generate a data record. However, once a person began the survey, they generated a data record. The first question in the survey asked if the respondent wished to participate or not, and if the person selected “No”, the responded exited the survey but left a mostly blank data record. The survey was also approved by the IRB at The University of Texas at Dallas.

The second question asked if the respondent was a student or a faculty member previewing the survey. Sometimes an instructor completed the survey, but this second question allowed us to excluded instructor data records from analysis.

After reviewing the data set, we decided to exclude all respondents’ data if they spent less than 4 min answering the questions. The responses from this set of students were mostly a jumble of generally incoherent values. In addition, we scanned the data looking for patterned responses such as 25252525 or 666666 and eliminated these data records as well. The number of surveys examined and excluded for various reasons is shown in Table 3.

Table 3. The number of surveys processed.
Full size table

Analysis of variance of the means was used to identify significant correlations between the learning skill responses and the little things. Using p < 0.10 as the test for significance, we show the p-values for the statistically significant correlations in Tables 4 through 6. Normally, p < 0.05 is the sacred value for showing significance; however, we believe relationships with p < 0.10 are meaningful for this analysis. Why? Recall that a p-value is the probability that the null hypothesis may be correct. This is referred to as “significance”. Although significant or not significant is a binary outcome, significance cannot be considered as true or false. We believe the selection of a p-value should be the function of a loss table and it should never be a hard and fast value. Where the cost is high, such as the cost of a person’s life, most would want a very low p-value related to failure of a surgical procedure. On the other hand, if the problem regarded how satisfying the taste of ice cream is, most would likely agree a much higher p-value is reasonable because the cost of failure is lower.

4 Results

The three tables labeled 4A through 4C, use the rows for learning skills and the columns for the behaviors, beliefs and opinions (little things) and the row intersections contain the significance of the interaction between each learning skill and each little thing question for “p” < 0.10. Blank cells indicate “p” >  = 0.10.

Table 4. ANOVA significance of learning skills vs. Little things showing “p” < 0.10
Full size table
Table 5. ANOVA significance of learning skills vs. Little things with “p” < 0.10
Full size table
Table 6. ANOVA significance of learning skills vs. Little things showing “p” < 0.10
Full size table

5 Discussion

Tables 4, 5 and 6 show many of the little things have significant covariation with the learning skills while others are far less important. In discussing these findings, the authors will hypothesize some possible rationale for the strong interactions.

5.1 Sharing Work Evenly

Little thing question A, “We shared the work as evenly as possible” is highly related to three learning skills: the ability to set goals, the ability to work well in teams, and the ability to learn how to work under uncertainty. Nevertheless, “sharing the work” was not closely related with the other aspects of learning that we measured. It suggests that these three skills are difficult without team interaction and the others can be developed individually.

5.2 Selected Our Leader

Little thing question B, “We selected our own leader” strongly co-varied with the ability to forecast outcomes, the ability to analyze results, the ability to analyze quality control measurements, and the consideration of possible competitors' decisions when making our firm's decisions. This covariance may be because these skills require high levels of trust among the team members and trust must be earned. When the team selects its own leader, that act demonstrates a high level of trust. Teams that are not allowed to select, or are unable to select, their own leader do not have a built-in expression of trust to the leader.

5.3 Few Team Members Did All the Work

Little thing question C, “only a few team members did all the work”, closely matches Question A, “we shared the work”. Although these questions purport to measure the same thing from the opposite perspective, only the abilities to work well in teams and work under uncertainty correlated strongly, the ability to set goals did not.

5.4 Most Effort Learned the Most and Becoming a Better Employee

Little thing question D, “those who put in the most effort learned the most” is an acceptance that hard work pays off. This belief highly co-varies for most skills with little thing question E, “the simulation's team experience will result in being a better employee”. This “better job” belief co-varies with the ability to understand the interactions among two or more decision variables and the ability to assess risk”. This suggests students believe these two learned skills will be important when they enter the work force. And the simulation supports these two skills.

5.5 Team Friendship

The last two little things shown in Table 4, “our team quickly became good friends” and “at the end, we were not friends”, are almost opposite sides of the same issue; however, both may be true. These answers strongly co-vary with most of the learning shills. This demonstrates the importance of creating a teaming environment in a simulation that goes well beyond the simulation itself. It may be impossible to control, but some of the teachers or game administrators may be able to influence this by incorporating team-building activities into the curriculum before the start of the simulation.

5.6 Difficulty Coordinating Efforts

The little thing question, “We had difficulty coordinating our efforts” co-varied with learning skills that require group learning and less on skills that can be learned by working alone. It is most highly related to the skill of “Work well in teams”, “: understanding decision variable interactions”. In all likely-hood, this item is the frequent topic of team interactions during team meetings. Other issues like “Analyzing reports and financial results”, Considering possible competitor’s decisions”, and “Forecasting” are often the major discussion in team meetings.

5.7 Prepared at the Start

Many skills that can typically be learned alone or prior to beginning a simulation, either in prior coursework or by strong preparation. The most important skill (not included in the list) is “Learning how to learn”. This “Learning how to learn” is essential when a student undertakes participation in a simulation. Preparation was essential with several of the little things question, “I felt well prepared when I started the simulation”, highly relates to one-fourth of the learned skills. These four learned skills all have “p” values less than 0.0005. These questions were “Analyzing Quality Control measures”, “Anticipating competitive reactions”, “Assessing risks”, and “Considering possible competitors decision” Three additional questions had very low “p” values. The ability to “Differentiate between important and unimportant information” had a “p” value of 0.005, “The ability to set goals” had a “p” value of 0.006 and “The ability to make competitive decisions” had a “p” value of 0.019.

5.8 Realism

The little thing questions, “The simulation was unrealistic” and “The simulation represented the ‘real world’” are essentially opposites of one another. This is shown by the degree of commonality of the responses to the entire set of learning skills co-variations shown by the responses to most of the learning skills questions. The unrealistic question had only two out of the sixteen learning skills had “p” values of 0.10 or higher in its relationship to the realism question.

5.9 Duration and Effort

“The simulation took entirely too much time” and “the simulation took entirely too much effort” were designed to measure different aspects; however, the response patterns to the duration question were similar to the effort question. Three learning skill questions had “p” vales equal to or greater than 0.1000. They were; “The ability to work under uncertainty”, “The ability to create budgets”, and “Assessing risk”.

“The effort was well worth it” strongly co-varied with fifteen of the sixteen learning skills. “Considering possible competitors’ decisions” had the high “p” value which was only 0.083. These findings support to old adage that “Hard work pays o0ff in the long run.

5.10 Simulation Represented the Real World

The belief that the simulation represented the working environment was very important. The differences between those who considered that the simulation represented the working environment was extremely important when it came to the learning skills of; “Being innovative”, “Creating budgets”, “Analyzing quality control measurements”, “Working well in teams”, and “Setting goals”. It tended to be unimportant to; “Making competitive decisions”, “Working under uncertainty”, “Forecasting”, “Analyzing reports”, “Anticipating competitive reaction,: “Assessing risks”, nor “Considering possible competitors’ decision”.

5.11 Team Spent More Time on the Simulation Than any Other Course

The responses for “I spent more time on the simulation than I did on any other courses” significantly co-varied with a little more than half of the learning shills. Those who agreed this this statement reported they gained greater skills at “Marginal analysis”, “Making competitive decisions”, “The ability to set goals”, and “Differentiating between important and unimportant information”, and “Making more accurate forecasts”. These five skills had “p” vales less than 0.05. The four shills of “Analyzing reports and financial results”, Understanding decision variable interactions”, Anticipating competitive reaction”, and the ability to “Work well in teams” all had “p” vales between 0.054 and 0.087.

5.12 Gender

Gender differences indicated few significant co-variation with skills acquisition. There were four exceptions. “The most significant differences were the skills of “Considering possible competitors’ decisions”, with a “p” value of 0.004, “Analyzing reports”, with a “p” of 0.015, “Differentiating important information from unimportant information”, with a “p” value of 0.020, and lastly “The ability to analyze reports and financial results”. Which had a “p” value of 0.034. All other learning skills had “p” vales that were equal to or greater than 0.100 0.

5.13 Team Rank at the Completion of the Simulation

Overall team performance on the simulation as measured by its ranking at the end of the completion co-varied with “p” values under 0.10 with 11 of the 16 learning skills. Mastering these, 11 skills determine the competitiveness or success of the teams. The authors have no explanation for why team rankings were not more related to the final team ranking. They may indeed be related, but if so, the relationships have “p” values of 0.100 or greater.

5.14 The Differences Between International Students and US Students

International students were defined as students who had been graduated from a high school located outside of the United States. The country at which a student was graduated from high school had almost no significant differences with the acquisition of skills. There were two exceptions; “Analyzing quality control measures” had a “p” value of 0.028 and “Being creative” differed between international students and US students with a “p” value of 0.056. The response to the “being creative” question may be a culture difference between the US students and international students, but why the difference in “analyzing quality control measures” was significant remains a mystery to the authors.

6 Conclusions

Learning is a multifaceted process. Many of the things associated with the amount of learning that takes place when individuals are participating in business simulations are, at least partially, under the control of the simulation administrator or teacher. He or she has the capability of enhancing the degree of learning that takes place when students participate in business simulations.

The male-female mix and the number of international students are not under the teachers’ control; however, our study indicates those issues are not particularly important to the set of learning skills by participating in a business simulation.

As these are preliminary results of our study, we make no overall conclusions. However, we can claim that many more interactions exist among the different groups of students when they experience a business simulation than we previously thought. We plan to examine some of these other interactions in the future and encourage other faculty to ask their students to participate in the survey.

Table 7. Contributing faculty
Full size table