Keywords

Managers are pivotal in their role as the link between individual contributors and higher-level leaders within an organization. Overseeing the majority of the workforce, they are responsible for ensuring effective operations, high performance, and a positive work climate. Finding the talent to fill these key roles cannot be left to chance. It is not enough to simply be strong technically; a manager needs to be able to inspire others and accomplish work through them.

Compared to the role of individual contributor, the role of manager presents unique challenges. There is acceleration in demands for prioritizing tasks and making quick decisions. Managers cannot lose sight of the organizational goals and strategies as they focus on day-to-day operations. High-quality work that got them noticed in the first place must be maintained while demonstrating nimbleness in dealing with new issues. They need to be people-focused: setting aside time for developing their employees, monitoring their performance, and managing disputes as they arise. To be truly effective in leading others, they also need to be an advocate for their team while interfacing with the rest of the organization. Based on numerous job analyses conducted by Corporate Executive Board Company (CEB), the following is a list of competencies and experiences that underlie successful performance in managerial roles:

  • Coaching and Development: effectively engaging with direct reports

  • Prioritization: identifying and completing critical tasks in the most efficient manner

  • Decision Making: identifying issues and drawing conclusions based on these issues

  • Monitoring: assessing the effectiveness of a team’s performance

  • Personality Characteristics, such as Achievement, Influence, Independence, Reliability, and Confidence and Optimism: setting and accomplishing challenging goals, taking actions and making decisions without direct support from others, directing others in situations that require leadership and leading others toward a group objective, taking responsibility for own actions, and having belief in one’s ability to get the job done

  • Problem Solving: efficiently and effectively using numbers and analytical reasoning to solve problems

Given the breadth of behaviors and experiences listed earlier, well-designed selection tools are necessary to identify the best person for a managerial position. However, finding the tools that can measure the more complex aspects of the position can be a significant challenge for an organization. Traditional selection tools such as multiple-choice cognitive tests, biodata scales, and personality assessments can be administered to measure problem solving, professional potential, and personality characteristics. However, these types of assessments are not ideal when attempting to measure complex competencies such as coaching and development, prioritization, or monitoring. For example, the extent to which a candidate can effectively prioritize information from numerous sources, make day-to-day strategic decisions, coach and develop employees, and monitor employees’ performance are difficult work behaviors to assess utilizing simple multiple-choice or Likert-type response formats. As such, even in combination, traditional selection tools will not thoroughly cover the domain of key competencies needed for success in managerial positions.

These difficult-to-measure competencies have traditionally been assessed in the context of an assessment center (i.e., sets of instruments and techniques used as part of managerial selection to judge the likelihood of a candidate’s success as a manager; Cascio 1998). Assessment centers require candidates to role play or perform in-basket exercises to simulate interpersonal or problem-solving tasks that are frequently performed by managers (Bray and Howard 1983). Although utilizing a managerial-focused assessment center may be the optimal choice when attempting to understand a candidate’s potential for success in a role that requires tasks such as monitoring employee performance and developing one’s employees, organizations do not often find them to be cost effective. Traditional assessment centers rely on human raters to judge the extent to which a candidate’s record of behaviors is related to the characteristics required for the job. Many organizations have a high number of manager positions spread across a variety of locations; sending candidates through an assessment center would require at least one assessor to be available at each location in order to conduct the assessment, a requirement that is quite costly.

An alternative approach, both more cost effective and time efficient, is to supplement traditional selection tools with state-of-the-art simulations designed to assess the more complex core competencies required for these important positions. Simulations offer assessments that appear more job-relevant to candidates than traditional selection tools, such as text-based assessments . The increased face validity is rooted in the fact that simulation item formats allow the information to be presented in a manner more similar to how the information would be experienced in daily life, which is a more authentic presentation of information to the candidate. For example, rather than reading a situational judgment item that describes an employee who is upset about a co-worker stealing his sales opportunity, a simulation item format will use a video to convey not only the message but also visual cues such as body language, and verbal cues such as voice inflection. As the candidate can better envision him or herself within the situation, improved measurement of these competencies may be realized. In other words, simulation content may provide a more direct presentation of information to candidates, leading to more accurate measurement and therefore stronger reliability and validity of scores (Zenisky and Sireci 2002).

Simulations may also hold the potential for better measurement of these constructs due to the positive reactions that are likely to be elicited from candidates. Measurement practitioners have shown that negative reactions to a test, such as viewing the content as irrelevant, can lead to poor motivation to do well (Macan et al. 1994). When candidates have a decreased motivation to exert effort, their test scores will not accurately reflect their ability on that construct. Since simulations have been shown to elicit positive reactions by candidates (Richman-Hirsch et al. 2000; Shotland et al. 1998), administering a simulation may result in increased motivation of candidates and lead to scores that more accurately reflect candidates’ true ability on the constructs of interest.

Finally, although assessment centers are job relevant and can offer much in the way of both content and criterion-related validity evidence (Winfred et al. 2006; Hermelin et al. 2007), these methods tend to be time-consuming and costly. They are often ineffective when the organization is global and manager positions are spread across a number of locations. Technological advances have allowed employers to bring much of this assessment content into an online format in which technology is leveraged to assess candidates via interactive, media-rich simulations that simulate a “day-in-the-life” of a manager while measuring job-relevant traits and abilities. The use of simulated assessment content contains immense potential for the automation and objective measurement of the core competencies needed to succeed in managerial roles that are difficult to assess otherwise. Utilizing simulations allows for the assessment of more candidates less expensively, with fewer resources, and does not require that candidates travel to a particular location to complete the assessment.

Although great resource-related benefits can be realized using simulated assessments, these assessments must also be held to the same psychometric standards as traditional selection tools—they must be reliable and valid predictors of performance. In order to develop high-quality simulation content for managerial roles that is likely to be valid and reliable, an understanding of the work behaviors and competencies that are necessary to perform the job well must first be achieved. The purpose of the current chapter is to provide a detailed framework of the development work that was conducted to successfully design and implement two managerial simulations, (1) Inbox Simulation, and (2) Coaching Simulation, that have been shown to be predictive of success in managerial roles.

1 An Overview of the Managerial Simulations

1.1 The Inbox Simulation

The Inbox Simulation consists of two sections. The first section measures prioritization, while the second measures decision making and monitoring. The interface of this simulation is designed to closely mirror the day-to-day managerial experience. In the assessment , candidates assume the role of a leader and are provided with background information and tools that are needed to complete the assessment , including a job description and organizational chart. Candidates are asked to prioritize demands from across the organization and their team, identify critical tasks, and ensure that those tasks are completed. Information is presented to candidates in the form of email, voicemail, phone calls, visitors, and calendar reminders. Emails are presented with text, the phone calls and voicemails through audio files, and the visitors to the office appear on screen in video format. Situations encountered in the Inbox Simulation include determining what resources employees need to do their jobs, responding to urgent questions from higher-level management, doing “just in time” problem solving, and working with managers of other departments or organizations. All questions are multiple-choice in nature. Examples of this interface can be seen in Figs. 10.1 and 10.2.

Fig. 10.1
figure 1

Inbox simulation interface—displaying email and voicemail stimuli

Full size image
Fig. 10.2
figure 2

Inbox simulation interface—displaying visitor to manager’s office

Full size image

1.2 The Coaching Simulation

The Coaching Simulation consists of various scenarios designed to measure coaching effectiveness. These scenarios contain fictitious subordinates, and candidates are measured on how they approach these conversations. Candidates are provided with opportunities to show how they would manage employees, display supportive leadership behaviors such as listening, probing, encouraging and empowering, and be directive and assertive when necessary. Scores on the Coaching Simulation are designed to predict the candidate’s likelihood of engaging in such behaviors on the job. The simulation design is similar to a situational judgment test. Candidates are presented with video-based scenarios along with alternative courses of action. They are then asked to select which response options are most and least effective given the situation. An example of this interface can be seen in Fig. 10.3.

Fig. 10.3
figure 3

Coaching simulation interface

Full size image

The following section is devoted to providing an in-depth review of the development efforts involved in designing and implementing the simulations described earlier in the chapter.

2 Simulation Development and Implementation

2.1 Job Analysis

Development of the Inbox Simulation and the Coaching Simulation began with a series of job analyses involving managerial roles across multiple organizations within a variety of industries, including Telecommunications, Retail, Financial Services, Insurance, Healthcare, and Medical Services. As the simulations were designed to be applied universally to most industries, it was important to have a combination of industries represented in this phase of development. The goal of the job analysis was to gather information about the present and future job requirements of manager positions in order to develop a set of highly predictive and face valid simulation content. The approach to the job analyses involved four activities not only to comprehensively study the managerial roles, but also to focus specifically on the development of the simulations: (1) job observations of incumbents, (2) focus groups, (3) interviews with incumbents, their managers, and executive leadership , and (4) job analyses questionnaires. The multiple methods of gathering input were utilized to obtain perspective and feedback from a variety of levels. These four activities are important parts of any job analysis and test development process, but even more so for simulation development. As one of the main goals of utilizing simulations is a more realistic presentation of information, test developers had to truly understand the context in which these managers were working. The test development team included five Industrial–Organizational Psychologists who held advanced degrees, all with extensive experience in the area of selection and assessment . Through job observations, onsite focus groups, and interviews, test developers were able to obtain important situational, environmental, and background information that helped shape the basis for the simulations.

2.1.1 Job Observations

The goals of the job observations were to (1) directly observe the actions performed by the managers , and (2) gather critical incidents about the types of issues presented to these leaders on a day-to-day basis. The critical incidents gathered in this portion of the job analysis process became the topic matter for the managerial simulation content.

The job observation process included a walk-through of a typical day on the job, followed by the job analysts observing the interactions between the managers and his/her team members as the manager performed basic job activities. Additionally, at the end of the observation period, the analyst asked the manager a set of questions regarding the tasks that had been observed. Following each observation, job analysts rated the importance of a variety of competencies in terms of their job relevance to the work behaviors performed by the manager. In addition, to aid in the development of the Inbox Simulation, job observations were used to determine the breakdown of time spent on tasks/activities performed by the incumbent. The job observations were typically completed at the same time as the interview, so additional questions were asked to gather critical incidents for the Inbox Simulation. Critical incidents focused on the following topics:

  • Typical distractions that occur during the course of the day that require switching attention from one task to another

  • Methods used to prioritize work when multiple things/people are competing for attention

  • Methods and frequency of monitoring work being completed by subordinates

  • Tasks that are typically delegated to direct reports

Furthermore, certain topics were the focus for the development of the Coaching Simulation. Questions for the Coaching Simulation targeted the following topics:

  • Frequency of coaching poor-performing employees

  • Common performance issues that require coaching, such as compensation, customer complaints, and interpersonal disagreements

  • Most difficult problems to coach

  • Important things to do when coaching an employee and common mistakes an ineffective or new manager might make

Finally, for both managerial simulations, participants were asked to rate incidents that were important, common, difficult, and challenging to handle. Then, they were asked to provide examples of how one might effectively or ineffectively respond to such incidents.

2.1.2 Interviews

The goal of the interview portion of the job analysis was to obtain information about the overall mission and operating procedures within the participating organizations. In addition to the interviews conducted with the managers , interviews were conducted with individuals in director, trainer, and HR manager roles. Interview protocols were structured to elicit the following information:

  • Overall mission of the organization and the role that front line managers and supervisors play in executing on the mission of the organization

  • Description of the performance ratings and metrics collected

  • Current selection process

  • Knowledge, skills, abilities, and other characteristics needed to be a successful manager

  • Anticipated changes to the business over the next 5 years

  • Characteristics that separate top managers from those who are just ’average’ performers

Some of this interview protocol was not directly relevant to the design of the simulation content per se (i.e., mission of the organization, anticipated change to the business). However, it was important for the test developers to gather such information in order to ensure that the simulations were supported by other job and organizational-relevant content that would be later incorporated in the final assessment solution.

2.1.3 Focus Groups

Focus groups were conducted with incumbents in manager roles and consisted of two phases: (1) continue to gather critical incident information, and (2) obtain feedback on potential test questions for the new managerial simulations. The goal, then, of conducting focus groups was to gather additional material for scenarios that could be used within the two new simulations. Specifically, critical incidents were gathered relating to work behaviors that were known to be difficult to assess with traditional selection tools, such as coaching and developing employees, monitoring, decision making, and prioritizing on the job. Additionally, the development team had created potential test questions prior to the start of the focus group process, based on the information gathered in the job observations and interviews. Focus group participants were asked to evaluate these potential test questions in terms of job relevance and appropriateness/plausibility of response options, and were also asked to generate ideas for alternative response options.

2.1.4 Job Analysis Questionnaires

Upon the completion of the job observations, focus groups, and interviews, the resulting information was compiled and reviewed. Analysts developed a comprehensive list of work behaviors that represented what typical managers do in the course of a work day. From this list, an extensive conceptual model was created that contained the competencies and work behaviors that led to successful performance of the requirements of the roles and was the basis for the job analysis questionnaire. As the model was designed to be encompassing all competencies required in these roles, this questionnaire addressed work behaviors outside of the behaviors that the two managerial simulations were targeted to measure.

The questionnaire consisted of job task and activity statements relevant to entry-level leadership roles, such as: “Strive to achieve departmental objectives despite challenges.” These statements were organized into 26 work behavior dimensions, nine of which were to be mapped to content in the two new simulations: Handling Conflict, Building Relationships, Developing Employees, Analyzing Problems, Identifying and Considering Alternatives, Building Teams, Planning and Monitoring Progress, Prioritizing Work Demands, and Monitoring Against Goals. Job task and activity statements were placed on a five-point scale that ranged from 1 (Unimportant) to 5 (Critically Important). High-performing job incumbents from across participating organizations completed the questionnaire. Results indicated that the conceptual model consisting of 26 work behavior dimensions aligned with the role of manager . This confirmed that the extensive four-prong approach to the job analysis of the manager’s position led to the identification and confirmation of the work behavior dimensions required for success in these positions.

The goal throughout the development of both simulations was to be able to assess a portion of competencies traditionally measured in an assessment center context using internet-based simulations. The phases of the job analysis described earlier were imperative to gain insight into the true “day-to-day” context in which managers functioned and served to aid the development of realistic and appropriate item content for the simulations. The critical incidents collected during the job analysis process directly served as the stimuli for the assessment . The process of transforming the critical incidents gathered during the job analysis into functioning simulation content is described in the next section.

2.2 Content Development

The development of the two simulations is described next.

2.2.1 Inbox Simulation

2.2.1.1 Stimulus Material Development

The Inbox Simulation was divided into two sections. The first section of the simulation was designed to focus on candidates’ ability to prioritize information, and is related to the Prioritizing competency identified through the job analysis . The emails, phone calls, voicemails, and visitor stimuli were written to vary in terms of their urgency of response. Focus groups conducted during the job analysis helped to refine and revise the stimuli. The second section of the Inbox Simulation was designed to consist of “mini in-basket exercises.” These exercises were designed to measure candidates’ judgment with respect to monitoring employees’ performance and decision making in the context of receiving information from multiple sources. The items are related to the Monitoring and Decision Making competencies identified through the job analysis. Related sets of content targeting specific themes were created, such as planning/administration, monitoring employee output, and responding to customer issues.

2.2.1.2 Item and Scoring Development

The format for all items in the Inbox Simulation is similar to situational judgment items. Candidates are presented with information through the various stimulus materials discussed earlier, and are asked to make judgments about the information. A range of response options were written for an item that would discriminate those with high levels of the competency from those with low levels of the competency.

Subject matter experts (SMEs) provided feedback on these response options by rating the relevance of the items to the dimension it was intended to measure, providing input on the quality/plausibility of the response options, and indicating the extent to which each response option would relate to successful outcomes. The SME rating/feedback process contributed to the creation of a priori scoring that was empirically tested during the criterion-related validation phase of development.

2.2.2 Coaching Simulation

2.2.2.1 Stimulus Material Development

Similar to the Inbox Simulation, content development of the Coaching Simulation began with the job analysis . The development goals included utilizing the information obtained through job analysis to design a highly predictive and face valid simulation by understanding the extent to which coaching was part of the job, and the nature and types of coaching that were performed.

Critical incidents gathered during the focus groups included an employee confiding in their manager about personal issues at home impacting their performance, providing critical feedback to an employee about poor job performance, having an employee make excuses and push back when critical feedback is given, resolving conflict between two employees, and having a junior-level employee complete his portion of the project and go home without telling the project lead. These critical incidents were used for developing the scenarios in the simulations. Each scenario designed for the simulation consisted of a hybrid of the incidents that were documented during job analysis . Site visits occurring later in the job analysis process were used to refine the scenarios and to develop new response options.

2.2.2.2 Item and Scoring Development

Four professional coaches were asked to rate the response options on their effectiveness. The coach read the background to each situation, viewed the video , and then rated the effectiveness of the responses. During the validation, data were collected that were later used to empirically investigate the usefulness of the score key provided by the effectiveness ratings of SME raters during development. The empirical scoring method was determined to be superior at predicting successful outcomes than the SME effectiveness ratings, and was thus implemented in the final version of the assessment.

2.3 Final Validation Versions of the Assessments

2.3.1 Creation of Scripts and Test Stimuli

For both managerial simulations, the stimulus materials, test questions, scoring, and dimensions were organized and developed into test scripts. These scripts served as the basis for the technical development of the simulations as well as the scripts for professional acting and voice talent. The test questions were created by engaging with a professional audio/video production company to record the audio and video portions of the assessments . Following the audio/video production, files and test scripts were handed off to a programmer for creation of the assessments.

Once programming was complete, the final beta versions of the managerial simulations existed on an online testing platform and were ready to be included in the validation work conducted by partnering organizations. The online testing platform allowed the managerial simulations to be deployed anywhere with an Internet connection, at any time of day.

2.3.2 Concurrent Validation

The Inbox Simulation and the Coaching Simulation were validated within a criterion-related study design with data collected by a consortium of six organizations. Incumbents working in the role of manager within the consortium organizations were asked to complete a set of assessment content that included the new simulation content along with problem-solving (cognitive ability) measures, a writing ability test, personality scales, and biodata scales. For each incumbent, the direct manager was asked to complete a job performance rating form. The form included a variety of items designed to obtain as much information about the incumbent’s job performance as possible. For example, the form included 27 performance dimension ratings—items that aligned with the job analysis work behavior dimensions such as Decision Making and Managing Talent. Seven items referring to the incumbent’s global/overall job performance were also included (e.g., incumbent’s long term career potential, the manager’s desire to rehire this incumbent, general effectiveness as a manager). From these items, several performance composites were formed to serve as the primary criteria for the validation. The data provided by the consortium were used to examine the statistical relationships between scores on the new simulation content and ratings of job performance.

Prior to investigating the test-level validities of the new simulation content, item-level analyses were performed for both the Inbox Simulation and the Coaching Simulation. Final forms of the assessments were created based on extensive review of the items and how they functioned both at the item- and the test-level. Based on the incumbent data provided during the study, scores were tabulated for each incumbent on these final forms of the assessment. These scores were then used for the examination of the validity of the simulation content.

Given the multiple samples of data provided by the consortium, the validation results were examined in the context of meta-analysis. Meta-analysis in this case allowed for the synthesis of information from multiple studies that used the same content in a variety of settings to judge the overall value of the new content. The results of the meta-analysis indicated that scores on both simulations were substantially related to various types of supervisor ratings of job performance, and provided support for the use of the new managerial simulations to assess some of the core competencies required to perform well in front line manager and supervisory roles.

2.4 Implementation

Following the development and validation of the media-rich assessments , a number of clients have implemented these assessments into their organizations’ hiring systems. A case study for a large retailer that has implemented the two managerial simulations into their selection process for Store Manager and Assistant Store Manager roles is discussed further.

2.4.1 Case Study: Retail Store Managers and Assistant Store Managers

2.4.1.1 Understanding the Role

For this particular organization, meeting customer needs and expectations was the most important goal of the business. It was up to the Store Managers and Assistant Store Managers to model this organizational core value in their everyday actions, leading their team members to do the same. Individuals in this role were expected to be ambassadors for the brand. These roles were critical to the organizations’ success as Store Managers were not simply team leaders or sales leaders, but were business leaders who were held responsible for the revenue and profitability of their branch.

The work behaviors expected in these positions actively encompassed many of the competencies outlined earlier in this chapter. Of specific interest was the Store Managers’ ability to coach and develop the branch sales team. Through an internal survey, coaching and development was identified as a weak spot within the organization. Individuals in these roles needed to be adept at monitoring and assessing the effectiveness of a team’s performance. Additionally, they needed to be able to prioritize competing tasks, identify issues, and draw conclusions in order to address such issues.

2.4.1.2 Understanding the Organizational Goals

Prior to implementation of the managerial simulation content, the hiring process involved a resume review, an interview, background check, and finally a reference check. This process was resource intensive, requiring multiple human touch points for every candidate applying for the role. As the Store Manager role gained visibility within the organization and more accountability for reaching revenue targets was placed on those in the role, the organization strived to improve the quality and standardization of their current hiring process. The organization desired to increase the objective data used to inform the selection process, relying less on subjective decisions. In particular, they wanted to be able to assess one’s ability to coach and develop subordinates in a less subjective manner.

Due to the decentralized nature of the organization, resource constraints (both budget and staff) limited the feasibility of the use of assessment centers to inform the selection process. Although paper and pencil assessment may have been an economical solution for the decentralized company, paper and pencil tests would not have been able to effectively cover the range of competencies required for success in these very important roles. Given the requirements of the role within the organization, along with the organizational goals to improve upon the current selection system/process, implementing the two managerial simulations offered an opportunity to assess core competencies and work behaviors that might not be assessed otherwise.

2.4.1.3 Implementation

CEB recommends the use of a comprehensive assessment program in order to measure the knowledge, skill, ability, or other characteristics of an individual. We feel that organizations should take a ’whole person’ approach when implementing an assessment program, utilizing a variety of test types and content to measure the set of competencies relevant to a job. This approach allows for a better understanding of an individual’s characteristics, and therefore will increase the likelihood of hiring candidates who will perform well on the job. In order to effectively implement a ’whole person’ approach, an organization must take the time to design an effective assessment battery (otherwise referred to as a “solution”) for that particular job role.

Based on Federal guidelines and professional standards, we approach the design and implementation of assessment solutions for clients in the following manner:

  1. 1.

    Perform a job analysis to understand and document the job requirements: this step allows for the identification of the primary work activities, competencies, knowledge, skills, and abilities required for successful performance on the job within a specific organization.

  2. 2.

    Assemble a tailored assessment solution: select a set of well-developed assessments that measure the core work behavior dimensions and competencies identified by the job analysis.

  3. 3.

    Collect validation evidence showing the ability of the assessment scores to predict performance in the job role.

  4. 4.

    Implement a final version of the tailored solution: utilize the validation evidence to determine the most predictive set of content and calculate an overall score, based on the underlying competency scores, which can be used for easy decision making.

The described process was followed by the focal organization of this case study to design a relevant assessment solution for their roles. This assessment solution was then implemented into the hiring process for the Store Manager and Assistant Store Manager roles and included the following flow: a high-level resume screen, part one of an assessment solution (this section does not contain the simulation content for security purposes as it is administered unproctored in the location of the candidate’s choice), an onsite interview and part two of the assessment solution (this is proctored, and includes the simulations), and finally a background check. The goal of the implementation was to put the assessment content as early in the process as possible—acting to screen candidates out prior to bringing them onsite for an interview. Implementing these technology-based assessments has reduced the amount of human judgment in the hiring process by effectively providing objective data points regarding key competencies. The ability to coach or monitor employees was previously determined through the interview process, whereas, assessment scores now exist to inform the selection decisions that are made. Additionally, the organizational goal to streamline the recruiting process was also met.

The effect of implementing such technology-laden assessment content did not increase the number of technological problems or user complaints that this organization typically encountered, beyond what was normally seen for traditional assessment content. When issues do arise, the problem is most often due to the logistical constraints under which the proctored tests must be delivered, and not with the managerial simulations themselves. Within the organization, the managerial simulations continue to receive positive reactions from business leaders due to not only their effectiveness in predicting successful candidates for these extremely important roles, but also the increased face validity of the assessment due to the media-rich simulations.

3 Conclusion

Simulations can meet an important need when selecting the best individual for a management role; without them, important and hard-to-measure competencies may not be effectively assessed. This chapter reviewed the development of two managerial simulations that can effectively replace two traditional assessment center exercises—the direct report role-play and the in-basket —with multimedia-based versions that can be delivered via the Internet . These simulations are most often supplemented with other types of online content such as personality , cognitive ability , and biodata assessments , or with in-person assessment exercises that are not currently available online. The future of the assessment of candidates for managerial roles may look very much like a traditional assessment center, but one that is delivered and scored completely online. Today, face-to-face interviews and other exercises such as the leaderless group discussion and presentation exercise, are still very often delivered in person. The promise of simulation technology is that it will offer a choice to an organization, to either hire managers with in-person assessment centers or to opt for a completely online version that is just as effective, more efficient, and less expensive than the traditional in-person assessment center. As a field, work in the area of simulation development is far from complete. Assessment professionals must look towards the future and answer the question, “What is next for this type of simulation?”

As technology advances, additional consideration should be explored and leveraged during the design and development of simulation assessment. For example, to the best extent possible, the ease of use should be optimized. This may include changes to the question type, the interface, and how the test is deployed (proctored vs. unproctored ). With advanced technological capabilities, future versions of managerial simulations may include the use of customization to enhance the face validity and representativeness of the simulation to the organization employing it. Avatars could be used instead of live actors to increase the ability for customization. By changing the avatar and color scheme behind the simulation interface, a simulation could be made to have the ’look and feel’ of the specific organization for which the candidate is taking the test. Additionally, the use of dynamic/random administration of items to increase the test security of the assessment could be considered.