Keywords

Manufacturing jobs vary widely. Some involve relatively straightforward, light industrial tasks such as sorting and packing lightweight materials. Others are much more physically demanding and involve heavy lifting, climbing, or working in awkward positions. And some others are more cognitively demanding because they create highly complex outputs and require workers to monitor equipment and fine-tune machinery. In many manufacturing jobs, exposure to dangerous materials and environmental hazards is common, and the chance for injury is inherent to the job.

As technology advances, manufacturing organizations are using more robust, predictive, job-related, and engaging simulations in their personnel selection processes. Leveraging these technology advancements and using high-fidelity, multimedia simulations can help to prevent these organizations from employing workers that will turnover, make costly mistakes, or significantly increase the risk of injury to themselves or others. Effective performance in many manufacturing jobs requires behaviors that are ideally suited to simulation-based assessments. Competencies such as attention to detail, safety orientation, multitasking, processing speed, and work pace are routinely found to be critical to success in high-performing manufacturing employees (O’Connell and Reeder 2008). These types of behavioral competencies are ideally suited for multimedia, simulation-based measurement methods.

Work sample testing is a form of assessment involving the use of hands-on performance measures, whereby a candidate or incumbent performs a given task or set of tasks under conditions comparable to those found in the position in question (Callinan and Robertson 2000). The primary philosophy behind this approach to assessment lies in the theoretical foundation set forth in the seminal works of both Wernimont and Campbell (1968) and Asher and Sciarrino (1974). Although addressing the concept of validity more broadly, Wernimont and Campbell (1968) encouraged a general shift from the traditional focus on traits and predispositions to a greater emphasis on observable forms of behavior.

Simulations , in general, can be thought to fall along different points on the sign versus sample continuum. One can think of the Wernimont and Campbell (1968) distinction as a continuum based on content (and face) validity and overlap with job tasks rather than two discrete entities. On the far “sign” side, you have personality inventories; on the far “sample” side, you have work samples ; in between, you would have situational judgment tests and interactive simulations.

Based on Wernimont and Campbell’s (1968) ideas, selection practitioners have built signs of performance (e.g., personality inventories), samples of performance (e.g., in-basket assessments), and all points in between along the continuum. In fact, over the past 40 years it has become common to see manufacturing assessments at various points on the continuum of content and face validity . However, in the past 5 or 10 years, there has been rapid advancement in the degree to which technology has been leveraged to enhance these signs and samples of performance. Obviously, computerized “signs” have become commonplace. Online “paper-and-pencil” tests are widely used, and technology has allowed these assessments to be administered more efficiently to larger samples.

Technology has provided assessment developers the opportunity to push manufacturing assessments closer and closer to the sample side of the continuum. Enhancements in technology allow assessment developers to change the way we measure behavior. While production simulation work samples have been used as far back as the 1950s, the approach to capturing and evaluating behaviors has evolved dramatically with advances in technology. This chapter will highlight not only technologically advanced testing formats, platforms, and simulations from a candidate perspective, but also, and more importantly, this chapter will address how advances in technology have allowed assessment developers to understand candidate performance in work samples to an amazing degree of specificity, consistency, and objectivity. For example, it is becoming increasingly commonplace to see a “day in the life” production simulation , where job candidates actually perform the job at one or more simulated work stations (e.g., building a vehicle dashboard), that have amazingly complex data collection points occurring automatically “behind the scenes.” These types of simulations are able to measure variables, such as fine motor skills, that it would be impossible for a rater to observe. It is possible to leverage technology to automatically measure variables such as processing speed, processing accuracy, task completion order, torque, etc. These variables are critical to success, and until recent technological advances, even the most robust production simulation could not capture these measures. Consequently, in this chapter, we will focus not only on innovative approaches to presenting test content to candidates from various points on the sign versus sample continuum, we will also discuss how technology advances have dramatically impacted approaches to collecting, understanding, and evaluating candidate behaviors. In many ways, the automation of data collection, scoring , and evaluation that occurs “behind the scenes” of today’s high-tech, multimedia production simulations represent the most complex and innovative developments in leveraging technology to impact how job candidates are evaluated.

As these complex approaches to measuring candidate behaviors evolve, evidence suggests that these procedures lead to incremental validity . By leveraging technology in simulations, the increase in fidelity and measurement precision is leading to positive outcomes. For example, a growing body of research shows that web-based interactive simulations measuring characteristics such as multitasking, processing speed, and attention to detail have high levels of validity in manufacturing situations (cf. Kinney and O’Connell 2012; Kinney et al. 2009; O’Connell and Reeder 2008).

A number of other potential benefits of simulations over traditional measurement methods (namely, cognitive ability and personality ) have been put forth in the academic literature. Although their popularity as selection tools continues to increase (Dudley et al. 2006), personality measures often produce only low to moderate validities when used in isolation (Barrick and Mount 1991; Salgado 1997; Schmitt et al. 1984; Tett et al. 1991). While cognitive ability is widely recognized as one of the best available predictors of overall job performance (Gottfredson 1997; Gottfredson 2002; Murphy 2002; Schmidt 2002), it, too, has its own recognized limitations. For instance, cognitive ability does not appear to predict subfacets of the criterion domain for low-complexity tasks as well as other constructs or methods (see Avis et al. 2002 for a brief review). In fact, evidence exists that interactive simulations, both moderate and high fidelity, add incremental prediction of job performance (Kinney and O’Connell 2012; Reeder et al. 2008) and safety (Kung et al. 2012), above both cognitive ability and personality .

The current chapter presents several examples of multimedia , interactive simulations currently in use by manufacturing organizations across a diverse variety of industries. These unique predictors range from computer-based simulations measuring targeted competencies (e.g., multitasking) to complex multi-work station, interactive production simulations involving the actual equipment and processes encountered on the job. An overview of these simulations will be presented along with research regarding validity in predicting job performance and safety as well as evidence of incremental validity , wherever available. Pros and cons of multimedia production simulations are presented, as well as a discussion of the evolution of production simulations over the past 30 years. Suggestions are also made about how to design and validate these types of multimedia simulations , including a discussion of the challenges faced in developing these types of simulations, such as avoiding the overreliance of cognitive loading, time constraints, realism, and the most appropriate types of dependent variables.

1 Manufacturing and the Challenge of Hiring the Best People

The United States has the world’s largest manufacturing economy, and roughly 12 million Americans are employed in the manufacturing industry (National Association of Manufacturers 2013). Manufacturing has a long history in America and, while it is often publicized when organizations move plants overseas or outside of the United States, manufacturing is still a major sector of the economy. We work with a broad range of manufacturing companies; they hire large numbers of hourly workers across multiple shifts and are constantly in need of finding good employees who can learn new skills, work both independently as well as part of a team, and work quickly, safely, and accurately. Hourly manufacturing jobs are very attractive to many workers because they do not require higher education, and the company will provide employees with the training they need to perform the job. In addition, compared to other hourly jobs, manufacturing jobs often pay well and offer competitive benefits.

The quality of the product that is manufactured by an organization is inextricably linked to the employees who make it. Ensuring that the best employees are hired helps organizations meet their production quotas and quality standards. Over the past two decades, we have toured literally hundreds of manufacturing facilities and studied the job of a “production worker” around the world. While different products are manufactured, the situation is generally the same. Products are made through a series of interdependent processes. Employees are typically under pressure to complete a specific set of tasks in a set amount of time. Their output is then sent along to someone else for the next step in the manufacturing process. Rarely does a manufacturing process have just one employee produce a complete product from beginning to end. With multiple hands touching the product, it is very important that each employee does his or her job effectively so that a high-quality product exists at the end of the process. From the employees’ perspective, these jobs provide attractive wages and benefits, but there are trade-offs. There are often unattractive shift times, the work itself can be monotonous and/or physically challenging, and the work environment can be unpleasant.

The average manufacturing company has the challenge of employing a large number of hourly workers who will show up to work every day and on time, work hard, get along with others, and produce a high-quality product. How do they find these people? What assessment tools make the most sense for doing this? The next sections describe two distinct types of tools: computer-based simulations and multimedia physical simulations .

2 Computer-Based Simulations

For purposes of simplicity, we refer to simulations administered by computer, whether they are web-based or not, as computer simulations. Computer-based simulations have clear advantages over paper-and-pencil assessments or even non-simulation-based online assessments. Computer-based simulations provide powerful behavioral samples of job-related performance. As mentioned earlier, whereas an online personality inventory may provide a “sign” of how a candidate may perform on the job, a well-developed simulation can capture actual behaviors on tasks very similar to those that will be encountered on the job. These computer-based simulations are often not as high fidelity as a robust multimedia production simulation (discussed later in this chapter); however, they are easy to administer to a large number of candidates with little to no resource expense in the recruiting process. These easily administered computerized work samples are not only efficient to administer, there are also clear advantages in using these types of measures over the typical online assessment measures in terms of predicting future job performance.

Over the last 20 years, more and more organizations have leveraged technology to impact how they measure the qualifications of candidates. Despite advances in technology, most online assessments are little more than paper-and-pencil tests recreated on a computer screen; however, the potential to leverage technology to measure candidates in amazingly sophisticated ways has existed for many years. While manufacturing jobs often do not take place solely in front of a computer monitor, many production operator behaviors are easily captured by computer simulations. Further, many of these behaviors cannot be reliably measured with less sophisticated approaches to measurement.

In developing or choosing an online behavioral simulation, it is important to think carefully about what types of competencies to measure. Certain competencies are ideal to measure with a computer-based simulation ; others are better suited to other modes of measurement. For example, consider three competencies often correlated with success in manufacturing positions: positive attitude, welding/machining skills, and multitasking. Let us look at these three competencies with regard to their best mode of measurement.

Positive attitude (also known as positive affectivity) is a personality trait. Specifically, someone high in positive attitude will likely react positively versus cynically, or view their world optimistically across time, setting, and situation. As such, this competency is not ideally measured in a computer-based simulation . Rather, a typical approach (e.g., online or paper-and-pencil self-report assessment) is more than adequate to capture this job-related individual characteristic. Research has shown that by asking candidates how they tend to behave and react to different situations can lead to a reliable and predictive indicator, or “sign,” about whether they will have a positive attitude in a production job (cf. Barrick and Mount 1991). Consequently, building a simulation to measure positive attitude or other personality traits is not typically a worthwhile endeavor.

Similarly, specific skills that require job knowledge are not ideally suited to computer-based simulations. For example, the ability to use a mig/tig welder or the ability to machine a part using a mill is required for many skilled positions in the manufacturing setting. These specific skills, however, are better demonstrated with a more robust, high-fidelity, physical simulation where a candidate may be asked to make a weld or machine a part. Through these hands-on, robust simulations, detailed measurements of job-related knowledge or skill can be collected. A computer-based simulation could ask the candidate questions about their experience with particular equipment or about the actual machines, but a high-fidelity simulation evaluates their level of capability using the actual machine.

We have worked with a number of organizations to develop this exact type of assessment . In every case, candidates have been required to “demonstrate the skill,” which is ultimately scored by a trained rater using an objective scoring format. These tend to be excellent ways of evaluating whether someone can actually “do” something as opposed to just “knowing how to do” it. The two are not always the same thing.

Multitasking, on the other hand, is a specific ability ideally suited to measurement in a computer-based simulation . With technological advances over time, the opportunity to encounter multiple tasks in the same general time period has become a key feature in many production jobs. How an employee maneuvers through this constantly shifting sea of tasks necessarily impacts that employee’s productivity. Consequently, identifying candidates that are better suited to these demanding and complex environments is critical to maximizing prediction in the selection system. Computer-based simulations of competencies such as multitasking can provide robust, accurate, and predictive measures of someone’s ability to perform on the job.

While it is true that multitasking on the shop floor does not occur at a computer terminal, but rather often involves monitoring processes, while checking the quality of the material on the line, and often involves moving from machine to machine and process to process, a computerized multitasking assessment can be valuable. A computer-based simulation does not recreate the exact task; however, a well-developed multitasking simulation can capture the underlying behaviors that make a person successful in these types of positions. If we consider the typical environment on the shop floor where an employee is required to multitask, there are some consistent similarities between the jobs. Many production jobs involve situations where (a) there is a heavy requirement to switch tasks frequently; (b) there is uncertainty about when to switch tasks; and (c) there is time pressure to switch tasks and complete work. While the actual tasks cannot be replicated, these characteristics of a multitasking environment can reliably be reproduced in a computer-based simulation. Consequently, if we can measure (via computer simulation) a person’s ability to switch tasks at uncertain intervals under time pressures, we are then able to capture an ability that will translate to on-the-job performance.

So, what does an online, computer-based simulation look like? The key to building a computer-based simulation is specifying the behaviors you are trying to predict. For example, consider multitasking again. In the previous paragraph, the behaviors critical to performance in a multitasking environment were identified; the key to building or selecting the right computer-based simulation is finding an exercise that mimics these behaviors. Figure 8.1 provides a screenshot of one such simulation.

Fig. 8.1
figure 1

Screenshot of multitasking simulation

Full size image

In this simulation, the job candidate must perform a variety of tasks that look similar to a typical production environment. This type of exercise is typically very well received by job candidates because the tasks they are asked to do “look and feel” like what they would expect to see on the shop floor. From a multitasking measurement perspective, all of the key characteristics of a manufacturing multitasking environment are present. First, the candidate is asked to switch tasks frequently. There are two main tasks that the candidate must accomplish. On the left side of the screen, there are a series of gauges that move toward the “red zone.” By clicking on the gauge, the candidate can reverse the direction of the needle to keep the gauge in the acceptable range. On the right side of the screen, the candidate must also make as many comparisons of number pairs as possible during the exercise. Consequently, the first condition is satisfied: the candidate must switch tasks frequently.

The second condition (uncertainty about when to switch tasks) is created with the gauges. The needles move toward the red zone at uncertain rates of speed. The candidate does not know when they will have to switch tasks to and from the number comparison task to keep the needles out of the red zone. Finally, there is inherent time pressure in this exercise. The candidate is aware that this is a timed task, and that he or she is being evaluated based on the number of comparisons that are made and how effective they are in keeping the needle out of the red zone.

By mimicking the multitasking demands on the shop floor, these computer-based simulations have led organizations to more effectively identify candidates who are not only able to do the tasks that are required of them, but also able to effectively switch among them. This is a key point—job tasks do not exist in isolation; rather, job tasks occur sequentially or even simultaneously. It is not enough to measure whether or not someone can do a particular task; to really understand job performance, measuring how well someone can switch between them is a critical consideration. For example, many people can monitor the quality of product on the line; but it takes a skilled person to monitor quality under demanding takt times (also known as cycle times), while also monitoring gauges and evaluating how well machines are running. These computer-based multitasking simulations provide a window into how well a job candidate might manage these demanding environments.

What are the results? Do these computer-based multitasking simulations really work to predict job performance in manufacturing settings? Further, how do these types of assessments predict performance relative to other more commonly used types of assessments for manufacturing jobs?

Until recently, with a few exceptions, not many organizations had high-quality computer-based simulations available to use. The evidence that has been collected, however, is in support of the use of these types of simulations. In a recent series of studies, the authors of this chapter have looked into these research questions. Kinney and O’Connell (2012) reported results not only showing that a computer-based multitasking simulation is an important predictor of manufacturing outcomes, but also, these simulations explain unique performance variance not explained by traditional predictors such as cognitive ability , personality , polychronicity (a personal preference for multitasking), and situational judgment tests .

In the first study, we investigated the criterion-related validity of a measure of polychronicity, cognitive ability, and a computer-based simulation of multitasking using 156 manufacturing operators. When these predictors were correlated with supervisor ratings of task performance, the multitasking simulation was the clear winner of the “horse race.” Polychronicity did not predict significant variance in task performance. Cognitive ability was a good predictor (r = 0.23; p < 0.05); however, multitasking was the strongest predictor (r = 0.36; p < 0.05). We also investigated the incremental validity of the multitasking simulation in this sample, and it was found that the simulation predictor did contribute unique variance in the performance ratings beyond polychronicity and cognitive ability, respectively (polychronicity and multitasking: ∆R 2 = 0.18; p < 0.05; cognitive ability and multitasking: ∆R 2 = 0.06; p < 0.05).

While predicting supervisor ratings of performance is important, in a second study Kinney and O’Connell (2012) reported on the impact these simulations have in predicting important objective manufacturing outcomes. In this study, the participants were 901 production team members. The focus was to investigate whether or not these simulations could predict workers’ compensation claims over a 12-month period. The results indicated that the multitasking simulation added incremental validity to the prediction of both workers’ compensation claims (∆χ 2 = 24.5, df = 3, p < 0.001) and costs (∆R 2 = 0.01; p < 0.01) above situational judgment and a personality composite. Clearly, these results suggest that a well-developed computerized sample of performance can be used as an effective predictor of manufacturing performance.

3 Multimedia Production Simulations

As previously discussed, online assessments and computer-based simulations are useful assessment tools for measuring important success competencies for production environments. Online assessments provide a “sign” of what a candidate can do, and computer-based simulations move closer to the “sample” side of measurement but are still moderate in their fidelity . Multimedia production simulations are the epitome of a work sample within the manufacturing realm and provide the highest fidelity measurement. This section of the chapter discusses the benefits of multimedia production simulations within the manufacturing industry, and then specifically discusses how advancements in technology have made it possible to design and implement multimedia production simulations.

The words “production simulation” might conjure an image of a candidate standing at a table and building or assembling a part. This is common in traditional production simulations, where the candidate is typically asked to participate in completing a hands-on task at a station for a defined period of time. The work is completed and then evaluated by an assessor or administrator, and then the candidate moves to another station for another task. In recent years, production workers use computers and computerized machines more often in their jobs, and those same advancements have made it possible to build computer technology into production simulations .

Let us first discuss the value of physical production simulations, computerized or not. Practically speaking, physical production simulations are often used in conjunction with other selection instruments. Before candidates get to this phase, they have often passed other assessment hurdles that evaluated them on experience and personality characteristics. What the production simulation provides is a chance to measure observable, physical performance on a simulated, job-related task. The data from a simulation can provide information on competencies that cannot be effectively obtained through other means such as process monitoring, work tempo, stability of performance over time, physical endurance, fine motor skills, and manual dexterity. Physical production simulations can add significant incremental validity to an already valid selection process because of the extra information they provide (cf. Kung et al. 2012; Reeder et al. 2008).

3.1 Benefits of Physical Production Simulations

3.1.1 Higher Performance

One of our manufacturing clients was using a physical production simulation as part of their hiring process for hourly assembly workers. Due to resource issues and a push to hire a large number of people in a short period of time, the organization stopped using the simulation. Once the new hires began training , human resources started to receive feedback from the managers on the floor. Suddenly, human resources staff was receiving complaints about the quality of the employees. Managers were frustrated because the new hires were not learning as quickly or performing as well as the previous employees. After a short hiatus, the organization reinstated the production simulation and the quality of their hires dramatically improved. Keep in mind, this organization was still screening individuals using valid non-physical assessment tools including personality , situational judgment, computer-based simulations , and cognitive ability measures. However, the additional validity and screening power that they had come to expect could not be realized without the physical simulation.

3.1.2 Reduced Turnover

Turnover is an issue for many manufacturing organizations at the hourly level. Whether the turnover is voluntary or involuntary, it is still costly for the organization. Organizations invest quite a bit of time and money into their employees by giving them all of the training they need to learn how to do the job. If an employee leaves or is terminated before becoming a fully productive employee, the organization does not recoup the money they invested in that individual. In addition, they are losing productivity by having an open position, and they must incur recruitment costs for replacing the worker. Our analysis of turnover in these organizations suggests that most manufacturing turnover, at least on the voluntary side, is related to poor job fit and/or dissatisfaction with the job or organization (cf. Lawrence et al. 2004).

Let us explore poor job fit in more detail. Employees who voluntarily decide to leave the organization shortly after they are hired often mention not liking the type of job task or work environment. Workers talk about the monotony of the tasks; they may not have realized that they would be doing the exact same task (e.g., installing electrical wire harnesses) all day long. Some organizations have workers rotate between several different tasks, while others do not. In addition, manufacturing facilities often have less-than-ideal work environments—they can be noisy, hot or cold, and even dangerous. For example, workers in a stamping facility operate 100-ton presses that stamp white hot metal ingots into axles or other components. There are bins of grey hot parts (over 1,000°F) going past them all the time, and the sound of the presses reminds one of the T-Rex’s footsteps in Jurassic Park. Employees in this plant deal with noise and the possibility of danger at every turn. If they are not paying close attention, they could easily be badly burned or crushed. Some people do not feel comfortable working in that environment every day.

Manufacturing jobs can be physically demanding as well. Tire manufacturers typically have a wide range of positions that require workers to lift 25– 50-lb tires hundreds of times throughout a work shift. It is not unusual for employees in such companies to walk the equivalent of over 5 miles and lift thousands of pounds during an 8-h work shift. Over the past 20 years, we have toured a number of automobile manufacturing facilities; their assembly workers are required to crawl in and out of the automobile cabs, stand, crouch, reach overhead, and lift greater than 25-lb parts over and over again on a moving assembly line.

Some workers are drawn to manufacturing jobs for the pay and benefits but realize that the actual work does not fit them. A key benefit of production simulations is that candidates have the opportunity to experience some of the potentially negative aspects of the job prior to accepting a position. In the literature, this is called a realistic job preview (RJP) (cf. Premack and Wanous 1985). An effective production simulation will provide a good enough feel for the job that candidates may self-select out of the hiring process, thus preventing the individual from leaving soon after accepting the job.

Several years ago, we designed a very comprehensive production simulation for a global automobile manufacturer at their plant start-up in the southwestern United States. They were focused on the “fit” part of the production simulation and made sure that the building that housed the production simulation was very similar to the conditions on the plant floor. Summers in that part of the country are very hot and the plant can get very warm—so, the candidates completing the production simulation were expected to complete a full 8-h day of work (four production exercises each lasting about 2 h) in a warm building, doing physically demanding work. We worked with this organization to process tens of thousands of candidates through the production simulation phase of their hiring process. On a regular basis, candidates would decide during the lunch break to withdraw from the process. When asked about it, candidates cited two main reasons for choosing to leave: (1) the tasks were boring, or (2) the tasks were too physically demanding. Because the individuals who participated in the production simulation had already passed several other selection hurdles, the candidates who withdrew from the process very likely would have been hired if the production simulation had not been part of the process. Both the candidates and the employer were grateful to have figured this out ahead of time.

3.1.3 Reduced Accidents/Injuries

Despite serious efforts on the part of dedicated safety professionals, injuries and accidents still occur regularly in the manufacturing industry. According to the most recent data from the U.S. Bureau of Labor StatisticsFootnote 1, slightly more than one-half of the 3.7 million private industry’s injury and illness cases reported nationally in 2010 were of a more serious nature that involved days away from work, job transfer or restriction—commonly referred to as DART cases. In other words, over 50 % of all injuries were severe enough to lead to loss of work, restricted duty upon return, and/or transferring out of the original job.

The nature of the jobs creates opportunities for workers to get hurt even though there are extensive safety policies and procedures in place to minimize risk. For example, manufacturing workers regularly work with and around machinery; they climb on ladders, work in confined spaces or underground, or work with hazardous chemicals, to describe just some of the dangerous activities that could lead to injuries. In some environments, a small mistake could lead to serious injuries. Injuries are not always a result of an accident; some of the injuries sustained by manufacturing workers are a result of repetitive motion. Because workers perform the same activity in exactly the same way repeatedly throughout the workday, every day, issues can arise with overuse of some muscles and joints.

Some workers are more able to tolerate the physical activity and follow the safety rules that are required to be safe on the job. Well-designed production simulations replicate the physical tasks required and build proper body positioning and procedures into the instructions and evaluation process. Participants are also required to wear the same personal protective equipment (PPE) during the production simulation that would be required on the job. That may sound trivial, but we were surprised by the number of candidates who actually removed their PPEs during the course of the production simulation even though they were explicitly instructed to keep them on. Imagine what these individuals would do on the job if they cannot even keep their PPE on for 8 h when there is a proctor watching them the whole time. It is much better to screen them out before they ever get to the shop floor.

Research has shown that using a production simulation as part of a selection process can help companies reduce incidents and injuries in the workplace. For instance, in a recent study, we looked at a sample of 130 manufacturing employees hired using a comprehensive selection process, including a physical production simulation. After employees had been on the job for 1 year, we gathered information on the number of safety incidents in which they had been involved during that time period. Both fine motor skills and work pace measured in the production simulation were significantly related to safety incidents, r = − 0.22, p < 0.05; and r = − 0.27, p < 0.05, respectively (Kung et al. 2012).

3.1.4 Positive Candidate Reactions

The reality of high-volume hiring processes, in manufacturing environments and otherwise, is that candidates will likely be screened out based on their performance on non-physical components of the selection process such as personality , cognitive ability , situational judgment, etc. In fact, they will most likely be screened out before they are able to talk with a representative from the hiring organization. Even though this might be the most valid, fair, objective, and efficient means of screening potential employees, this has the potential of leaving them with the feeling that they were not able to demonstrate their true ability.

Candidate reaction research has shown that giving candidates an opportunity to perform leads to more positive reactions regarding the perceived fairness of the selection process in particular and favorability of the company in general (cf. Truxillo and Bauer 2011). Work sample tests such as production simulations tend to elicit positive candidate reactions and are generally perceived as being fair (Robertson and Kandola 1982; Schmidt et al. 1977). Cross-cultural research on candidate perceptions of selection techniques suggests that work sample tests are viewed as among the most favorable within the United States and much of Europe (Bertolino and Steiner 2007; Hausknecht et al. 2004; Marcus 2003; Moscoso and Salgado 2004; Nikolauo and Judge 2007). Such broad findings of positive candidate reactions to work samples may be due to the high level of opportunity to perform, as well as perceptions of job-relatedness by candidates, properties that are not always evident in other types of measurement methods (Nikolaou and Judge 2007).

Consumer-oriented manufacturing companies are particularly concerned about their candidates’ reactions to their experience with the company, whether they are hired or not. The automobile facility mentioned earlier was building a new plant and needed to hire about 2,000 assembly workers. They received over 75,000 applications for these positions. Only about 3 % of the people who applied would actually get a position. The local newspaper wrote a story about the hiring process and said it was more difficult to get a job at this facility than it was to get into an Ivy League college. That may or may not be true, but with 97 % of the candidate pool being screened out, the organization wanted to have the fairest and most comprehensive process in place that would not negatively impact their brand in the minds of these candidates. Basically, they still wanted people to look positively on their products even though they were not offered a job. We find this to be a common feeling among large organizations that have large candidate pools.

Granted, many candidates never made it to the production simulation because they were screened out earlier in the process. Nonetheless, well over 20,000 candidates did go through the 8-h physical production process. Informal interviews and follow-up surveys of these individuals revealed that the opportunity to go through the production simulation did have a positive impact on their perceptions of fairness of the entire selection process.

3.2 Drawbacks of Production Simulations

After reviewing the many benefits of production simulations, one may ask why every company does not use them. It may not make sense for all manufacturing organizations to use simulations. Listed below are some of the reasons why companies may choose not to include simulations in their selection processes.

3.2.1 Expense

Probably the main obstacle for companies is the expense associated with designing and implementing a production simulation . Typically, experts are consulted with regard to the design of the tasks, equipment, and scoring of the simulation. The equipment used in physical production simulations is usually custom fabricated and manufactured using real materials used in the manufacturing process. All equipment must be uniform and ensure that every candidate has the exact same experience, regardless of the equipment being used. Even if a company does have the materials and resources to build a simulation, they still need a place to put it. Physical production simulation equipment can be large and difficult to transport. The company needs to have a space to put the equipment where it can remain for extended periods of time (typically months or even years), and also allow easy access for candidates and administrators. For the automotive plant start-up mentioned earlier, the organization leased a building near the plant to set up 10 complete work stations of three simulated tasks each. The equipment alone covered approximately 2,000 sq ft of space. They hired a full staff of assessors/proctors to process candidates daily for several years until their hiring needs were met. In fact, the production simulation process is still running almost 10 years after start-up because they continue to hire employees into the facility to cover additional shifts, attrition, etc. This might seem like a lot of work, but it should be noted that many of the plants using this robust system routinely win high quality awards by J. D. Power and Associates.

3.2.2 Labor Intensive

Because one of the main benefits of the physical production simulation is observable, work-related behavior, trained assessors or administrators are needed to observe and evaluate candidate performance. The more automated the simulation and the scoring , the less an assessor is needed for collecting and evaluating work behavior. Depending on the level of automation and type of activity being evaluated, the ratio of candidates to assessor could range from 2:1 to 20:1. More on the evolution of automation in the production simulation process is discussed later in this chapter. Given the nature and purpose of a production simulation, a typical production simulation lasts for several hours, requiring trained assessors to also be available and dedicated during that time period. Compared to other selection tools such as personality scales, cognitive ability tests, online simulations, etc., the resources needed to run a production simulation are significantly higher.

3.2.3 Increased Time to Hire

One of the important metrics tracked by most human resource departments is the time to hire. This is the time it takes for a candidate to apply and move through each phase of the selection process until they are hired—the more steps, the longer it takes. They want to put into place an accurate process that will allow them to process candidates quickly. Including a production simulation in the process adds a lengthy step that requires administrative resources and takes time. In addition, the number of candidates that can be processed at one time is often limited due to the equipment and space limitations. Therefore, companies must factor in the extra time it will take to fill an open position when the production simulation is added to the process. For companies that are focused on hiring the best employees up front, a production simulation definitely helps with that goal. The automobile start-up felt that taking the extra time was important to ensure that they had a high-performing and qualified workforce from the very beginning. As mentioned previously, one organization decided to remove the production simulation from their process because they needed to fill their open positions more quickly and could not budget in the time for it. They felt the impact afterward when they had lower performing employees, poorer job fit, and higher turnover among the new hires.

3.2.4 Adverse Impact

As with all selection procedures, it is important to understand how subgroups of candidates perform to determine how its use might affect the diversity of the workforce. With computerized simulations that have a large cognitive component, there would likely be race differences on the performance of the simulation. For physical production simulations , there is often a gender gap, where men pass at a higher rate than women. When the majority group passes at a substantially higher rate than a minority subgroup, then adverse impact exists. While this does not mean that the assessment is discriminating against a subgroup, it does allow for the possibility of a legal challenge. Therefore, it is important for a company to document the design, development , and validation process used to support the production simulation before it is used for decision-making purposes. As long as the tasks included in the production simulation can be clearly linked to tasks required on the job, the likelihood of a successful legal challenge to the process is dramatically reduced. When designing production simulations, it is critical to determine the physical demands required, upon entry into the job, of individuals in the target positions and ensure that the simulation does not exceed those levels. If a job requires most individuals to lift 10-lb weights repeatedly during the shift and the simulation requires them to lift 20-lb weights, it may be difficult to defend that simulation.

Because production simulations are highly face valid and candidates react more positively to them, it makes logical sense to presume that candidates are less likely to challenge the validity of the production simulation as compared to other parts of the selection process. Regardless, physical production simulations are more likely than personality and other non-cognitive and non-physical components to show adverse impact .

3.2.5 Limited in Measurement

After reading the literature and seeing the strong arguments for the validity of work samples and simulations, one might conclude that it is the only predictor needed in the selection process. While there are many benefits to simulations, they are limited in the kinds of skills and characteristics they can measure. As mentioned earlier, most manufacturing simulations measure competencies around physical ability, attention to detail, quality focus, multitasking, processing speed, work tempo, and, to some extent, cognitive ability . When we implement a simulation for an organization, it is always in conjunction with other selection measures to ensure that we are getting a comprehensive look at a candidate. The automobile plant alluded to an earlier simulation of an 8-h workday. If that organization had only used the production simulation , they would not only have had to process tens of thousands more candidates through the production simulation, but also they would have obtained less information about their candidates. They would have had a good understanding of who might do well on the job tasks, but they would have been missing information about work ethic, teamwork, positive attitude, responsibility, and leadership potential—all competencies they were also very interested in measuring. Production simulations are best used as one of the final steps in the selection process, after other measures have screened candidates for non-physical competencies.

3.3 The Evolution of Production Simulations

As discussed earlier, production simulations have been around for a long time. Our experience with them goes back to the late 1980s but they were used for decades prior to that. This section describes the evolution of production simulations over the past 30 years. The evolution has coincided with two major changes in the workplace. The first has been the changing nature of manufacturing jobs themselves, especially in the United States, Canada, and certain parts of Western Europe. The second has been the incredible expansion of digital technology over that time period.

3.3.1 Production Simulations in the 1980s

The idea of providing a RJP during the selection process had already gained acceptance and was quite popular during the 1980s. In manufacturing environments, that often took the form of plant tours during the interview process along with a description of what it was really like to work on the floor. Production simulations greatly expanded the concept of a RJP. Instead of just showing someone where they were going to work or telling them about what it would be like to work there, why not actually have them go through something that would replicate a “day in the life”? As discussed elsewhere in this chapter, the idea then, and now, is to identify the areas that cause new employees the most problems. What are the hardest things for new employees to master? What are the things that cause the biggest complaints or reasons that people cite as reasons for leaving?

3.3.1.1 Tough, Physical, and Repetitive

One thing that has not changed much over the years is the repetitive nature of most manufacturing jobs. Many organizations like to rotate employees among a group of jobs to increase flexibility in filling in for people who are sick or who leave, as well as to reduce repetitive motion-type injuries, and to some extent improve the job environment. Nonetheless, because of the nature of most production line jobs, they end up rotating from one repetitive task to another one. Some people perform well in repetitive tasks and some do not, whether it is because they have a hard time maintaining focus over time, that they are easily bored, or that they just do not like doing the same thing over and over again.

To simulate the repetitive nature of work, most production simulations , then and now, tend to have job candidates perform several simple tasks for an extended period of time. For instance, one common task in many production simulations, particularly in automotive facilities, is to have candidates take metal tire rims from a rack, carry them for about 10 ft., and then place them onto a metal peg on a grid. It is not a particularly difficult task to learn. For instance, the instructions might tell them to do the following: (Task 1) Red rim onto A4; (Task 2) Blue rim onto B2; (Task 3) White rim on D3, etc. Assume that you have three different colored rims, red, white and blue, and a 4 × 4 grid going from A1 (lower left corner) to D4 (upper right corner). Most people can perform this task on the first trial. The issue is not that it is difficult to learn, but when you have to lift, carry, and place a 15-lb rim 10 ft. over and over again for an hour or two, it is not only tiring but certainly repetitive.

Compared with today’s manufacturing jobs, jobs in the 1980s tended to be more physically demanding than they are today. Increasing the level of automation and reducing the physical nature of manufacturing jobs is a trend that has continued steadily since the 1980s. This is particularly true in the United States, Canada, United Kingdom, Germany, and several other countries. As an example, we helped open up a cold-rolled steel facility in the late 1990s that had only 128 full-time operators. It was very technologically advanced and a majority of the work involved monitoring sophisticated equipment and little, if any, physical labor. A similar plant that opened 20 years earlier required over 500 employees performing much more physical, less cognitively demanding tasks. It is often said that more jobs are lost to automation than any other cause, and this is likely to be true. Nowadays, robots perform many of the most repetitive, physically demanding jobs that used to require teams of skilled individuals.

When manufacturing jobs were more physically demanding, the focus of the production simulation tended to be guided by a theme of “make it tough and see if they can handle it.” The purpose was twofold, give candidates a solid feel for the job and see if they can make it through the simulation and measure key predictive competencies for making selection decisions. As the nature of work has evolved, so too has the focus of the production simulation. For instance, when we analyzed the most difficult things for new employees to get used to at several automotive assembly plants, it was less about lifting heavy weights than it was about working in awkward positions.

Specifically, one of the more challenging tasks involves working inside a car chassis and making wiring connections, such as connecting electrical harnesses under the dashboard. It requires a lot of twisting, bending, and fitting into very tight places. Particularly for a larger individual, it is very difficult to do on a regular basis. Because of that, some people have a hard time keeping up with the pace of the work, or they start making mistakes, such as missing connections, not securing connections, or making incorrect connections. Therefore, we focused a lot of our simulation on doing just that sort of activity. It does not require heavy lifting but it does require the candidate to get inside a chassis and make a series of connections. Then they disconnect them, replace the wire harness onto the rack, and then do it again.

Usually, production simulations are designed in stations. We have already discussed two stations that are found in many automotive plants. Those two, tire mounting and wire connections, in and of themselves, might form the basis of a 4-h production simulation. It would not be unusual to have candidates train for 5 min in each activity and then rotate between them every 30 min for the 4-h period. That way, they are performing two very different, yet repetitive, physically demanding activities over an extended period of time.

3.3.1.2 Replacing Assessors with Automation

One of the clear challenges of production simulations is their resource-heavy nature. The other challenge is how to accurately score them. Fortunately, as computers, digital cameras, video displays, and digital technology in general have become smaller, more powerful, and less expensive, both of those problems can largely be eliminated. In the 1980s, it was common to have three participants assigned to one assessor, a 3:1 ratio. Now, there are very sophisticated centers that can assess a group of 20 candidates with one assessor, or a 20:1 ratio. The scoring of the simulations in those centers is completely automated and done in real time with much higher levels of accuracy and information than was previously possible.

To provide a better idea of how production simulations have evolved over the past 30 years, we will walk through two different exercises, which we will refer to as “bolt mount” and “weight mount,” and discuss how they have gone from being assessor-intensive to almost completely automated.

3.3.1.3 Low Tech—Assessors Do It All

Traditional production simulations required assessors to: (a) explain and demonstrate to candidates how to perform the tasks; (b) observe candidates as they went through the exercise, usually making notes on a structured rating sheet; (c) score candidates’ performance, usually by checking what they had done for a particular phase; (d) direct candidates to different phases, stations, etc.; and (d) make final calculations to assign final simulation or competency scores.

This original version of the “weight mount” exercise is similar to what was described above. Essentially, candidates read an instruction sheet that told them which rim to place on which grid spot, they picked up the appropriate rim, carried it, and placed it on the rack. They continued to do this until the grid was filled up. At that point, the assessor recorded what the candidate did and would then tell them to continue onto the next set of instructions. The candidate then removed the rims from the grid, replaced them on the rack, and began the process anew. Candidates earned points for both speed and accuracy. How many rims was the candidate able to accurately place on the grid over the allotted time period? How many were incorrectly placed? In addition, the assessor typically watched the candidate to make sure that they wore their PPE and followed instructions correctly. Typically, if candidates did not follow established procedures, such as carrying a rim with one hand versus two, or taking off their PPE, then the assessor rated them negatively either on “following instructions” or “safety.”

This approach was neither high-tech nor multimedia in any way. Nonetheless, it was effective. For example, based on a study of 153 manufacturing employees, the work tempo component of the production simulation had the highest correlation of any competency in the assessment process with regard to supervisor ratings of work pace (r = 0.46; p < 0.01) and was also significantly related to overall performance ratings (r = 0.30; p < 0.05) (O’Connell and Smith 1999).

One challenge that this approach presents is using the assessor time effectively. The assessor was required to monitor two to three candidates, and often these candidates finished their exercises at approximately the same time. In that case, the candidate stood around waiting while the assessor recorded information for another candidate. There is essentially built in “down time,” which ultimately takes away from the physical nature of the exercise.

In the “bolt mount” exercise, candidates are presented with a number of bins containing different bolts. Some are black, some silver, some white, etc. They are instructed to put a specific bolt into a spot on a metal plate that might be mounted onto a rack of some type. These plates typically have a grid pattern with a set of pre-drilled holes. There is usually either a number or letter assigned to each hole. So, candidates might be instructed to insert a “black” bolt into hole “A2.” Usually, candidates are mounting these bolts using either a cordless drill or pneumatic drill. The pneumatic drills tend to be a bit heavier and better to simulate, in terms of both feel and sound, what they will experience on the line. The bolt mount exercise is more about manual dexterity and precision than it is about physical endurance.

As far as the assessor requirements, it is similar to the weight mount exercise. After candidates have mounted 20 or so bolts, the assessor comes over and checks their work. They are interested in how many bolts were correctly mounted, how many mistakes were made, and whether the candidate followed instructions appropriately. In addition, they might penalize candidates for stripping the threads of the hole, which tends to occur when bolts are put in haphazardly. The same issues described earlier in terms of the time required for the assessor to record the candidate’s work apply to this exercise.

3.3.1.4 Automated Multimedia Production Simulations

Our most recent production simulations , those developed since about 2002, have gone to an almost completely automated administration and scoring process. We have leveraged audio, visual, and computer technology to increase the efficiency and accuracy of the physical production simulation . Technology was able to reduce the number of administrative resources needed, increase the realism of the work by reducing downtime, and provide real-time scoring and instant decision making.

Instead of assessors explaining and demonstrating to candidates, in a group, how to perform an activity, candidates now watch a video on a small liquid crystal display (LCD) right at the work station. This ensures consistent instructions for everyone, and if they need to watch it more than once, they can. It also removes the need for an assessor to be involved at this stage. In addition, each participant watches while using headphones, and they are in control of how many times they watch the instructions. Once they feel comfortable with the task, they have control over when they start the exercise and receive the first set of instructions. This also allows candidates to begin the production simulation activities at varying times without disrupting other candidates.

In the bolt mount example described earlier, all of the individual bolt mount plates are self-contained units with pressure actuators that are connected to a computer server. This allows us to place plates throughout the interior of a car or truck chassis so that candidates must sit inside and mount bolts above them, to their side, below them at awkward angles, etc. The same LCD set in front of the windshield of the cab provides all of their instructions. Once they mount a bolt into a hole, the computer senses it and records whether it is correct or not. No assessor is needed. A simulated example of the bolt mount exercise is shown in the figure below (Fig. 8.2).

Fig. 8.2
figure 2

Simulated bolt mount exercise

Full size image

A similar process is used for all of the other simulations in the center. This new automated approach provides several key advantages. The first is that one assessor can now proctor a group of 10–20 candidates as opposed to just 2 or 3. The second is that instructions are consistent and provided to candidates in real time. The third is that scoring is now completely objective and the level of information available is significantly higher. In earlier versions, the information gleaned from the simulation was almost exclusively limited to productivity (number of completed tasks) and quality (number of errors). The addition of computer sensors to the production simulation provides information not available with the other designs. For instance, the multimedia production simulation allows us to chart an individual’s fatigue level and correlate that with performance on the job. One thing we found was that individuals who had steeply declining curves in terms of productivity and increased errors as time went on, were rated significantly lower by their supervisors. We are currently evaluating additional approaches to better utilize this information to help predict injuries, both acute and those associated with repetitive motion. The final advantage is that all scores are computed in real time and then uploaded to a server where they can be integrated into the candidate’s profile, and decisions can be made immediately.

The one real downside of the automation approach is that the initial setup costs for the equipment (e.g., servers) is significantly higher. However, as long as the volume is high and the assessment runs for an extended period of time, the increased accuracy and the dramatically lower administration costs more than offset this one-time cost.

Consider another multimedia physical production simulation that focuses primarily on the monitoring and recalibration of equipment, multitasking, taking measurements, and working quickly and accurately with very few physical demands. This particular simulation was designed for a new plant start-up that had a highly automated manufacturing process. Individuals were hired into one of two primary positions: assembly station operator or machine operator. Although neither of these jobs was particularly physically demanding, making a mistake, such as not ensuring that the machines are operating within proper specifications, could cause serious consequences in terms of quality, productivity, and ultimately, costs.

For this particular organization, we designed two distinct workstations. The first mimicked the assembly station. A photo of the multimedia assembly workstation is shown in Fig. 8.3.

Fig. 8.3
figure 3

Assembly workstation

Full size image

Candidates who work on this station are provided with standardized instructions from the video monitor on the right-hand side of the unit, which is also where they receive all of their work instructions during the course of the 2-h simulation. Candidates are required to screw on metal caps, make connections with a wire harness, and screw in and remove bolts using a handheld pneumatic gun. The entire station has built-in sensors throughout, and the computer-based scoring system automatically scores candidates on their pace of work and whether they have followed the instructions appropriately. The three primary functions covered in this particular station accurately simulated approximately 70 % of the core activities required for all production positions at this organization.

The second station was developed primarily to assess performance in the most advanced position on the floor, that of machine operator. While this was still an entry-level position, the cognitive demands required to be successful in the position were higher than in all other positions on the floor. Figure 8.4 provides a quick snapshot of one of the core activities required of candidates in this particular simulation.

Fig. 8.4
figure 4

Example of interactive machine operator station

Full size image

This was a truly interactive, multimedia simulation . Candidates moved continuously between three different work stations. Work instructions were provided to them by a flat-screen monitor located between the stations. One of the core activities was to keep the machines within acceptable tolerance limits. They did this by pressing various buttons that moved the readings within various gauges up and down. The individual in this figure is pressing a button to lower the reading on one of the three gauges on the screen. Gauges moved automatically and in an unpredictable manner based on several algorithms built into the system. Candidates needed to stay constantly vigilant to keep them within the tolerance limits. They were also required to take physical measurements and record their readings into the computer. At any given time, they would be monitoring and adjusting six sets of gauges, resetting their machines and recording product counts that appeared on the screen, and also making and recording measurements of a variety of articles, all based on the instructions presented to them on their computer monitor. As with the other automated simulations described thus far, all scoring was done in real time and automatically by the computers hooked up to these stations.

Reactions both from candidates as well as the organization were very positive regarding the realism of the simulations and, most importantly, to the quality of hires made based on these systems.

4 Developing a Multimedia Production Simulation

Developing an effective multimedia physical production simulation takes a lot of time and effort, but companies should experience a return on their investment in terms of higher productivity, reduced turnover, accidents, and injuries. The recommended steps involved are described in detail below:

  1. 1.

    Define the job family—Given the time and expense associated with the design and development of a multimedia production simulation , using it to hire for just one position is costly. Typically, the simulation is used for positions within a larger job family. For instance, it is the norm rather than the exception that a manufacturing plant will assign the title of “production team member” or “production employee” to a broad range of positions covering different functions and departments. This applies to skilled or maintenance positions as well as non-skilled positions.

    This is a relatively dramatic shift when you compare it to traditional manufacturing facilities and job titles. If you were to look at the number of “job titles” in a traditional automotive facility, you would likely see over a hundred individual job titles. Compare this to newer facilities that may cover those hundred job titles with three to four primary positions. It still remains that unionized facilities typically have more job titles and non-unionized ones have fewer job titles, although there are plenty of exceptions to that rule. Many newer unionized plants have moved toward broader positions with fewer job titles. This broadening of the job families has salutary implications for both employers and employees. It allows easier cross-training, more flexibility in allocating resources, and more task variety for workers.

    To use the example of an automotive manufacturer, a production employee in “assembly” is likely putting parts on vehicles as they move along the assembly line. This may include installing the dashboard, installing and securing the steering assembly, electrical wiring harnesses, seats, etc. A production employee in the “stamping” area helps fabricate, or stamp parts out of metal (e.g., doors) that are then put on the car either by robots or by workers in assembly. In this job, there is often a higher level of equipment monitoring, such as stamping presses, required to make sure they are working correctly and that they are properly loaded, etc. Other production employees work in the paint, body weld, final quality inspection, and conveyance (i.e., moving parts and supplies around the plant) departments. They are all in the same job family but do not do the same physical tasks. When candidates apply, they do not apply for a specific department, and they can move around within the organization to other departments throughout their career. With so much task variance within the job family, the production simulation should measure skills needed by all of them. Therefore, it is important at the beginning of the design process to know exactly which jobs will be filled by individuals who complete the production simulation .

  2. 2.

    Analyze—Once you have a good understanding of the jobs involved, it is important to better understand them from the standpoint of the tasks involved, reasons for failure, reasons for turnover, and common injuries or accidents. This information can be valuable when deciding what aspects to build into the design of the simulation.

    1. a.

      Job analysis —Learn as much as possible about the jobs in the job family. Tour the facility, observe the jobs, shadow incumbents, work the job yourself, and talk to incumbents and leaders. The more you know, the better the final simulations will be. If the plant is a start-up, visit a different facility with similar jobs or talk to the leadership about the vision for the position.

    2. b.

      Turnover analysis—Certain aspects of the job itself can lead to turnover. If available, gather information about why individuals have left the organization—voluntarily and involuntarily. Identify the reasons that speak to job design and determine if the simulation can and should include activities to address them. For one manufacturing company, one of the main reasons for turnover was the monotony of the work. Because of the pay and benefits, individuals who were trained for other careers (e.g., school teachers) were applying for manufacturing positions. Once hired, they had difficulty being tied to the production line for hours without being able to leave for a break. They also missed the intellectual stimulation they were accustomed to in previous jobs. As a result, a production simulation was built that required candidates to do the same task for close to 2 h and then rotate to two other tasks both for another 2 h each. The job rotation and length of time resembled the job as closely as possible. Instead of screening out individuals without previous manufacturing experience, this company allowed the experience with the production simulation to help make the hiring decision. Turnover was greatly reduced after the production simulation was introduced.

    3. c.

      Injury/accident analysis—Knowing as much as possible about common injuries and the types of accidents that have occurred can aid in the design of the production simulation . A Canadian manufacturing facility actually included their occupational therapist as part of the design team. She was responsible for working with engineers and technicians to design the work processes in a way to reduce physical strain and repetitive injuries. Not only did she want to ensure that new hires could perform the physical tasks, she wanted to make sure that candidates did not get hurt while completing the stimulation.

  3. 3.

    Identify minimum physical requirements—As mentioned earlier in the chapter, the complexity and difficulty of simulated tasks should not exceed those required on the job. It is important to identify the minimum requirement for a specific task before building it into the production simulation . For example, it is common for manufacturing workers to lift heavy parts, boxes, or other objects as part of their job. However, not all jobs require heavy lifting or lifting of the same amount of weight. At one manufacturing facility, there were a few jobs that required employees to lift up to 50 lb regularly as part of their job, but most of the jobs never required lifting more than 20 or 25 lb on a regular basis. When designing the production simulation , an exercise was designed where candidates were instructed to move weights back and forth from one location to another. During the design phase, it was decided that the heaviest weight would be 25 lb and it would be the least moved amount of weight by the candidates during the exercise. If the company had chosen to make the heaviest weight 50 lb and instructed candidates to move it often during the exercise, they would have been requiring candidates to work harder than is typical on the job. By doing so, they would have screened out individuals who could have adequately performed the vast majority of the jobs. In addition, because the individuals who would have been screened out were more likely to be women and older workers (both protected classes), there would have been a greater likelihood of a legal challenge. It is difficult to defend using a higher standard than what is required for the most likely target positions. Be aware of this issue when deciding what to require, and be sure to tie the equipment back to the job and have job-related rationale for the decisions being made.

  4. 4.

    Identify common tasks—Similar to the previous step, it is important to identify the tasks most common to the jobs in the job family. You want to simulate the jobs as closely as possible and measure the skills that will get you the best information for predicting a candidate’s performance once hired. Because you are looking across jobs within a job family, there will be very few tasks that are present in every single job; however, there will be skills that are common to a large group of them.

    Like the minimum requirements, be sure to identify tasks that are job related and do not require more of the candidate than the jobs in the plant. All decisions should be documented with job-related rationale.

  5. 5.

    Determine skills and method of evaluation—All the steps up to this point are recommended for production simulation design—multimedia or not. It is at this stage that the multimedia technology makes an impact because it allows the developer an opportunity to easily measure some skills that were difficult, if not impossible, to measure previously (e.g., time per task), and to do so more accurately.

    In general, production exercises evaluate speed (number of tasks completed) and accuracy (percent completed accurately). Computers can easily calculate those measurements very accurately and provide the ability to look for data trends as well (e.g., stamina or accuracy over time). Our experience has been that it is better to limit the number of competencies measured in the production simulation . Attention to detail (also known as quality orientation) and work tempo (speed) are the most commonly evaluated measures in a production simulation. We have included other competencies, but when we analyzed our results, they typically gravitated to those two primary factors. Other competencies such as multitasking, fine motor skills, following instructions, or safety orientation are also competencies that lend themselves to assessment in such simulations. Nonetheless, it is better to get solid, accurate, and reliable measurement on a few areas than trying to force fit too many competencies into the equation just for the sake of covering more competencies.

    After identifying the general skills to be measured, the next step is to decide how to score the exercises. Instead of requiring an assessor to observe, record, and score each exercise, a computer does it all. Investigate the technologies that exist to be able to give you the feedback that is needed throughout the exercise. In most cases, simple electric switches that feed data back to a server are adequate for telling the computer if a candidate was right or wrong. Knowing these things ahead of time is important because it will affect the design and fabrication of the exercises.

  6. 6.

    Design—There are many factors to consider when designing a multimedia production simulation . Some of the factors to consider are listed below:

    1. a.

      Administration—When designing the exercises, keep in mind how the candidate will receive instructions. Using instructional videos and on-screen instructions provides each individual with clear, consistent instructions on how to complete the exercise. It is also recommended to build in a short practice time for the candidate on each exercise before the scored exercise begins.

    2. b.

      Number of exercises—The number of exercises to be developed is often driven by time and budget. If the target job(s) offer job rotation, the simulation should also offer this opportunity. However, if the target job requires workers to complete the same task for the entire workday, the simulation would be more beneficial by not rotating candidates.

    3. c.

      Type of exercises—Remember that employees in the organization have been trained on how to complete their job tasks. In some cases, that training lasts for days or weeks to ensure that they are able to effectively perform the tasks. In a production simulation , candidates have a very short period of time to learn the task and then perform it. The tasks should be straightforward enough that every candidate, regardless of knowledge and experience, has the same understanding of the task at hand. It is not necessary for the production exercise to simulate an exact job within the plant. For example, let us examine the task of fastening screws and bolts. The manual dexterity needed to perform the task is what should be measured in the production simulation. The screws can be fastened onto a metal plate on a table. However, if screws and bolts need to be fastened in awkward positions or overhead, the simulation should build this into the task as well by placing metal plates high and low and in hard-to-reach places. Keep in mind the underlying skill that is being measured, and keep the task simple enough for candidates to learn easily. In addition, once the exercises have been determined, the next step is to investigate the technology available to be able to build a simulation that is able to collect the important data needed to measure the skill or competency of interest. One point to consider when using technological equipment for assessment is that parts can malfunction or break. When designing the system, ensure that there are “checks” in place to identify switches or other equipment not working prior to assessing a candidate, as broken equipment could affect the final score of a candidate.

    4. d.

      Length of simulation—The amount of time that the candidate should take to complete the entire simulation can be difficult to determine. The shorter the simulation, the more candidates that can be processed and the less time resources spend administering and scoring the assessment . However, if the simulation is too short, then it does not adequately provide candidates a good preview of the actual work. When a task is new, monotony and boredom do not set in right away. It is important to give candidates enough time to get tired of the task before switching them to a new one or ending the simulation.

    5. e.

      Ergonomics—When building the equipment for the simulation, it is important to allow for adjustments to accommodate different heights. If a candidate is particularly short or tall, making incorrect motions over time on some tasks can lead to injury or put them at a disadvantage in terms of performance. A simple rule is that if the organization offers accommodations on the job, then you should incorporate similar accommodations during the simulation.

  7. 7.

    Pilot test—After the tasks have been designed and the equipment has been fabricated, pilot testing should be conducted to test the accuracy of the design and establish a reference point for scoring candidates. For this step of the process, individuals who are unfamiliar with the simulation should act like candidates and complete the simulation. Some companies have even hired a temporary agency to provide a group of “fake” candidates. Job incumbents are not the best population to use for this step of the process because they have on-the-job experience, which might give them an advantage when completing the exercises. The data gathered from the pilot allows the individuals involved in the design to identify errors and improvements and set scoring (including initial cut scores) for the assessment. The pilot testing is also an ideal time to train administrators and assessors on their roles.

  8. 8.

    Validate—As is often the case, a predictive validation study is recommended because it allows you to examine the relationships between the simulation and job performance, turnover, safety incidents, and other outcome variables. The drawback is that you may need to wait 2 years or more before conducting such a study. Conducting concurrent validation studies, using incumbents in the job, is also an appropriate strategy. While incumbents may have an advantage over job candidates, we have found that you typically still obtain acceptable levels of variance of performance in the exercise. You will not be able to measure turnover, but you can look at other criteria measures and establish validity linkages.

    As a solid starting point, production simulations lend themselves to content validation strategies. If designed well, a production simulation epitomizes a content valid approach. This requires drawing clear linkages between tasks on the job to tasks in the simulation. If you are not able to establish the content validity of the production simulation then you probably have not designed a good production simulation.

5 Summary

Manufacturing environments require a broad range of skill sets, abilities, and motivations that clearly lend themselves to simulations in general and production simulations in particular. The results are consistently strong in terms of the validity of such simulations. The perceived fairness and candidate reactions to such simulations also appear to be positive.

Another benefit of such simulations, that is not widely discussed or published, is their face validity and perceived fairness by internal stakeholders. On the surface, that may not seem to be that important, but it has a number of beneficial outcomes. The first is that it helps ensure that higher validity solutions, such as comprehensive assessment center/production simulations, are given appropriate consideration in the hiring process compared with less valid, more subjective methods such as hiring manager interviews. While interviews are certainly valid predictors of performance, our experience has been that production managers tend to overemphasize previous manufacturing experience to the detriment of almost anything else. While research does suggest that previous manufacturing experience is moderately related to reduced turnover (cf. O’Connell and Kung 2007) it is only modestly related to performance, especially compared with a well-designed production simulation . During times of plant start-ups, we have found that production and other line managers tend to be overly cautious in accepting candidates onto the shop floor. This is not without reason. It is in their best interest to have the best possible employees in the plant from the moment the plant starts production, and even before. However, in their zeal to “hire the best,” they unfortunately tend to fall back on bad habits and implicit biases and ultimately screen out some individuals who would be predicted to be fantastic employees because they “have never worked in manufacturing” or “have worked in manufacturing but not like ours… ” Because of factors such as financial incentives and lower labor costs, many plants open up in areas that do not have well-established manufacturing bases and, therefore, it is difficult to find individuals who have relevant manufacturing experience. That is not a problem as long as the company has a fair and accurate methodology for screening candidates. The production simulation and other manufacturing simulations provide such an approach. It usually takes some education and time to break old habits, but when production managers see what candidates have to successfully complete in order to make it to the final interview they are more likely to trust the system than go by their “gut feel.” This is a huge benefit to the human resource professionals at the plant tasked with staffing the facility.

Leveraging recent technologies and applying them to multimedia production simulations , as well as interactive computer/web-based simulations, is a smart way for organizations to gain predictive power to their selection processes without extra resources. They have proven themselves many times over to be effective, fair, and accurate methodologies for assessing candidates and making predictions of their likelihood of success in a manufacturing environment. The future of these types of simulations is likely to mirror the environments they are designed to simulate. As described in detail earlier in this chapter, manufacturing jobs continue to move away from single task activities and instead are shifting toward those that require more decision making, multitasking, and collaboration. Manual dexterity, processing speed, work pace, stamina, and attention to detail will continue to be important competencies in almost any manufacturing job. Well-designed multimedia production simulations are the most fair and accurate method of measuring such competencies.