Data Collection Methods

Abstract

The underlying foundation for several commonly-used data collection methods for evaluating wage and hour compliance is provided in this chapter. The resolution of nearly all wage and hour disputes covered in this book requires detailed measurement of the work employees perform, the amount of time spent on that work, and the context in which work is performed. In the past 15 years, consultants and experts have been successfully applying job analysis techniques to collect data to address these issues. The data collection methodologies discussed in this chapter fall into two general categories: observational methodologies and self-report methodologies. Observational methods include live “time and motion” observations and video-based observations. Self-report methods include job analysis questionnaires or surveys and structured interviews. Guidance is also provided for choosing an appropriate method. These methods are customized to address the unique legal questions associated with each wage and hour issue.

The original version of this chapter was revised. A correction to this chapter can be found at https://doi.org/10.1007/978-3-319-74612-8_10

This chapter describes the underlying foundation for several data collection methods that are commonly used to evaluate wage and hour compliance . The resolution of nearly all wage and hour disputes covered in this book requires detailed measurement of the work employees perform, the amount of time spent on that work, and the context in which work is performed. In the past 15 years, consultants and experts have been successfully applying job analysis techniques to collect data to address these issues.¹ The core methodologies are applicable to multiple wage and hour disputes. However, the general approaches described in this chapter are often customized to address the unique legal questions associated with each wage and hour issue. In later chapters, methodological considerations specific to a particular wage and hour issue are addressed.

As Guion and Highhouse (2006) put it, “fundamentally, all job analysis consists of observing what can be seen and asking questions about what cannot.”² Consistent with that framework, the data collection methodologies discussed in this chapter fall into two general categories: observational methodologies and self-report methodologies. Both approaches are grounded in well-established job analysis techniques.

2.1 Measurement

The concept of measurement plays a significant role in the context of wage and hour litigation. Within the sciences, researchers generally strive to maximize the precision of measurement , as more precise measurements are more useful for making accurate inferences.³ Historically, measurement has served one of two purposes: assign numeric quantities to objects and define whether an object falls within a certain category or group.⁴ Both are applicable within the realm of wage and hour compliance . It is generally accepted that measurement accuracy can have consequences for the quality of decision-making in various areas of Human Resources (HR).⁵ What is unique about the wage and hour context is that even small measurement errors can have major consequences in litigation; that is, small measurement errors can make the difference in the outcome of a lawsuit and potentially millions of dollars in financial liability . Without precise and accurate measurement of relevant factors, the data collected may be unable to answer key questions or, worse, lead to erroneous conclusions.

It is important to draw a distinction between the quality of the data used in a statistical analysis and the quality of the statistical analysis itself. The meaningfulness of results from a statistical analysis is dependent on the quality of the underlying data. Many wage and hour cases, for example, involve extrapolating statistics from a small sample of employees to a large population of employees. Thus, small measurement errors are magnified when they are extrapolated to the population. Issues related to sampling , statistical analysis, and extrapolation are covered in Chap. 8. The focus of this chapter is to provide strategies for gathering valid and reliable data that address key legal questions. The methods described in this chapter were chosen because they tend to produce the most precise measurement of concepts directly related to the legal questions in wage and hour disputes.

2.2 Job Analysis

Knowledge of the work employees perform, time spent performing work, and the context in which it’s performed are critical components in the resolution of most wage and hour disputes. For more than a century, various job analysis techniques have been developed to gather this information in a systematic manner. There are many excellent texts that provide extensive detail on various job analysis techniques.⁶ I do not intend to provide a complete summary of all job analysis techniques. Rather, the discussion below is focused on job analysis methods that are applicable in the wage and hour context.

The term job analysis refers to “a wide variety of systematic procedures for examining, documenting, and drawing inferences about work activities, worker attributes, and work context.”⁷ Job analysis is one of the most commonly used organizational data collection techniques⁸ and can provide a basis for variety of HR applications (e.g., selection, training, performance appraisal) and non-HR applications (e.g., ergonomics, human factors).⁹ In recent decades, job analysis techniques have become commonplace when evaluating many different wage and hour disputes.¹⁰

2.2.1 Toward an FLSA-Relevant Job Analysis

Conducting a job analysis requires many methodological choices throughout the process. There are many different uses for job analysis data, and the purpose of the study drives these methodological decisions. Sackett et al. (2013) highlighted many of these choices including two that are especially relevant in the wage and hour context.

The first is whether the focus of the job analysis is the work performed (“work-oriented” approach) or the attributes required to perform that work (“worker-oriented” approach). Addressing wage and hour disputes typically requires the researcher to take a work-oriented approach such as a task inventory.¹¹ Information about worker attributes, such as knowledge, skills, and abilities (KSAs) required to successfully perform the job, are relevant in many applications (e.g., employment selection) but are typically not applicable to resolve wage and hour disputes.¹²

The second choice is the degree of specificity or generality at which data will be collected. Many job analyses methods are not designed to collect data at a sufficient level of detail to determine compliance with wage and hour laws. An analysis of exemptions from the Fair Labor Standards Act (FLSA), for example, may require a calculation of the percent of time that an individual employee spends performing exempt tasks. A job analysis questionnaire which shows that employees in general perform many exempt tasks “frequently” does not allow this required calculation. In addition, class certification decisions are usually based on the degree of similarity between employees on factors such as the job duties they perform and time spent in performing them. Overly generalized descriptions of work can make dissimilar employees appear similar, while overly specific descriptions of work can make similar employees appear different.¹³

An additional methodological issue when conducting a job analysis for wage and hour purposes is the unit of measurement . Most job analyses have the goal of describing the work performed by a “typical” employee, not an individual employee. This is preferred in many situations, such as when decisions are made based on job title (e.g., selection criteria, performance criteria). Wage and hour cases, on the other hand, typically require decisions to be made at the individual level,¹⁴ creating the need for an individual differences approach to job analysis. It does not matter if employees in the job generally perform exempt tasks, for example; it matters if each employee performs exempt tasks.

Along similar lines, it is important to consider how differences between employees should be treated. There has been debate in the literature regarding whether variability in job analysis ratings represent inaccuracy or meaningful differences.¹⁵ When job analysis data are used in the context of a class certification decision, a critical question is whether meaningful differences exist between employees. Methods that describe a typical employee or treat within-title variability as measurement error do not provide information valuable to this inquiry. The method used must at least acknowledge the possibility that meaningful differences exist between employees and be able to describe the degree to which employees differ. This approach is consistent with recent literature that suggests actual differences in work can be the sources of differences between employees, rather than measurement error.¹⁶

In the following section, I describe many considerations that frequently influence methodological choices when designing and executing a job analysis to study wage and hour issues.

2.3 Choosing an Appropriate Method

One of the most important decisions to make when studying a wage and hour issue is determining the appropriate method for collecting data. The decision will have significant impact on the execution of the study and may impact the legal defensibility of the study results. The methodology should be driven by the goal of collecting valid and reliable data that can address relevant legal questions. Often, more than one method can achieve this goal, and practical or logistical factors such as cost, time, or client preferences are considered. Below, I list several factors that often impact the choice of method:

• The specific violation at issue. Some methods are better suited to capture relevant data for a specific violation than others. In the next few chapters, these potential violations will be described along with the methods that are used to address them.

• Stage in litigation. As described in Chap. 1, class action lawsuits typically go through three phases: class certification , merits/liability, and damages . The legal questions at each stage of litigation differ, and it’s important to design a study that will address relevant questions. Studies that are conducted prior to class certification are typically conducted with the goal of determining the degree of similarity or variability between putative class members. A study conducted after the class has been certified typically has the goal of determining whether violations occurred. These two goals are related but require a slightly different focus that may impact the methodology. In addition, direct contact with incumbents may be prohibited post-certification which can limit some methodological options.

• Type of job. Jobs that involve high complexity or primarily consist of mental tasks are not as well suited for an observational approach.¹⁷ An employee who works on a computer for a large portion of their day, for example, is challenging to observe because it may be difficult to reliably determine what task they are performing at any given time.

• Size of the putative class. The size of the class may impact the amount of data desired. Generally, more data can be collected using self-report methodologies such as questionnaires. When a class consists of a relatively small number of people, an observational approach can gather data from a significant portion of the class members. When the class includes thousands of employees, it can be extremely costly to gather observational data from a large proportion of the class. If this is desirable (in some cases, the portion of class members sampled is not important), self-reported questionnaires are often the preferred option.

• Geographic disparity of class members. Some methods require job analysts to be physically present at the workplace to collect data (e.g., observational approaches), while others do not (e.g., questionnaires, structured interviews) . The geographic disparity can have implications for travel time, travel expenses, and the speed with which data can be collected. All of these factors may play a role in the choice of method.

• Degree to which the job has changed over time. Another consideration is the degree to which the job being studied has changed within the relevant time period (e.g., different processes, different staffing models, reallocation of responsibilities). Observational methods are capable of describing how work is performed currently. For jobs that have not changed significantly, these data are informative about how the job used to be performed. However, for jobs that have undergone significant changes, information about how the job is performed now will be less informative as to how the job was performed before the changes. Self-report approaches are often designed to collect retrospective data and therefore may be able to provide reliable information about how a job was performed in the past.

• Existing company policies and practices. Some companies regularly conduct job analyses observations or administer internal questionnaires and surveys. Employees may have developed a comfort with these approaches, and systems are already in place to communicate about the study and execute the data collection. It may make sense in these situations to use a method familiar to employees to minimize operational disruption.

• Language fluency of incumbents. In some companies, the language ability of the employees from whom data will be collected plays a role in the methodology. Questionnaires require a minimum level of reading ability, and structured interviews require a minimum level of verbal ability. When employees are not fluent in English, these methods are more challenging.¹⁸ Questionnaires can be translated into other languages, but this can introduce new challenges. Professional translations might be problematic in some companies because employees tend to use informal terminology to refer to various work processes. This is often the case in the restaurant industry, for example. In some companies, employees have low levels of language ability, regardless of their native language, which requires self-report instruments to be designed with low complexity.

The appropriate method is often based on these and many other factors unique to the company. In the following sections, I describe the methods that are typically used to collect data relevant to wage and hour compliance .

2.4 Observational Approaches

One of the most commonly used methods to collect data to evaluate wage and hour compliance is through direct observation. This method is a systematic process in which a job analyst directly observes incumbents performing their work and documents detailed information about that work such as the tasks performed and the duration of tasks. Direct observation is a well-accepted technique to learn about the work employees perform, especially for jobs that involve physical or otherwise observable work.¹⁹

There are different types of observation methodologies that are applicable to wage and hour cases. Broadly, these can be categorized as either “live” observations or video observations. Live observations require a job analyst to be physically present to observe and record tasks performed and time spent on tasks. Video observations involve analyzing video recordings of employees performing work. There are advantages associated with each type of observation. The choice of observational approach is typically driven by the specific wage and hour issue and therefore the relevant legal questions. For example, live observations are well suited for situations that require detailed information about what an employee does on the job including the sequence of tasks, the content of conversations, and the work context. Alternatively, video observations are well suited for situations that require precise timing of employee movements within the same physical location. Each is described in more detail in later sections.

2.4.1 Sampling Considerations

Observations of all varieties are time-intensive and costly, thus making sampling necessary. In addition to the factors that are used to determine who should be observed,²⁰ observation studies may also require attention to when they should be observed, such as day of the week and shift time. As an example, consider an observation study of managers at a chain of restaurants. At most restaurants, weekends tend to have more customers (i.e., more tasks related to customer service), and midweek days tend to have fewer customers (i.e., more administrative tasks). In addition, morning shifts tend to involve different tasks (e.g., setting up tills, receiving deliveries, preparing the bank deposit, checking food temperatures) than a mid or closing shift (e.g., cashing out servers, inspecting side work, and completing accounting reports). Because restaurants typically differ in customer flow across days of the week and times of the day, and therefore require employees to perform different tasks across days of the week, all of these factors should be reflected in the sample of restaurants observed. In general, objective differences in the sample observed (e.g., shifts, days of the week) should mirror the range of differences found in the population . Disproportionality in observed days or shifts may result in a biased view of how the job is performed.

2.4.2 Live Observations

Live observations capture a detailed description of a “day in the life” of incumbents by adapting time and motion methods that have been used since the 1890s.²¹ Time and motion methods were initially developed to determine the time required to perform a repetitive task such as assembling a part. However, time and motion methods adapted for wage and hour compliance have some key differences. Although both involve an observer tracking the duration of tasks, the goal of a wage and hour observation study is to describe what work an employee performs across an entire day or workweek, as opposed to describing how much time it takes for a group of employees to perform a single task or set of tasks. This technique may also be called the continuous clock, continuous workday, or continuous observation²² method. Full-day observations almost always result in the description of unique tasks and time transitioning from one task to the next (e.g., walking to the office to get a report), information that would not be included in a traditional time and motion study. The adapted time and motion method is now regularly used to capture all the tasks performed by a single employee and the duration of each task across a fixed period of time.

Usually this technique involves one observer being assigned to “shadow” a single employee and track the tasks they perform either for an entire workday or some portion of a workday. However, there may be some circumstances where multiple observers are used simultaneously or one observer can observe multiple employees simultaneously. For example, in a large facility, multiple observers can be used to avoid observers following employees throughout the facility. In other circumstances, a single observer may be able to reliably observe multiple employees such as when only a small number of tasks are being tracked which always occur in the same location (e.g., clocking in/out, donning, and doffing) .

Through observation, observers are able to capture highly detailed descriptions of the work incumbents perform and the amount of time spent performing categories of tasks (e.g., exempt vs. non-exempt tasks) . Observers follow the employee wherever they go during the shift. Observers are also close enough to the employee to capture detail regarding the tasks performed such as the reports being reviewed or what is being said to other employees. Without that level of detail, coding a task into legally relevant categories (e.g., exempt or non-exempt) is challenging. Observers also capture important contextual information because they can see and hear what is going on around them which may be important for properly interpreting the task performed and, thus, the proper coding of that task. Observers also ask clarifying and probing questions when it is necessary for understanding what the incumbent is doing. However, interaction with the incumbent is minimized to avoid influence the observer may have on the work the incumbent performs. Therefore, observers interact with the incumbent only when it is crucial for properly understanding the work an incumbent is performing. The key steps of a live observation study are listed in Table 2.1, and some are described in more detail in the next section:

Table 2.1 Steps in a typical observation study

Communication Process

In most observation studies, particularly when the observer is in close proximity to the employee for an extended period of time (such as when conducting a full-shift observation), it is advisable to inform the employees selected for observation about the study in advance using a structured communication plan. A structured communication plan is helpful for notifying key employees about the study and for ensuring standardization of information received by those being observed. Formally scripted communication can help to avoid incumbents speculating about the reasons for, or implications of, the study due to the absence of complete information about the study. The value of the data is dependent on the job analyst’s ability to observe the incumbent’s behavior on the job as it is normally performed; a properly scripted communication plan helps ensure that this occurs. In particular, it should be clearly communicated to those being observed that their performance is not being evaluated and that they should perform their job normally during the observation. This helps to minimize the likelihood that incumbents will purposely distort their behavior during the observation to project a favorable image. This message is most impactful when it is repeated multiple times by several company representatives and especially by their direct supervisor and the job analyst.

Though generally recommended, there could be situations in which prior communication with study participants may not be necessary or appropriate. In a study where job analysts observe employees from a distance and do not interact with them, the absence of formal notification might not impact the ability to collect reliable data. In other situations, employees could be aware of the active litigation and may have a desire to purposely changing their behavior to manipulate the data and influence the outcome of a lawsuit (which could have a direct benefit to them financially). The decision not to formally notify employees about the study in advance may be advisable in some circumstances to maximize the reliability of the data.

Conducting an Observation

The observation requires a trained job analyst to observe an employee for a pre-determined period of time, often an entire workday which can last 8 or more hours. The observer records every task the employee performs along with the start and stop time of each task. Despite technological advancements such as software that allows data to be recorded electronically using a mobile device or tablet, there are distinct advantages to recording data using an old-fashioned pen and paper. Most important is the ability to record detailed task statements that describe exactly what the job analyst observes—information that cannot be pre-programmed into electronic devices. Recording data using smartphones or tablets may enable greater precision in the time stamping of tasks, but the use of these technologies generally requires observers to report what tasks are performed by selecting from a pre-defined list, thus not allowing the observer to report precisely what was observed. The trade-offs are an important consideration when deciding which method will generate the most useful data. Table 2.2 contains portion of an example observation record for a restaurant manager.

Table 2.2 Example observation record

The level of desired precision in the timing of tasks in an observation study must also be determined. This largely depends on the issue being studied. When studying FLSA exemptions (such as the example in Table 2.2), observations typically last 8 or more hours. For this issue, recording data in 10-s increments is common and usually provides a sufficient level of detail to answer relevant legal questions. Recording data in 10-s increments (e.g., as opposed to 1-s increments) enables the observer to capture a greater level of detail in the tasks performed, which is often important for purposes of classifying tasks. In other circumstances, such as a study of off the clock work , greater precision in timing may be preferred over more detailed task statements. In some off the clock cases, the amount of time in dispute may only be a min or less, and it may be desirable to record activities to the nearest second. Greater precision in measurement is usually at the expense of detail about the tasks performed, so it’s important to consider these options when determining how to collect data that will answer relevant legal questions.

Coding and Analyzing Observation Data

For most wage and hour issues, the observed data is coded to generate meaningful statistical results. For an exemption analysis, observed tasks are coded into exempt or non-exempt groups, yielding a total time spent for exempt and non-exempt work. To determine how much time is spent performing work off the clock, tasks observed before clock-in or after clock-out are coded as compensable or non-compensable.

Coding observation data should follow a clear and systematic approach to maximize reliability . A written protocol to guide data coding is useful in many studies to standardize the process. This often includes guidelines that clarify potential ambiguities. Coding is typically completed by multiple independent coders to minimize coding errors, increase standardization, and evaluate the reliability of the coding. Once the observation coding is finalized, an observation record is generated for each incumbent, and this record contains all tasks performed, the duration of each task, and the coding of all tasks. Coded data can be analyzed to generate numeric estimates that address relevant criteria.

Controlling for the Hawthorne Effect

A potential concern when conducting an observational study is the well-known psychological phenomenon called the Hawthorne effect ²³ or, alternatively, the Heisenberg effect.²⁴ That is, without proper controls, the observation itself may influence the employee to alter his or her behavior. It is advisable to implement controls to minimize if not entirely eliminate these potential effects. First, as noted, it is helpful to communicate to employees that their performance is not being evaluated, and they are expected to perform their job as they normally would. It may also be useful to ask incumbents at the end of the observation whether they would have done anything different if they hadn’t been observed. In my experience, very few incumbents are able to identify anything they would have done differently. Second, observers should minimize interactions with the incumbent and to stay out of the incumbent’s line of sight as much as possible. By implementing these controls, incumbents generally habituate to the observation and go about their typical job duties .

Advantages and Limitations

One of the primary advantages of an observation study is that it results in a record of work performed that is extremely rich in detail. Some attorneys and judges find data collected using this method particularly persuasive because it paints a very clear picture of what employees actually do. The method also does not rely on the memory or language ability of incumbents to gather reliable and valid data. Moreover, it is difficult for a motivated employee to purposefully distort the data resulting from an observation study. This is because it would require the employee to make significant changes to their behavior while working with other employees and being expected to accomplish work tasks. Also, this method of data collection does not take employees away from their jobs, a feature that is very important to operations managers from a cost perspective.

There are a few limitations associated with observations. Observations provide a “snapshot” of the work an incumbent performs at one point in time. To the extent that the job an incumbent performs changes significantly over time, the observation record may not be generalizable to all other periods of time. Another limitation is that an observer can only record tasks that can be observed and cannot record most mental tasks. Observation studies in the context of exemptions tend to underestimate the amount of exempt time as a result of this limitation, as most mental tasks are considered exempt. Still another limitation is that this method is focused on tasks performed on the job and does not indicate directly the role of the incumbent in hiring, firing, or exercising discretion on the job—other criteria that would be important to know in evaluating exemption status. Observational data alone are unlikely to be sufficient to establish whether employees do have this authority.

2.4.3 Video Observation

Observational data can also be collected using video technology. This method involves capturing video of employees performing work, coding the activites performed, and analyzing the data. Video observations have several advantages. Video observations can be used to capture data regarding employee tasks and activities over a designated period of time. Two contexts where this method is particularly useful are determining meal and rest break compliance and occurrence of off the clock work .

Video data also can be collected to precisely measure the duration of certain activities. This information is especially useful when determining the amount of time that employees spent donning or doffing uniforms and personal protective equipment (PPE) . This is important when time data are required to resolve whether the amount of paid compensated time allocated by the company is sufficient to cover the actual time it takes to don and doff uniforms and PPE. Data can be collected from many employees simultaneously to capture the range of time it takes to don and doff, giving the court the information it needs to make a decision about the occurrence and duration of off the clock work and whether the amount of uncompensated time is de minimis .

Depending on the physical layout of the work location, a small number of cameras could capture the movements of many or all employees. After the data are captured, it must be reviewed, coded, and then analyzed. The coding process is made easier by coders’ ability to review the video as many times as needed for coding accuracy. Multiple coders can also code the same video to verify the reliability of data coding.

There are practical advantages to video observations. Unlike live observation, the costs associated with collecting additional data are minimal. Once the video cameras are purchased and installed, no other significant costs will be incurred by letting the cameras run over time. This is an advantage when a large amount of data is desired. The marginal costs associated with additional coding time are minimal as coders can be deployed relatively inexpensively.

There are also limitations to video observations that should be considered. Most importantly, data can be collected only when employees are in the cameras’ view; when employees move outside of the cameras’ view, data are lost. Even when employees are within view of the camera, it may be difficult to record much detail about what work they are performing. For example, it is difficult to tell what information an employee is reading (e.g., sales report or personal email) or the content of their conversations (e.g., taking a customer’s order or evaluating customer service)—both of which may be important pieces of information, depending on the issue being studied. In addition, privacy laws in some states may restrict the use of video and audio recording in the workplace by placing restrictions which prevent the placement of cameras in specific locations (e.g., changing rooms) or require the posting of a notification that the area is under video surveillance. All of these limitations should be considered when applying this method as it may compromise one’s ability to collect crucial data.

2.5 Self-Report Approaches

Asking employees or other knowledgeable persons to self-report their work experiences can be an effective way to collect reliable data relevant to wage and hour disputes. There are two broad methodological approaches described in this section: (1) job analysis questionnaires and (2) structured interviews . Both of these approaches involve employees self-reporting their work experiences but in a different format. Questionnaires allow employees to report their experiences nonverbally (hardcopy or online), whereas structured interviews allow employees to report their experiences verbally. Each approach is described in more detail in later sections.

A particularly useful reference to consult when collecting self-report data in the context of litigation is the Reference Manual on Scientific Evidence (2011), currently on its third edition.²⁵ The chapter titled “Reference Guide on Survey Research” offers detailed guidance for conducting a study through self-report in a legal setting.²⁶ The chapter covers several broad topics including the purpose and design of the study, population definition and sampling, questions and structure, use of interviewers, data entry, and reporting. The chapter combines perspectives from social sciences and law to provide specific guidance on the design of a self-report instrument that is legally defensible. Many of these recommendations are incorporated throughout this section.

2.5.1 Biases and Limitations in Retrospective Reports

An underlying assumption of all self-report methods is that participants will accurately recall and report past events. However, research has shown that in certain situations, one’s ability to accurately report retrospective information is limited.²⁷ In order to collect meaningful data, these limitations should be addressed to maximize data quality . In many cases, issues related to memory can be minimized through the design of the study. There is a vast body of research literature on the topic of human memory as it relates to one’s ability to accurately self-report information.²⁸ I do not intend to cover that body of research here. However, one of the most useful practical implications from that literature to enhance accurate recall comes from research on the event history calendar.²⁹ The event history calendar taps into the hierarchical nature of human memory by tying less memorable events to more memorable events that occurred around the same time. For example, an event history calendar exercise can be inserted at the beginning of a questionnaire to improve memory for the relevant time period. The calendar can include well-known events that occurred during specific years, and participants can be asked to recall memorable events in their personal lives during the relevant time frame. Many studies have demonstrated that exercises like that can increases the accuracy of self-report data.³⁰ In addition, strategies such as reducing the referenced time frame and breaking questions down to their constituent parts, called “decomposition ,”³¹ have also been proposed as strategies for overcoming memory limitations and increasing accuracy.³²

In addition, several studies suggest that survey participants tend to overestimate the amount of time they spent performing work tasks. For example, researchers have compared the number of hours employees reported working per workweek to employer reports³³, time records³⁴, and time diaries³⁵. Each study found that employees tend to overestimate the number of hours worked per week. The magnitude of the error was larger for professional and managerial employees and for employees who reported working more total hours. These findings are significant in the wage and hour context because, often, the amount of time spent performing certain tasks is a critical issue in the dispute. One way to avoid this error is to ask participants to report relative time spent performing tasks, rather than absolute time. That is, in an exemption study, participants can report the percent of time they spent performing groups of tasks, rather than the actual number of hours. Percentage estimates are unlikely to suffer from the same bias because percentages cannot be consistently overestimated and still sum to 100%. In other circumstances, estimates of absolute time are necessary. When possible, verifying self-reports of absolute time with external data will strengthen the amount of confidence in the accuracy of the data.

2.5.2 Confidentiality and Anonymity

Study designs may differ based on whether data will be collected confidentially or anonymously. Many survey participants do not recognize a difference between these two terms, so it may be useful to clearly describe how their identities will be protected. Anonymity means that the identity of the participant is not known to the person collecting data. Even if the researcher wanted, they would not be able to determine who provided the response. In contrast, confidentiality means that the identity of each participant is known to the researcher, but they do not disclose that information. Confidentiality is often protected by taking precautions such as reporting results in aggregate, not sharing individual responses with the employee’s supervisor, securely storing data, and using numeric codes rather than participant names to conceal identities. In either case, the researcher is responsible for protecting the identities of participants, especially when revealing their identity would harm them in some way.³⁶

The choice between the two forms of participant protection can have important consequences when study results are involved in litigation and should be discussed with clients to make an informed decision. Although anonymous surveys may increase honesty and response rates, confidentiality is typically preferred in the wage and hour context when possible. One reason is that circumstances in the litigation may change, creating an unanticipated need to identify certain individuals in the sample. As an example, suppose that data are collected anonymously from a group of employees in response to a nationwide lawsuit. After data are collected, the class definition is revised to only include employees in Texas. Data collected from employees in all other states are no longer relevant to the case and should be excluded . However, the results cannot be updated to reflect the response of Texas employees only because the data were collected anonymously. A potential solution in this example would have been asking employees to report the state in which they work as part of the study. However, it is not always possible to anticipate which factors will become relevant to the litigation prior to the study. Similarly, any factors used in sample selection would need to be included in an anonymous survey . Without this information, it is not possible to know whether the sample is representative when the response rate is less than 100%. This may add length and time to the survey.

In some circumstances, self-report data are required to determine the amount of economic damages owed to individuals. In this situation, class members’ self-reports (e.g., hours worked per week) can result in personally receiving large sums of money. Anonymity may exacerbate this bias because participants are not accountable for their responses³⁷.

Another factor to consider is whether an anonymous survey is acceptable to the court. Researchers are generally expected to produce all questionnaires so that the opposing party has an opportunity to evaluate the raw data. Conflicts often arise when one party also requests the names of those who provided data in the study, a request that researchers often resist.³⁸ This presents a difficult dilemma for the researcher to which there is not really a satisfactory solution. Though promising confidentiality to participants is common, those promises cannot prevent a lawful inquiry.³⁹ It is possible that a judge could require the names to be provided or rule the entire study inadmissible. At the same time, the use of the study in litigation does not relieve the researcher of their responsibility to participants.⁴⁰ As a practical matter, the names of participants are often provided to both parties in the lawsuit, but not made publicly available. However, instances in which disclosing participant identities raises legitimate concerns about harm to the participants may justify a decision not to disclose names.⁴¹

2.5.3 Threats to Data Quality

The quality of the data and the accuracy of inferences made from the data are dependent on individuals’ ability and willingness to provide accurate information. There are several ways in which data quality can be compromised and precautions should be taken to eliminate or minimize these issues. In this section, I discuss some of the most common threats to data quality in collecting self-report data in the wage and hour context along with potential strategies to address them.

The first threat is purposeful distorting of data by participants. Employees may be motivated to provide false job analysis data for many reasons, but in the context of litigation, one concern is that participants have a desire to influence the outcome of the lawsuit. Participants directly involved in the litigation might have financial incentive to provide data that favors plaintiffs. Employees with strong loyalty to the company may be motivated to provide data that favors defendants. Purposeful distortion is only possible when (a) participants are aware of litigation, (b) participants are willing to provide false data, and (c) participants know enough about the legal issues in the case to be able to distort data in one direction. It is useful to investigate whether there is a general awareness about the litigation among employees early in the project. In larger companies, it is rare that a typical employee is aware of the litigation, making this issue moot. In addition, the typical employee may not know the legal issues at a level of detail that would enable them to distort data, even if they were motivated to do so. Nonetheless, it is wise to implement controls into the study design to detect and eliminate this issue. One strategy is to embed “lie items” (sometimes called “distractor items” or “validity checks”) into the questionnaire . These are items, typically multiple choice, in which the correct answer is known to the researcher in advance. An example may be, “How frequently do you review and analyze the Red Report?” In this example, the Red Report does not exist in this company, so participants who report performing this task frequently may be providing false data. It is advisable to include multiple lie items throughout the questionnaire, around 4–6 depending on the length of the questionnaire. Of course, it’s critical to ensure the answers are known in advance and will not be misinterpreted by participants. Using the previous example, store managers whose weekly performance scorecard appears in red text if they are not meeting their targets may think the “Red Report” is another term for the scorecard and truthfully report that they perform that task.

Participants may also provide false information, not because they are motivated to distort the results, but because they are not motivated to devote sufficient attention to the questionnaire , called “satisficing.”⁴² This may occur to varying degrees, based on the effort required by the participant, the participant’s ability, and their motivation.⁴³ These participants may respond carelessly or randomly to complete the task as quickly as possible. Lie items are also an effective way to identify these individuals. In addition, some participants simply select the same rating for all questions to complete the questionnaire as quickly as possible. Reponses can also be analyzed to determine the frequency with which the same rating was selected. If 250 tasks are all rated on a 7-point Likert scale and an employee selects a rating of 5 on more than 95% of the tasks, that suggests the employee may not be responding in a careful manner which could justify removing their responses from the dataset.

Another threat to data quality is the language ability, specifically reading comprehension, of the participants. This is more likely to be an issue when studying an issue that impacts non-exempt employees such as off the clock work or meal and rest break compliance because reading comprehension is often required to be an effective management employee. For example, plaintiffs in a lawsuit against a fast food chain in California alleged that non-exempt employees (e.g., cooks, cashiers) were not provided meal and rest breaks. The majority of this population had limited reading comprehension, especially in English. To preserve data quality, the questionnaire was translated to Spanish, and the wording of both version was simplified to maximize comprehension. Prior to administration, feedback from employees confirmed the revised questions were clear and understandable. In the end, a questionnaire was administered that was clear to participants and still able to directly address the legal issues.

There are a variety of sources available that offer guidance on survey design.⁴⁴ This literature covers far more topics than can be included here. Most important is that questions are clear, unbiased, and not leading⁴⁵ and properly address a relevant legal question. For example, consider a survey intended to assess meal break compliance in California. The survey includes the following question: “Do you take a meal break on every shift that you work.” An affirmative response seems to indicate compliance, while a negative response seems to indicate non-compliance. However, the question wording does not properly address the legal questions for at least two reasons. First, employees may only be only eligible for a meal break if they work five or more hours. An employee who works less than 5 h would respond negatively even though no violations have occurred. Second, compliant meal breaks often must be at least 30 consecutive min. An employee who routinely takes breaks less than 30 min or is interrupted during their breaks would respond affirmatively to the question, even though their meal breaks on non-compliant. This example demonstrates how the questions themselves can compromise the ability to make accurate inferences about relevant legal questions.

2.6 Job Analysis Questionnaires

Job analysis questionnaires are used to collect systematic self-report data about tasks and activities performed, individual attributes required to perform those tasks, working environment, and other characteristics about the job or the workers performing the job.⁴⁶ This method offers several advantages. First, a large amount of data can be collected more quickly when compared to other methods like observations. Also, information can be collected retrospectively. That is, employees can report their work experiences currently but also can provide valuable information about prior work experiences. This is an important feature when the study results may be used in litigation because the relevant time period typically goes back several years. Another advantage is that information can be collected about non-observable features of a job such as role in decision-making processes, reasons for performing certain work, or company policies and procedures.

2.6.1 Administration Method

Job analysis in this context is usually administered using hardcopy questionnaires or online. Surveys in other contexts may also administered by phone.⁴⁷ However, for most wage and hour issues, the type of detailed information being elicited in the questionnaires requires the employee to view the instructions and the questions and refer to definitions and examples as needed. An interview that requires assistance from visual materials is not feasible by phone.⁴⁸

Hardcopy surveys are printed and participants complete them by hand (also called “paper and pencil”). Questionnaires can either be mailed or hand-delivered to individual employees, or employees can meet at a centralized meeting location to complete the questionnaire . The latter offers some important advantages when the study results may be used in litigation. An in-person administration can be closely controlled by a proctor. Proctors receive training along with a written protocol and scripts to standardize their behavior during administration. Part of the process can include reading all instructions aloud to participants and demonstrating how to complete each section. Proctors are also able to answer questions that participants have about the questionnaire. This process is time-consuming but ensures that all participants in all locations have received the proper instructions and understand how to complete the questionnaire.

Alternatively, administering surveys online has become increasingly common. There are various online survey vendors that have made the task of programing an online survey relatively simple. Even surveys that require branching and logic to determine which questions are asked can be implemented fairly easily. There are several advantages to administering the questionnaire online. First, it is much easier to collect data from a large number of employees using this method. The survey URL can be emailed to employees in any location with internet access. Another advantage is that all data are electronic and no data entry is required. This is not only faster and less expensive but also eliminates the possibility that data entry errors introduce unreliability into the dataset. In addition, questions or response options can be randomized to avoid potential order effects.⁴⁹ However, participants for an online administration are perhaps less likely to read all instructions and less likely to put forth the effort to contact the researcher if they have a question. A person completing an online questionnaire may also be more likely to multitask rather than devoting their full attention to the questionnaire. Online questionnaires are only an option when employees have access to computers with internet access and a minimum level of comfort using computers.

When implementing an online questionnaire for current employees, it is advisable to work with the internal IT department to ensure that employees will have access to the survey from their work computers. Many companies, especially larger companies, have network security protections that may prevent employees from accessing an online survey. It’s recommended to address these issues early in the process as they may take time to resolve.

Questionnaires relevant to most wage and hour issues tend to be very detailed and may require a significant time commitment to complete them. It is not uncommon for questions to take 1–2 h for employees to complete them. Typically the questionnaire includes multiple sections, each designed to address different components of the legal issue. For example, a questionnaire designed to study an FLSA exemption will usually include a section measuring relative time spent on individual tasks, a section measuring percent of time spent on groups of exempt or non-exempt tasks, and a section measuring decision-making authority. Other sections may also be included, such as demographics to the extent they are relevant.

2.6.2 Job Analysis Questionnaire Development

The process to design and execute a job analysis questionnaire contains several broad steps, and some modifications to the process could be necessary depending on the specifics of the company or the issue being studied. Table 2.3 provides an outline of typical steps.

Table 2.3 Steps in a typical questionnaire study

A preliminary step in most job analyses is a review of existing company materials. In some companies, it is also helpful to talk to SMEs and physically see the workplace and observe work being performed. This background research is helpful for designing a questionnaire that asks appropriate questions and uses appropriate terminology that employees will clearly understand and will interpret consistently. For example, some restaurants refer to their customers as “guests” and employees as “team members.” Using these terms in the questionnaire will help ensure that participants are able to understand the questions.

The next step is to prepare the questions. The content of the questions will vary greatly not only based on the legal issue being studied but also on the company. When studying whether employees are exempt from the FLSA (see Chap. 3), the “questions” often include a task list with an instruction for the participant to indicate the amount of time spent on each task. When studying independent contractor classification (see Chap. 4), questions would likely ask about the forms of control exerted by the company over different aspects of the work. Depending on the industry and the company, the way in which control is exerted varies, and questions should be crafted to be most relevant to the participant.

Because of the detailed nature of these questionnaires, it is important that instructions are sufficient for employees to clearly understand what they are being asked and how they are supposed to respond. It is often helpful to also provide definitions of key terms when there is potential that they could be misinterpreted.

An important step in a high-quality survey is pre-testing or “pilot” testing.⁵⁰ This step helps identify any issues that exist prior to the full administration, such as questions that are confusing, instructions that are not clear, or response options that are inadequate. For online surveys, the pilot test may also uncover technical issues that prevent employees from the completing the survey. Usually, a small sample of employees are selected to participate in the pilot. My experience has been that a sample of 6–10 employees who are diverse with respect to key factors such as geography, location type, sales volume, etc. is sufficient to gather quality pilot data.⁵¹ The pilot participants are told they are participating in the pilot program and will be asked for their feedback after they complete the questionnaire. For an in-person questionnaire , a pilot administration is completed, and participants can provide feedback in a group discussion. For an online questionnaire, a job analyst contacts each participant, often by phone, to collect their feedback on many different aspects of the questionnaire. It is useful to prepare a list of questions in advance to guide these feedback discussions and ensure all aspects are covered. To maximize the utility of the pilot, the pilot version of the questionnaire should be as close as possible to the actual administration.

Based on feedback from pilot participants (or others who have reviewed the questionnaires), it is common that some changes are necessary, which often include rewording questions and adding or clarifying instructions. Not every suggestion received from the pilot should be accepted. Participants generally do not have expertise in the legal issues being investigated or survey design. Additionally, one participant’s feedback may conflict with others. Incorporating feedback requires professional judgment to balance competing needs and ensure the final survey is able to accomplish its intended goal.

Details regarding the sample selection process are discussed Chap. 8. Please refer to that chapter for a discussion of this process. Once the sample is selected, the selected group must be notified about the study. Similar to the process used in an observation study, a prescripted communication plan is helpful for this process. This ensures that the correct people are notified about the study and that each party receives complete, accurate, and standardized information prior to participation.

Once participants are notified, the administration can begin. When using an online platform, administration effort is minimal as it consists primarily of providing the URL to participants. The trade-off is the loss in control over the administration such as whether participants are reading the instructions or multitasking while completing the questionnaire . Most of the effort in administration of an online questionnaire are addressing participant’s questions or technical issues. In addition, the administration often occurs over a period of time which may require regularly tracking participation rates and following up with those who have not completed a questionnaire . Participation rates tend to spike when the administration begins and right before the deadline. Participation can often be increased by setting an artificial deadline (e.g., 2 weeks) and then extending it a week at a time until participation rates are acceptable.⁵²

The final step in the process is to analyze and summarize data. The analysis of questionnaire data mostly involves descriptive statistics and frequency counts. In addition, measures of reliability and validity are important to demonstrate the quality of the data. This may involve calculating statistics such as Cronbach’s alpha to test the internal consistency of responses,⁵³ calculating similarity in responses to similar items or groups of items, or comparing responses to external data (e.g., comparing self-reported start time to timeclock data).

2.7 Structured Interviews

The second self-report approach that can applied in wage and hour cases is structured interviews. Interviews are a frequently used job analysis method⁵⁴ and have recently begun to be applied to address wage and hour issues. A structured interview allows a researcher to systematically collect employees’ verbal reports of their work at a high level of detail. This is useful when studying jobs that are highly technical (e.g., silicon chip development, financial analysis) or involve tasks that are complex and vary widely person to person. Structured interview methods involve asking the same set of questions in a specific order to a group of participants. Typical self-report questionnaires preload questions about the work performed; a structured interview may better reflect the potentially large range of tasks involved and skill sets required to perform such complex work.

The same foundations of job analysis practice are used to form the basis of the structured interview. However, unlike questionnaire which typically contains mostly closed-ended (i.e., fixed-scale) questions, the structured interview contains mostly open-ended questions. Open-ended questions have several notable advantages⁵⁵ such as giving the employee an opportunity to provide unlimited detail about their work to provide rich, in-depth information about tasks performed that otherwise would not be captured by standardized, fixed-format methods, thus enabling the job analyst to capture each employee’s job uniquely and precisely. These types of interviews can result in interesting and illustrative examples of different scenarios and circumstances. In addition, follow-up questions can be built into the tool to capture the drivers that lead to different employee behaviors. Some interviews may incorporate both open- and closed-ended questions which can expedite the interview and provide numeric data to analyze.

It may be easier to collect data that directly addresses certain legal questions using this method. Evaluation of the administrative and professional exemptions or employment status , for example, often requires contextual information about the work performed, such as the purpose that tasks are performed, the impact of the work on the company’s business operations, and the specific KSAOs required to perform the job effectively. This information is often easier to communicate in an interview because of the open-ended format and the ability to ask probing questions to clarify responses.

Given the large number and detailed nature of questions , structured interviews can take a significant amount of time to execute. Depending on the environment, this time requirement can limit the number of employees who can be interviewed.

2.7.1 Elements of Structure

Interviewing is a commonly used technique is various areas of HR management. Perhaps the most widely studied application of interviewing is for the purpose of employment selection.⁵⁶ The literature on employment selection interviewing places a strong emphasis on the degree to which the interview is “structured.” Interview structure can be loosely defined by the degree to which the process is standardized. For example, Campion, Palmer, and Campion (1997) list a variety of factors that increase structure in an employment selection interview. Many of these features apply in this context as well, such as asking the same questions, limiting prompting, and controlling ancillary information.

Generally, structured interviews are preferred to unstructured interviews⁵⁷ as they are consistently shown to increase the quality of the data and inferences that can be made from the data.⁵⁸ Many different aspects of the interview process can be standardized including the questions asked, the visual materials shown, and the method used. In addition, written guidelines and interviewer training can increase standardization of interviewer behavior during the interview.⁵⁹ Written scripts including responses to frequently asked questions can be used to ensure all interviewers provide the same information to participants. Guidelines for the interviewers’ appearance (when in person), attitude, and demeanor can also be standardized to the extent possible as these factors may also impact responses.⁶⁰ Interviewer guidelines for asking follow-up or “probing” questions during the interview may also be important.⁶¹ Participants may provide responses to open-ended questions that are unclear, are incomplete, or do not actually answer the question. In these cases, it is appropriate for the interviewer to follow up to get additional information. Providing written guidelines improves structure by ensuring that interviewers are asking follow-up questions at the appropriate time and asking appropriate questions.

Standardizing the process serves two primary purposes. First, it allows meaningful comparisons to be made across employees. This is often necessary when conducting a study before a class is certified.⁶² At this stage in litigation, the primary legal question is related to the degree of similarity or variability between putative class members. Standardizing the interview process helps rule out the possibility that the interview itself resulted in different responses, as opposed to actual difference between employees. Second, standardizing interviewer behaviors minimizes the possibility of biasing the results by “leading” participants to a particular response. This is a common critique from attorneys when the study is used in litigation and documenting the process is one way to demonstrate the validity of the data.

2.7.2 Documenting Interview Responses

The manner in which interview responses are documented can play an important role in the legal defensibility of a study. In particular, the degree to which interview responses are summarized should be carefully considered. Responses that are overly summarized are more vulnerable to critiques that study results are biased by the interviewer’s interpretation of the actual responses. To avoid this concern, two features can be included in the interview design. First, responses should be recorded as close to verbatim as possible. Second, interviews can be designed to allow the participant to review and edit all answers recorded by the job analyst. This allows the researcher to verify that answers are recorded accurately and in the interviewee’s own words. This can be accomplished by conducting the interview in person or using online meeting software that allows the interviewee to view and comment on the job analyst’s recorded answers in real time. Computerized data collection during survey interviews has become a common practice⁶³ and falls within the definition of what has been labeled as a computer-assisted interview (CAI) or computer-assisted telephone interview (CATI).⁶⁴ Using this platform avoids problems with opposing parties’ objections to the data, alleging that answers were not recorded accurately. The questions can be presented, and the data can be recorded using a variety of applications, including customized software, online survey tools, or even widely used applications such as Microsoft Excel or Word. Generally, using an application that produces a database with all interview responses is preferred because data in this form can be filtered, categorized, and analyzed much more quickly and efficiently. These applications sometimes require a time investment on the front end, but this is usually outweighed by the time savings on the back end.

2.7.3 Analyzing Interview Data

Responses to open-ended questions can be detailed and lengthy. Depending on the number of interviews conducted, the amount of interview data can be substantial. Whereas the amount of detail is one of the primary advantages of this method, the data typically need to be summarized in order to be communicated to a client or the court. This can be accomplished using a content analysis approach, a technique for extracting quantitative data from qualitative data.⁶⁵

Generally, a content analysis involves identifying relevant information within the detailed responses and assigning numeric codes. The coding process should be executed in a way that produces high reliability , typically measured by interrater agreement. Training raters is one way to increase rating reliability by ensuring that each rater has a consistent understanding of the responses they are coding , the coding scheme and rules, and how to assign and record their codes. Training usually includes a calibration exercise to ensure that raters are applying consistent codes. All raters review and code the same responses which are then compared and discussed. This process can be repeated until all raters are assigning consistent codes.

To code the interview responses, multiple raters are typically assigned to independently code each response. After both coders have completed their coding , the codes can be compared and interrater reliability can be calculated. There is no widespread agreement on the minimum level of acceptable agreement. For coding schemes that are less complex, greater than 90% agreement is usually a reasonable expectation. For more complex coding schemes, agreement of 70% or higher may be acceptable. When there is a coding discrepancy between the two raters, a structured process can be used to resolve them. There are several strategies for resolving coding discrepancies. The two raters can have a discussion to reach a consensus code, or a third rater could be used to resolve the discrepancy. In reality, the method of resolving discrepancies usually has a minimal impact on the study results, especially when interrater agreement is high. This is consistent with research on similar rating schemes.⁶⁶ Of course, the actual responses are preserved and will not be impacted by the coding.

2.8 Conclusion

This chapter provides the foundations for several data collection methods that are commonly used to evaluate wage and hour disputes. The basic methods are typically customized to address specific legal issues, and discussions about what modifications are necessary are described in later chapters. In addition to the type of issue being studied, methods are typically customized to best fit a particular industry, company, and job included in the study.