1 Introduction

Crowdsourcing—the employment of ad hoc online labor to perform various tasks—has become a popular outsourcing vehicle. Digital platforms like Mechanical Turk (http://www.mturk.com), CrowdFlower (http://www.crowdflower.com), MobileWorks (http://www.mobileworks.com), and Crowdcrafting (http://crowdcrafting.org) are in part responsible for the emergence and popularity of crowdsourcing. These popular platforms have been dominated by micro-tasks—standalone decomposed tasks (Schmitz and Lykourentzou 2018). This arrangement—micro-tasking through digital platforms—has been successful at providing organizations with access to affordable labor available 24 h a day (Ye et al. 2017).

Our current approach to crowdsourcing—focusing on micro-tasks—fails, however, to leverage the potential of crowds to tackle more complex problems. Addressing complex problems requires collaboration among individuals who hold multiple perspectives and diverse expertise. Crowdsourcing affords the opportunity to assemble individuals with a diversity of knowledge and skills that is not often available to a single individual or organization. However, employing this collective knowledge to tackle complex problems requires the shift from standalone micro-tasking to more collaborative macro-tasking. Macro-tasks are complex crowd work that is sometimes but not always decomposable to micro-tasks and requires collaboration among crowd workers to accomplish (Schmitz and Lykourentzou 2018).

Crowdsourcing controls are mechanisms used to align crowd workers’ actions with predefined standards to achieve a set of goals and objectives. These goals and objectives are often set by the requestor, organization, or platform but can be set by the crowd itself. Crowdsourcing controls can be classified as those that influence the inputs, behaviors, and outputs of crowds and their workers. In the crowdsourcing literature, issues of control are usually addressed indirectly through individual financial incentives (Ye et al. 2017). Financial incentives used in crowdsourcing are designed to influence the effort and attention of crowd workers. This makes sense when crowds are performing individual standalone micro-tasks. This makes less sense for macro-tasks, which require group cooperation. Unfortunately, we know very little about the topic of crowdsourcing controls directed at groups (Daniel et al. 2018).

To address issues associated with crowdsourcing controls for macro-tasks, this chapter has several objectives. First, it presents and discusses the literature on control theory. This includes behavior–output control systems developed by Ouchi and the integrative model developed by Cardinal. These frameworks represent the most widely used control theories in the organizational behavior literature (Cardinal et al. 2017). Second, this chapter presents a scoping literature review that surveys the conceptualization and operationalization of crowdsourcing controls in the HCI/CSCW, information systems and organizational behavior literature. In doing so, this chapter highlights current approaches to controls used in crowdsourcing with an emphasis on what is needed to support macro-tasking. Finally, the chapter identifies gaps and puts forth a research agenda to address these shortcomings. The research agenda focuses on understanding how to employ the controls needed to perform macro-tasking in crowds and the implications for crowdsourcing system designers.

2 Background

2.1 Micro-tasking Versus Macro-tasking Controls

The first question one might ask is: Why not employ controls used in micro-tasking to accomplish macro-tasking? In other words, what makes macro-tasking so different that we need to rethink our approach to controls in crowdsourcing? Micro-tasks are different from macro-tasks in the following ways. First, micro-tasks are already decomposed. Decomposition allows for crowd work to be transparent and predictable. Both transparency and predictability reduce the complexity associated with controls. Second, micro-tasks are standalone independent tasks that require little to no cooperation among crowd members. This narrows the problem of control to the actions of a single individual rather than a group. Third, micro-tasks are homogeneous with similar goals—multiple crowd workers are often performing the same task or set of tasks with the same or similar goal. This decreases the possibility of crowd members having conflicting goals and allows the same control to be used across the crowd.

Macro-tasks, on the other hand, are not decomposed, and in some cases cannot be decomposed. Therefore, crowd work for macro-tasking is often not very straightforward or predictable. This requires crowd workers to negotiate what needs to be done, and in some cases, this happens in real time. This introduces the problem of determining not only which controls to employ but also who should employ them. Second, macro-tasks are not standalone independent tasks but instead interdependent tasks requiring cooperation and coordination among crowd members. As such, the problem of controls requires understanding how to control the actions of a group—not just individuals. Third, macro-tasking requires crowds to undertake a diverse set of tasks, each with its own goals and objectives. Therefore, workers in the same crowd can have different goals associated with their part of the macro-task. This makes it much harder to align goals using a single control. As such, one control might be effective for one component of a macro-task but not another. Issues related to the use of multiple types of group controls in crowdsourcing have largely been ignored.

2.2 Control Theory in the Organizational Behavior/Science Literature

Control is viewed as one of the four primary functions of management (Carpenter et al. 2010). This is often embodied in the planning, organizing, leading, and controlling (PLOC) framework used in most basic management books. Controls are goal-oriented in that they direct employees’ actions to a specific goal, and controls are multifaceted in that there is a diverse set of ways to implement them (Cardinal et al. 2017). Generally, research on the employment of controls has been directed at understanding effective approaches to aligning workers’ attitudes, intentions and behavior with an organization’s goals and objectives. Next, the chapter presents the various approaches to classifying controls.

2.3 Formality of Controls

The actual procedures or practices used to implement controls can be viewed as either informal or formal. Informal controls are implemented by workers. They represent a shared set of beliefs and values among workers driven in part by their social relationships (Eisenhardt 1985; Ouchi 1979). Informal controls are often implicitly understood as a set of acceptable and unacceptable actions (Ouchi 1980). The consequences of violating them often include being expelled or ostracized from one’s social group (Liu 2015). On the contrary, formal controls rely on explicitly stated rules or procedures that outline acceptable and unacceptable actions (Eisenhardt 1985; Kirsch 1997; Ouchi 1979). They are often driven by the management, and workers may or may not agree with them. In fact, workers often have little to no influence on determining formal controls. The consequences of violating a formal control involve the official actions by the organization.

Ideally, informal and formal controls should be aligned, but often they are not. It is possible for an employee to conform to a formal control and violate an informal control. Likewise, it is possible to conform to an informal control and violate a formal control. For example, workers who cross picket lines during an illegal strike might be violating an informal control while conforming to a formal control. These workers might keep their job but be expelled from their social group (i.e., union workers).

2.4 Control Systems

There are four types of control systems, i.e., configurations of multiple formal and informal controls. These include market, bureaucratic, clan, and integrative control systems (Cardinal et al. 2010). Market control systems are designed to focus on evaluating transaction outcomes such as the cost to perform a job. Market control systems do not rely heavily on either formal or informal control mechanisms. Bureaucratic control systems instead focus on specifying, monitoring, and evaluating the performance of workers (Ouchi and Price 1978). Bureaucratic control systems rely heavily on formal control mechanisms such as organizational rules, regulations, and procedures. Clan control systems emphasize aligning workers’ motivations, beliefs, and values with those of the organization (Kirsch et al. 2010; Liu 2015). Clan control systems rely more on informal control mechanisms such as appealing to workers’ personal pride or their identification with the organization. Finally, integrative control systems leverage both formal and informal control mechanisms (Cardinal et al. 2004; Jaworski and Kohli 1993; Sitkin and George 2005). For example, integrative control systems might employ formal controls such as rules and procedures along with informal controls such as appealing to workers’ pride.

2.5 Control Focus

Controls can also be classified by the areas they are designed to influence: input, behavior, and output (Cardinal et al. 2017). Input controls focus on selecting the inputs (e.g., people and materials) that go into the work processes (Cardinal et al. 2010). This is often done by filtering out inputs that are seen as substandard. Typically, input controls are embedded throughout the hiring process of many organizations. For example, this would include requiring specific entrance exam scores or educational achievements before a person could be hired. Other examples include requiring potential suppliers to be certified before they can bid to provide manufacturers with raw materials. Input controls assume that if the inputs are of a certain quality it is more likely that the process will produce an acceptable output.

Behavior controls focus on aligning behaviors used to transform a set of resources such as labor and raw materials to a specific output such as the completion of a task or set of tasks. Behavior controls are directed at work processes needed to accomplish work (Robert 2016; Tiwana 2010). Behavior controls assume that if employees align their behavior to a predefined behavior or set of behaviors they are likely to perform a given task well (Dennis et al. 2012). Behavior controls include creating plans, defining work assignments, explicating work processes, and providing status reports on work (Piccoli and Ives 2003; Robert 2016). Behavior controls are effective when workers align their behavior to act in accordance with the established rules and procedures (Dennis et al. 2012; Robert 2016).

Output controls focus on influencing workers by holding them accountable to a predefined output metric (Choudhury and Sabherwal 2003; Kirsch 1997; Maruping et al. 2009). Output controls are directed at the final products or services produced and ignore the processes needed to accomplish the work. Output controls assume that if workers are held accountable for a predefined output they will align their behavior to achieve this output. Examples of output controls include paying factory workers for the number of correctly completed products rather than for the number of hours worked to complete the products. Output controls also include yearly, monthly, and quarterly goals for sales and production volumes.

Input, behavior, and output controls have advantages and disadvantages. In many cases, output controls can be very costly. This is because discovering that the final product is below standards means in many cases that resources that have been allocated were wasted. It is also costly in that any other task dependent on the final output is now held up. On the contrary, behavior controls allow for the continuous evaluation of work, which allows for problems to be identified and corrected sooner. Input controls are often the least costly when one considers the resources involved later in the transformation process, but this varies by industry. Input controls are often necessary but not sufficient to ensure successful output. The use of unqualified personnel is likely to lead to poor outputs, but the use of qualified personnel does not ensure high-quality outputs.

Input and output controls also have advantages. They do not require knowledge of the work process itself, nor do they require detailed planning to implement. This is important in creative knowledge work, where the work processes are often not understood or cannot be seen. Hiring the most talented people and holding them accountable for what they produce and not how they produce it is an example of employing input and output controls. However, behavior controls do require knowledge of the work processes to create a predefined set of behavior standards. Behavior controls also require the ability to monitor the work processes. This can be problematic for creative knowledge work where work processes are less known and work is less visible.

2.6 Control Source

Who determines what controls are needed and how they should be implemented? This question speaks to the source of control. Sources of control include any entity that can impose controls. For example, in crowdsourcing, there are at least five sources of control: (1) platform providers, (2) requestors, (3) crowds, (4) sub-crowds, and (5) individuals within sub-crowds. Platforms provide the digital labor markets that connect workers to requestors who want to employ them. For example, Mechanical Turk and CrowdFlower are two popular digital platforms. Digital platforms can impose controls on crowd workers. Many platforms require crowd workers to maintain minimum performance standards. Requestors are another source of control. Requestors hire crowd workers and can employ controls to influence their behavior (Ye et al. 2017). Crowds themselves can exert control over their members. It is quite possible that controls can be exerted by multiple sources simultaneously, each with pros and cons. For example, Robert (2016) demonstrated that controls imposed by the group itself lead to better performance when compared to controls imposed by someone outside the group.

2.7 Control Unit of Analysis

Controls can be designed to influence organizations, groups, individuals within groups, and individuals. Controls directed at groups hold groups accountable rather than any individual within the group. For example, a group project completion date would be a group output control, whereas a task completion date for a specific individual would be an individual output control. This chapter differentiates between controls directed at individuals and controls directed at individuals within a group. Controls directed at individuals within a group are focused on aiding the collaborative work of the group, whereas controls directed at individuals who are not within a group are not focused on aiding collaborative group work. Therefore, controls directed at individuals within groups could be used to help promote macro-tasking, whereas controls directed at individuals outside of groups tend to be used to promote micro-tasking.

3 Scoping Literature Review

The authors of this chapter employed a scoping literature review to identify the various approaches to employing controls in crowdsourcing. The purpose of a scoping review is to rapidly map out the underpinnings of a research area (Mays et al. 2001). Scoping reviews provide an overview of a broader topic, whereas systematic reviews tend to have a narrow focus with an emphasis on depth (Peterson et al. 2017). The purpose of this scoping review was to survey the topic of controls in crowdsourcing and map out the various approaches used in the literature.

3.1 Literature Review Search

The literature review was conducted using Google Scholar. Google Scholar ranks articles by their relevance to the search topic and covers a wide and broad set of literature. This allowed the review to cut across several research areas covering controls in crowdsourcing. The search keywords were “controls” and “crowdsourcing” and the search was conducted in September 2018. The initial search identified 58,000 articles. The authors of this chapter evaluated article abstracts against the following inclusion and exclusion criteria.

Inclusion criteria. Studies were included if they (1) were empirical crowdsourcing studies and mentioned the use of controls and (2) were published in English-language journals/conferences.

Exclusion criteria. Studies were excluded if (1) they focused on types of controls that did not apply to the crowd or its members, (2) controls were in reference to variables such as age and gender (i.e., control variables), (3) they focused on control as an experimental procedure, or (4) they were nonempirical papers.

The literature search stopped at the first 370 articles identified by Google Scholar, for two reasons. First, articles beyond the first 300 became less and less relevant to the topic of control in crowdsourcing as outlined by the inclusion criteria. In fact, at the mark of 370, the articles met very few, if any, of the inclusion criteria. Second, the articles that were relevant did not add new knowledge to the scoping literature. In other words, the papers that were relevant employed controls no different from those in the papers already included in the review corpus.

The initial screening of the article abstracts produced 192 articles from the 370. Further analysis showed that 30 articles fell under exclusion criteria 1, 2, or 3, while 52 others fell under exclusion criterion 4, so they were dropped from the analysis. The remaining 110 articles met all inclusion criteria. Appendix 1 presents a summary table of the included articles.

3.2 Publication Venues

The publication venues of the 110 included articles were as follows: 69 (63%) were published in journals, 29 (26%) were published in conferences, 8 (7%) were workshop papers, 3 (3%) were book chapters, and 1 (1%) was a research report. Although the journal and conference listings were diverse, many were published by ACM or IEEE (Table 3.1).

Table 3.1 Publication venues, sources of control, and task type
Full size table

3.3 Sources of Control

Reviewers identified sources of control in each paper. Three sources of control were identified: platform, requestor, and crowd members (i.e., peers). Articles that employed platform controls relied on a predefined control embedded within the platform. An example of the use of a platform control would be to only include master turkers (MTurk crowd workers) in a study. The criteria used to determine who is or is not a master turker are set by the platform. The requestor was by far the most widely used source of control, employed in 101 (92%) papers; this was followed by the platform and requestor controls, used in 6 (5%) papers, then requestor and crowd members (peers) controls, used in 3 (3%; Table 3.1).

3.4 Macro Versus Micro

Reviewers determined whether the controls in each paper were focused on micro- or macro-tasks. Generally, studies that required participants to engage in simple standalone tasks without any need to coordinate with others were identified as micro, while studies that employed tasks that were not broken down and required coordinating with others were labeled as macro. The controls employed in crowdsourcing overwhelmingly focused on micro-tasking. One hundred four (94.6%) articles focused on micro-tasking while only 3 (2.7%) focused on macro-tasking. Three (2.7%) articles focused on both micro- and macro-tasking controls. See Table 3.1.

3.5 Level of Analysis

The paper findings on the level of analysis were consistent with those by Daniel et al. (2018). As stated by Daniel et al., “the quality and benefit of group work are still not fully studied and understood” (p. 29). Only 3 (3%) the articles focused on controls directed at individuals within groups, or groups, whereas 107 (97%) focused on controlling individuals (Table 3.2). This fully supports Daniel et al.’s additional conclusions that in failing to address issues of group control we also fail to fully leverage the potential of crowds.

Table 3.2 Level of analysis and control type
Full size table

3.6 Control Type

The authors of this chapter reviewed articles to determine the types of controls employed: input, behavior, or output, or any combination. Output controls were used the most, with 91 (83%) of the articles employing some type of output control (Table 3.2). Originally, the evaluation of crowd members’ output was done by humans; more recent work has shifted toward the use of advanced forms of artificial intelligence (AI). These approaches vary from relatively simple to more complex and are designed to better predict and evaluate worker outputs (e.g., Kajino et al. 2014). Yet, other approaches have sought to use both human and artificial intelligence systems (e.g., Haas et al. 2015).

Input controls and behavior controls were used less often than output controls. Input controls were used in 25 (23%) articles. The most common use of input controls was entrance tests to participate in the crowd work (Bozzon et al. 2013). Behavior controls were the least employed type of control, appearing in 18 (16%) articles. Types of behavior controls included real-time feedback on task performance, which allowed crowd workers to redo and improve their work, and design of better user interfaces to reduce error (e.g., Ashikawa et al. 2015; Gadiraju et al. 2015). See Table 3.2.

Nearly half of the articles (50, or 45%) employed more than one type of control. The most popular combination was input and output controls (39 articles, or 35%). This combination was typically employed by requiring an entrance test to participate in the work, then performing quality checks on the work performed (e.g., Eickhoff and de Vries 2013; Hutton et al. 2012). Five articles (5%) employed both behavior and output controls, and 6 (5%) employed all three controls (Table 3.2).

3.7 Formality

The review found no evidence of informal controls. Because it was a scoping review, this does not mean that there was no use of informal controls but rather that they were rarely used when compared to formal controls.

3.8 Major Findings

Three major findings were derived from the literature review. Although the review also showed empirical evidence of other findings, the following insights represent the most consistent and generalizable results.

  1. 1.

    Crowdsourcing literature has focused primarily on the individual engaging in micro-tasking, with little attention directed at groups engaging in macro-tasking. As a result, we know very little about controls for macro-tasking involving groups.

  2. 2.

    The requestor has been the source of control and has relied heavily on output controls, with some efforts to leverage platform controls. On one hand, this approach does not require the requestor to have any knowledge of the work process. On the other hand, output controls alone are not enough to help the crowd manage and coordinate the work of its members. To accomplish this, the crowd itself must be leveraged as a source of control.

  3. 3.

    The literature on controls in crowdsourcing has focused mainly on formal controls. Yet informal controls can be as effective, if not more so, than formal controls (Kirsch et al. 2010). Informal controls also have the additional benefit of being more effective at promoting group cohesiveness.

4 Recommendations for Future Research

This section outlines a research agenda as a roadmap for future research by giving specific suggestions on how to shift toward the study of crowdsourcing controls for macro-tasking. Our research agenda is based on three assumptions:

  1. 1.

    Macro-tasks are not decomposed when assigned to a crowd; therefore, they require the crowd to decompose the task. In many cases, the tasks are not decomposable.

  2. 2.

    Macro-tasks require some degree of interaction and coordination among crowd members.

  3. 3.

    Macro-tasks require crowd members to undertake a diverse set of activities to accomplish their work. In other words, all crowd members do not perform the same task (i.e., little redundancy).

Given these assumptions and the gaps in the literature, this research agenda focuses on informal as well as formal controls for groups. The research agenda for formal controls not only includes input and output controls but also emphasizes the importance of behavior controls. To capture the effects of the group, this chapter conceptualizes crowds as a higher order structure that can exist on a given platform. Please see Fig. 3.1 for a visual depiction. Platforms are the digital technology that can host multiple crowds. In macro-tasking, crowds are groups of individuals working to achieve an overall common or shared goal. Crowds can be composed of multiple subgroups or sub-crowds. The term “sub-crowds” has been used by other scholars to represent smaller groups within the crowd (Malhotra and Majchrzak 2014). This chapter defines sub-crowds as crowd members who work independently to accomplish an objective that helps the crowd achieve its overall goal. Sub-crowds have boundaries in that there are members and nonmembers of sub-crowds. This boundary requirement applies even if membership is fluid. Sub-crowds can vary in size ranging from at least two crowd members. Macro-tasks that cannot be decomposed to micro-tasks are likely to be assigned to sub-crowds. Therefore, this chapter asserts that sub-crowd controls are a missing but vital component to understanding macro-tasking in crowds. In all, the research agenda’s focus on informal as well as formal controls, the inclusion of crowds and sub-crowds as sources of control, and increased attention on behavior controls are expected to help address core shortcomings in the literature.

Fig. 3.1
figure 1

Levels of crowdsourcing control

Full size image

4.1 DE-CoRe Control Framework

To help identify the steps involved in the developmental of controls, this paper introduces the Defining, Evaluating, Correcting and Redefining (DE-CoRe) control framework. The DE-CoRe framework consists of four activities, listed next.

  1. 1.

    Defining involves developing and setting standard(s) that will be used later to compare against actual actions. These standards could refer to input, behavior, or output standards. Prescribed standards are the backbone of any control system. Standard setting for crowdsourcing input controls would focus on defining the selection criteria for potential crowd workers. For behavior control, it includes defining the behavior standards needed to perform the crowd work. Standard setting for output control would involve defining what constitutes a quality output.

  2. 2.

    Evaluating involves assessing the actual inputs, behaviors, and outputs against those prescribed standards. For input controls, this would involve evaluating potential crowd workers against the established selection criteria. Evaluation via behavior controls would involve comparing actual crowd worker behavior with the predefined behavior standard. Output control evaluation would determine whether the outputs produced met the predefined standard.

  3. 3.

    Correcting, if needed, involves identifying why and how inputs, behaviors, and outputs failed to meet the standards. This information provides feedback to explain what needs to be done differently to meet the prescribed standards. Correcting activity is distinct from the evaluation activity. Evaluation determines whether actions meet or fail to meet a predefined standard. Correcting activity focuses on why or how the actions failed to meet the predefined standard.

  4. 4.

    Redefining, if needed, is the final activity. For input control, this could entail changing the selection criteria. This might occur when new knowledge or skills are needed by crowd workers. In case of behavior controls, the need to redefine standards might be driven by new technology. For output control, quality standards can be redesigned based on new requirements.

In all, the DE-CoRe control framework provides a simple model to help organize and better communicate the research agenda presented in the next sections. Figure 3.2 depicts the developmental process and the iterative nature of the defining, evaluating, correcting, and redefining activities.

Fig. 3.2
figure 2

DE-CoRe control framework

Full size image

5 Formal Controls Research Agenda for Crowdsourcing Macro-tasking

Crowd: Input Controls

Research Question 1a: What are the most effective ways for crowds to employ input controls to promote crowdsourcing macro-tasks?

Definition. Crowd input controls are directed at the selection of the inputs (e.g., people and software) that go into the work processes of its sub-crowds. Crowd input controls ensure that the crowd inputs meet the predefined standards needed to support the achievement of the overall crowd’s goals and objectives.

Examples. Examples include knowledge, skills, personality, and experience selection requirements, and minimum reputation scores.

Challenges. The problem of input control for macro-tasking in crowds is threefold: First, the set of knowledges and skills needed to complete macro-tasks might not be known because all the macro-task requirements might not be immediately identifiable. Second, the knowledge and skills needed might vary greatly depending on the task requirements of the assigned sub-crowd. This makes it difficult to determine whether one set of selection criteria should be used for all crowd workers or a different set of selection criteria should be used for each particular sub-crowd. For reasons one and two, the crowd selection criteria should be more general, focusing on basic requirements for crowd workers. Finally, who should determine the selection criteria needed to employ the input controls: the requestor, the crowd, or the sub-crowds?

Design requirements

Defining. Systems must be able to help crowds determine the selection criteria for potential crowd workers. Such systems could allow crowds to leverage information from other crowds. For example, new crowds could use the work requirements from similar macro-tasks to determine the knowledge and skills needed by their crowd workers.

Evaluating. Going beyond filtering potential crowd workers by attributes, systems should be able to aid crowds in their decision-making process. A system might produce a list of recommended crowd workers based on the selection criteria. However, the system could go beyond this by rank ordering the list of crowd workers from most to least qualified. To promote diversity, the list could highlight the underrepresented minorities. To avoid problems of bias, the system could also alert the crowd when the selection criteria produce a list with no underrepresented minorities. Of course, what is and is not an underrepresented minority and whether a list should consider such factors is beyond the scope of this chapter.

Correcting. After crowd work has started, systems should be able to help crowds determine whether the selection criteria are being employed correctly. This would involve answering questions such as this: Are the selection criteria being ignored or incorrectly applied?

Redefining. Systems should support the redefining of selection criteria by using actual crowd worker performance. Crowds need answers to questions such as, “How predictive were the selection criteria in determining actual crowd worker performance across sub-crowds?” To this end, systems should produce reports that identify predictive selection criteria against actual performance data. Crowds could also leverage what they learned from the correcting activity to employ more effective selection criteria. For example, crowds might discover that their selection criteria were being ignored because they were ineffective.

Crowd: Behavior Controls

Research Question 1b: What are the most effective ways for crowds to employ behavior controls to promote crowdsourcing macro-tasks?

Definition. Crowd behavior controls direct the behavior of sub-crowds toward the achievement of the crowd’s goals and objectives. At the crowd level, behavior controls should be focused on ensuring effective interactions among sub-crowds. Therefore, crowd behavior controls should be directed at establishing standards to help govern how sub-crowds engage with one another.

Examples. Examples of behavior controls include sub-crowd status reports and lists of completed or uncompleted work.

Challenges. The biggest challenge with regard to crowd behavior controls is to determine how much autonomy should be afforded to sub-crowds. This is particularly problematic when many of the work requirements are not initially known. Therefore, crowd behavior controls should foster cooperation among sub-crowds while providing them with the needed autonomy to develop their own behavior controls after work requirements become known. Specifically, crowd behavior controls should be directed at creating standards for communication and interaction among sub-crowds. Crowds should pay attention to work dependencies that require handoffs among sub-crowds. Crowd behavior controls should be developed to avoid or resolve problems that slow or hinder the transfer of work among sub-crowds.

Design requirements

Defining. Going beyond basic communication requirements, systems should help identify work dependencies across sub-crowds. This would help crowds understand the requirements needed to ensure effective handoffs of work among sub-crowds.

Evaluating. To help with evaluation, systems should support the creation of digital boundary objects. Boundary objects are artifacts employed to track activity across group boundaries (Star and Griesemer 1989). Within crowd work, digital boundary objects are electronic artifacts employed to track work across multiple sub-crowds. Digital boundary objects are vital to assisting crowds in monitoring and tracking the work of sub-crowds. Although boundary objects are common to most work, such as “to-do lists,” some boundary objects are context-dependent. Therefore, systems must have the flexibility to allow crowds to construct their own digital boundary objects when needed.

Correcting. To support the correcting activity, systems must produce work reports that highlight where sub-crowds went wrong and how to correct their actions. These reports could focus on identifying the sub-crowd that failed to meet requirements. This would answer questions such as this: Which sub-crowd failed to report what information when?

Redefining. Systems should allow crowds to redefine work standards when needed. After employing behavior controls, crowds might realize that their current reporting requirements are: (1) simply not enough to promote effective communication and interaction or (2) too cumbersome for sub-crowds to follow. Systems that could help to diagnose either problem and allow crowds to leverage this information in redefining their behavior standards would be invaluable.

Crowd: Output Controls

Research Question 1c: What are the most effective ways for crowds to employ crowd output controls to promote crowdsourcing macro-tasks?

Definition. Crowd output controls ensure that sub-crowd outputs meet the crowd’s predefined output standards or set of standards. Crowd output controls are used to hold sub-crowds accountable by making it clear what is and is not an acceptable output. Crowd output controls should ensure that sub-crowds are supporting the crowd’s overall goal and not engaging in suboptimization at the expense of other sub-crowds.

Examples. Examples of this include sub-crowd goals for completed work, sub-crowd goals for correctly completed work, and deadlines for completed work.

Challenges. The interdependent nature of macro-tasking across sub-crowds presents several challenges. First, macro-tasking often requires output from one group to be used by other groups. Such dependencies must be identified before group output controls can be designed and employed. Crowds would also need to build consensus among groups on what such output controls should be when such dependencies exist. The second and related issue is that group output controls must align across groups. An example of misalignment is when one group in the crowd is being evaluated on quantity but the group receiving the output is more concerned about quality. The group producing the output might ignore issues related to quality to achieve more quantity. Yet this would be all for naught, because the output would be useless to the receiving group if the quality was not acceptable.

Design requirements

Defining. Systems must allow crowds to define output standards by identifying quality criteria and assigning value weights to such criteria. Systems with advanced capabilities might provide cost–benefit calculations. This would allow crowds to understand trade-off between decisions regarding quality and quantity. Going beyond this requirement, advanced systems would need to help crowds deal with issues related to the task interdependence among sub-crowds. To avoid problems related to suboptimizing, systems should aid in the identification of work dependencies.

Evaluating. Systems should provide tools to assess or help assess the quality of crowd outputs. These tools could be designed to help crowds manually assess quality or be completely automated.

Correcting. If needed, systems should produce reports that help crowd workers understand why and how they are failing to meet output standards. For example, are the failures related to quantity or quality or both? Should sub-crowds focus on doing less but better?

Redefining. Similar to behavior control, systems should allow crowds to redefine their output standards. Systems could help leverage the information generated in the correction activity. If many sub-crowds are failing to meet deadlines, maybe the deadlines should be changed. If the sub-crowds are meeting output standards regarding quantity easily, maybe such output standards should be increased.

Sub-crowd: Input Controls

Research Question 2a: What are the most effective ways for sub-crowds to employ input controls to promote crowdsourcing macro-tasks?

Definition. Sub-crowd input controls focus on the inputs that go into the sub-crowd’s work processes. Like crowd input controls, sub-crowd input controls would primarily focus on selection criteria for membership. However, they could also include the selection of software or other collaborative tools. Sub-crowd input controls offer another opportunity to employ controls that help promote macro-tasking.

Examples. Examples include knowledge, skills, personality, and experience selection requirements over and above those required by the crowd, and minimum reputation scores over and above those required by the crowd.

Challenges. Several issues arise when considering sub-crowd input controls. First, it is important to determine what additional selection criteria might be needed for sub-crowd membership above those required for crowd membership. This entails determining the sets of knowledge and skills needed to complete the sub-crowd’s work. This could also include increasing the required scores needed on the crowd’s selection criteria. For example, sub-crowds might require higher technical skills depending on the nature of their work. Second, it would be necessary to determine whether the sub-crowds’ selection criteria superseded the crowd’s selection criteria or vice versa. It would also be important to know whether sub-crowds could completely bypass the crowd’s selection criteria. For example, could sub-crowds select individuals who had been rejected by the crowd? This is important because sub-crowds might have the opportunity to hire unqualified crowd workers and provide training that would eventually make them qualified. Sub-crowds could evaluate the performance of such crowd workers after a trial period to determine whether they should be retained.

Design requirements

Defining. In addition to the design requirements outlined for defining crowd input controls, systems must be able to help sub-crowds identify any conflicts between their and the crowd’s input controls. For example, such systems would need to identify potential conflicts between the crowd and sub-crowd selection criteria.

Evaluating. The evaluating requirements for sub-crowd input controls should be similar to those for crowd input controls.

Correcting. In addition to the correcting requirements outlined for crowd input controls, systems should be better designed to provide more flexibility in allowing sub-crowds to override their selection criteria. These systems should require an acknowledgment and a thorough explanation as to why the selection criteria are being ignored. Unlike the crowd selection criteria, which are likely to be more general and stable, the sub-crowd selection criteria are likely to be more specific and dynamic. Sub-crowd selection criteria are likely to change rapidly as work requirements become clearer and work progresses. Therefore, sub-crowds might not have the luxury to wait for the redefining activities to change selection criteria. In fact, depending on the work duration, sub-crowds might disband before they ever reach the redefining activity.

Redefining. Processes for redefining sub-crowd input control requirements should be similar to those for the crowd input control requirements.

Sub-crowd: Behavior Controls

Research Question 2b: What are the most effective ways for sub-crowds to employ behavior controls to promote crowdsourcing macro-tasks?

Definition. Sub-crowd behavior controls focus on aligning the behaviors of the sub-crowd workers with the behaviors needed to achieve the sub-crowd’s goals and objectives. Although sub-crowd behavior controls are concerned with effective interactions among sub-crowd workers, they also specify work standards needed to accomplish work tasks. Therefore, when compared to crowd behavior controls, sub-crowd behavior controls should be more detailed and task specific.

Examples. Examples of sub-crowd behavior controls include work instructions, crowd worker status reports, lists of crowd workers’ completed or uncompleted work, shared calendars, and work assignment spreadsheets.

Challenges. It would be difficult to assemble sub-crowds with no common work history and expect them to work together to develop behavior controls without any guidance. In other words, newly formed sub-crowds would need behavior controls to begin to work together to develop behavior controls. This chapter proposes conceptualizing behavior controls as those employed before and after the sub-crowd workers develop knowledge of their work requirements. To address this challenge, this chapter introduces Layer 1 and Layer 2 behavior controls.

Layer 1 behavior controls are standards directed at helping the sub-crowd determine the work requirements. Layer 1 behavior controls can be imposed by the crowd or quickly agreed upon by the sub-crowd. In the first approach, the crowd could dictate initial basic sub-crowd behavior controls. This approach could be referred to as the template approach to behavior controls. Templated behavior controls should be generic and light and apply broadly to any sub-crowd. These template behavior controls can be viewed as basic rules of engagement for crowd workers. Sub-crowds could then develop their own behavior controls later when work requirements became clearer. In the second approach, sub-crowds could engage in swift planning via a sub-crowd charter. A sub-crowd charter is a document that outlines the sub-crowd’s objectives and communication protocols, and crowd workers’ basic roles and responsibilities. Sub-crowds could add or remove requirements to their charter as work progressed. The differences between the first and second approaches to developing Layer 1 behavior controls are a matter of degree. Simply put, the two approaches vary on the degree to which the crowd or the sub-crowd has an initial influence on the Layer 1 behavior controls. Therefore, the third approach would be for the crowd to provide a template in line with the sub-crowd character and enlist the sub-crowd to decide which aspects to keep and which to remove.

Layer 2 behavior controls are directed at defining standard behaviors needed to perform work. There are two approaches to developing Layer 2 behavior controls. The first approach is to have the sub-crowd workers determine them as their work requirements become clear. Layer 2 behavior controls provide instructions on how crowd workers should accomplish their job. The degree of detail associated with the instructions depends on the effort and time needed to specify such detail. Ideally, sub-crowds should weigh the benefits associated with such specification against the time and effort needed. The second approach is to provide sub-crowds with work standards already developed based on best work practices. Like the template approach to Layer 1, these best work practices would be generic and light and apply broadly. However, they could also be very detailed if the new work requirements were similar to previous work requirements from another sub-crowd or crowd. Like the two approaches to Layer 1, the two approaches to developing Layer 2 can also be combined. Therefore, the third approach would involve the sub-crowd starting with a template based on best practices and customizing it to the sub-crowd’s needs.

In either case, Layer 1 behavior controls should be removed or changed if they prevent the actual work from being accomplished. At the same time, Layer 1 behavior controls might be sufficient to accomplish the sub-crowd work; if this occurs, there is no need to define Layer 2 behavior controls.

Design requirements

Defining. Going beyond basic communication requirements, systems should provide tools to help sub-crowds break down, structure, assign, and aggregate crowd work. Such systems could provide digital workflow diagrams, shared calendars, and work assignment spreadsheets.

Evaluating. To help with evaluation, systems should afford the design of digital artifacts such as to-do lists and crowd worker status reports. These digital artifacts would be similar in concept to the digital boundary objectives but different in at least two ways: (1) these artifacts would not be designed to be used by other sub-crowds and (2) they would be focused on evaluating the behavior of sub-crowd workers rather than the sub-crowd itself.

Correcting. Systems must produce work reports that show where sub-crowd workers went wrong and how to correct their actions. These reports should be more detailed than those produced for crowds.

Redefining. After employing behavior controls, sub-crowds might realize that they were: (1) ineffective even when followed correctly or (2) too difficult for crowd workers to follow correctly. In either case, sub-crowds would have to redefined work standards. Ideally, sub-crowds should be able to leverage the same system capabilities used in the defining phase. However, new system capabilities might be needed when new work standards are vastly different.

Sub-crowd: Output Controls

Research Question 2c: What are the most effective ways for sub-crowds to employ output controls to promote crowdsourcing macro-tasks?

Definition. Sub-crowd output controls ensure that the output of crowd workers in a sub-crowd meets the sub-crowd’s predefined output standards or set of standards. Sub-crowd output controls hold crowd workers accountable to a predefined outcome or set of outcomes identified as vital to achieving the sub-crowd’s overall goals and objectives. Note: Output controls are likely to be very important to sub-crowds engaging in complex and creative macro-tasks. In such cases, output controls are often preferred over behavior controls. This is because specifying detailed instructions for complex and creative tasks is very difficult. In addition, creative work is often not visible; as such it is hard to monitor and track the progress of creative work.

Examples. Examples of sub-crowd output controls include crowd worker lists of correctly completed tasks, the number of completed tasks, and due dates for completed tasks.

Challenges. The degree of task heterogeneity and its corresponding output control heterogeneity is likely to be a major challenge. The tasks of crowd workers within a given sub-crowd are likely to be related and interdependent—related in that all tasks performed by crowd workers in the same sub-crowd would be directed at achieving a common goal, and interdependent in that the output of every crowd worker within a sub-crowd would need to be aggregated before the sub-crowd could achieve its goals.

Yet, crowd workers’ tasks are likely to be different. Task heterogeneity might require a diverse set of output controls among crowd workers within the same sub-crowd. For example, for one task, the quantity might be far more important than quality. But for another task, deadlines might be the most important factor. Finding a way to harmonize the output controls needed to avoid conflicts within a sub-crowd is likely to be problematic. In addition, incompatible output controls are likely to lead to low sub-crowd cohesion.

Design requirements

Defining. In addition to the design requirements outlined for defining crowd output controls, systems supporting sub-crowds should place more emphasis on issues related to task heterogeneity. More specifically, how can such systems help sub-crowds harmonize output controls to avoid controls conflicting with one another?

Evaluating. Systems should provide tools to assess or help assess the quality of individual crowd workers. In addition, such systems should be able to evaluate small groups of crowd workers who perform a similar task, yet be flexible enough to evaluate individual crowd workers across a wide range of tasks.

Correcting. For correcting sub-crowd output controls, systems should be able to provide detailed reports on a range of tasks.

Redefining. Like crowd output controls, systems should allow sub-crowds to redefine their output standards.

Table 3.3 summarizes the formal controls research agenda.

Table 3.3 Formal controls and DE-CoRe design objectives
Full size table

6 Informal Controls Research Agenda for Crowdsourcing Macro-tasks

Research Question 3: What are the most effective ways to promote informal controls in crowds for macro-tasking in crowdsourcing?

Many of the challenges and design requirements for informal controls are similar to those of formal controls. The biggest difference is the role that social relationships play in the employment of informal controls. Generally, informal control is a type of social control exerted by members of the collective. Informal controls influence actions by exerting normative peer pressure on crowd workers. A more specific definition of informal controls can be derived from Kirsch et al. (2010). According to Kirsch et al., informal controls are exerted when shared norms, values, beliefs, and vision influence the behaviors of the collective. This is consistent with literature identifying the need to facilitate social bonds, identification, and common values among members of a collective to help establish and strengthen informal controls (Weibel et al. 2016). However, social bonds, identification, and common values are normally associated with groups with a long history of working together (Robert et al. 2008).

Therefore, the biggest challenge associated with informal controls relative to formal controls is determining how crowd workers with little history can develop the social bonds, identification, and common values needed to employ informal controls. In this section, the discussion on informal controls is focused on addressing this issue only. However, some of the same challenges and design requirements identified in the discussion on formal controls are also applicable. In addition, this chapter acknowledges that depending on the task duration and task complexity, crowd workers may or may not have an opportunity or a need for informal controls. Yet, without informal controls, macro-tasking complex and creative work is likely to be difficult. Consequentially, informal control is likely to be difficult to establish but nonetheless very important in the crowdsourcing of macro-tasks. Next are several approaches to promoting informal controls in crowdsourcing macro-tasks. They are summarized in Table 3.4.

Table 3.4 Informal controls and design objectives
Full size table

One approach is to understand how to help crowds build common norms, values, beliefs, and vision through the promotion of a shared identity. Research has shown that a shared identity can facilitate the establishment of common norms, values, beliefs, and vision (Chatman 2010; Robert 2016). Windeler et al. (2015) provided an example of how this approach could be operationalized. They studied ways to reduce conflict and promote a shared understanding and ultimately improve performance in online teams. They designed a system that provided one set of teams with profiles of each team member that only listed similar attributes among team members. This was done to promote perceptions of similarity—a shared or common identity among team members. Another set of teams received no such information regarding their similarities. The online teams that received the similarity information experienced less conflict, had a better shared understanding, and performed better as a team. A similar approach could be used in crowdsourcing. Questions like how to best promote similarities or which similarities to promote still need to be addressed. Nonetheless, designing crowdsourcing systems to promote similarities among crowds or sub-crowds holds much potential.

Another approach is helping crowds establish shared work norms and values. In traditional organizations, new employees go through a socialization process that both introduces and facilitates preexisting shared norms, values, beliefs, and vision (Turner and Makhija 2006). Organizations often leverage orientation and training programs to help establish prototype norms, values, and beliefs. Similar approaches have been done in groups. For example, Homan et al. (2007) conducted a lab study and found that teams trained to value diversity were able to establish norms that led them to better leverage diversity to perform better. Crowdsourcing systems can be designed to not only train crowd workers but also orient workers to a specific crowd climate or culture. This could be done by building crowdsourcing systems that walk crowds or sub-crowds through series of group-building exercises. Although there are many unanswered questions related to finding effective team-building exercises and designing such a crowdsourcing system, this avenue holds the potential to promote informal controls.

Another approach to promoting informal controls is to select crowd workers who already have shared norms, values, beliefs, and vision. This could be accomplished by selecting crowd workers who worked together in the past. For example, a crowdsourcing system could be programmed to select crowd workers from a GitHub project. This system could be designed to assess the success of a group of crowd workers based on a specific metric. Then the system could invite all crowd workers who participated in a specific project or part of the project. These crowd workers would likely have been indoctrinated into a system of shared norms, values, beliefs, and vision. Salehi et al. (2017) provided an example of this approach. Their systems selected crowd workers based on whether they were familiar with one another. Familiarity is a strong predictor of shared norms, values, beliefs, and vision. By selecting specific online communities like GitHub, organizations could ensure they hire crowd workers who are competent in a specified domain. Questions about which parameters to use to select crowd workers along with the actual design of such systems needed to operationalize the selection criteria are important issues to be addressed.

7 Future Research and Limitations

The next section presents several limitations as well as future research opportunities. While these areas complement and overlap the research areas identified and discussed earlier in the chapter, these areas could themselves constitute their own research agenda. Although they could not be sufficiently discussed in detail in this chapter, they are important areas that should be acknowledged.

7.1 Meta-control Theory

To accommodate the use of multiple types of control inherent in the crowdsourcing of macro-controls, this chapter introduces the meta-control theory. Meta-control theory focuses on comprehending the impacts of controls on controls. Meta-control theory is concerned with understanding how controls reinforce or undermine one another. The goal of meta-control theory is to avoid controls conflicting with or undermining one another. Meta-control theory also recognizes that controls must be dynamically managed throughout their use. Meta-control theory acknowledges that controls make up a complex system that might not lead to linear, well-understood effects but instead could lead to nonlinear effects that are difficult to understand. Understanding how to ensure that controls align across levels of analysis is one example of meta-control theory.

The theoretical development and empirical validation of the study of how controls impact controls could significantly contribute to control theory in general as well as its specific application to crowdsourcing. Yet, we have not begun to scratch the surface in this area. Although we have empirical examples of the use of multiple controls, little theory or reasoning has been offered as to why these particular controls were chosen or how they are expected to align with one another or, better yet, when they are expected not to align with one another. This is almost certainly a result of the micro-tasking nature of most crowdsourcing work. Nonetheless, as we move toward macro-tasks, meta-control theory, or the study of how controls impact controls, is becoming increasingly important.

7.2 Temporal Effects on Control

Generally, things change over time. This is not surprising or profound—the impact and importance of time have been increasingly recognized by many HCI/CSCW scholars and others (You et al. 2015). Yet no studies of control examine the impact of time. At this stage, the evidence of the importance of time on controls is more anecdotal than scientifically verifiable. For example, platform companies like Uber update their controls based on dimensions such as time. For instance, by implementing surge pricing, Uber charges higher driving fares during peak demand times.

A less popular example of the impact of time on the effectiveness of control relates to Uber’s driver assignment algorithm. Uber’s driver assignment is a type of behavior control the company imposes on drivers. However, many drivers learn how Uber’s algorithm assigns which drivers to which routes. Drivers then attempt to manipulate their assignment to more lucrative routes. Uber responds by changing the assignment algorithm to prevent such manipulation. Hence, over time Uber’s behavior control has become less effective. A more systematic research agenda might not only investigate how time impacts the effectiveness of controls but why, when, and how. What is certain is that we know little if any with regard to the impact of time on the effectiveness of controls in crowdsourcing.

7.3 Artificial Intelligence Control Systems

The use of artificial intelligence (AI) to control workers is becoming popular in many industries. AI—the ability of a computer system to sense, reason, and respond—holds many potential uses for controlling crowd workers for macro-tasking. Artificial intelligence control systems (AICS) are intelligent computer systems that seek to align and dynamically realign workers’ actions to predefined standards to achieve a set of goals and objectives. AICS can dynamically evaluate, correct, and redefine controls in real time. AICS can be used as input, behavior, and output controls. There are several examples of researchers employing automated quality assessments (Hoßfeld and Keimel 2014) or automating work processes (Schmitz and Lykourentzou 2018). However, these systems fall far short of employing the full capabilities of AICS currently used in many digital platforms (i.e., Uber and Upwork). Future HCI/CSCW research needs to explore both the development and implications of AICS in crowdsourcing.

8 Conclusions

The conditions needed to design effective controls for micro-tasks represent an approach to control that is typical of the Industrial Age. But as crowd work becomes increasingly more complex, interdependent, and less decomposable, focusing more on innovation and learning than performing, HCI scholars must ask ourselves how we can design controls that better meet the demands of macro-tasking. The need to rethink controls for new ways of working is not a particularly new problem, nor is it confined to HCI scholars examining crowdsourcing. Organizational scholars have warned of the need for dramatic changes in our approaches to organizing and they have decried the lack of progress toward newer approaches to designing controls (Cardinal et al. 2010). As such, this chapter should help organizational scholars begin to rethink the design of controls in traditional organizational settings.