Understanding the Crowd: Ethical and Practical Matters in the Academic Use of Crowdsourcing

Abstract

The driving force behind digital crowdsourcing are its workers: working, hidden behind the scenes, churning out data in experiments, participating in research studies, completing little tasks to accomplish HITs online. Understanding workers and crowdwork better is therefore key to develop a more effective and fair use of crowdsourcing for research. This chapter attempts to help develop an understanding of the various aspects of the crowd by drawing parallels between workers of different platforms (AMT, Microworkers and Crowdee) through quantitative and qualitative analysis of current and newly collected data. A picture of the crowd is drawn by uncovering their motivations, workplaces, skills and infrastructure, issues and perspectives about the design of microtasks, the employers and the microtask-based platforms. Legal and ethical perspectives on crowdwork are also discussed, and online resources are reviewed that researchers can use as a primer to employ crowdworkers in an ethical and fair way. The chapter provides information, a review of internationally recognised ethical principles and practical advice to those who would like to use crowdsourcing for experiments and to carry out research studies as an informed researcher and crowd employer.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Take the fake novelty of a term like “crowdsourcing” – supposedly one of the chief attributes of the Internet era ... “Crowdsourcing” is certainly a very effective term; calling some of the practices it enables as “digitally distributed sweatshop labor” – for this seems like a much better description of what’s happening on crowdsource-for-money platforms like Amazon’s Mechanical Turk – wouldn’t accomplish half as much” [34].

In his recent book “To Save Everything, Click Here” [34] Evgeny Morozov produces a sustained critique of what he calls technological solutionism. His key argument is that modern day technology companies, often situated in Silicon Valley, are increasingly touting technological innovation as the quick route to solving complex and thus far relatively intractable social and societal problems. He documents how in many cases the technologies simply do not deliver the wished for result, meaning that this trend ends up as little more than a marketing exercise for new technologies and gadgets. A related phenomenon is one where a technology innovation, that may be of little or ambiguous social merit is presented in a way that it is value-washed – given a positive social-spin, and marketed as something inherently virtuous when, once again the situation is far from clear. It is from this perspective that he criticises the use of ‘crowdsourcing’ to describe the labour situation in relation to Amazon Mechanical Turk (AMT)¹, which in reality often equates to low-paid piece-work.

Crowdsourcing refers to accessing a diverse and large workforce via the web and several platforms have emerged to facilitate this, with AMT being the best known. The term crowdsourcing captures ideas like voluntarism, altruism, and community which really do seem relevant in some cases, such as the crowdsourced nature of surveys done as citizen science². However, in other cases it can mask the reality of what the work is really like and why people are doing it. So while it provides an accessible and cheap source of labour for various purposes, does it provide reasonable employment for those workers doing the microtasks? In this chapter, we clarify who these crowdworkers really are and why they do this type of work as it applies to the work through AMT and other similar microtask platforms.

We review studies of the people who work on microtask platforms: who they are; what their motivations are; and how they organise and carry out their work. Our key focus is the AMT platform because within academic literature and in more public and mainstream coverage of crowdsourcing it is the best known, most used and most researched platform. We draw on qualitative and quantitative research in the literature as well as providing some new data and analysis to provide a more up-to-date picture. We also provide some comparative analysis, particularly in terms of demographic information, using data we have gathered from two other crowdsourcing platforms, Microworkers³ and Crowdee⁴. Our contention is that it is important to understand who the workers are, why they work on these platforms, and what their perspectives on the market and employers, their skills and expertise, and their difficulties are. We use this understanding to outline how academic researchers can use AMT or other similar platforms and work with crowdworkers in a way that is both ethical (in terms of respecting them and their work, through positive and polite communication, decent pay and so forth) and successfully productive (i.e. how things can be best managed to try and ensure good quality, timely work).

While we mainly focus on AMT it is important to understand that there are different platforms available that can have more ethical modes of operation built-into them, although a large part of ethical responsibility within crowdsourcing relationships necessarily lies with the parties involved. Other platforms can also allow access to a different demographic or a more global workforce, people with different types of skills, expertise, and so forth. Crowdsourcing projects and the work they entail vary in terms of their complexity, what the work or activity is about, whether and in what way they are collaborative, whether they are paid (and if so how they are paid) or voluntary. In this way crowdsourcing is best seen as the form and mechanism whereby work projects, campaigns or individual microtasks are handed out to a large distributed workforce. We are interested in digital crowdsourcing, whereby the form is electronic and the mechanism is computers and the Internet. The range of work is essentially all that which is possible in this form and through this mechanism, which will be dependent on the skill and ingenuity of those designing and carrying out the work. The terms, conditions, pay and so forth are primarily determined by the participants.

Crowdsourcing, in the cases we examine (microtask, paid, non-collaborative work), is a form of work and it is remunerated as piece-work. Our key aim is to provide information and advice to current and potential academic requesters who use crowdsourcing for carrying out tests and experimentation with datasets. Ideally, this advice will make it easier to make better decisions about which platform to use, about how to design and target the right microtasks, and about how to communicate and manage the relationship with workers. Thoughtful use of these labour markets may lead to a more ethical approach while at the same time maximising the chances of receiving good quality, timely work.

A deep understanding of the work that crowdworkers do is important ethically and socio-organisationally, since questions have been raised about the ethics and efficacy of current crowdsourcing practices [2, 11, 28, 38, 41, 45]. Felsteiner [11] provides a comprehensive summary of research on workers and their legal situation, highlighting the legal ambiguities surrounding AMT, and workers’ difficulties in ensuring fair pay, and recompense for bad treatment. Bederson and Quinn [2] outline a series of design and policy guidelines to provide more transparency and fairness for workers, suggesting amongst other things that requesters should be clear about hourly pay, payment procedures and policies, and should offer grievance procedures. Kittur and colleagues [28] consider how crowdwork might be developed technologically and organisationally such that it could be desirable and productive for both workers and employers. They recommend better communication between requesters and workers, and that opportunities should be provided for learning and career progression. Silberman, Irani and colleagues created a ‘Turker’s Bill of Rights’ [44], which illustrated the issues faced by Turkers⁵ – primarily, unfair rejection of work, uncertain or slow payment, low wages, and poor communication [44, 45]. Recently Salehi et al. have been involved in a project called Dynamo [43] that attempts to support Turkers in an initiative to form a workers guild for organising various campaigns aimed at securing more workers’ rights.

The structure of this chapter is as follows: in the opening sections we provide quantitative and qualitative data analysis to show who the Turkers and other crowdworkers on Microworkers and Crowdee are – particularly focusing on their demographic information. We then move onto why and how, looking at why they are working on crowdsourcing platforms as opposed to other sources of work or labour markets, how they got into the work, and what makes or allows them to stay. We can think about this in terms of a set of motivations, reasons or explanations. Then we look at how they organise their work and workplaces, their relationships with employers (requesters) and their participation in communities (both on-line and through their regular social networks). In the closing sections we focus on how this understanding we have provided can be translated into ethical and practical guides for using crowdworking as part of academic research.

2 Who: Understanding Who the People Who Do Crowdwork Are

A number of crowdsourcing platforms exist nowadays, each with its own features, and each populated by a different crowd. Depending on the platform, crowds present a different degree of diversity in the country of origin (e.g., AMT poses constraints on the country of residence of both workers and requesters). Thus diversity can vary in gender, socio-economical background, education, and motivation. In addition, research has shown that only a fraction of the workers registered on a platform are highly active: on AMT, it is estimated that 80% of the HITs are carried out by the 20% of the most active Turkers [13]. Furthermore, different parts of the crowd may specialise in certain microtask types and may thus be highly trained [7].

Given the above, the risk of serious sampling issues exists for crowdsourcing-based studies. Highly active workers and highly specialised workers may belong to specific groups exhibiting distinctive characteristics and behaviours, and these characteristics may vary across platforms. In addition, as illustrated by Kazai et al. [27] demographic traits such as gender and location are related to differences in the workers’ performance and data quality. Hence, it is important to know who the workers are in order to design microtasks properly (e.g. by setting up pre-qualification questions, adjusting the compensation, balancing the task complexity). Knowing who the workers are will also help in choosing the right platform where to deploy such microtasks, and in taking precautions in analysing results to make them as general as possible.

There is a growing body of research that seeks to understand the operation of the crowdwork market and the people who work within it. Survey-based demographic studies [22, 23, 41] in 2010 show that the majority of Turkers (${\approx }$50–60%) are U.S. based, with Indian workers forming the second largest population (${\approx }$30–40%). US Turkers are more likely to be female and are 30+ years old on average. Indian Turkers are more often male and a bit younger, 26–28 years old on average. Both groups are reasonably well educated with the vast majority having at least some college experience. In November 2009 Indian Turkers on average earned $1.58/hour, as opposed to $2.30/hour in the US [41]. Over 50% of Indian Turkers reported an annual household income of less than $10,000, while 45% of US Turkers reported one of less than $40,000 [22, 23].

Although informative, the studies available on crowd (demographic) characterisation have limitations. First, data was collected some years ago and due to the rapidly changing world of crowdworking may not be accurate anymore [46]. Second, the vast majority of studies focus on AMT [22, 23, 30, 33, 39, 41]. Very little is known about the crowds from other platforms, with the notable exceptions of Berg’s study [3] who compared Turkers, both from India and the US, to CrowdFlower workers, Peer’s study [40] who compared CrowdFlower workers and Prolific Academic workers with Turkers and a study by Hirth et al. [21], who investigated the locations and home countries of workers and requesters of the Microworkers platform.

2.1 Method

In this work we set out to collect and analyse demographic data of crowds of different platforms, so as to provide the reader with a timely characterisation of crowdworkers, their background, motivation and the ways in which they organise their work and personal life. Figure 1 provides a summary of these different aspects of the crowdworkers discussed in this article.

Fig. 1.

Summary of these different aspects of the crowdworkers discussed in this article.

To complement the existing literature on crowd characterisation, we study the demographics of workers on three crowdsourcing platforms, namely AMT, Microworkers and Crowdee. AMT is by far the most popular and the most researched crowdsourcing platform. Thus, the results from AMT may be understood as the reference point to which the other platforms are compared. AMT connects a wide variety of requesters with over 500,000 Turkers (although the analysis of [13] suggests that the real number of active Turkers is between 15,059 and 42.912. Microworkers is an international crowdsourcing platform, active since 2009. With over 600,000 registered workers from over 190 countries, it provides an excellent environment to study workers’ demographics and motivation. Contrary to AMT, which imposes restrictions on the geographical location of both workers and requesters, Microworkers gives access to workers and requesters from everywhere across the five continents. In our investigations concerning who is part of the crowd and why s/he is part of it, having access to a diversity of workers, with a wide variety in geographical origin, is core. The third platform, Crowdee, is a German based crowdsourcing platform; in contrast to the other platforms, Crowdee focuses on mobile microtasks with a small but growing worker community from west Europe [35].

Questionnaire. In order to investigate workers’ demographics and motivation, we conducted surveys. We posted a questionnaire-based task on AMT, Microworkers and Crowdee; the questionnaire to be filled in contained a similar set of questions for all three platforms, yet customised depending on the specific platform. The questionnaire was created to investigate the following items:

• Demographics (gender, age, size of household and education level). As mentioned above, our intention was to demographically characterise the crowds in order to be aware of potential biases in task⁶ execution [7], to be accounted for in the platform selection and task result analysis. Here we focused on gender and age, size of household (also related to the economic status characterisation, see below) and education level. The latter was especially of interest to investigate the potential of crowdsourcing platforms for tasks requiring special skills.

• Economic status (yearly income, employment status, expenditure purpose of money earned through crowdwork). A second point of interest was related to the economic status of the workers. Martin et al. [31] have shown how the primary motivation of workers is monetary, and how that affects their perception of the marketplace and their preferences in terms of task execution. To this purpose, we focused on gaining knowledge on whether crowdwork was a primary source of income for workers on the different platforms, and on the use they would make of the money earned through it (either primary expenditures such as bills and rent, or secondary expenditures such as hobbies and gadgets).

• Crowdwork conditions (time spent on the crowdsourcing platform weekly, number of tasks completed per week, location from which microwork is carried out, equipment and software tools used to support crowdwork, usage of other crowdsourcing platforms). Here, we were interested in characterising working conditions and attitude of crowdworkers, following the findings of Gupta et al. [17]; we were specifically interested in quantifying weekly workload (number of tasks and hours spent), and in investigating working environment conditions, including the physical place from which the crowdwork was carried out (e.g., home, Internet cafe, office), and the devices (mobile or not) from which tasks were executed.

To check the reliability of the answers, at least one gold standard question, also called a trapping question or a honeypot (e.g. [18, 36]), was employed in every survey. The question had a straightforward answer which did not require any specific background knowledge (for example, workers had to indicate how many letters were included in the word “crowdsourcing”).

Different platforms use different terminologies for crowdwork and the tasks created by the requesters. The work task is called a “Project” or a “HIT” in AMT, “Campaign” in Microworkers and “Job” in Crowdee. Questionnaires were also customised per platform; for example, the AMT questionnaire was adapted to Microworkers by replacing “HIT” by “task” and “AMT” with “Microworkers”.

2.2 Data Collection

For AMT, the demographic study was conducted in March 2016 with 100 workers from the US and 100 workers from India. The HITs were created at 9 AM PDT and within 56 min for US workers, and 62 min for Indian workers, all answers were collected. As a result of the reliability check, 10 responses from US workers and 29 responses from Indian workers were removed. The job was extended for Indian workers to gather more data. Overall, 90 responses from US workers and 87 responses from Indian workers were collected. For every survey, the US workers were rewarded $1 and $0.7 was paid to the Indian workers, following guidance from the Dynamo project⁷.

In Microworkers, we set up a number of campaigns to have our questionnaires filled in April 2016. We targeted workers from five continents (North and South America, Africa, Europe, Asia, and Oceania). Previous work has shown that Microworkers’ tasks are completed mostly within working hours [21]. As a consequence, launching a single campaign for all continents may have led to collecting responses mostly from workers in time zones for which the campaign started within office hours. To overcome this limitation, we launched independent campaigns in the different continents. In addition, because most continents span a large number of time zones, we took the further precaution to run the campaigns at minimum speed in the beginning. This allowed us to minimise the speed at which the campaign would be completed, and maximise the probability that workers from any time zone in the continent would fill in the questionnaire. Targeting separated continents also allowed us to customise the monetary reward for the questionnaire completion, following the recommendations of the platform.

Table 1 summarises our experimental setup. Note that for some continents, we limited the number of targeted workers, as we knew that the pool of workers in those areas was limited [21]. It is also interesting to note that the recommended monetary rewards vary quite substantially across regions (the suggested payment for US workers is almost three times as high as that suggested for Asian workers), and that in general, they are higher paid than those typically used for AMT HITS. The campaigns were launched simultaneously in the different continents. The fastest was completed within a few hours (Eastern Europe) and the slowest took over a week (South America). We collected data from 474 workers. Again, the reliability of workers’ responses was checked using a gold standard question. Eventually, 380 of the original 474 responses were retained.

The third study was conducted using the Crowdee platform. The study took place in March 2016 and was open for 250 participants. The survey was divided into two jobs, which were published one after the other with some hours delay. Overall 242 participants filled in the survey completely. All of them answered the obvious gold standard question correctly; however, inconsistent answers to the repeated birth year question led to the removal of 6 responses. As a result, responses of 236 participants were used for further analyses.

Table 1. Overview of the data collected and analysed in this study. The “Data (acronym)” column reports the origin of the data (survey or MTurk tracker) as well as the acronym used in tables and figures throughout the rest of the chapter.

In addition to the studies explained above, we included in our analysis raw data from the MTurk Tracker [10, 23], as a reference for the AMT survey. The data covers the time range from April 2015 until January 2016. The data set contains responses to a five-item demographic survey (gender, year of birth, household size, household income, and marital status); 23,839 of these responses are from US crowd workers and 4,627 are from Indian crowd workers. The demographic API of the MTurk Tracker creates a survey job in AMT every 15 min to capture time variability. The survey is compensated with 5 cents; participation is restricted, i.e. each worker can participate once a month [24].

2.3 Results

In the following, we report the outcomes of our data collection on crowd characterisation, in order to answer the question “Who are the crowdworkers”? Detailed numbers are presented in the Appendix. To capture the geographical diversity of Microworkers, we differentiate the analysis for two separate groups, roughly identified based on GDP: (1) a group of Western countries, i.e., those included in North America, Europe and Oceania, and (2) a group of developing countries, included in Africa, Asia and Latin America. Although this separation is somewhat artificial, we deemed it sufficiently realistic to provide a good term of comparison to the AMT results, for which US (Western) and India (developing) workers were analysed separately.

Gender. In general, male workers outnumber female workers (see Fig. 2). For AMT, in line with the data reported in recent studies (e.g. [3, 5]), we find gender to be more balanced for US Turkers compared to the Indian Turkers. According to our survey results, more than 60% of Indian Turkers are males, as also confirmed by the MTurk tracker data (although in this case more females are observed than in the survey data). On the other hand, for US Turkers, the data obtained from the MTurk Tracker indicates that the majority of Turkers are females, which is in contrast to our survey results, as well as to those of other recent surveys reporting numbers similar to ours [3, 5].

The gender distribution of both Crowdee and Microworkers Western countries is close to that found for Indian Turkers: male workers form more than 60% of the population (or at least of the participants in our study). In Microworkers, the prevalence of male workers is more prominent in developing countries as compared to Western countries. In developing countries, our survey captures a ratio of one female to every five male workers.

Fig. 2.

Gender distribution of crowd workers in different platforms observed in survey studies.

Age. As shown in Fig. 3, the age distribution differs considerably across platforms (note that to be a crowdworker, a minimum age of 18 is required in all platforms, hence we set 18 as the starting age for which we analyse data). Within AMT data, there are differences between the outcomes of the survey and the data collected by the MTurk Tracker. Regarding the US Turkers, the group aged between 41–55 years is larger in our survey than in the MTurk tracker data, while the opposite is observed for Turkers in the youngest age group (18–26 years). Discrepancies between the survey and MTurk tracker data are smaller for the Indian population, as was observed for Gender. However, also for the Indian population, we find younger Turkers to be more numerous in the MTurk tracker sample compared to our survey data. These discrepancies make it difficult to properly characterise US and Indian Turkers according to their age.

Crowdee and Microworkers workers seem to be younger than Turkers. For Crowdee this is possibly due to many of workers being students as the platform is developed and maintained by a university team [35]. For Microworkers, independent of the region, the vast majority of workers are 32 or less. Workers in developing countries seem to be younger than their Western counterparts.

Fig. 3.

Distribution of crowd workers’ Age in different platforms observed in survey studies.

Household Size. Figure 4 shows how both the MTurk tracker and the survey data indicate that most of the US Turkers (${>}{70}\%$) live in a household of utmost two other people. A similar trend is found for Western countries in Microworkers, with big households (${>}4$ people) accounting for about 40% of the total. For Crowdee, the majority (${>}{55}\%$) of Crowdee workers lives either alone or with one other person, which is in line with expectations, being that the Crowdee crowd composed for a large part by students (as will be detailed below).

The data for the Indian Turkers is not as homogeneous across the two data sources as it is for the US Turkers; nevertheless, the data clearly indicates that the household sizes are larger. In contrast to the US Turkers, most of the Indian Turkers (${>}{58}\%$) live together with three persons or more. Even larger sizes are found for Microworkers in developing countries, with only 30% of the total number of workers living with at most two other people. Based on these data, we can see a clear distinction between the composition of the crowd in Western countries (AMT data for US, MW Western and Crowdee) and in developing countries.

Fig. 4.

Distribution of Household Size (counted including the worker), for the different platforms considered.

Educational Level As reported in previous studies (e.g. [22]), and visible in Fig. 5, the education level of the Turkers is rather high. Only very few Turkers have no high school degree. Most of them have at least some college education with the Indian workers reporting a higher education level compared to the US workers. More than half of the Indian Turkers have a Bachelor’s degree⁸ and more than 65% report to even have a Master’s degree. However, holding a degree⁹ is not a good measure of one’s foreign language or computer skill in developing countries¹⁰. Note that for this item, MTurk Tracker data are not available.

The distribution of the education levels of the Crowdee workers is similar to that reported by the US Turkers. Most of them report to have some college education and about 30% have a Bachelor’s degree or higher. Again, it should be noted that Crowdee workers are for a large part students, thus they may still be in the midst of their educational path.

Workers using Microworkers have, in general, achieved higher education levels than Turkers and Crowdee workers. More than half of the workers have a bachelor’s degree or higher qualification. This resembles the education level distribution of Indian Turkers. When looking closer at the two regions we are analysing, we find that workers from developing countries have significantly higher education levels than workers in Western countries, in a way that recalls the differences in distribution of education levels between US and Indian Turkers.

Fig. 5.

Distribution of crowd workers’ Education Level on different platforms, observed on survey studies.

Household Yearly Income. For the US Turkers, survey data are in line with the data from MTurk Tracker. Around two thirds of the Turkers report a household income below $60,000, as visible in Fig. 6. Regarding the Indian Turkers, the data are also somewhat consistent: more than 90% of the Turkers report their household income to be less than $60,000. A large proportion of Indian Turkers state that their household income is $10,000 or less. The proportion of this group having the lowest income is considerably higher in our own survey data compared to the tracker data.

As for the US Turkers, more than 60% of the Crowdee workers have a household income below $60,000; however, compared to the US Turkers, a higher proportion of Crowdee workers belongs to the lowest income group. This may, in combination with the low household size and young age, be explained by the fact that students make a consistent part of the Crowdee crowd; as students, they have no or very low income. Note that 22.18% of participants did not report their household income.

Finally, the Microworkers workers come, in general, from low income households. Almost half of the respondents to the questionnaire earn less than $10,000 per year, and albeit developing countries workers contribute to this number for the most part, still almost 40% of the Western workers claim to have such a low income. This is in contrast with US Turkers, whose average income seems to be much higher (the majority earn $40,000 a year or more), and diverges from the previous similarities observed for US Turkers and Microworkers workers in western countries. In interpreting the data, it should be taken into account that several Eastern European countries were included in the data collection for MW Western countries (15% of the respondents included in the MW Western groups were Serbian, for example); those countries have a significant lower GDP per capita with respect to the US, which may also partially explain these findings. In addition, as illustrated below, about 10% of MW Western workers are students, which may explain the low-size households with low income.

Fig. 6.

Distribution of Household Income of crowd workers in different platforms observed in survey studies.

Employment Status. Independent of their location, a large proportion of Turkers (${>}{44}\%$) have a full-time jobs besides their crowd work (Fig. 7). This is more pronounced in the Indian population (${\approx }{56}\%$). Also the proportion of the workers working part time is higher in India¹¹ (${\approx }{26}\%$) as compared to the US (${\approx }{15}\%$). A fairly large number of the US Turkers – almost 25% – are keeping house.

Also in the case of Microworkers, most workers either have a full-time or part-time job, in addition to their crowdsourcing job. A significant percentage (more than 10%) are students, which is also reflected in the young age of the workers. Compared to (US) Turkers, Microworkers workers, are more likely to have a part time job and being students, less likely to be housekeepers.

As for all other platforms a large proportion of the workers are working full-time additionally to the crowdwork they carry out. Compared to all other platforms, the proportion of Crowdee workers who are students is large.

Fig. 7.

Distribution of Employment Status of crowd workers in different platforms observed in survey studies.

Time Spent on Crowdsourcing Platforms. The majority of Turkers (${>}{60}\%$) stated that they spend more than 15 h per week on AMT; of these, a large number is even working more than 25 h per week (US: ${>}{37}\%$; India: ${>}{47}\%$, Fig. 8). This is noteworthy as many of the workers reported to have either a full-time or a part-time job in addition to their crowdwork (cf. previous section). Interestingly, whereas most Turkers dedicate a high number of hours to their crowdwork, for Microworkers we find a binomial distribution. A large percentage of workers (${\approx }{50}\%$) spend relatively few hours on Microworkers (less than 10) and a lesser but also large percentage spends more than 25 h in crowdwork (${\approx }{25}\%$). For Western workers, the distribution is more skewed towards a smaller number of hours spent on crowdsourcing, whereas workers from developing countries spent more time on platforms, which could be due to factors such as geographical time differences between posting and accessing HITs or availability of Batch HITs on platforms like AMT, that although were available in large numbers, paid only modestly¹². In Crowdee, this trend is even more pronounced. Crowdee workers spend very little time on the platform. This might however be due to the smaller number of jobs available on Crowdee as compared to AMT and Microworkers.

Fig. 8.

Distribution of Time Spent on Crowdsourcing Platforms observed in survey studies.

2.4 Discussion on Who Crowdworkers Are

The results show that the demographics of the workers differ considerably between the platforms, as well as within the platform, depending on the workers’ location. For a number of items (household size, educational level), the Indian Turkers have a profile more similar to that of the Microworkers workers from developing countries, than to their US colleagues. The same is true for the US Turkers, the Crowdee workers and the Microworkers from western countries. In fact, it is interesting to note again that Indian Turkers and Microworkers in developing countries report much higher educational levels than their western counterparts. In developing countries, like India, it is probably linked to the fact that the higher the socio-economic status, the more educated a person is in the Western sense of the word, and the more likely they are to have English and computer literacy and access¹³, whereas, in the US, the Internet access has penetrated further down the socio-economic class ladder¹⁴. This should be taken into account by requesters willing to post tasks which require special skills related to the educational level. Note that high education level, i.e. holding a degree, may not correspond to advanced English language levels in the developing countries as most of education programs are in their mother tongues or local, regional languages, discussed in the ‘education level’ in Sect. 2.3.

With respect to gender, AMT seems to attract more female workers in the US; requesters seeking gender-specific information, or looking for a diverse pool of workers to carry out a task (e.g., when performing studies related to gendered innovations), should take into account that female workers are more scarce in Microworkers, and especially in developing countries. Conversely, requesters looking for a younger crowd (e.g., requesters investigating new trends among young people), should prefer Microworkers and Crowdee to AMT.

When it comes to working conditions and attitude, we found a high platform-dependency. Our data showed Turkers to be more dedicated to crowdwork, spending longer hours on the platform, which may suggest higher specialisation and possibly efficiency in completing jobs. On the other hand, for the most part, Turkers have a full-time job, and perform crowdwork as a second job. On one hand, tiredness due to excessive workload (and consequent unreliability in task performance) may be a risk in this case. On the other hand it may indicate that crowdworkers have a potential of performing more advance tasks, than the type currently asked of them, as they are qualified enough to have a full-time job.

Finally, it is worth noticing that US Turkers set aside from the other workers and platforms when it comes to income per household: their income is consistently higher than their counterpart of Western workers in Microworkers. This is probably due to comparative living standards: $10,000 per year does not even make minimum wage in the US¹⁵ but is a reasonable income in India¹⁶. Workers in developing countries have low incomes and large households; in many cases, as also noted by Gupta et al. [17], crowdwork is their primary source of income.

A further finding is related to the methodological validity of crowd-based demographic survey. The discrepancies between our survey data and the data obtained from MTurk Tracker indicate that data from one-shot surveys are not necessarily in line with the results from surveys which collect data repeatedly over a year. Therefore it is not advised to generalise demographic results by using a survey job: it may be the case that a very specific group of workers will participate in the job. As jobs often have a short time frame, also the participants for the same job type (e.g. surveys vs. annotations) may differ from job to job. This means that participants who are taking part in one specific survey job may not be representative for the crowdworkers who are normally performing this type of job.

Regarding the incomes, it is noteworthy that we do not know how much of these incomes are generated by Turking, and differential costs of living limit the interpretative power of direct dollar comparisons.

3 Why: What Motivates Crowdworkers?

Why do crowdworkers do crowdwork, why do they work on particular platforms and why do they choose particular types of tasks, working for particular requesters? It is rare to find people who would continue happily doing their job if they were no longer paid for it. It is also quite rare to find people who can find no other positive thing about their work than the fact that it pays a wage. Modern crowdwork is a relatively novel type of work if looked at through the prism of technology and the Internet but it is also in many ways simply the modern twist on home-based (or sometimes mobile) piece-work. In this section we draw heavily on our previously studied in-depth ethnographic studies of US and Indian crowdworkers on AMT [17, 31, 32]. When crowdworkers are studied in depth it is obvious that so much of what they talk about and how they talk about it is classic ‘shop talk’ – i.e. work talk. How to earn best, how to maximise earnings, what jobs pay what amount per hour, how many HITs and how regularly batches are posted, what are the best paid HITs and so forth are the topics that monopolise forums, groups and interviews – money and how to best earn it is the over-riding theme, with workplace relations and topics around managing work, learning and so forth related secondary topics. The obviousness of the work dimension may even lead to crowdworkers to provide other reasons why they do crowdwork in response to why-do-you-do-this questions from researchers while lacing their answers with words like work, earn, money, job, pay, employer and employee, as we sometimes saw in our interviews of Indian Turkers.

The problematic other side of this coin is that researchers have often found it hard to believe that people could be possibly doing crowdwork as a job – ‘how could anyone accept such a low wage?’ This disbelief seemed to be part of a distancing and insulating move – exaggerating the secondary benefits and positive aspects for Turkers – and making academic researchers feel better, since Turkers could be conceived of as doing things for fun, passing time, enjoying helping out in academic research. This trend reached its apotheosis with the publishing of a couple of papers [1, 26] that employed rather dubious techniques in order to re-interpret their own questionnaire-based results on reasons and motivations. Both these studies had indicated that money was the primary factor but sought to minimise this result by employing ‘social desirability bias’ detection to essentially suggest that respondents had answered in this way because they thought they should. We can see no technical or analytic justification for their decisions – and given the overwhelming evidence to the contrary across the media and other academic work, and in the forums, web-resources and so forth – we feel their work can be discounted.

3.1 US Turkers

In our in-depth study of the Turker Nation forum [31] we gathered many materials to demonstrate as strongly as we could just how clear it was that participations in doing HITs on AMT was for the vast majority of Turkers a form of paid work, where the pay was of key importance. As evidence we produced a variety of material from a number of threads, beginning with a thread titled “Turkers Turking for Fun” where the opening post questioned whether money was the primary motive in all cases of Turking or whether sometimes Turkers chose work according to other criteria. The replies to this essentially took two forms. The first form was reactions with opprobrium:

danturker: “This attitude would be requesters dream come true. The workers come here to have fun and play and the lousy pay for work is not an issue. This attitude helps create low pay for the MTurk work force that does care about fair pay.”

In these cases the Turkers made it clear that they believed that even discussing such issues promoted the discourse that pay was not important, and that people worked for fun or charity, and undermined the fight for respectable wages. In the second form of response the reaction was milder, simply stating that pay was clearly the most important factor, secondary factors like interest or fun could figure in the decision if there was no pay difference:

larak56: “I agree with most everyone here. While I do find some of the HITS fun and actually learn an incredible amount by doing HITS, I do it for the cash.”

The idea that this perspective is not shared by the overwhelming majority of Turkers does not seem credible. One of the clearest features of all of the forums and resources dedicated to serving Turkers, and the tools and scripts they use is that the massive preoccupation is on how to find and do the best paid HITs, which good HITs come at the biggest frequency in the biggest volume, how to maximise earnings and who can be trusted to pay, quickly. As soon as new HITs come on the market people try them out and post their projections on what their pay rate is, whether their work has been accepted and how long it took them to be paid. They talk about how often requesters post tasks in what volume, and how they arrange their work around the periods where there will be high availability of good tasks. They are concerned about their ratings and HIT count as these are passports to more work. They worry about being suspended or banned as AMT is such a valuable source of income. They talk about how much they make in a day or in a year. Their targets are either purely financial or when they talk about ratings or HIT targets it is as the means to improve earning. The weight of evidence on this matter is massive. These are people who identify as workers working in a low paid labour market where it seems that some of the best workers can earn around $15,000 per year, which represents around the minimum wage income for US workers in a 40 h per week job. It is quite likely that many earn less than this for doing longer hours.

When we consider whether US Turkers work full-time, part-time, alongside another job and more generally how many hours of work per day or week, it is important to separate what they would ideally be doing from what they are actually doing. In fact Turking is not often a job of preference even though they like the fact that they can work from home, be pseudonymous and can choose when and where they work. The problems of the general precariousness of the work, the fact that income can fluctuate by large amounts, low pay and lack of rights (their right to participate can be taken away at any point and there are no specific procedures for dealing with worker grievances) all mean that most Turkers would rather have a secure job in a more conventional labour market. In terms of wages, we can see it is hard to attain more than the US minimum wage and for many it is considerably less. The fact is that many earn what they can, but for many that is not enough to live on; they need to have other income. It is clear, however that some people do carry out the work as supplementary income, which may allow them to buy non-essential luxuries from time to time, but that is a function of their needs rather than indicating it is not a serious job.

3.2 Indian Turkers

Indian Turkers do crowdsourced tasks for the money too. They can earn comparatively more given lower living costs in India – you could support a family to a reasonable standard of living in a reasonably-sized town on $10,000 a year. But once again their level of earnings (or earning potential) is a function of (1) available volume of HITs they are able to do and the earnings paid on those HITs, (2) how crowdwork lines up alongside their other responsibilities, e.g. other work they may do, and (3) how much they need the money, i.e. do they (or their family) have other sources of income? One can think of these as adjustable sliders whereby greater consistent availability of good paying HITs may mean they put in more hours or even go full time. If they really need money to survive they will put in time crowdsourcing even after working a full day in another job. If they have other reasonable sources of income they are more likely to pick and choose the crowdsourcing work they do, and are less likely to do long hours and are more likely to spend their money on treats. In many cases the Indian Turkers (as with all Turkers) are limited in the amount they can earn due to lack of availability of work they can do, and also the pay level of that work. For example, if they can only do relatively simple tasks where the English language is particularly clear in the instructions, and these tasks are limited in number, pay approximately $0.01, and take 5 min to complete, they will not be able to fully support themselves in their living costs with this.

While US Turkers in general are realists or even cynical about the other benefits of Turking – over and above the money earned (see above) – they do prefer tasks that pay well and are interesting, provide learning opportunities, are engaging, funny, creative and so forth. They do feel that they can learn some things. The contrast with the Indian Turkers is that for the Indians the opportunities for learning are more strongly stressed and they place a strong emphasis on ‘timepass’ which in our study was not just ‘passing the time’ but rather passing the time doing something of value as opposed to wasting time. There was a moral component focused on using your spare time in a good way; earning money, self-improvement, and developing skills. While it was clear that US Turkers also indicated in various ways that it was seen as a more productive way to spend spare time they tended to view this through the lenses of necessity far more than moral improvement.

3.3 Complementary Survey Findings

In addition to the qualitative studies reported above, Naderi et al. [37] developed the Crowdsourcing Work Motivation Scale (CWMS) for measuring motivation of crowdworkers. This scale is based on the Self-Determination Theory (SDT) of motivation [9, 42]. The SDT not only differentiates between intrinsic and extrinsic motivation but also considers a spectrum of different types of extrinsic motivation. These different types of extrinsic motivation vary depending on the level of internalisation of the goal, i.e. how much a person can identify with the activity and its outcomes, and therefore how much personal investment or enthusiasm they have for the task.

Internalised extrinsic motivations share similar consequences with intrinsic motivation, i.e. the more people believe a task to be of value, whether in a purely personal way or to have, for example, societal worth, the more effective performance is in complex tasks, the higher participation rates are, and this also leads to higher well-being and satisfaction scores amongst the workers [14]. Results from the study by Naderi et al. [37], show that US Turkers have very high external motivation (i.e. earning money). However, it was also shown that their levels of intrinsic and internalised extrinsic motivation are positively correlated with their participation rate and the reliability of their responses. These results back up the position argued above. Money is the highest motivator but both features of the work and what it is for and features of the task are also important in recruitment and quality, as well as making crowdworkers feel more positive and satisfied. In citizen science projects people gladly give their time for free due to personal interest and an idea of giving something positive to nature and society. In micro-task markets workers often talk more positively if they feel they are contributing to research and the task itself is interesting and engaging, and they may learn something themselves. These are ways to get more people interested and get higher quality output but the bottom line is still work for pay.

3.4 What Does Identifying Motivations Tell Us About Task Design?

The most important thing to take away from this discussion is that there is a strict hierarchy of importance in motivations – pay comes first, and all other motivations are secondary. Really interesting badly paid tasks will not be as attractive as boring well paid tasks. This does not mean that a really interesting but badly paid task will not get done but there will be less of the Turker population who will be attracted to it, and so it may well take longer to, for example, have the batch completed, the quality may be lower, and it may attract more bad behaviour. However, it must also be noted that it is very clear that simply paying more and more does not guarantee more success, better quality work, or faster batch completion times. Crowdsourcing platforms are markets and being as such there are norms and standards of market rates (as well as other things like behaviour and etiquette). If prices outlie the norms by too great a margin people will be suspicious of the task (occasionally ‘earn 100s of dollars quickly’ scam tasks are posted). You should, however, pay at the upper level of market rates – and certainly have an eye to paying an amount that given reasonable/average completion times would pay roundabout minimum wage per hour. This is not necessarily a calculation that is easy to do precisely but that is not really necessary. If you try to do this honestly it will be transparent to the Turkers.

There are another series of features of tasks that are not really Turker motivations – so to speak – but they are elements that motivate them – or attract them – to do your tasks, and they are: how well your task is designed (does it work well, is it clear, are the instructions good); how quickly do you pay, and overall how do you conduct yourself in your dealings and interactions with Turkers (are you fair, polite etc.?). In short, your reputation will impact how easy it is to get good quality work, quickly. A good reputation, earned over time, makes your tasks very attractive and they will be looked out for, picked up quickly and generally done to a high standard. If you have all of these components and can add interest, engagement, creativity – these will serve like the icing on the cake but good pay, good design and good conduct are the most important aspects.

In the survey (see Sect. 2.1), workers were asked about expenditure purposes of the money that they earned through crowdwork. Although similarities in patterns based on countries of crowdworkers (Western vs. developing) was expected, differences based on platforms are observed (see Fig. 9). Crowdee and Microworkers workers mostly use their earnings for ‘secondary’ expenses (${>}{65}\%$) or as pocket change (for hobbies, gadgets, going out etc.). The majority of Indian Turkers (59%) and half of US Turkers (51%) use their earnings for ‘primary’ expenses (like paying bills, gas, groceries etc.). As a result, Turkers rely on their crowdwork income for everyday living expenses which can be indicator of crowdwork being taken more seriously, and used to support workers and even their families, on the income from AMT.

Fig. 9.

Distribution of Expenditure Purposes of Income through crowdwork observed in survey studies.

4 How: Social and Organisational Aspects of Doing Crowdwork

In this section we want to review the data and findings we have available on how crowdworkers organise and manage their work. For the most part this section will deal with qualitative insights about how crowdworkers organise their working lives, their worksites and their actual crowdworking. To do this we will draw again most specifically on the ethnographic studies of Indian and US Turkers [17, 31, 32]. The study by Gupta et al. [17] of Indian Turkers is our richest source of observational data as the lead author actually visited a number of Turkers, saw their worksites and observed them Turking. In the work by Martin and colleagues [31, 32] we draw on forum discussion of these matters to provide insights into the ways in which people work, the circumstances of their work, how they manage work and life and what other resources and technologies they use in organising their work. We cannot make statements with absolute certainty about the correlation between our findings of crowdworkers with that of workers using other platforms that we have not directly studied, but given that a number of these crowdworkers do crowdwork on other platforms with a similar social and organisational set up, and given other studies (e.g. [29]) indicating similar circumstances, we believe that our findings have general wider application than just for AMT, Microworkers or Crowdee.

4.1 Workplaces

In order to do crowdsourcing work, workers require an account, a computing device and an Internet connection. Therefore technically crowdwork can be done in a wide range of places, on the move, in public or private, and it is. In our qualitative studies of Turkers we have examples of a wide variety of workplaces, varying at an individual level too. More generally there is a preference towards having a dedicated place, often in the home with some degree of privacy; quite simply to allow a degree of concentration for the worker and to not disturb other members of the household. This can be thought of as the aspiration of most crowdworkers: a home office. The extent to which this is possible depends to a large degree on peoples’ living conditions. Quite a large amount of Turkers have access to some private space. We know from our material that some Turkers live in crowded conditions where they may well be doing their work in a shared space – living in small family apartments, flat shares or co-habiting in a hostel. In these cases they try to find a quiet corner. This also assumes they have access to a computing device and Internet connection. Sometimes they can work on a shared device while other times they need to go to a place where they can access a device; and this may be work, college or an Internet cafe.

We have examples from our US study where some people are specifically allowed by their work to do Turking during periods of work down time. We have an example in India where employees in a small business process outsourcing company do Turking as a part of their work. Turking in Internet cafes appears more common in India, and this is likely a feature of the depth of penetration of computing technology there. That is, it is more common to have to go to an Internet cafe in order to access more traditional personal computing (PC) technology and the Internet. In some of our examples from India, turking is much more fluid and social in Internet cafes, where people work cooperatively on HITs and may share accounts or do HITs on one another’s behalf. Although mobile phone penetration in India is massive with over a billion mobile phone subscriptions for a population of ${\approx }1.2$ billion¹⁷, the same is not true for PC penetration and the number of people with smartphone, while impressive at 220 million¹⁸ only represents ${\approx }{22}\%$ of mobile phones, and it is clear that smartphones are only suitable for a proportion of microtasks.

A final point to note is that people may prefer certain types of microtasks according to their current workplace (and device and Internet connection). Certain microtasks may be easily done on the move, and using a smartphone (simple tagging and clicking tasks) while others may require a set up with a better connection, keyboard and bigger screen (requiring research, writing, sustained concentration). As such, places and technologies can dictate what microtasks are doable and desirable. Finally it should be noted that sometimes places and devices are used for different sub-tasks in Turking – we have a number of examples of people using their smartphones to search for and book out work (to be done later) or to check up on job status, payment etc. while on the move and then to do the saved work later when they get to their workplace.

Results from our survey studies (see Sect. 2.1) show that crowdworkers work on microtasks from their homes most of the time (${>}{80}\%$) and secondly, from their offices (see Fig. 10). On the other hand, Crowdee workers also work on the move (31%). This is easily explained by the fact that the platform takes the form of a mobile application (as of June 2016). As shown in Fig. 11, crowdworkers mostly used their desktop computer or laptop for crowdwork except Crowdee where workers had to use their mobile phones to do crowdwork. Microworkers workers from developing countries worked more (${>}{15}\%$) using their phones than their colleagues from Western countries. Similar patterns were observed between Indian and US Turkers, which could be due to high mobile phone (see Footnote 17) use in developing countries.

Fig. 10.

Distribution of crowdworkers’ Place of Work observed in survey studies.

Fig. 11.

Distribution of Type of Devices used for crowd working observed in survey studies.

4.2 Informational and Communal Resources

When you study Turkers, one thing that quickly becomes clear is that a lack of information is their biggest problem and that knowledge is a key component in earning power. Crowdsourcing platforms and markets, particularly as exemplified in AMT, are information poor and rather opaque, by design, even if we do not know if this was a thought-through design decision. An individual crowdworker, through trial and error, can clearly learn a certain amount about different types of jobs, what suits them, what pays better and more reliably, who is a good requester for them and so forth, but limited to personal experience their view will be narrow and their learning experiences will be limited to their imagination and abilities.

It may be striking but should not be surprising that they depend upon a number of on-line resources and social and familial networks in order to ameliorate this information deficit and access community support. These sources allow them to acquire much useful knowledge and information, find out what the good resources and technologies are, get tips on how to learn and acquire new skills, and learn what the best strategies and techniques are for finding and completing particular HITs. These resources also help them to understand what to avoid, how best to comport themselves in their dealings with platform owners and requesters and other practical and emotional support. These sites are crucial for many Turkers and other crowdworkers in helping them reach a stage where they can earn a reasonable amount and sustain those earnings. There are a number of websites, forums and Facebook groups that provide a number of informational and supporting services for Turkers, such as Turker Nation (the forum studied by the authors), mTurk Forum, mTurk Grind and Reddit groups¹⁹. Crowdsourcing can and could be a form of synchronous or asynchronous cooperative work but it is not supported on platforms like AMT, but one thing that requesters should understand is that there is a very strong community outside the platform – online and even offline. They work together on ‘the work to make the turking work’ [31] – i.e. the work that they do in order to manage their crowdsourcing work and career, like searching, configuring their system and practices for particular jobs, understanding and judging jobs and requesters, learning and so forth. One of the key features that requesters should understand is that both they and their jobs will get discussed and rated.

4.3 Technology Configurations

Technology set-ups like workplaces vary widely, and may even do so for the same crowdworker since they may be doing crowdwork in multiple places with multiple device and Internet configurations. When we talk about technology configurations this takes in hardware, software and Internet connections. Again, as with workplaces (with the private, dedicated space), there is an ideal set-up that involves high performance, ergonomically comfortable hardware technology, a sophisticated (but often cumbersome) bricolage of apps and browser plug-ins and a reliable high-speed Internet connection.

In some of the forum threads on Turker Nation (or other forums and lists) there are dedicated discussions about the best hardware and technology set-up, including discussions of the fastest most durable keyboards, best screens, set-ups and so forth. Speed is of the essence, but durability for repetitive HITs involving speed of key strokes means some may use special keyboards or configure their set-up with better ergonomic properties to avoid repetitive strain injury. A powerful computer with a reliable high-speed Internet connection is obviously desirable to enable faster downloads, searching, better handling of images, sound files and videos. This set-up also should be more dependable meaning that fewer jobs will be lost half-way through with slow Internet speeds or crashing. Large screens, multiple screens, good headphones, and peripherals like transcription pedals all are ways in which equipment can make a difference to how easy HITs are to do. In general better hardware and network connections extend the variety of HITs that can be done and the speed at which they can be completed, i.e. it increases earning power.

In the US there is a wider spread of Turkers who have good quality device set ups, although there is clearly diversity in quality depending on peoples’ material circumstances. In India, however, the general trend was that there was more variability and often poorer quality device and Internet configurations. We observed people working on older computers, smartphones for HITs not ideally suited to such devices, poor or intermittent Internet connections, and even problems with their electricity supply. These problems were common to the extent that Indian Turkers had worked out various back-up electricity sources and Internet access possibilities [16].

Through our studies it also became clear that software, scripts, apps and plug-ins are crucial to carrying out the work and optimising it. When Turkers can access more, better quality information through their networks and on-line resources this allows them to be more effective in learning and operating in the market. In participating in the forums and so forth they can gain a lot of information, but then another problem is posed; how do they marshal that information to their best advantage? One answer is through the development of tools. A well-known tool in the Turker and academic community is TurkOpticon²⁰ [25]. This is a simple to use qualitative, community-based rating tool, where Turkers provide their ratings out of five on four categories relating to each HIT; communication, generosity, fairness, promptness. The tool comes as a browser extension such that when looking at specific HITs they can view the ratings and be assisted in their decision making. TurkOpticon therefore is a conduit for the type of information on HITs that is informally shared on forums every day. It is also implemented in an embedded fashion in the Turker workflow, i.e. they can see the rating during the search rather than having to look at the HIT then search on a forum for information on that HIT. The tool therefore helps both decision making and productivity.

There are a wide range of tools and these generally come in two forms – (1) tools that help you to navigate to and grab the HITs that you want quickly, and (2) tools that help you to optimise the speed with which you can do HITs. In the case of TurkOpticon they may help with both. A few academic researchers have developed tools – and they are mainly centred around how Turkers can find good paying, reliable jobs, more quickly. Of note is Crowd-Workers [6] that cooperatively collects and aggregates ratings and information on pay. A small application called TurkBench, was also designed to automatically create dynamic work schedules based around the best paying work [19] – the concept was well received but the technology unfortunately ran into technically insoluble issues.

However, the vast majority of tools are designed and produced by Turkers themselves. Whole suites of tools that, for example, notify them when certain HITs become available (e.g. Turk Alert²¹), automatically grab those HITs, enable better search, enable shortcuts for quick navigation and form-filling, and other browser enhancements as well as tools that help assess pay rates and keep track of earnings etc. These can help change the value of HITs (i.e. make them more worthwhile because they can be done quicker) but also help Turkers to operate in the market, they can filter the good from the bad and spend less time searching and more time working on good HITs. Detailed discussions and links can be found on forum pages²². A few important points come out of understanding the role of tools. Firstly, the amount and ingenuity of tools and scripts underlines the fact that Turkers are knowledgeable and inventive people. Secondly, it shows that pretty much anywhere in the workflow where time can be saved through tool use and where technical development is possible it seems to have been done – once again demonstrating that saving time increases earnings power. Thirdly, however, AMT does not have a Turker API and is not configured to support these tools. And these tools form a fragile ecosystem (i.e. they are not fully integrated nor fully compatible with browsers, AMT etc.) and they are liable to cause application or system crashes from time to time. And finally, and importantly, it should be noted that Turking experts using these tools have a market advantage in being first in the market to take the good jobs, and some HIT batches consequently disappear in a matter of seconds. Novice Turkers stand little chance of accessing those HITs.

4.4 Managing Work and Life

As should be becoming clear, one of the elements we wish to emphasise in regard to crowdsourced work is that the motivations, concerns and problems of crowdworkers are very similar to those of any group of workers, but particularly those of more precarious²³ workers. It is however, important to look at what is different to other forms and sectors of work and what the implications of these differences are. Some of the key differences between crowdworking and many other sectors are the fact that it is often anonymous work, with little direct communication between employer and employee, it is a form of radical freelancing in that crowdworkers can work for various different employers in a day and may have no stable relationships, and finally, the market is open all hours, and crowdworkers can choose whatever hours they like. The lack of rights, protections and grievance processes separate crowdwork from many other forms and sectors of work, although things are different in different markets, once again with AMT being an example of a market with little legal coverage [32].

When it comes to managing work and life Turkers are very similar to many other workers. They have a life outside of work with desires, needs, responsibilities and relationships just like everyone else. Their personal life and their work life have to be juggled, but how does that organisation work, and what are the effects of having an always-on market? In theory an always-on market would provide a positive flexibility, i.e. work could be accommodated to life. When you were available, on your terms, you could pick and choose the work to do. This would allow you to accommodate a complex personal life (e.g. with uneven demands on time), you could go on holiday when you wanted, sickness would not have to be reported. In reality things are a little more complicated.

First of all, you can choose your schedule, but if it is not possible to earn a decent wage without putting in long, monotonous hours, suddenly you do not have the same flexibility. Secondly, while you do not need to get approval for holidays or sick leave, you are not paid for them, and you have no long term benefits like pension rights or healthcare. Finally, if you work in a market where there is in general an over-supply of labour and an under supply of good (well-paying regularly available) jobs you are in stiff competition and need to take the good work if and when it is available, thus finding the flexibility in your personal and family life to accommodate doing the work when it is good. Flexibility, as others have pointed out (e.g. [4]), while touted as good for all often leads to a situation where workers compromise their personal lives to accommodate the variable needs and fluctuations of their work. The testimony across all of the Turkers we studied was that very often they organised their hours based on when better work was available, or they hoped would be available (guessing market dynamics is not always an easy task), and when good jobs in large batches came along they were willing to drop everything to make the most of this work.

Another interesting set of observations that come from our studies is some similarities to gambling. In the forums and amongst Turkers there are various stories about $100 HITs or ones where the hourly rate turned out to be $50 or something similar. When this is added to the uncertainty over whether HITs are genuine, whether you will be paid or not, and the slightly addictive desire to keep looking, checking out whether you have been paid or not, or whether a great job has just been posted, one can appreciate the draw on attention it has, that promotes a certain set of compulsions. This does not make it much different from so much of the World Wide Web these days in the attention economy (see [8] for a sustained discussion and critique of this), where much is designed and delivered to keep grabbing your attention, encouraging you to click on links and keep you returning to particular sites but it is nevertheless an interesting feature of the work. However, much more importantly, it should be understood that the casino aspects of the work and the market are not what the majority of workers seek. They instead seek stability and predictability; regular, dependable, decent paid work for people they can trust and who will communicate with them politely and productively when required. One of the biggest secrets of the AMT market is the amount of stable working relationships going on under the hood, managed either informally as certain Turkers always try to work for certain requesters or more formally through the qualification system or through direct contacts. This really should be no surprise because it makes the work and earnings less unpredictable and more trustable for all involved.

4.5 So What Does This Knowledge About Turking Tell Us?

The knowledge about how Turkers organise their work brings forth issues about task design and relationship management. Firstly it suggests that requesters should be tolerant and understanding. There are a set of reasons why jobs may be poorly done, terminated without being finished and so forth that may be due to technical difficulties or an inappropriate set-up for the microtask. Also, and this relates to other aspects of Turker profiles and demographics, it can be helpful to make HITs as intuitive as possible and include visual instructions, to make it easier for people with lower levels of English comprehension and even computer skills complete your tasks properly. There is a notable degree of variability in physical infrastructure and resources required to carry out crowdwork, not just globally, but within the same country. Where possible and sensible requesters could try to design tasks that are ‘light-weight’ and more ‘accessible’ in terms of technology, which would make crowdwork more ‘inclusive’ – for people of different quality of resources and different abilities; and reduce the time and effort spent ‘managing’ work that was not completed due to infrastructure failure issues [16]. Microtasks can also be designed specifically for mobile devices or with workflows that would mean that the HIT would not be failed if there was a loss in Internet connection or similar problem.

Secondly, be aware that in the vast majority of HITs, for the vast majority of Turkers (and this may well apply to crowdworkers in general) speed-to-HIT completion (and often HIT-to-HIT speed) are crucial to their earning power. They will do HITs as quickly as possible to acceptable quality. This is testified in their use of tools, shortcuts and scripts that buy them small amounts of time, and it is also clear from their many discussions on how to optimise for particular tasks. The nature of the work is that unless it is somehow required and enforced (e.g. if you want a crowdworker to reflect before answering, you might best not allow them to answer straightaway) you cannot expect the worker to take any more time than the minimum. This is a key feature of the work that requesters need to think seriously about in the design of their experiments, questionnaires and so forth.

Thirdly, try to build up a good reputation but be aware of who is doing your HITs, how quickly. If you have a good reputation in general you will attract the more expert workers with the tool set-ups that allow them to grab your jobs almost instantaneously. This will be more pronounced if your pay is at the high end and your jobs come in large batches. The effect will be less pronounced if each worker can only do the job once, as in the case of a questionnaire, and/or your pay is lower. It is important that you understand as to whether your research is compromised in terms of sampling if you have too much of your work done by a smaller or more stable returning group of workers. Whether these aspects of the market create issues for you will depend on your research and your experimental design.

Finally, it is worth considering that you may want to develop professional working relationships with a corpus of crowdworkers, particularly if you want to do repeated work over a period of time. You could establish a group of workers through a system of qualifications for accreditation and post jobs directly to them through the qualification system. You could even collect enough workers that you could sub-sample within them. Another feature of this is that you could notify them in advance of upcoming work and use them in the beta and usability testing of your experimental design. If you have large projects that you intend to crowdsource over time this type of model, that involves some organisation without it being too laborious, would have a number of key benefits in terms of much less uncertainty over their work products while also being an ethical approach that was generally welcomed by the crowdworkers.

5 Leveraging Our Understanding of the Crowd for Research

As has been stated on a number of occasions, crowdsourcing currently operates in a legal grey area – i.e. pre-existing laws do not clearly map onto the territory it occupies, and therefore, which legal principles should apply and how they should be applied are still open questions [11, 12, 32]. The producers and owners of platforms are keen to remove and excuse themselves from the labour relations of those employers and employees, or requesters and providers (as independent contractors). The discussions over how these entities should be named and known is also a reflection of a desire to carve out a new legal territory in a number of ways. The ambiguity about how to categorise this work legally – while providing platform owners and requesters with opportunities to take advantage of lower standards regarding pay and conditions – is also reflected in the way in which crowdsourcing has been treated by the academic research community. Was crowdsourcing simply a cheap data service, was experimentation with the crowd somehow different, and in which cases, which ethical policies procedures should apply, what about codes of conduct, pay and so forth?

Initially much of crowdsourcing use – most often AMT use – by the academic community basically fell under the radar in terms of ethics and legal considerations, most likely because it was new and unknown, it was anonymous and lacking a human face, academics generally followed market rates, and it was mostly used for data services (rather than testing or experimentally manipulating the workers themselves). Now, the legal discourse – if not yet fully the law itself – is catching up (e.g. [11, 12, 32]) and so are institutional and national ethics committees. This work seems to be trending in particular directions but it is not unanimous in position. In this section we will try to provide navigation through the key features of the ethics and legal aspects of crowdsourced piece-work.

Crowdsourcing platforms clearly have their own sets of rules – or policies ²⁴, although these are both wide-ranging and vague, and it is not clear just how stringently or fairly they are enforced. More importantly, however, Turkers themselves have very few rights and no clear and transparent means of complaint and restitution if they think they have been mistreated. If they are understood as being independent contractors entering into a series of singular HIT-based contracts with essentially anonymous requesters they can only really appeal to Amazon at the moment about bad behaviour on the part of requesters. However, Amazon does not have a transparent and binding grievance process and the evidence from Turkers suggests that Amazon is not particularly responsive to complaints. It would not make sense practically for a Turker to pursue a requester legally over a single non-payment given the small amounts of money involved. As has been stated on a number of occasions (e.g [1, 35, 45]) in order to deal with this issue Turkers essentially follow a social policy of ‘name and shame’ through their forums, groups or technologies like TurkOpticon. This does not recover the money unpaid but it helps them avoid losing more.

In light of this current situation we would suggest that researchers navigate the situation as follows. (1) as we wait for the law to fully catch up with crowdsourcing it seems like we should heed the argumentation of legal experts already writing about the topic and should follow their suggested classification of the situation. This can set standards for how this work should be viewed and managed, what sorts of principles might apply and so forth. (2) Professional bodies and academic institutions are now beginning to subject research applications involving the use of crowdsourcing to ethics committees and are writing about it in their ethics guidelines – for many in e.g. universities this is becoming an institutional ‘fact of life’ – they will need to submit their work for prior approval, and so this will impact the way they do research. We want to discuss the key features coming out of both of these domains without offering an exhaustive or precise position, but rather mapping out the general territory and pointing out the key issues at stake. (3) As a pragmatic counterpoint to these professional, technical, and official responses and guides we would also like to offer some comments on best practice and ethical conduct from a mundane perspective, i.e. what should the everyday, human, interpersonal ethics be that operate in this situation? There are now a number of on-line resources that offer best practice guidelines from this perspective, the most detailed of which is provided by the Dynamo project²⁵, in their guidelines for academic requesters. Their site provides detail on such matters as pay and conduct and links to external resources on ethics and other guides as well as having a considerable amount of academic and Turker signatories to their guidelines. We believe that by following ordinary practical ethical approaches like these researchers’ actions will be compatible with legal and ethical requirements and that these ordinary ethical principles should be followed simply as a demonstration of good faith and courtesy.

When we explore what is good or best practice in relation to crowdsourcing one important strand of discussion is about labour rights, working conditions and employment ethics, the second strand concerns academic ethics, treatment of participants and ‘subjects’, and national ethics documentation and review boards. The first strand should apply in all cases of crowdsourcing use while the second strand should apply as additional principles and practices to a subset of crowdsourcing use; for academic or professional use where the situation is covered by institutional or professional rules regarding research. We would argue strongly that within academic/professional research there are clear differences between cases of crowdsourcing use for data services (e.g. image tagging) and use for, for example, psychological experiments, and thus some differentiation is most likely needed in terms of required principles and practices. However, there will always be marginal cases – and indeed ruling on these cases is a key aspect of ethics committees’ work – but there is a general difference in situations in academic or professional research where someone is being employed in a job as opposed to someone being employed as what used to be termed a ‘subject’ (but now more often as a participant) in an experiment. As stated there is also a third strand – and that is the one of ordinary ethics – which we will discuss as well. Firstly, we think it is a set of foundational principles that should apply in these situations, and therefore it underpins both the legal and formal ethical strands. Secondly, we think it is important to have guidelines that can be used in the current absence of a rigorous legal framework and for situations where ethical scrutiny and approval is not in play, even for research. In these cases we will provide a set of straightforward values and principles to apply to how the work and the relationships are managed, as well as pointing researchers towards relevant detailed resources.

6 The Legal Position

Crowdsourcing companies such as Amazon, with their AMT platform generally have sought to define the situation where they remove themselves from the labour relationship, simply acting as the market facilitator in enabling employers (requesters) to connect with independent contractors (providers). However – the lack of responsibility in this relationship seems a bit more tenuous when you consider that they extract a fee of between 20 and 40% per HIT. Configured (or viewed) in this fashion Amazon basically carries little legal responsibility apart policing the market for illegal use. It does not accept any legal responsibility over the functioning of the labour relationship.

In the US independent contractors are not covered for the following laws: minimum wage, compensation for overtime, antidiscrimination, family and medical leave, social security, unemployment compensation, protection for unionisation and collective bargaining, and more [12]. The anomaly, here though, as pointed out by Felsteiner [11], Finkin [12] and others, is that the only reason for these exemptions being applied to independent contractors is that it was aimed at highly qualified and paid professional people mainly doing consultancy work, with the thinking being that they would be more than enough compensated for their work and this would ease any bureaucratic burden on either side that would serve as a barrier to their doing business. The law was never intended to further penalise those in precarious badly-paid work.

The alternative analysis of the legal situation offered by both Felsteiner and Finkin is that crowdwork should be viewed as home-based piece-work in the same way that filling envelopes, craft industries and routine administrative work was ‘put-out’ to a mass distributed workforce, mainly working from home. While this work has always been a welcome source of income for part of the workforce, with various benefits (working at home and not incurring travel and other expenses, diminished supervision and control, some flexibility on working hours and so forth) it has historically also been favoured by employers due less legal protection combined with the ability to scale production up-and-down easily according to demand without extra cost – both of which can save them a considerable amount of money. Finkin (ibid.) points out that digital crowdsourcing also offers (to the employer) a few advantages over traditional home-based piece-work: there is no need to provide any equipment, surveillance can be much greater, and there is less opportunity for sharp practice.

In consideration of home-based piece-work it took a lengthy legal battle until the right to minimum wage and other protections were established, so while the precedent seems sound overall it may take some time for the legal situation to be clarified. As Felsteiner (ibid.) points out, one of the main sticking points may be the fact that crowdworkers work for multiple employers on a HIT per HIT basis, rather than a single employer, who would assume legal contractual responsibility as sole employer. However, in the case of Crowdflower, a company that contracts workers to do crowdsourcing work on different platforms (including previously AMT), there has been a definitive ruling that minimum wage should apply²⁶. In Europe there are already much stricter labour laws which is one of the reasons why AMT does not operate there, and on balance the evidence suggests that at some point minimum wage will apply to crowdsourcing in the US. All of this suggests that crowdsourcing employers should pre-emptively look to price work at at least minimum wage level of the most developed country targeted – in this case that would be $7.25/hour.

7 Professional Ethics

Most nations have developed ethical principles for conducting research basically derived from the following documents: the UN Declaration of Human Rights²⁷, the Nuremberg Code²⁸, the Declaration of Helsinki²⁹, and the Belmont Report³⁰. In fact, often the terms used in documents about research ethics relate to the key terms in the Belmont report (1979) [20]:

• Respect: which covers that people must be treated as their own agents and that people who have diminished capacity in some manner (are very young, do not speak the language, are coping with some other challenges, etc.) have the right to some protection. The notion of respect in research studies is usually implemented through informed consent.

• Beneficence: that treating people in an ethical manner goes beyond not just doing no harm but includes the importance of some effort into ensuring their well-being. This concept requires the thorough assessment of the type and extent of the associated risks.

• Justice: builds on the idea that injustice can occur from over-burdening a person or a group of people or from denying access to a person or a group of people. The idea of justice is usually reflected in the use of fair distribution and selection of participants.

One can see how these concepts are reflected in most ethics guidelines. Through the 1980s and 1990s most nations have developed ethics boards and associated procedures to ensure that research involving humans is conducted ethically.

To look slightly more closely at one particular national example, in Canada there are three primary federal research funding bodies – one each for health, natural sciences and social sciences. For ensuring that research is conducted ethically they have joined forces forming a tri-council. In August 2010 they released the 2nd version of what they term a living document: Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS). They call it a living document because the intention is that it continues to be a work in progress that can be adjusted. The formulation of this document deeply involved the active research community including over 2,000 interviews and 370 briefs. The basic premise is to preserve human dignity through adhering to principles of respect, concern for welfare and justice. In Canada adherence to TCPS is strictly enforced. One cannot get research funding if one does not follow these ethical guidelines. Respect includes factors like making sure that all participants give consent prior to the study starting and that this is open and informed consent. Concern for welfare of participants covers not increasing the risks beyond the risks of everyday life. Justice includes concerns about treating all people equally – so that no one group is either unfairly receiving either more possible harms or more rewards by being involved with the research. This very brief summary does not get into the innumerable subtle details such as the possibility of obtaining ethics board consent to run a study where deception is part of the study design but the board authorises the research if it feels that the long term benefit to humans outweighs the short term effects of the deception. For any research to be conducted a detailed ethics proposal must be approved by an ethics board. In Canada running a study with crowdworkers falls under TCPS and experimenters are required to get TCPS approval before commencing. In many countries there will be similar guidelines and procedures to follow, particularly in academic or professional work.

However, if one steps through the normal ethical experimental process, the process for a crowdworker, of necessity, differs.

• 1. Informed consent: the process of informed consent could be thought of as comparable in that similar information can be provided on-line as in person. However, on-line one does not really know that the participant has read the information but one can know that they had the opportunity to do so.

• 2. Task or activities: when the experimenter is there in person they can ensure that the conditions are reasonable with respect to such factors as adequate lighting, reasonable temperature, comfortable situation, etc. For a crowdworker, the experimenter does not really know under what conditions they are completing the activity.

• 3. Voluntary activity: it is important for ethical experiments that the participant knows that they can stop any time. This probably does hold up for on line experiments.

• 4. Risk: since the actual situations are not known, associated risk is not known either.

• 5. Debriefing: while ethical experiments always include the possibility of debriefing where the participant can ask the experimenter for more information or explanations, this is often not a part of the crowdworker experience.

However, there is a fundamental difference in that the crowdworkers are doing the HIT as a job. Their motivation is the pay. In a lab, while there might be some reward, it could be juice and cookies or a gift certificate or possibly some money but is not often thought of as pay and people do not usually make a living being participants. This puts quite a different colouring on the Turkers’ activities. Different rights emerge such as the right for a minimum wage (see legal section above). This is further complicated by the fact it is piece-work, and further complicated by the fact it is ‘perfect’ piece-work. That is a Turker is quite likely to not get paid if the requester does not like the work they have done. The important point to note – is that employment law and minimum wage considerations apply in the vast majority of crowdsourcing situations (especially in microtask based crowdsourcing) just as much as the ethical guidelines and procedures of any given national or professional jurisdiction.

8 Practical Relationship Ethics

Human interaction and relationships are morally ordered and organised all the way down. This is not to say that interaction and relationships continually involve elevated moral questions, seen in the local micro-organisation of talk up to the topics of everyday conversation, although serious moral and ethical questions can and do arise. The key point we want to make here is that participants orient to the organisation of interaction and therefore the management of relationships at the lowest level as having a moral order to do with norms, convention, expectations and trust. When these are broken without apparent due clear reason, people react with moral indignation. Garfinkel [15] brought this into clear view through getting his students to conduct a series of ‘breaching experiments’ in which they were to break the often unarticulated ‘rules’ of forms of interaction and relationships. For example, by asking for continual clarification of open but accepted norms of interacting such as responding to ‘how are you feeling?’ by asking ‘mentally or physically?’, then when answered ‘physically’ to again ask for clarification ‘muscle ache or in your bones?’ and so forth until the interlocutor became often rather annoyed. Or in another case the student was to behave as if they were a ‘boarder’ in their own (family) home – again causing real consternation to their family. The purpose of these ‘breaches’ was to demonstrate what normally was simply taken for granted in the ways in which interaction and relationships were managed – what was the trusted context and set of norms that people we expected to adhere to.

The reason for introducing this material in relation to crowdsourcing is that one of its strongest selling-points was the idea that the need for human interaction and relationships could be removed from the work situation – everything could be handled, anonymously and digitally, in cold ‘mechanical’ transactions. Looking beneath the surface, away from the sparse, emotion-free interface we find that even in these stripped-down interactions and relationships which may be simply comprised of pseudonyms, the microtasks themselves, the speed and form of acceptance or rejection and the pay (or not), there is a whole moral order of interactions and relationships. We see this most clearly in the forum and group discussions concerning how good or bad behaviour, respect or the lack of and so on are inferred often from very small details. Trust in crowdsourcing comes through human relationships even though they are diminished by the distance and the technology: the qualities of the exchanges determine the level of trust from the most basic ways in which impersonal transactions are handled, up to direct communication (it is important and even necessary to communicate at times), and even on to regular employer-employee relationships³¹. The other key point to note is that in general a lot of what we need and expect in terms of relationships and interaction in more conventional work carries over in principle, although sometimes differently in detail, to crowdsourced work: people still want to be valued, have their work appreciated and to be treated with politeness and respect – although they do generally like the fact that there is a degree of privacy. This really should not surprise us since it is integral to healthy well-functioning human relationships.

9 Conclusions

As a grassroots campaign, two popular initiatives have emerged from a practical approach to ethics: the Dynamo project (see Footnote 26) [43] and the Code of Conduct³². The Dynamo project provides a set of guidelines and links to further resources detailing best practices and professional ethics for research, with the added bonus of Turker and requester signatories. The Code of Conduct is an example of a more concise set of principles and guidelines developed in Germany as a joint production between crowdsourcing platforms – Testbirds, clickworker and Streetspotr³³. For researchers planning to use crowdsourcing, following the guidelines offered in these two campaigns would be a good place to start. Researchers should make themselves aware of the rules and policies of the platform that they are aiming to use, and the rights for different parties involved before they set out to get the crowd to do any crowdwork. Conducting research or supporting a business through crowdwork should serve by promoting respect and trust in the marketplace. For the requesters this has a straightforward practice advantage: it would attract more of the better workers to their microtasks, as well as enhance their reputation as a requester.

We believe developing an understanding of the broader context of ethics in research is important. It is of practical value for researchers and academics using or wanting to use crowdsourcing platforms for their research, or simply to get work done. The information given in this chapter should help them make choices about platforms, microtask design and how they interact with and treat crowdworkers. In this chapter we have discussed in detail who the crowdworkers are, why they do this form of work and how they organise it. We have then set this in context of legal, research and labour ethics. We finished the chapter with an in-depth discussion of legal and ethical issues that are relevant to both the use of crowdworking platforms and the way relationships are managed. We want to conclude this chapter by providing six clear take-away points for people interested in using crowdsourcing:

• 1. The people completing microtasks on crowdsourcing platforms are overwhelmingly workers providing labour, rather than people volunteering for a good cause or doings microtasks for leisure. Accordingly, they should be paid a decent wage/fee for their services and should be treated with respect and communicated with politely.

• 2. Building a good reputation as a requester will make microtasks attractive to skilled, trustworthy crowdworkers. A good reputation comprises of a selection of the following attributes: good pay, prompt pay, is fair, polite, good at communicating, has well designed tasks (nice interfaces, function well, clear instructions), posts batch tasks regularly.

• 3. Following ethical guidelines and engaging in productive communication and relationships with crowdworkers should help maximise the effectiveness in using these platforms and gaining high quality results for tests, experiments and any microtasks in general. In this chapter we have described some ways in which this can be put into practice.

• 4. Different platforms have different orientations towards governance, care of those using the sites, worker rights and pay. You may wish to choose a platform on an ethical basis, however, you may also use a platform like AMT – which has a laissez-faire approach to governance – in an ethical manner.

• 5. You can choose between platforms as a means of accessing workforces with different demographic constitutions and different skills sets or different task capacities. You may want to look further than simple cost in choosing a platform such as technological support and expertise offered by platforms. This is crucial as this can help you avoid hidden costs associated with design problems as well as avoiding frustrating the workers.

• 6. Academic work making use of crowdsourcing often requires ethics approval, and even when this is not the case researchers should still follow professional ethics and use the ideas from ethical research to interact with their participants in an ethical manner. A number of sets of codes and guidelines on ethical use of platforms are readily accessible and are referenced in this chapter.

Appendix: Survey Data

See Tables 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 and 13.

Table 2. Distribution of Gender observed for AMT workers (survey studies and on MTurk Tracker).

Table 3. Distribution of Gender observed for Microworkers and Crowdee workers using survey studies.

Table 4. Distribution of Age observed for AMT workers (survey studies and on MTurk Tracker).

Table 5. Distribution of Age observed for Microworkers and Crowdee workers using survey studies.

Table 6. Distribution of Household Income observed for AMT workers (survey studies and on MTurk Tracker). Please note that for the MTurk tracker data, the data is not available for all income classes (rows) and is therefore aggregated over two classes.

Table 7. Distribution of Household Income observed for Microworkers and Crowdee workers using survey studies. 22.18% of Crowdee participants did not report their household income.

Table 8. Distribution of Household Size (including the worker) observed for AMT workers (survey studies and on MTurk Tracker).

Table 9. Distribution of Household Size (including the worker) observed for Microworkers and Crowdee workers using survey studies.

Table 10. Distribution of highest Education Level achieved observed for all platforms using survey studies.

Table 11. Distribution of Employment Status of crowd workers from all platforms using survey studies.

Table 12. Distribution of Times Crowd Workers Spent on All Platforms (per week).

Table 13. Distribution of Stated Task Approval Rate of crowd workers on all platforms. For the Crowdee platform, no data is available for the stated task approval rate.