Keywords

1 Introduction

Advances in communication technology and internet created possibilities for people across the globe to increasingly join into mass collective projects and share their contributions to create value. Emerging of mass collaboration and its application to different domains enabled multitudes of humans to build powerful hubs of resources, skills and knowledge, helping to find solutions for a wide variety of problems. Indeed, it opens a wide range of opportunities to truly harness the power of groups, leveraging resources and driving profound societal changes. In comparison with other forms of collective action, it is amazing that even when each contributor pursues his/her own interests in a mass collaborative project, still a coherent product may come out of it.

There are many applications of mass collaboration. In education for example, it emerges in the light of collaborative learning where a large number of uncoordinated contributors give themselves the chance to learn, adapt, and achieve impact together. It refers to a mechanism in which learners at various levels of performance not only proactively acquire and share a wide variety of materials but also autonomously contribute to knowledge creation and consolidation. It is, indeed, a great shift from a formalized and centralized to an informal and self-directed form of learning [1]. Mass collaborative learning accommodates recent changes in technology and methods of learning leading to a new paradigm of education. Opposite to formal learning delivered by instructors in a systematic intentional way within an educational setting, in this case knowledge is created, revised, and shared in large scale within informal collaborative communities.

Knowledge and information can appear in a different variety of kinds (e.g., stories, interpretations, opinions, and facts) and created for various purposes (e.g., to sell, to inform, to present a viewpoint, to encourage). For each one of these diverse kinds and purposes, knowledge and information can enormously vary and differ in terms of value, reliability, nature, granularity and lifespan. It can range from high to poor quality and include every shade in between unlike traditional printed materials in newspapers, magazines, books and academic libraries which are somehow regulated for quality and accuracy [2, 3]. Therefore, along with informal learning in mass collaborative projects, there is a strong need for being able to discern the quality of knowledge or information created in whatever format on the internet and social media. Forasmuch as the degree of learners’ skills and proficiency in creating and sharing right information and knowledge is varied so the accuracy and reliability of created and shared content then becomes a main concern. Even though large amounts of high quality information and knowledge are available on the internet and social media, there is also countless incorrect knowledge (mistakenly created and spread by honest people), false knowledge (deliberately generated and diffused by dishonest people), half-truths, fallacies, distortions, exaggerations, urban legends and plain old lies [4]. Hence, it is for learners indispensable that instead of easily “accepting” or “rejecting” each knowledge claim, they can adopt a skeptical attitude towards all received piece of knowledge or information, and as a critical and savvy user put it in a “this is claimed” pile, and neither accept or reject it upon receiving. Moreover, they should not be deceived by the appearance of knowledge or information that looks just as professional or reported as highly credible and reliable sources.

The rapid development of knowledge and information repositories, proliferation of web-based knowledge applications and services, and easy access to diverse sources by knowledge users and learners have augmented the awareness of, and the need for high quality knowledge sharing in communities. In this context, quality is, indeed, a buzzword which evaluation is quite complicated. Many concerns have been raised recently about it, particularly in created and shared knowledge on the internet and social networks, and the possibility of detrimental effects that emanate from unreliable knowledge or information. That is the case, for instance, of the impact of fake news on politics. Learners require being certain that acquired knowledge is up to date, reliable, accurate, relevant, objective, and the degree of its quality is high. Nowadays, quality is considered a crucial issued for education in general and for mass collaborative learning. Although it remains an open challenge, over the last years it has been increasingly the focus of attention for many researchers to meet the needs of communities trying to evaluate and promote the quality of knowledge [5,6,7]. The literature also shows that there has been a great deal of effort to identify and/or introduce mechanisms for evaluating the quality of knowledge and information in different domains. As result, various mechanisms start to appear today for this purpose [8,9,10], nevertheless, the issue is far from being solved.

This paper results from a systematic literature review aiming to identify what kind of knowledge quality assessment mechanisms and Supportive tools could be developed to make a mass collaborative learning community more resilient against unreliable knowledge. A summary of the survey is included to address some of identified factors and constituents namely: the type of organizational structures, collaborative learning approaches, adopted technologies, and adopted methods for evaluating the quality of information and knowledge that have influential impact on mass collaborative learning.

It is expected that the findings of this study provide a comprehensive overview of the affecting elements on mass collaborative learning and help developing a better insight into how to evaluate the quality of created and shared knowledge in communities combined with supportive tools that can make their members more resilient in face of unreliable sources. It is also envisaged that the research findings of this study build a solid foundation for better developing the next research phase, which is generically guided by the following question:

Q: What kind of methods and supportive tools can provide an appropriate basis to help evaluating the quality and reliability of co-created knowledge or information in mass collaborative learning projects?

The reminder of this paper includes a short review of community resilience in Sect. 2. A synthesis of the state of the art is then presented in Sect. 3, and the research directions and plan are the topics of discussion in Sect. 4, followed by concluding remarks in Sect. 5.

2 Relationship to Resilience

Resilience has become a significant issue in many fields, reflecting the capacity of an economy, organization, city, forest or individual to deal with perturbations, and cope with all kinds or traumatic experiences. It represents the ability to successfully maintain a stable healthy level of physical or psychological functioning. In psychological terms, resilience can be employed to give us a scanty sense of hazard and let us get back to feeling normal again after any shape change [11]. Positive adaptation to development, reorganization, and renewal is also another essential aspect of resilience, but it has been less in the focus of attention [12]. Evidences in the literature show that resilience is in fact an ordinary, not uncommon function. People normally demonstrate resilience, for example, they respond to destructions caused by a storm, and make efforts to rebuild their houses that were destroyed by that stressful event. Resilience is in fact important for several reasons [13, 14], including:

  • Provision of opportunities to protect people against conditions that might be overwhelming;

  • Enabling people to develop mechanisms for managing extreme events during disasters;

  • Helping governments, communities, and people to create more prepared and safer environment;

  • Helping people to use resources and assets quickly;

  • Promoting physical, psychological health and well-being, and mitigating the rate of mortality;

  • Helping to decrease the stress on health care and the rate of risk-taking behaviors such as addiction, overuse of drugs, excessive smoking and drinking;

  • Helping to promote studying and learning achievements, and

  • Enhancing engagement in family activities and community collaboration.

A community that is resilient can harness, utilize, and develop nearly all possible resources to properly react and withstand against adverse situations to mitigate the rate of risk, and recover from emergencies. Resilience, indeed, enables community members to come together, intentionally promote their personal and collective capabilities, raise awareness of sustainability, respond effectively to turbulent changes, minimize impact of disasters, implement required plans and pay needed attention to urgencies, return to normal situations, and build development trajectory for future success [15].

Governments, organizations (specifically charitable ones), academics and communities show growing interest on programs that enable building resilience in face of turbulent changes and mishaps although the process of resilience-building is not precisely clear and there is not much understanding about what are its constituents. Nor is also there as a single approach or good model that could be used to build resilience for communities of all kinds. In addition, the literature is scant about what hinders or helps the community to be resilient in a disaster context. Evidences show that it is essential to consider what the community is used for; what is the vision of community; what are available resources and what are needed, who are members, how much understanding do members share about self-resilience, community-resilience and the risks they face, how much members are active and integrated, what factors cause community growth and decay, how much community is dynamic, and so forth. Nevertheless, it is notable that resilient communities share certain characteristics such as involving active participants, communication, cooperation, collaboration, loyalty, defined roles, diversity, sufficient resources to meet community needs, etc.

To make clear the process of building resilient communities in the scope of collaborative projects, some steps are proposed in [16, 17] which are summarized in Table 1.

Table 1. Proposed steps to clarify the process of building resilient communities in collaborative projects
Full size table

It is a widely held view that the more a community can leverage disasters as an opportunity to improve, the more resilient it is. To build and bolster community resilience, and thus augment the capacity to cope with perturbations, various tools, mapping methods and guidelines are suggested that can streamline the process. For instance, creating networks, sharing knowledge, and utilizing diversity of ideas and experiences [18] by developing collaborative approaches [19] can improve the levels of community learning to tackle complicated problems [20, 21]. It is assumed that incorporating specific kinds of approaches such as, extending the size of community to mass level, shifting to collaborative networks, exploiting mass knowledge co-production, finding proper mechanisms for evaluating the quality of co-created knowledge (e.g., creating nodes of expertise, and feedback loop), and adopting quality measuring tools can be helpful in making a community more resilient.

It is expected that by taking the advantages of fundamental properties of the collaborative communities such as, adaptability, efficiency, diversity, and cohesion we can leverage the opportunities to design a system with inherent resilience. Such resilient system creates the possibilities to reduce the risks associated with the attack of false inputs and their adverse impacts. That is, a developed sustainable system can help collaborative-networked learning groups to constantly maintain high level of preparedness against unreliable materials. Besides, it can strengthen communities and their members specifically those that are more vulnerable to withstand major threats related to wide spreading of unreliable knowledge or information in online environments.

In this context, our work focuses on the impact that untrustworthy information and unreliable knowledge can have on a community through mass collaborative learning processes. We are particularly interested in contributing to a better understanding of the mass collaborative learning concept and identifying approaches to deal with unreliability. In this way, we expect to contribute to more resilient communities.

3 State of the Art Overview

Literature shows that there is growing tendency in response to the need of communities for fostering collaborative learning in effective ways. Several contributing factors, e.g., pedagogical approaches, ICT-infrastructures, educational programs [22], learning environments, learning designs, and learning interactions [23, 24], etc., are highlighted in recent years as vehicles to better engage learners in collective learning, incite their passion for constructive social impact, and develop a foundation for next generation of learning approaches.

The main findings of the literature survey, which included reviewing about 100 papers in mass collaboration context and related areas, whose findings are succinctly presented in the following subsections.

3.1 Organizational Structures and Mass Collaboration

An organizational structure acts as an “instruction” for decision makers to more easily assign plans, strategies or decisions which are useful for their group [25]. Small or large, every community must operate with an appropriate organizational structure because, for example, it assists better identifying responsibilities and roles, utilizing and controlling resources, binding group members and pointing them common goals, facilitating decision making processes, making easier communication, etc. [26]. The type of structure indicates in which ways internal works can be carried out at all levels of the community. Basically, the goals and strategies of the community, and the type of members’ or customers’ needs are the main determinants for selecting a structure.

The organizational structure has profound impact on collaborative networks, and largely builds the level of autonomy and collaboration with and amongst the members. Evidences show that as the communities are more and more evolving from small and medium size to large scale collectives, and from non-computerized to a digital-based model, there is a need for structural adaptability. That is, shifting away from traditional structures (e.g., hierarchical, centralized, etc.) towards unconventional models (e.g., informal, self-directed, etc.). Although there are vast amounts of literature on organizational structures, there is very little work trying to specifically evaluate the role of organizational structures in large-size networked collaborative learning. Having reviewed several suggested models in relevant areas, from which no suitable organizational structure for mass collaborative learning could be found, the taxonomy recommended in [27] was selected as the closest fit with the nature and type of structures applied by virtual communities in collected papers on survey. As illustrated in Table 2, two main forms of collaboration in networks - collaborative networked organization and ad-hoc collaboration - root the main classes of this taxonomy.

Table 2. Taxonomy of collaborative network [27]
Full size table

Considering this taxonomy, an analysis of all collected papers shows that the issue of organizational structure was addressed in 32 papers. Details on the percentage and type of applied structure in those papers are illustrated in Fig. 1.

Fig. 1.
figure 1

Percentages of applied organizational structures in collected papers

Full size image

It is noteworthy that the type of organization structure which was used in three articles in collected papers falls between VBE and PVC categories in above taxonomy (a kind of hybrid). As it can be seen in the figure, informal networks and business ecosystem structures were the least and virtual team structure was the most addressed structure in the selected papers. From the findings it can be inferred that the organizational structure should be adopted based on community purpose, type, size, needs, communication devices, and environment. Furthermore, as the level of collaboration is shifting to large scale, structures should be dynamically reconfigurable. In other words, it needs blending available resources, skills and competencies across the community to take advantage of collaboration opportunities. Finally, in mass collaborative projects the more the organizational structure is dynamic and the better members’ network position fit with the organizational context, the more knowledge they can acquire.

3.2 Collaborative Learning Techniques (CoLTs) and Mass Collaboration

Collaborative learning in broad sense is a type of education approach in which learners in a group attempt to accomplish a common goal. In such group members are responsible for both their own tasks. CoLTs can make straightforward the process of discussion, and sharing knowledge, opinions and experiences for learners. Applying these techniques can provide supportive directions for development of learning and collaboration in communities of all sizes. Different areas of study over the years have benefited from using a variety of CoLTs [28]. However, their application to mass collaboration and learning has not received much attention. Therefore, there are not enough evidences in the literature showing that what types of CoLTs are exactly required for this purpose, and how these techniques can facilitate learning where the process is entirely self-directed. Despite such limitations, many proposed techniques from relevant domains were reviewed in order to pick up the ones that are most promising. To this end, the taxonomy offered by [28] was selected as it is reasonably comprehensive. It encompasses 5 major categories of general learning activities and 31 CoLTs (see details in Table 3).

Table 3. Collaborative learning techniques [28]
Full size table

The analysis of collected papers reveals that CoLTs were considered in 23 papers. In this regard, techniques for discussion received the most attention and techniques for reciprocal teaching received the least attention in those papers. More details about the percentage of applied CoLTs are depicted in Fig. 2. Although this taxonomy is not specifically designed for mass collaborative learning, findings reveal that some of the techniques such as group problem solving, peer editing, and paired annotation have seemingly potential structures to guide the development of mass collaborative learning projects. Moreover, techniques like note-taking, which provide elaborated explanations and reflective feedback from partners, can enhance the chance of learning in mass collaboration.

Fig. 2.
figure 2

Type and percentage of adopted CoLTs in collected papers

Full size image

3.3 Supportive Tools and Mass Collaborative Learning

Recent interest in technology support to collaborative learning represents a confluence of trends such as the emergence of constructive approaches for learning [29], the aim to build more influential learning environment [30], and the advance of new technologies to support collaborative learning [31]. However, neither every form of collaborative learning necessarily needs the same type of technology, nor can a single tool provide all required features.

Literature shows that new supportive tools such as CSCL, social media, web-based and mobile technologies along with Internet, have equipped large number of learners around the world to comfortably communicate anytime and anywhere, and empowered them to exchange their resources, knowledge, and experiences. However, the real use case for technology in mass collaboration is still evolving, and as such, improvements in learning outcomes for self-directed learners are yet to be proven. There are also several open issues, such as: how can supportive tools efficiently process the massive load of content?; or how can needed training or information be provided for a single learner in the community who does not have enough technical knowledge? As such, a comprehensive list of specific applicable tools for this purpose is not yet proposed. Therefore, considering different related models in the literature, the Project-Based Collaborative Learning Model [32] was selected to check which of proposed tools in this model are also used in the analyzed papers. Seven distinct phases of this model and more details are exhibited in Fig. 3.

Fig. 3.
figure 3

Technology support for project-based collaborative learning model [32]

Full size image

From this analysis, it was evidenced that nearly one third of collected papers (35 papers) evaluate the role of technologies in support collaboration and learning. It also shows that communicative tools are the most, and consensus building tools are the least employed tools. More detail and percentage of each applied tool are shown in Fig. 4. Therefore, it can be inferred that resource management tools can bring some opportunities for mass learners to access, evaluate, utilize, and share their resources more readily. These tools can also help transforming complex tasks into easy-achievable works. Moreover, tools such as Routing, Milestones, and Calendaring seem to not have as high chance as Wiki, Discussion board, and Blog for application in mass collaborative projects.

Fig. 4.
figure 4

Type and percentage of adopted technologies in collected papers

Full size image

3.4 Evaluating the Quality of Created Knowledge in Mass Collaboration

Since knowledge is recognized as one of the most strategic assets for every organization and community, evaluating the identified, acquired, created, shared and/or retrieved knowledge influences community’s prosperity. For every community it is important to know how to identify the quality and reliability of acquired/possessed knowledge. Evaluating the quality of knowledge can help in, for example, identifying strengthens and weakness, providing guidelines that could be helpful for future plans and development, improving effectiveness, and so forth [33]. The intangibility of knowledge, however, makes the process of evaluation somehow difficult in both practice and research. On the other hand, when large amounts of knowledge and information are exchanged through the Internet and social networks by known or unknown users serving different purposes, make the evaluation crucial. It becomes more complicated as online knowledge and information can be easily altered, misrepresented, built up, or plagiarized.

Despite quality of knowledge has been a topic in different fields of study and businesses, and various types of strategy, methods, and question have been proposed for this purpose, it has not been well studied yet particularly when integrated with mass collaboration. Therefore, we could not find a comprehensive mechanism or list of applicable methods in the literature which can be applied for gauging the quality and reliability of co-created and shared knowledge or information. Furthermore, no systematic research for addressing this issue has been already conducted. Hence, we collected a number of general methods, namely available in Wikipedia (Table 4) and compared them with solutions proposed in the analyzed papers in order to gain better insight about this scope of study.

Table 4. Suggested methods for evaluating the quality of knowledge in collected papers
Full size table

It is worth noting that for appraising the quality of articles in Wikipedia, eight major criteria are under consideration including, accuracy, comprehensiveness, stability, well written, uncontroversial, compliance with standards of Wikipedia, having appropriate style, and having appropriate images. Furthermore, in this approach some methods are commonly used for example, nominating qualified articles, reputation mechanism, peer review, and feedback, to name a few [34].

From the findings of collected papers it can be concluded that:

  • User feedback and expert evaluation were the most suggested methods;

  • The role of top contributors is deniable;

  • Publishing the result of evaluation could be helpful for all learners;

  • Both qualitative and quantitative approaches should be considered; and

  • Combination of machine learning and human factors seems work better.

4 Research Direction

In this section, we propose a research approach for tracking the reliability and quality of online knowledge or information in the context of mass collaborative learning. The goal of our approach is to integrate human and computer support to reach an optimal balance between simplicity and speed on one hand, and validity of result on the other. Hence, at the current stage, we envisage a prototype comprising two main parts: human part and computer part. The human part involves an individual phase and a community phase. The community phase benefits from the contribution of both ordinary and expert members. The computer part provides supportive tools (e.g., fake news detection, website or resource detectors) which can raise red flags on unreliable and questionable contents. More details are presented in Table 5.

Table 5. Suggested issues for evaluating the quality of online knowledge/information
Full size table

The planned approach for the collaborative evaluation consists of the following steps:

  • Step 1: individual phase - a community member quickly checks the knowledge or information manually to decide whether it is worthy enough to warrant further evaluation or not. After manual filtering, a defined check list will be completed for those items that are accepted to investigate in more detail. By applying cognitive skills (critical thinking and critical appraising) the member assesses by self-checking the reliability of knowledge or information based on suggested criteria (mentioned in Table 5) and gives each of them an emotional rate. The individual decision can then be made based on the given rates.

  • Step 2: once individual decision is made, that is the time to take advantage of crowd sourcing that takes place in both levels (ordinary or expert members) of community phase. In this step, at first ordinary and expert members separately but in parallel complete the check list and give emotional rates. The reasons and evidences for given rates are then shared among contributing members. Afterwards, developing formal argumentation and collective evaluation not only enables community members to gain common sense about the findings but also helps reaching results that are beyond individual’s ability. The results of decision in this phase should be visible for all.

  • Step 3: the final decision about the quality and reliability of knowledge or information evolves from the evaluation of results in both phases.

It is worth mentioning that detector tools can be used to support community members throughout the evaluation process. Figure 5 exhibits these three steps.

Fig. 5.
figure 5

Three steps for evaluation of knowledge or information in mass collaborative learning

Full size image

5 Conclusions and Future Work

Emerging collaborative forms of learning in open networks and communities provide new opportunities for joint learning. With the objective of gaining understanding of current state of the art in mass collaborative learning, an extensive literature survey was conducted. As a result, various findings on the organizational structures, collaborative learning techniques, and support tools were highlighted.

However, mass collaborative learning also confronts community members with the problem of dissemination of unreliable knowledge or information through Internet and social networks. To prevent the negative side effects of such problem, an important goal is making community members more secure and resilient against vulnerabilities caused by online fraud. To this end, in this study a preliminary approach is suggested aiming at a combination of human and computer support to enable contributors taking advantage of collaborative evaluation in dealing with threats.

This work is still at a preliminary level, but it is expected that collective evaluation in different steps and along with support tools can provide learners and communities with helpful guidelines for achieving a high level of consciousness about the quality of acquired knowledge. In next phases of this work we intend to investigate what organizational structure for the suggested approach should be established? What kind of mechanism does the community need for evidence acquisition and combination? What kind of mechanism can help appropriately inferring the final evaluations and decisions?