1 Introduction

Social computing has become a transversal issue that affects not only collaborative systems, but also information systems in general. The relevance of considering socio-technical aspects in the design of these systems has been highlighted by their users, the scientific community and the industry (Baxter and Sommerville 2011; Grudin 1994; Oinas-Kukkonen et al. 2011; Patnayakuni and Ruppel 2010; Wang et al. 2007; Whitworth and Ahmad 2013). Several previous studies indicate that adding social interaction capabilities to information systems can improve user experience and performance (Barjis et al. 2011; Choi et al. 2013; Shami et al. 2014; Wang 2011; Yang et al. 2012), and also the suitability of the services provided by these systems (Geyer et al. 2011; Grgecic and Rosenkranz 2010; Hovorka and Larsen 2005; Shami et al. 2011; Zhang et al. 2010). However, embedding social interaction services in information systems requires a paradigm shift, not only in the way in which software applications are perceived by users, but also in the type of support provided to users’ activities.

The suitability of these services affects user participation, which is the cornerstone of any community (Leimeister et al. 2006; Preece 2001). A wrong intervention on the supporting system could negatively impact on the users’ participation, thus damaging the community, and even destroying it (Gutierrez et al. 2012a).

User participation is a temporal variable, and its value depends on how well-aligned are the services provided by the supporting system and the users’ interests and needs. In order to keep the community in a sustainable condition and react on-time in case of need, this variable could be monitored and eventually controlled by community managers. Social services can be added or removed to try dealing with the users’ needs, and thus improve their participation. Following the same goal, the participation strategy can be adjusted or periodically redefined by the community managers. Therefore, these managers must be aware of the community status, and eventual changes in its behavior, before taking any action.

The main challenges to keep user participation under control in a community are two: (1) determine how healthy is currently the people participation, in terms of volume and quality of contributions, alongside the community lifecycle (Iriberri and Leroy 2009); and (2) identify suitable strategies to positively impact such participation (Preece and Shneiderman 2009).

This article reports a case study, in which student participation in an undergraduate course community was intervened using extrinsic mechanisms. The main goal was to see the impact that these interventions have in the community members’ behavior. The consequences of these interventions can be identified and monitored by analyzing the community structure. Therefore, community managers can use this mechanism to monitor the community before and after an intervention. This can also be used to detect changes in the behavior of community members. These results help designers to address the first challenge (i.e., diagnose the participation of community members) and suggest strategies to deal with the second one (i.e., positively impact member participation).

During the study, students of three sections of an undergraduate course community were exposed (through the use of a learning platform) to different participation strategies. Each course section represents a partially virtual community (PVC). In these PVCs, members typically know each other and have the opportunity to interact among them through both, a virtual and a physical space in order to help reach an individual or common goal (Gutierrez et al. 2012b).

The first course community did not use a predefined strategy to promote participation; i.e., community members did not receive any external motivation to contribute with others using the platform. This community was used as the control group, since its general behavior adheres to what designers would expect (in terms of participation) when there is no intervention. The second community promoted the quantity of student contributions, and the third one promoted quality over quantity. All studied communities were partially virtual and comparable among them.

We monitored student participation in these communities during 15 weeks. The results show that keeping a permanent monitoring of the community structure and activity is almost mandatory, if participation is meant to be kept under control. The results also show that participation in these communities should be motivated using extrinsic reinforcements, such as social recognition, prestige or visibility of member contributions. Based on the obtained findings, we raise a number of implications to design community supporting platforms, considering the needs of both, community managers and members.

Next section reviews the related work. Section 3 describes the case study, the strategies used to promote participation, and the metrics used to analyze the community structure. Section 4 presents and discusses the obtained results. Section 5 is dedicated to show the implications to design derived from the study. Finally, section 6 presents the conclusions and further work.

2 Related work

Partially virtual and online communities need to count on a software platform that eases and promotes the participation of their members, and allows their managers to diagnose and intervene the community. Therefore, the supporting platform should address various design aspects in order to allow smooth interaction among community members, as well as to monitor and intervene the community by their managers. Next sections discuss the related work in these areas, particularly focused on student communities.

2.1 Supporting computer-mediated interaction in PVCs

The software supporting these communities must ease the interaction among their members; therefore their services should be aligned with the users’ needs. Provided these needs evolve with the time, the platform should also evolve to fit the new needs of the users.

Many community supporting platforms embed asynchronous interaction tools (e.g., discussion boards), because they provide flexibility and effectiveness to the interactions among community members; particularly in educational scenarios (Hammond 2005). Researchers and instructors claim that discussion boards reinforce the learning experience by increasing student commitment in their courses, resulting therefore in significantly better results (Martyn 2005; Davies and Graff 2003; Ochoa et al. 2012). Vonderwell and Zachariah (2005) found that technology, user interface design, content-area experience, roles and tasks, and information overload, influence the students’ participation and interaction patterns.

Schummer and Lukosch (2007) propose a set of patterns that should be considered when designing tools for supporting computer-mediated interactions. These patterns involve community and group support, which can be used by software designers as guidelines for conceiving and modeling new community supporting platforms. Similarly, Gutierrez et al. proposed a reference architecture (2014) for assisting the modeling of such systems, based on the authors’ experience designing and developing software platforms to support partially virtual communities. Typically, software architects and designers need to make design decisions in three criteria: (1) elements that will shape user identity in the community, (2) interaction mechanisms to be used for engaging community members and fostering participation, and (3) the structural sustain of the community in terms of motivation, governance and metaphors. Furthermore, when trying to involve physical members into virtual participation, it turns crucial to embed appropriate synchronous and asynchronous communication mechanisms (Neyem et al. 2011), conceive strategies for facilitating physical encounters (Herskovic et al. 2012; Westerlund et al. 2009), and provide efficient community awareness to its members (Acquisti and Gross 2006). Particularly, Antunes et al. (2014) provide a checklist of awareness mechanisms suitable for the development of collaborative applications.

All of these research works represent meaningful guidelines, which are potentially useful depending on the community current status (e.g., its level of maturity, stage in its lifecycle, and closeness of its members) at the moment to deploy these services to its members. Therefore, the community status should be captured and understood by the managers, before deciding to deploy a service into the supporting platform.

2.2 Determining the status of a PVCs

There is consensus that the current status of an online or partially virtual community can be captured by analyzing its structure and activity through time (Rowe and Alani 2012; Danescu-Niculescu-Mizil et al. 2013). Social scientists have historically used techniques from social network analysis to identify the underlying interaction patterns that emerge from a community (Wasserman and Faust 1994); and based on that, it is possible to diagnose the community current status. These interaction patterns also allow us to understand the impact of such social structures on other variables (Wellman 1996).

Social network analysis manages social relationships in terms of network theory; therefore, it models individual actors within the network as nodes, and the relationships between them as ties. Therefore, course communities can be understood as graphs where the students represent the nodes and the edges indicate the relationship among nodes. In the particular case of learning communities, several approaches for social network analysis have been successfully used to understand participation and interaction aspects during learning processes (Harrer et al. 2007; de Laat et al. 2007; Martinez et al. 2006; Reffay et al. 2011).

Provided that the structure and interaction in a community are represented by a graph, several metrics can be used to characterize and study them; for instance: degrees, centrality, density, clustering, cliques, and cohesion (Scott 2012). For diagnosing, it is important to detect network motifs, which are defined as recurrent and statistically significant sub-graphs (or patterns) that are present in the graph. Although network motifs help provide a deep insight into the network functional abilities, their detection is computationally challenging (Kashtan et al. 2004). Therefore, it is important to count on a visual representation of this information, in order to understand the network data and convey the result of the analysis (D’Andrea et al. 2009). These visual representations also allow managers to not only diagnose member participation, but also analyze its evolution over time.

2.3 Promoting participation in PVCs

Regulating the quality and quantity of user contributions, and therefore ensuring a sustainable level of user participation in an online community, requires an adaptation of the participant rewards for particular forms of participation, depending on the user reputation and the current needs of the community (Cheng and Vassileva 2006). In general terms, the problem of encouraging participation in online communities has been observed mainly from a social psychology point of view. For example, Cheng and Vassileva (2005) proposed a motivation strategy based on persuasion, in order to reinforce the value of quantity and quality in user contributions. Harper et al. (2007) studied the effects of social comparisons (i.e., displaying how community members can compare to others in the system, e.g., in terms of performance, participation and interaction). Janzik and Herstatt (2008) proposed a set of incentives to motivate community members (using peer recognition, status, reputation, and identification). Current alternatives to encourage participation involve the use of gamification (Zichermann and Cunningham 2011); i.e., game mechanics into the design of non-game contexts to increase users’ self-contributions (Hamari et al. 2014; Herzig et al. 2012; Kapp 2012). These works provide an overview on how participation can be modeled and structurally considered as a key component in the basic definition of the community.

Preece and Shneiderman (2009) followed user life cycles through their evolution in a community and listed strategies for motivating their participation according to their evolving role within the group. Gutierrez et al. (2011) proposed a framework for motivating user participation based on intrinsic motivation, which included several strategies such as displaying rankings, proposing challenges, and displaying feedback.

Several authors (Butler 2001; Kim 2000; Raban et al. 2010) claim that communities have to achieve a certain critical mass, i.e., a minimum number of users in order to sustain activity and information exchanges within the group. Dabbish et al. (2012) studied the effects of turnover in online communities, i.e., the dynamics of user entrance and exiting in a particular group. In online and partially virtual communities, both participation and member commitment tend to increase when there is a noticeable turnover. This is translated into group members understanding the community activity as a dynamic entity, whereas turnover affects information exchange and content generation within the group. In some communities, it is more important for its sustainability to achieve a critical mass of contributions rather than a critical mass of users.

This know-how about participation and promotion helps managers to choose candidate strategies for intervene the community when needed, but they do not ensure the result or the way in which the community is going to react to this intervention. Therefore, managers should also monitor the community structure and evolution to see if the intervention impacted the community in the expected way. Therefore, providing monitoring and diagnosing capabilities to community managers is mandatory for the supporting platforms.

While the design of online community supporting systems has been widely explored in the literature, little attention has been devoted to understand the underlying participation mechanisms and design concerns aimed to support PVCs. For instance, Schummer and Lukosch (2007) define a pattern language for computer-mediated interaction that can be used to design several aspects of community support, such as user identification, contacts, and mechanisms for reciprocity and rewards among community members. However, PVCs are particular socio-technical constructs, in which it is needed to explicitly support the physical interactions that are required by their members (Gutierrez et al. 2014).

In the present article, we aim to extend the current research literature in online communities by presenting the findings of a case study that explores the implications of quality and quantity of contributions in participation strategies. PVCs are a particular case of online communities, where the main concerns of physical (i.e., as how they are understood in psychology and sociology) and virtual communities (i.e., as how they are understood in human-computer interaction and computer-mediated communication) need to be considered.

3 Methodology

We conducted a case study aiming to understand how student participation impact the interaction patterns in a partially virtual course community. The following subsections detail the study settings and a description of the collected data.

3.1 Settings

The study scenarios involved three communities of third-year undergraduate students enrolled in an information technology course offered by the Information Systems Department at the Business School of the University of Chile. The three groups were independent among them and each intervention lasted 15 weeks (one regular course semester). Students knew each other beforehand in their particular community. In each scenario we put in service a dedicated online discussion board, which runs on the learning platform that students regularly use to support their activities.

Scenario 0 served as control group, in which we did not neither motivate nor control participation. Scenarios 1 and 2 served as experimental groups, in which the dependent variable is the strategy used to motivate participation in each case. All of the students participating in the study were affected to one and only one scenario. The three scenarios are characterized as follows:

  • Scenario 0. This community was composed of 41 students, where 20 of them were men and 21 were women.

  • Scenario 1. This community was composed of 30 students, where 16 of them were men and 14 were women.

  • Scenario 2. This community was composed of 42 students, where 19 of them were men and 23 were women.

The instructional team was the same for the three sections of this course (i.e., for the three communities). Before the intervention, the course instructors agreed the instructional approach to be used, and also the number and type of assignments to be given to students, making thus comparable the study scenarios.

The discussion board supporting each course community allows students to publish new topics, reply to others’ contributions, and receive notifications concerning user availability and recent activity. In order to ensure the usage of the software platform, students of each scenario were asked to publish at least one contribution every 2 weeks, related to the topics covered in the lecture sessions. These contributions were based on recent news found in diverse media and had to include a personal opinion elaborated by the student prior to publication. Once this contribution is made available in the software platform, other students had the chance to rate the article (according to their own perception on quality and pertinence) and comment on it. It is important to note that ratings could only be made after a student commented on the contribution in order to address typical free riding situations. Furthermore, in order to limit the influence of student personality issues (e.g., peer pressure, shyness and peer identification) on the interaction network structure and user activity, participation was remained anonymous as opposed to traditional activity patterns in online discussion boards. Student participation and interaction in the online discussion boards was monitored by a teaching assistant, but only in a role of observer.

The design of the discussion board in this case study encourages participation through extrinsic mechanisms. The embedded motivation strategies are grounded in the research conducted by Cheng and Vassileva (2006), which have shown to be effective for designing sustainable online communities in educational scenarios. Some of the used design mechanisms include: explicit labels and participation indicators for displaying the level activity of different members; call-to-action triggers due to group dissimilarity and peer ratings; the establishments of participation goals; and gamification-based incentives (Zichermann and Cunningham 2011), such as ranking stars.

The discussion board used different algorithms to calculate user participation in scenarios 1 and 2. In the former, participation was calculated based on the number of students’ contributions, and in the latter, such a metric was focused on the quality of contributions. An indicator showing this metric was visible in the user interface of the tool, and it was also used to rank the students according to their participation (Fig. 1). Scenario 0 served as baseline, where we did not include a participation metric nor a visible indicator to students about their current activity.

Fig. 1
figure 1

User interface of the online discussion board

Full size image

The user interface is divided into two modules: (1) a main page where users can read the different contributions published in the site, and (2) a detailed view of one of these contributions. The first module displays a list of the 10 most recent contributions, a tag cloud and a panel of links pointing to other articles classified by categories and relevant tags (Fig. 1). This element, alongside with the search bar, helps users identify and find relevant documents, facilitating thus the interaction between the author and the reader. Users can access to the detailed view of any contribution by either clicking on its title, content, or dedicated icon at the bottom of the box. Other articles can be found by navigating through different pages at the bottom of the site.

The detailed view of each contribution displays the complete text (citing the source from where it was taken), the personal opinion of the author regarding the content of the article, and a list of reactions made to the contribution by other students (Fig. 2). Once a student publishes a comment linked to a particular contribution, the system proposes a rating system for grading the perceived quality of the article on a scale of one to seven stars. We chose to set this metaphor, since students are graded in a similar way in their regular courses at the University. Ratings and voting options were available and visible on the user interface, only for students participating in scenarios 1 and 2.

Fig. 2
figure 2

Detailed view of the user interface for comments

Full size image

During the 15 weeks in which this tool was used, three milestones were established for gathering traffic data from each site. The milestones were roughly placed every 5 weeks in order to make results comparable not only between communities, but also to analyze the evolution of the interaction patterns over time.

In each milestone, we identified: (1) the number of published contributions of each student in the discussion board; (2) the number of comments to other articles made by each student; and (3) the number of comments received by the other students.

With these values, in the case of scenarios 1 and 2, we computed a participation metric for each student, according to the strategy used in each situation. For these scenarios, we also computed the perceived quality of each contribution by the other students in the community.

The user interface for students belonging to scenarios 1 and 2 showed their participation value every time they logged in. This value and the corresponding label that situated each student within the community could be checked on demand. We classified students in three categories: “high participation” (top 20 % of the whole group), “low participation” (bottom 20 % of the group), and “medium participation”. The maximum and minimum values to set up the three categories were calculated in real-time. At each milestone, we published the participation values for all students and we reinitialized the counters for all metrics. The detailed results of the participation metrics obtained in these two cases, at each milestone, are presented and discussed in (Gutierrez et al. 2013).

The obtained results help explain the influence of participation strategies in user activity under the presented intervention context (i.e., small undergraduate course communities, interacting mainly through an online discussion board). In particular, we acknowledge that user behavior depends on several other variables, whose study goes far beyond the scope of this article. Therefore, we report the obtained results in scenarios 1 and 2 with regard to the activity registered in scenarios 0, which serves as baseline.

3.2 Measuring user participation

Communities in scenarios 1 and 2 were intervened to use a particular participation strategy to motivate and rate student contributions. As a design decision, students could not directly reply to one of their own contributions. This was intended for avoiding self-referencing when computing the participation metric for each student, and for encouraging interaction with other course members.

We computed a participation function with the values gathered in each milestone, considering the number of published articles (A), perceived quality of the contributions (Q), number of published comments to other students (PC), and number of received comments from other students (RC).

In scenario 1, we highlighted the quantity of contributions rather than the quality of them. With this strategy, our aim was for students to increase the number of contributions in time. Considering these four metrics, we computed the value of participation (P) for the first community as follows:

$$ \mathrm{P}=\mathrm{A}+\mathrm{P}\mathrm{C} $$
(1)

In this case, the participation value is a function of the number of published articles and the published comments to other students. We purposely did not consider in Eq. 1 the value of received comments and the perceived quality of the contributions by other students.

On the other hand, scenario 2 also included quality as a dimension of how participation is measured. With this strategy, we also aimed to increase the number of contributions, but also to improve the perceived quality of them by the other community members. In this case, we computed the value of participation (P) as follows:

$$ \mathrm{P}=\mathrm{A}\ast \mathrm{Q}/2+\mathrm{R}\mathrm{C} $$
(2)

In this case, the participation value stresses the quality of the contributions, since those that are perceived as more “useful” or “pertinent” by other members, will weigh more in the participation value of a student. In Eq. 2 we have also considered the number of received comments instead of the published ones. This was done for two reasons: (1) students will tend to comment on those articles that they find interesting or useful, therefore they might be of better quality; and (2) when a student posts a comment on the contribution of another student, s/he helps increase the other’s participation value instead of his/her own.

Following the intervention conditions detailed in the previous section, participation in scenario 0 was not manipulated using any external mechanism.

3.3 Relevant network metrics to analyze

We modeled the interaction network as a weighted directed graph where the nodes are the students, and the edges between nodes represent the number and direction of comments that one student published to another. Figure 3 shows an example of the network representation. Alice, Bob and Charlie are students in the course and published at least one article each. Alice posted three comments to Bob, Bob commented four articles written by Charlie, but Charlie only returned one comment to Bob. The arrow size indicates the cardinality of the relationship in that direction.

Fig. 3
figure 3

Example of an interaction network

Full size image

In order to understand the interaction network, we analyzed these interactions considering the following metrics:

  • Indegree: This metric represents the number of edges that arrive to a given node. It can be understood as the number of students who write to a particular node.

  • Weighted indegree: This indicates the number of edges that arrive to a given node, weighted by the number of comments. This metric can be understood as the number of comments that a given student receives.

  • Outdegree: This metric shows the number of edges that emerge from a given node. It can be understood as the number of students that a particular node is writing to.

  • Weighted outdegree: This represents the number of edges that emerge from a given node, weighted by the number of comments. It represents the number of comments that a student posts in the community.

  • Average path length: This metric corresponds to the average number of hops along the shortest paths for all possible pair of nodes in the interaction network. It represents the average number of students that are linked within a chain of contributions.

  • Diameter: This metric indicates the greatest distance between any pair of nodes in the interaction network. It corresponds to the maximum value for the path length; i.e., it represents the largest chain of contributions among students.

  • Density: This metric indicates the number of connected edges with regard to the total number of edges. In other words, it represents how many authors have interacted with each other, out of the total number of links possible.

  • Modularity: This is a factor between −0.5 and 1.0 that reflects the division of the network into groups within which the network connections are dense, but between which they are sparser. If this value is positive, the number of edges within groups exceeds the number expected on the basis of chance. When this value approaches 1, it means the strength of division of a graph structure is high (e.g., clear and distinct groups within the community).

  • Number of nodes: This metric indicates the number of people participating in the community.

  • Number of edges: This indicates the number of links between nodes considering both, the edges arriving to and emerging from the nodes.

Finally, we analyze the different triads that coexist within the network in the form of 3-node motifs. There are 13 different isomorphic 3-node motifs, and they are presented in Fig. 4. It is worth pointing out that among these motifs, seven of them are complete (or partially complete), since they tend to form cliques (i.e., a subset of three nodes in a graph, such that every two nodes in the subset are connected by an edge). On the other hand, six of the motifs are partially incomplete, since two nodes remain unconnected given that there is no interaction between them.

Fig. 4
figure 4

Isomorphic 3-node motifs

Full size image

4 Analysis of results

This section reports the main results obtained after analyzing participation metrics and the interaction network structure of the three scenarios used in the study. For analyzing and visualizing the interaction networks, we used the software Gephi v.0.8.

We first analyze the main participation metrics defined in section 3, and then we show the main metrics of the graph structure. Later, we identify and quantify the different 3-node motifs that compose the structure of each network. Finally, we comparatively discuss the results obtained in each scenario to better understand how the different participation strategies affect the several interaction networks, and hence, the interaction patterns among students in their respective communities.

4.1 Participation metrics

Table 1 presents the relative number of articles and comments published in each scenario, with regard to the total number of students in each case.

Table 1 Participation metrics
Full size table

We note that user participation in the baseline scenario is smaller than participation in scenarios 1 and 2. This is quite probably due to the introduction of an explicit strategy for triggering user activity and interaction among users in the intervened scenarios. Furthermore, the average number of published articles in scenario 2 was lower than in scenario 1, which is also a result that we can expect, since creating good quality contributions requires more effort than just doing a contribution.

Finally, the average rate of comments per article is also greater in scenario 2 than in scenario 1. This can be explained because we induced a quality factor in the participation strategy, and this could have triggered more interest to generate better and more appealing contributions. Nevertheless, in order to properly conclude this fact, we need to carry on further research regarding this situation.

4.2 Analysis of the network structure

Even if we continuously monitored student interaction independently in each community throughout the observation period (i.e., 15 weeks – a regular semester course), we analyzed their network structure in three milestones placed every 5 weeks. The reason behind this decision was to dispose of enough data to compare between two observations for tracking the network structure evolution through time. However, while we found differences regarding the volume of participation in terms of number and quality of contributions, we did not detect statistically significant differences in the network structure within each community (Gutierrez et al. 2013). Table 2 presents the interaction network metrics for each study scenario, taken once the activity in each scenario formally ceased (i.e., at the end of the semester).

Table 2 Network metrics
Full size table

Aiming to identify statistically significant differences in the values measured for characterizing the network structure, we conducted pairwise Wilcoxon tests. Given that the measured data do not follow a normal distribution (formally verified conducting Kolmogorov-Smirnov tests of data against the normal distribution), the assumptions for applying parametric tests do not hold.

In scenarios 1 and 2 the number of edges, density, average degree (i.e., the mean number of students that are connected through comments) and average weighted degree (i.e., the mean number of published comments in the platform) are significantly greater (p < 0.001) than in the baseline scenario. This can be explained due to the existence of a particular participation strategy motivating the interaction among users in the community (i.e., an extrinsic reward (Tedjamulia et al. 2005)). In order to be sure about this assumption, we verified the impact that the gender has in the participation rate. We found no statistically significant differences between the number of contributions of male and female students; therefore, such an effect can only be explained by the existence of an extrinsic rewards in scenarios 1 and 2.

Moreover, the average weighted degree in scenario 1 is significantly greater than in scenario 2 (p < 0.05). This can be understood as a consequence of a “snowball effect”, since the goal of the participation strategy in scenario 1 was merely to increase the number of contributions in the community. Therefore, the perceived measure of success is linked to the number of contributions published by the students. In addition, posting a comment requires less effort than selecting and publishing a new article. This can be a plausible explanation for this particular difference.

Although there are noticeable differences in the values of the activity metrics in the studied scenarios, these differences are not statistically significant. In the case of the baseline scenario, the average path length is even larger than in scenarios 1 and 2, meaning that the interaction among users forms longer chains, and therefore the community is not as tightly linked as in the other ones.

Finally, the value for modularity in scenario 1, being smaller than the baseline and scenarios 2, it can be understood as a tendency to not forming independent groups within the community. In other words, this means that the community in scenario 1 is more tightly connected than in the other scenarios. This seems to occur since in the formed scenario the participants can obtain rewards with low effort.

In order to have a closer look at what happened in these scenarios, Fig. 5 depicts a Fruchterman-Reingold visualization for each interaction network. The size of nodes represents the number of published contributions, and such size is comparable only among nodes of the same networks. The colors of the nodes represent the different modularity classes (i.e., eventual subgroups that can be identified within the community), and the thickness of the edges represents the number of comments that are posted in that particular sense.

Fig. 5
figure 5

Interaction network: (a) Scenario 0, (b) Scenario 1, (c) Scenario 2

Full size image

In the interaction network of scenario 2 (Fig. 5c) we can identify a clear subgroup in the community (i.e., black nodes). This subgroup consists of 12 nodes (25 % of the network), and it has a noticeable central leader who acts as a hub linking this subgroup with the rest of the community. Regarding the distribution of posts per user (i.e., the size of nodes), we identify that in this scenario they seem to be comparable to those to the baseline scenario. In other words, this means that the participation strategy, while it motivated to publish more articles in scenario 2, it did not have a major impact on producing more active members than the rest of the community.

However, in the case of scenario 1 (Fig. 5c), we remark that most nodes have an interesting participation level and there are few lurkers (i.e., nodes that just browse and consume the information of the community (Tedjamulia et al. 2005)). In particular, even if in the baseline scenario there is a quite low number of edges (i.e., interaction between community members), we note that there is clearly a node acting as a hub; i.e., his/her participation is rather low, but gets involved in receiving and posting comments to others’ contributions.

Moreover, this scenario does not display a clear leader within the community, but rather a set of central nodes that gather the attention and drive the interaction of the other students. The distribution of published contributions per user is also uneven, meaning that some community members became regular contributors and engaged into producing new content in the group, while others only contributed with the expected levels agreed at the beginning of the intervention. In this case, the interaction network structure is rather closed and it does not display clear subgroups.

However, structurally it is possible to acknowledge the existence of similar patterns in the intervened scenarios. In fact, there is a small number of big central nodes (i.e., those who post the highest number of contributions to the community), that tend to drive the interaction with the other nodes in the network. These nodes are usually known as veterans (Tedjamulia et al. 2005), and the community sustainability typically depends on them.

There are also sets of nodes that do not appear to interact with the others, represented in the borders of the graph. These nodes show similar interaction patterns to lurkers in a community, i.e., participants that do not strongly participate within the community, but appear to act like passive entities consuming information and resources from the social structure, rather than contributing. The number of these node types seems to be larger than in scenarios 1 and 2, probably to the lack of a proper reward that motivates the participation of these community members.

At this point, it turns out relevant to analyze if it is worth considering a community structure that is composed of two or more independent subgroups, or if it is better to have a closed and tightly connected group. Both situations have their own pros and cons, and the answer to this dilemma is rather unclear, since it depends on the context where the community is going to be established. In this case, since we are supporting a small partially virtual course community, we would like to benefit from having discussions in small groups. However, to some extent we do have to maintain the size of these subgroups, avoiding that they become independent and generate traffic that will be eventually irrelevant to the rest of the community.

Therefore, having a visualization that displays the dynamics of group generation over time would give signs of how the community is evolving, and also if it turns out to be necessary to put some control mechanisms in order to prevent the community break into independent subgroups. In other terms, this kind of analysis can be used for designing monitoring strategies that allow determining, in real-time, the dynamics of a community alongside its life cycle. In relation to this proposition, one way to affect the interaction patterns in the community would be to influence central nodes in the network (e.g., those that gather interest from other members and generate relevant and important traffic). In terms of affecting the participation strategy, this would be related to motivating the activity, aiming to integrate the different groups that appear to be in different modularity classes.

4.3 Identifying and quantifying motifs

Figure 6 shows a histogram representing the different isomorphic 3-node motifs in a directed network (as shown in Fig. 4). By identifying the different 3-node motifs we can structurally analyze, more in-depth, the network representing the community. Unfortunately, given that the interaction in the baseline scenario is not high enough to even notice the existence of triadic relationships, we note that most interactions are dyadic and asymmetric; i.e., they are only established between two nodes in one direction. This helps us to realize the type of interaction that we can get among community members if we do not motivate the participation.

Fig. 6
figure 6

Three-node motifs found in each interaction network

Full size image

Unlike the baseline scenario, in scenario 1 (i.e., where participation was stressed in terms of quantity), the interaction patterns tend to have closed groups. Thus, the possibility of completing 3-node motifs is high. In this latter scenario, the motifs 12 and 13 count for about 50 % of the total, and they are almost-fully connected (motif 12) and fully connected (motif 13) 3-cliques. Therefore, this is an alternative way to conclude that the community was tightly closed. This also indicates that strategies rewarding low effort contributions (like in scenario 1) can be used to motivate the participants and get a more cohesive community.

Regarding scenario 2, motifs 3, 7 and 8 count for about 50 % of the total, and they all correspond to disconnected 3-motifs. Indeed, when looking at the distribution of motifs between both scenarios, it is possible to note that complete triads are predominant in scenario 1 when compared to scenario 2, while unconnected triads appear in scenario 2 in a larger number than in scenario 1. This fact indicates that in scenario 2, the community is partially connected (as opposed to what happened in scenario 1), and community managers could take some actions into identifying why this is happening. Eventually, they can try to integrate the community, if desired.

In summary, by analyzing the histogram of 3-node motifs, community managers can get an overview of how connected is the community, and how the different interaction patterns are distributed in the whole group. In other words, this technique can be used as an alternative tool for monitoring the evolution of the community alongside its life cycle.

5 Implications for design

Next we present a list of considerations for software designers when they have to define the social services to be embedded in applications that support the activity of a community.

5.1 Providing participation rewards

Most communities, particularly those that are part of formal organizations, need to promote participation of community members through extrinsic rewards (Tedjamulia et al. 2005). Recent literature has shown good results with the use of game mechanics to promote self-promotion and encourage new contributions, such as leaderboards, trophies and stars (Zichermann and Cunningham 2011), particularly in learning (Kapp 2012) and organizational scenarios (Herzig et al. 2012). This happens because in these communities there is usually no intrinsic motivation to participate, or such motivation is only temporal; therefore, these communities quickly tend to diminish due to the lack of activity. As we have seen in scenarios 1 and 2, an extrinsic motivation strategy can considerably increase participation, as compared to scenarios where it is not motivated (like in scenario 0). Indeed, the baseline scenario helps us understand the participation that we can observe in a community when the interaction among members is not promoted in any way. In a previous study, the authors identified that the extrinsic motivation in these scenarios also helped to keep a permanent interaction among community members (Gutierrez et al. 2012a).

5.2 Reducing participation effort

Participation is also affected by the effort required to accomplish it. This becomes evident if we analyze the average weighted degree in scenarios 1 and 2 (Table 2). As expected, if rewards keep being stable, the number of interactions decreases when increases the effort required to contribute to the community (Cheng and Vassileva 2006). Software designers must keep this issue in mind when modeling the participation supporting services for a community. Gamification mechanisms can also be used to help reduce the participation effort.

The extrinsic mechanism for promoting participation can also be combined in order to get a more important commitment of community members. For instance, an indicator (i.e., an awareness component, as those shown in Figs. 1 and 2) based on a low effort contribution strategy, can be used to get people involved and active in the community (like in scenario 1), while a quality based contribution strategy can motivate and recognize to members that produce valuable content for the community (like in scenario 2). Thus, participants will become more active and committed if they have a good score in one of these two aspects. As shown by other studies, a high participation in a community tends to encourage by itself the participation of the members (Dabbish et al. 2012). Multiple motivation strategies also contribute to increase the social representativeness of the community members.

5.3 Supporting social representativeness

It is important that all social groups forming the community are represented in the supporting system, and that users feel that they belong to such community (Lampe et al. 2010). If such a requirement is addressed, the inclusion of social services in information systems will have a chance to improve the members’ experience and performance. Otherwise, people will continue working as members of a physical community, and the social services will probably be more an obstacle than a contribution for the community. Figure 5 shows samples of the subgroups that can be present in a community, even if it is small.

Concerning our study results, in all scenarios we have identified lurkers; i.e., people that feel no represented by the community or motivated by the rewarding mechanisms used to promote participation. However, in scenarios 1 and 2 (but mainly in scenario 1) the interaction graph shows a more involved and cohesive community, with almost no lurkers. Furthermore, in scenario 0 (baseline), it is possible to identify a very active node in terms of interaction with other community members, but with a quite low number of new contributions. This situation exemplifies those community members who are eager to engage in interaction with other users and help sustain the community (Farzan et al. 2011; Dabbish et al. 2012), but who fail to achieve it due to the group not reaching a minimal threshold of contributions (i.e., critical mass).

This probably means that representativeness tends to increase if there is an extrinsic motivation to participate, and such participation involves low effort. Contrarily, if the effort to participate increases, the interaction graph tends to show sub-groups (as in Figs. 5 and 6), that jeopardize the representativeness of the members in a community. Anyway, these situations are identifiable by the community managers, who can design intervention strategies to try reunifying the community.

5.4 Providing monitoring services

Communities are dynamic entities that need to be monitored in order to understand their current situation and evolution. Services that show the structure of the interaction graph help community managers to make on-time diagnosis and also corrective actions in case of need. For instance, several community risks (e.g., due to the dependency of veterans) and existence of sub-communities can be identified in an early stage using these services, as shown in Fig. 5. Moreover, services providing temporal information of member participation are required to identify periods of inactivity that could increase the vulnerability of the community.

In the presented study, managers did not intervene the community to improve participation, although they were able to do it. The goal of this monitoring process was to establish a diagnosis of the community based on its structure, which was accomplished according to the managers’ opinion. The interaction graphs shown in Fig. 5 and the histogram depicted in Fig. 6 can be obtained in real-time, and they provide a visual representation that describes the community current status. Based on it, the manager can perform a diagnosis and determine strategies to intervene the community when needed. Therefore, counting on services for monitoring the community status and evolution tends to be mandatory for community managers. Otherwise, the probability to make late or wrong interventions grows considerably.

5.5 Enhancing community governance

In PVCs that group people belonging to the same organization (e.g., employees or students) or people with professional ties (e.g., customers and service providers), community members usually do not have an intrinsic motivation to participate, since their participation corresponds to a formal link among members instead of a personal decision. The lack of intrinsic motivation makes that counting on community managers turns to be mandatory to keep the community on the move (Gutierrez et al. 2014). Therefore, the social interaction system supporting the community activity must consider the services required by its managers. Moreover, the community member capabilities (or their roles) cannot be assigned according to the place that they have in the organizational structure.

As we can see in Fig. 5a, b and c, these communities usually have people that are recognized by other community members as veterans or leaders, and that are highly valuable for the community. Their presence and participation influence the sustainability of the community; therefore, any newcomer must have to change his/her behavior to become veteran based on their contributions to the community (Tedjamulia et al. 2005).

The interaction mechanisms supported by the software platform must allow community members to contribute and self-regulate such contributions. Thus, regular participants can help managers govern the community (Shea and Bidjerano 2010).

5.6 Enabling participation awareness

Provided that people participation is the cornerstone of a community (Leimeister et al. 2006; Preece 2001), member contributions must be visible for the rest of the community. Usually these contributions are represented through awareness mechanisms, as those implemented by the discussion board supporting scenarios 1 and 2. These mechanisms can be passive (i.e., the user goes to the information) or active (i.e., the information goes to the user). Typically the use of active awareness mechanisms triggers more interactions among community members, and make them aware of community current activities. However, the number and way to deliver these notifications must be designed carefully to avoid disrupt and overwhelm the participants (Mark et al. 2008; Toninelli and Khushraj 2008).

Figure 1 shows the participation level of the user, and Fig. 2 shows the rating of a contribution of a participant. These awareness components represent a clear concept and they must be designed to promote the participation of community members in terms of both, quantity and quality of contributions (Cheng and Vassileva 2005), as shown in scenarios 1 and 2. These components should not only illustrate the performance of community members (Harper et al. 2007), but also be used to integrate people that in the border of the community; for instance, adding and publishing metrics that are relevant for them. It is clear that a person is going to participate in an online community only if s/he feels that belongs to such group (Lampe et al. 2010).

6 Conclusions and future work

This article presents a study of people participation in three partially virtual course communities, trying to understand the impact that different participation strategies have on the community activities and structure. The findings of this study inform to the design of new information systems that embeds social interaction services to support the activities among community members; particularly in the case of undergraduate course communities.

The study was performed on homogeneous communities of information technology students, and it involved a period of 15 weeks. The students participating in the baseline scenario were not externally motivated; therefore, their participation levels correspond to those generated as consequence of the intrinsic motivation of these community members. As expected, participation was low.

In scenario 1, community participation was motivated by stressing the value of the quantity of contributions, and then enhancing the quality of the contributions in scenario 2. The activity within these communities was comparatively analyzed in terms of: (1) participation metrics, (2) structural network metrics, and (3) 3-node motifs that reflect the interaction among members.

Even if we derived some relevant observations regarding how participation can be triggered in terms of quantity and quality, it is worth recognizing that neither of the motivation strategies used in scenarios 1 and 2 were perfect. In the case of scenario 1, the community tended to follow a snowball effect, where publishing new contributions and generating traffic became the center of the community, rather than the interaction itself. In the other case (scenario 2), the community tended to split into two subgroups that interacted independently. Considering these results, we can infer that the participation strategy clearly affected the community structure and the interaction patterns among its members.

By analyzing the different structural network metrics, and more precisely, by having a visualization that displays the dynamics of group generation over time, we can get a first idea on how the community evolves, and also if it is necessary to install some control mechanisms to prevent the community from breaking into independent subgroups. In other terms, this kind of analysis can be used for designing strategies to monitor (in real-time) the dynamics of a community alongside its life cycle. Another alternative for monitoring the community activity is analyzing the structure of its interaction graph, in terms of motifs. It reflects the inner interaction patterns within the community.

Partially virtual communities are a particular case of online communities, where the key differences are that the former require mechanisms to actively support face-to-face interaction, awareness and coordination mechanisms. The case study findings then allow raising several design issues, which should be considered when modeling platforms or services to support the activities of partially virtual and online communities. We also suggest some solutions to address these design issues, which are grounded in the literature of online communities, and are extended to the case of PVCs. Provided that everyday more and more the information systems embed social interaction services to improve the user experience and performance, designers of these systems could take advantage of this study results, not only when extending the existing solutions, but also when conceiving the new ones.

There are three major limitations in this study. First, we modeled the interaction network as a directed weighted graph. However, the presented motif classes are based on directed unweighted graphs. Therefore, in the case of detecting weighted edges that outnumber the frequency of a particular motif class, we would need to analyze further in detail the resulting patterns, and eventually decide if they need to be considered as independent objects. Thus, the global motif distribution of the network would be altered. Second, some of the presented observations might be due to the differences in the even homogeneous student communities. These limitations will be further analyzed and possibly mitigated in a second stage of this study. Finally, the experimental design followed a between-groups (independent) approach instead of within-subject (repeated-measures). The reason behind this decision is the difficulty of collecting meaningful data in the specific scenario covered in our study. Indeed, we were limited on an observation period of just 15 weeks, thus making it impossible to introduce two independent interventions to a same group of students. Therefore, even if we cannot completely generalize the study results due to the possible presence of environmental factors, we found that the findings are indeed backed by the literature, and they provide more evidence to support the design and sustainability of participation in partially virtual course communities.

As future work, we are interested in identifying the correct metrics to monitor the evolution of a community alongside its life cycle, and how to design visualizations aiming to help community managers understand the dynamics of an online community in real-time.