Hybrid Ranking Method for E-Learning Platform Selection: A Flexible Approach

Abstract

E-Learning platforms comparison helps users select the most suitable platform according to their individual pedagogical needs and objectives. However, from decision support perspective, selecting the optimal platform in terms of tools and services that meet user’s requirements still remain difficult to achieve. Thus, we investigate in this paper an e-Learning evaluation method based on a symbolic approach using preference operators and Qualitative Weight and Sum method (QWS) [1] providing a total order among the considered e-Learning systems. However, even the totality ensured by the preference operators we developed, they are still not sufficient from decision making perspective, since they can return a ranking in which several alternatives are all equally and indistinguishably (un)satisfactory. Therefore, we combine our symbolic approach with a flexible ranking method based on linguistic quantifiers and fuzzy quantified propositions along with two new parameters for quality assessment refinement, called least satisfactory proportion and greatest satisfactory proportion, denoted by lsp and gsp respectively, to be able to discriminate among alternatives evaluated as equal. The hybrid method obtained can significantly refine the ranking providing users with valuable information to help them make decisions.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction and Motivation

The past decade has seen many changes in educational and industrial training methods along with the increasing of the number of users having diverse needs and objectives, and the large adoption of communication and information technologies. Thus, a large number of free and commercial e-Learning systems have been developed in different areas such as education [2], language learning [3], business training [4, 5], medicine [6, 7] and public administrations [8], etc. which provide on/off-line and remote training making user training more flexible and easier. The multitude of e-learning platforms developed for a single domain (such as in language learning, for instance, we can distinguish tens of e-Learning applications and on-line platforms like babel, busuu, duolingo, ef, tell me more, Pimsleur, etc.) makes it difficult to objectively select the more suitable one according to one’s pedagogical needs and objectives.

The choice of a suitable system based on criteria describing the considered e-Learning platforms is of great help for users. Standards and norms such as SCORM¹, QTI², IMS³, etc. define e-Learning quality properties, such as adaptability, sustainability, interoperability and reusability. We refer the reader to [9] for an e-Learning platform evaluation based on the SCORM specification. In addition, many other evaluation approaches have been proposed such as [10], in which the framework considers two models: one addresses the different ways to produce learning processes in an e-Learning system, which has been reused in [11], and the other characterizes the different evaluation criteria of learning models as introduced in [12].

Qualitative methods have also been considered for e-Learning systems evaluation. The most commonly used approach is Qualitative Weight and Sum, denoted by QWS [1]. It relies on a list of weighted criteria [13, 14] for the evaluation of e-Learning systems. In practice, it is based on qualitative weight symbols expressing six levels of importance, namely: E for essential, $*$ for extremely valuable, $\#$ for very valuable, $+$ for valuable, | for marginally valuable, and 0 for not valuable. Hence, e-Learning system performance is measured by symbolic weights attached to some criteria [13], such that low-weighted criteria cannot overpower high-weighted ones. For instance, if a criterion weighted $\#$, the platform can only be judged $\#$ or lesser ($+$, | or 0) but not $*$ or higher. To obtain a global evaluation for a platform, QWS approach aggregates the symbols attached to criteria through a simple counting, which is finally used to rank the considered e-Learning systems. Because of the naive aggregation function used by the approach, the result may be counterintuitive and not clear to explain and justify. For example, let us suppose three e-Learning systems, denoted by $e_1$, $e_2$, $e_3$ respectively, for which the aggregation function delivers the results as summarized in Table 1. It is easy to conclude that $e_1$ is better than $e_2$, since $e_1$ is better than $e_2$ on symbol $*$ and both tied the score for the other symbols. But, it is not that easy to say whether $e_1$ is better than $e_3$ or not, because even though $e_1$ performs well on symbol $*$, $e_3$ is much more better than e1 on symbols $\#$ and $+$. Therefore, further analysis has to be conducted to conclude. As some e-Learning systems are not comparable, then the approach delivers a pre-order over the evaluated platforms.

Table 1. Example of e-Learning system aggregation results.

To deal with this issue, one can consider the Analytic Hierarchy Process (AHP) method [14]. AHP is used to deal with complex decision-making processes. It translates the symbols defined in QWS into values as detailed in Table 2, borrowed from [1]. Thus, AHP captures both subjective and objective values, checks their consistency and reduces bias decision making in evaluating e-Learning systems [15]. The criteria are gathered up by category and sub-category. Results of feature category or subcategory evaluation computed by the weighted calculation functions are percentages of the form of a real number as described in [14]. For example, let us say that the percentage returned for the feature Chat is 14%; then, according to Table 2, the judgment of this result is between “marginally valuable” and “valuable”, but which the user should consider. The percentages returned can be difficult to interpret for the comparison of e-Learning platforms when several attributes have to be dealt with.

Table 2. QWS symbols translated into AHP weights.

These above methods return results which can be less expressive and non-intuitive enough from a user standpoint for system quality assessment and ranking. Therefore, we investigate in this paper methods for e-Learning systems assessment and ranking combining QWS values, symbolic preference relations and comparison operators, which have been proved to be total orders allowing to rank a set of e-Learning platforms from the most to the least optimal, helping the end user to make a decision. Notwithstanding, the practical application of approach we proposed revealed a limitation from decision making standpoint, in the sense that a large set of alternatives can be evaluated as equally (un)satisfactory owing to its purely qualitative nature. A need for an hybrid approach mixing both qualitative and quantitative behaviors is then underlined and a flexible solution is also developed based on fuzzy set theory [16] as a first attempt to provide a possible feasible solution.

Indeed, fuzzy sets approach is considered here as a formal theoretical basis since fuzzy sets are simultaneously quantitative and qualitative, and provide a general model for user preference expression and interpretation based on fuzzy predicates such as high, expensive, young, etc. Particular kind of fuzzy conditions, called quantified propositions, involving linguistic quantifiers have also been defined. A quantified proposition is based on a linguistic quantifier, which expresses a compromise between the universal quantifier ($\forall$) and the existential quantifier ($\exists$). A linguistic quantifier refers to expressions such as most, almost all, around 4, etc. For instance, the sentence “find colleges that have most of their alumni are professionally successful” involves a quantified proposition defined by the linguistic quantifier most of, and expresses both quantity and quality in terms of what extent the expression is evaluated as true. The closer to 1 the evaluation is, the better. We consider the criteria describing e-Learning platforms as a fuzzy set of important criteria (from the user perspective) then we compute for each platform the truth value of the following fuzzy quantified proposition: “almost_all the criteria are important”. This truth value expresses a quantitative evaluation of the alternatives in the sense that it corresponds to the extend platform’s criteria are evaluated as important by the user. After that, we define two new parameters to assess the quality of the subsets of criteria satisfying the truth value of the quantified proposition. They are denoted lsp and gsp and stand for the least satisfactory proportion and the greatest satisfactory proportion respectively. These parameters are used to break ties among equally evaluated alternatives (based on the truth value of the considered fuzzy quantified proposition) from qualitative point of view. The hybrid approach obtained shows a significant refinement of the final ranking which can greatly help users make decision for e-Learning platform selection.

It is note worthy that this paper is an extended version of our previous research paper [17].

The remainder of the paper is structured as follows. Section 2 details our symbolic-based approach for e-Learning systems evaluation. Section 3 presents an illustrative example of our approach to evaluate and to rank a set of open-source e-Learning systems. In Sect. 4, a reminder of linguistic quantifiers and the interpretation of fuzzy quantified propositions for preference expression and modeling are introduced. Section 5 details the application of the fuzzy quantification based approach to our application context (ranking e-Learning platforms), while introducing the new parameters for ranking refinement. Finally, Sect. 6 concludes the paper and introduces some future work.

2 Hybrid E-Learning System Evaluation Approach

In this section, we detail our approach for e-Learning platform evaluation and ranking relying on symbols borrowed from QWS method and qualitative preference relation and comparison operators. In Subsect. 2.1, we introduce our evaluation approach and in Subsect. 2.2, we show the use of our approach for e-learning platform ranking.

2.1 Symbolic Approach for E-Learning Platforms Evaluation

We define the evaluation symbols as follows.

Definition 1

(Evaluation Symbols). The evaluation symbols as defined in QWS approach are: E = essential, * = extremely valuable, # = very valuable, + = valuable, | = marginally valuable and 0 = not valuable. We denote by $\mathcal {S} = \{E, *, \#, +, |, 0 \}$ an ordered set of evaluation symbols.

We define a preference relations more preferred than or equal to, denoted by $\succeq$, and less preferred than or equal to, denoted by $\preceq$, over the evaluation symbol set $\mathcal {S}$ as follows.

Definition 2

(Preference Relations $\succeq$ and $\preceq$ ). Let $\mathcal {S} = \{E, *, \#, +, |, 0 \}$ be an ordered set of evaluation symbols such that: $pos_\mathcal {S}(E)=1$, $pos_\mathcal {S}(*)=2$, $pos_\mathcal {S}(\#)=3$, $pos_\mathcal {S}(+)=4$, $pos_\mathcal {S}(|)=5$, $pos_\mathcal {S}(0)=6$, where $pos_\mathcal {S}(\centerdot )$ stands for the position of symbol $\centerdot$ in set $\mathcal {S}$.

We define the preference relation more preferred than or equal to, denoted by $\succeq$, over $\mathcal {S}$ as:

$$\begin{aligned} \forall (a, b) \in \mathcal {S}^2:a \succeq b \text { iff } pos_\mathcal {S}(a)\le pos_\mathcal {S}(b) \end{aligned}$$

(1)

The preference relation less preferred than or equal to, denoted $\preceq$, is defined as:

$$\begin{aligned} \forall (a, b) \in \mathcal {S}^2:a \preceq b \text { iff } pos_\mathcal {S}(a)\ge pos_\mathcal {S}(b) \end{aligned}$$

(2)

We can easily prove both preference relations $\succeq$ and $\preceq$ are total orders.

Property 1

(Total Order Properties). The preference relations $\succeq$ and $\preceq$ are total orders.

Proof

The proof of property 1 is detailed in Appendix A.

Based on the above defined preference relations, we define two comparison operators named prefMin and prefMax, so that it will be possible to compare systems on each criterion describing them. These operators will serve as means to aggregate the evaluations obtained for system criteria.

Definition 3

(prefMax and prefMin Comparison Operators). prefMax and prefMin operators are defined by formulas (3) and (4) respectively.

The function prefMax is defined by the following formula (3).

$$\begin{aligned} \mathcal {S}\times \mathcal {S}\rightarrow & {} \mathcal {S}\nonumber \\ (a,b)\mapsto & {} max(a,b)= {\left\{ \begin{array}{ll} a &{} if\,\,(a\succeq b)\\ b &{} otherwise. \end{array}\right. } \end{aligned}$$

(3)

The function prefMin is defined by the following formula (4).

$$\begin{aligned} \mathcal {S}\times \mathcal {S}\rightarrow & {} \mathcal {S}\nonumber \\ (a,b)\mapsto & {} min(a,b)= {\left\{ \begin{array}{ll} a &{} if\,\,(a\preceq b)\\ b &{} otherwise. \end{array}\right. } \end{aligned}$$

(4)

When we apply the comparison operators prefMax and prefMin over our symbolic set $\mathcal {S}$, we obtain Table 3.

Property 2

(prefMax properties). prefMax is associative, commutative, idempotent, it has E as absorbent element and 0 as neutral element.

Proof

Proofs of prefMax properties are detailed in Appendix A.

Property 3

(prefMin properties). prefMin is associative, commutative, idempotent, it has 0 as absorbent element and E as neutral element.

Proof

Proofs of prefMin properties are detailed in Appendix A.

Table 3. The operators prefMax and prefMin table.

2.2 Using Our Comparison Operators to Rank E-Learning Systems

The evaluation of e-Learning platforms is based on categories, each of which defines some criteria as defined in [18], for example the category Communication tools, and their criterion such as Chat. Categories and their criteria are summarized in Table 4. The five categories considered in platform evaluation are:

• Communication tools

• Software and installation

• Administrative tools and security

• Hardware presentation tools

• Management features

To evaluate each category, we use the comparison operators prefMax and prefMin. But to evaluate a considered e-Learning system, we need the evaluation of the five categories. For that purpose, we define two aggregation operators, called prefMinMax and prefMaxMin, which are based on our comparison operators.

Table 4. Overview of the hierarchy categories and their criteria.

Definition 4

(prefMinMax). Let $\mathcal {A}$ be a matrix of n lines and m columns of evaluation symbols of $\mathcal {S}$. We define the minimum guaranteed satisfaction value as follows.

We denote a matrix from $\mathcal {A}$ as:

$$\begin{aligned} A = (a_{ij})_{1 \le i \le m \atop 1 \le j \le n }\ and \ a_{ij}\in \mathcal {S} \end{aligned}$$

We define prefMinMax of A as:

(5)

Definition 5

(prefMaxMin). We define the maximum possible satisfaction value of $\mathcal {S}^{m\times n}$ as prefMaxMin:

(6)

The prefMinMax operator computes the least optimistic value amongst the criteria, whereas prefMaxMin operator computes the greatest pessimistic value amongst the criteria.

3 Illustrative Example

We apply our e-Learning systems evaluation approach to a set of nine open-source e-Learning platforms below, and which have been tested and compared [19,20,21,22].

• 1. Claroline: version 1.9.2, http://www.claroline.net

• 2. Dokeos: version 2.1.1, http://www.dokeos.com/fr

• 3. eFront: version 3.6.11, http://www.efrontlearning.net

• 4. ILIAS: version 4.1.3, http://www.ilias.de

• 5. Open ELMS: version 7, http://www.openelms.org

• 6. Ganesha: version 4.5, http://www.ganesha.fr

• 7. Olat: version 7.2.1, http://www.olat.org

• 8. AnaXagora: version 3.5, http://www.anaxagora.tudor.lu

• 9. Sakai: version 10.4, https://sakaiproject.org

Each criterion takes a symbolic value from the set $\mathcal {S}$ based on users opinions community. To obtain the evaluation of each criterion, we have carried out surveys in our university involving under-graduated students (small group of 10 students), who have tested each e-learning platform during a training session (2 h). We are aware that the process is subjective and a different panel of students or users can express different opinions about the e-Learning platforms. We recall that this data collection aims at illustrating the use of our approach. The values obtained for each criterion in its category are summarized in Table 7. The application of our approach on the set of considered systems is performed as follows.

Table 5. prefMin and prefMax results for Communication Tools category.

• 1. for each category in Table 4 we compute values of prefMin and prefMax for all functionalities based on Definition 3. In Table 5, we display the results obtained by applying our approach on the category “Communication Tools” for our considered set of e-learning platforms.

• 2. for each category in Table 7 we compute values of prefMaxMin and prefMinMax. Results of both operators are displayed in Table 6.

According to Table 6,

Table 6. Results of prefMaxMin and prefMinMax computation over the considered set of e-Learning platforms.

Table 7. E-Learning platform’s features symbolic evaluation.

• we obtain the following ranking if we consider prefMaxMin operator: (Claroline $\simeq$ Dokeos $\simeq$ eFront $\simeq$ Ganesha $\simeq$ Sakai) $\succ$ (Ilias $\simeq$ Open ELMS $\simeq$ Olat $\simeq$ AnaXagora) such that Claroline, Dokeos, eFront, Ganesha, Sakai equally ranked at the top and outperform Ilias, Open ELMS, Olat, AnaXagora systems, which are ranked at the second position.

• we obtain the following ranking if we consider prefMinMax operator: Sakai $\succ$ (Claroline $\simeq$ Dokeos $\simeq$ Olat) $\succ$ (eFront $\simeq$ ILIAS $\simeq$ Open ELMS $\simeq$ Ganesha $\simeq$ AnaXagora) such that Sakai is the best e-learning system and ranked at the top; then Claroline, Dokeos and Olat are equally ranked at the second position, and also better ranked that eFront, ILIAS, Open ELMS, Ganesha and AnaXagora systems, ranked at the third position.

Furthermore, users can make a choice based on either prefMaxMin or prefMinMax operators or can combine the results returned by both operators. For instance, in our illustrative example, Claroline, Dokeos, eFront, Ganesha and Sakai are equally optimal platforms according to prefMaxMin operator, whereas Sakai is the optimal one according to prefMinMax operator. If we consider both operators, we notice that Sakai performs better since it is ranked at the top according to both operators.

For decision making support point a view, it would be interesting to have the possibility to distinguish between the similar or equally ranked options (e-learning systems in our case). The symbolic preference relations used in our case divide a set of items into classes of equivalent items (having the same ranking position). But, what if we combine to the approach a flexible quantitative dimension which refines the ranking in each class of equivalence and permits distinguishing among systems belonging to the same class.

We explore in what follows the opportunity to make use of a fuzzy approach for e-Learning platform ranking. Fuzzy sets theory and fuzzy logics have been widely proved to be powerful theoretical basis to model flexibility and to compute user preferences [23].

4 Fuzzy Quantifier Based Ranking Method

In this section we briefly recall the basis concepts upon which our approach is built. Subsection 4.1 reminds the main definitions related to fuzzy sets. Subsection 4.2 recalls the definitions of linguistic quantifiers. Subsection 4.3 summarizes concepts about fuzzy quantified propositions. Finally, Subsect. 4.4 details the decomposition-based approach as our main tool for the evaluation of fuzzy quantified propositions of the form “Q X are A”.

4.1 Fuzzy Set Theory

Fuzzy set theory introduced by Zadeh [16] to express the gradual membership of an element to a set.

Formally, a fuzzy set F is defined on a referential U by a membership function $\mu _{F}:U\mapsto [0,1]$ such that $\mu _{F}(x)$ denotes the membership grade of x in F. In particular, $\mu _{F}(x)=1$ denotes the full membership of x in F, $\mu _{F}(x)=0$ expresses the absolute non-membership and when $0<\mu _{F}(x)<1$, it reflects a partial membership (the closer to 1 $\mu _{F}(x)$, the more x belongs to F).

The core of F is $Core(F)=\{x\in F:\mu _{F}(x)=1\}$ and its support is $Support(F)=\{x\in F:\mu _{F}(x)>0\}$. If a fuzzy set is a discrete set then it is denoted $F=\{(x_1,\mu _{F}(x_{1})),...,(x_n,\mu _{F}(x_{n}))\}$, otherwise, it is characterized by its membership function, in practice often a trapezoidal function.

The union $\cup$ and the intersection $\cap$ operators are defined by a couple of a t-norm and a t-conorm, such as (min, max). Let F, G be two fuzzy sets, $\mu _{F\cup G}(x)=max(\mu _{F}(x), \mu _{G}(x))$, $\mu _{F\cap G}(x)=min(\mu _{F}(x),\mu _{G}(x))$, and the complement of F, denoted $F^{c}$, is $\mu _{F^{c}}(x)=1-\mu _{F}(x)$.

The logical counterparts of $\cap$, $\cup$ and the complement are respectively $\wedge ,\vee$ and $\lnot$. Other operators have also been defined such as fuzzy implications [24].

4.2 Fuzzy Linguistic Quantifiers

Linguistic quantifiers express fuzzy quantities such as most, almost all, around 4, several, much, not many, few, etc. They express a flexible quantification between the existential quantifier ($\exists$) and the universal quantifier ($\forall$). A linguistic quantifier Q can be absolute or relative:

• Absolute quantifiers such as “at least 3”, “at most 5” and “around 4” express a number and their interpretation do not depend on the cardinality of the set X. Such quantifiers are defined as follows:

  $$\begin{aligned} \mu _{Q}:\mathbb {R}|\mathbb {N}\rightarrow & {} [0,1]\\ x\mapsto & {} \mu _{Q}(x) \end{aligned}$$

• Relative quantifiers such as “few”, “almost all” and “around half” express a proportion and their interpretation depend on the cardinality of the set X. Their generic definition is as follows:

  $$\begin{aligned} \mu _{Q}:[0,1]\rightarrow & {} [0,1]\\ x\mapsto & {} \mu _{Q}(x) \end{aligned}$$

  where $\mu _{Q}$ is the truth-value of the quantifier Q applied on x elements; x can refer to either a quantity or a proportion.

A linguistic quantifier can also be increasing or decreasing. Q is an increasing quantifier if the truth-value of the quantified proposition in which it is involved does not decrease if the satisfaction to fuzzy predicate A by elements of X increases. “at least 3”, “almost all” and “most of” are examples of increasing quantifiers. A quantifier Q is a decreasing quantifier if the truth-value of the quantified proposition in which it is involved does not increase if the satisfaction to fuzzy predicate A by elements of X increases. “at most 3”, “few” and “at most half” are examples of decreasing quantifiers. A quantifier is monotonic if it is either increasing or decreasing. Non-monotonic or unimodal quantifiers could also be pointed out. They refer to quantities such as “around 5”, “around a quarter”, etc.

The evaluation of a quantified statement amounts to determine its gradual truth-value (in the unit interval [0, 1]). Propositions of the form “Q B X are A” raise an interpretation issue when the fuzzy set B tends to the empty set. Therefore, we only detail hereinafter the evaluation of quantified propositions of the form “Q X are A”.

4.3 Fuzzy Quantified Propositions

Linguistic quantifiers allow the definition of fuzzy quantified propositions by combining fuzzy predicates and quantifications. Let Q, X, A and B be respectively a linguistic quantifier, a set of elements and two fuzzy predicates. A fuzzy quantified proposition can have one of following forms “Q X are A” or “Q B X are A”. The former means that among elements of X, there are Q elements that satisfy the fuzzy predicate A, as in the proposition “most of e-learning platforms are well-designed”, and the latter means that among elements of X that satisfying B, there are Q elements that satisfy A, as in the proposition “at least 5 of big e-learning platforms are expensive”.

Many approaches have been proposed to evaluate statements of type “Q X are A”, but any evaluation of fuzzy quantified propositions should verify some properties to express user preferences. The following four properties are the most important ones to be satisfied by any interpretation and evaluation approach (see [25] for more details about properties).

Property 4

(Existential Quantifier Definition). The interpretation of fuzzy quantified propositions should be able to derive the interpretation of the existential quantifier-based proposition: “$\exists X$ are A”, such that “$\exists$” is an absolute quantifier that is defined as follows:

$$\begin{aligned} \mu _{\exists }(x)={\left\{ \begin{array}{ll} 0 &{} if\,x=0\\ 1 &{} otherwise \end{array}\right. } \end{aligned}$$

(7)

The evaluation of a proposition of the form “$\exists X$ are A” is $sup_{x\in X}\mu _A(x)$.

Property 5

(Universal Quantifier Definition). The interpretation of fuzzy quantified propositions should be able to derive the interpretation of the universal quantifier-based proposition: “$\forall X$ are A”. “$\forall$” is a relative quantifier that is defined as follows:

$$\begin{aligned} \mu _{\forall }(x)={\left\{ \begin{array}{ll} 1 &{} if\,x=1\\ 0 &{} otherwise \end{array}\right. } \end{aligned}$$

(8)

The evaluation of a proposition of the form “$\forall X$ are A” is $inf_{x\in X}\mu _A(x)$.

The other properties are related to monotony. A quantified proposition evaluation should be monotone along with both linguistic quantifiers and fuzzy predicates involved.

The following interpretations of fuzzy quantified propositions are suitable for user preference expression and evaluation, since they verify the regular properties introduced in [25]:

• the decomposition based approach [26],

• OWA-based approach [27],

• fuzzy integral (Choquet/Sugeno) based approach [25]. It has been shown in [25] that the Choquet (resp. Sugeno) fuzzy integral-based approach is equivalent to the decomposition-based approach (resp. OWA-based approach).

In the following subsection, we introduce the decomposition-based approach because it is the basis of our flexible ranking methods. The interpretation based on OWA approach is left for future works.

4.4 The Decomposition-Based Approach

The decomposition-based approach has been proposed by Yager in [26] for the evaluation of fuzzy quantified propositions of the forms “Q X are A”, in which Q is an increasing quantifier. Let “P : Q X are A” be a fuzzy quantified proposition such that Q is an increasing linguistic quantifier. The decomposition-based approach computes its truth-value, denoted by $\theta _{P}$, based on the best regular (crisp) subset $E\subseteq X$ which contains Q elements that satisfy the condition A. Therefore, we obtain the following formula (1) if Q is an absolute quantifier and formula (2) if Q is a relative one:

$$\begin{aligned} \theta _{P}=max_{i\in \{1,...,n\}}min(\mu _{A}(x_{i}),\mu _{Q}(i)) \end{aligned}$$

(9)

$$\begin{aligned} \theta _{P}=max_{i\in \{1,...,n\}}min(\mu _{A}(x_{i}),\mu _{Q}(\frac{i}{n})) \end{aligned}$$

(10)

where $\mu _{A}(x_{1})\ge \mu _{A}(x_{2})\ge ...\ge \mu _{A}(x_{n})$.

Remark 1

The decomposition-based approach considers only increasing quantifiers. When Q is decreasing, the evaluation of a proposition of the form “Q X are A” could be deduced from the evaluation of the derived proposition “$Q'$ X are not A”, where $Q'$ is the antonym of Q; such as the proposition “At most 3 platforms are expensive”, which could be evaluated using the equivalent proposition “At least 4 platforms are not expensive”.

5 Application to e-Learning System Ranking

We apply our linguistic quantifier based ranking method to our context of e-learning platforms ranking.

Let us model an e-Learning platform $p_j$ as a set of couples of (criterion, value) as follows $p_j = \{(c_{j1}, v_{j1}),(c_{j2}, v_{j2}),...,(c_{jn}, v_{jn})\}$ where $c_{ji}$ with $i\in \{1,...,n\}$ is a criterion and $v_{ij}$ its value given by a user to express the importance of platform functionalities. Therefore, we define a platform satisfaction degree, denoted by $\theta (p_j)$ as the truth value of the following fuzzy quantified proposition:

“Almost all criteria are important”

Its evaluation is given by using the decomposition-based approach through the formula (11) as follows:

$$\begin{aligned} \theta (p_j) = max_{i=1,...,n}(min(\mu _{imp}(c'_{ij}), \mu _{allmost\_all}(\frac{i}{n}))) \end{aligned}$$

(11)

$\mu _{imp}(c_{ij})$ is the degree of importance of criterion i of platform j, and $\mu _{imp}(c'_{i1})\ge \mu _{imp}(c'_{i2}) \ge ...\ge \mu _{p}(c'_{in})$ are the ascending raking of the criteria according to their evaluation $\mu _{imp}(c_{ij})$.

The ranking rule is as “the closer to 1 $\theta$ is the top ranked the platform will be”. The underlying preference relation is defined as follows:

Definition 6

(Preference Relation $\tilde{\succeq }$ ). Let $p_i$ and $p_j$ are two e-learning platforms. Then, $p_i \tilde{\succeq } p_j$ if and only if $\theta (p_i) \ge \theta (p_j)$.

Remark 2

It is easy to prove that the preference relation $\tilde{\succeq }$ is a total order.

To apply our fuzzy quantifier based ranking method on the set of considered platforms, we need to map the evaluation symbols of Table 7 to the unit interval [0, 1]; for instance, we can consider the following mapping: $(E, 1),(*,0.8),(\#,0.6),(+,0.4), (|,0.2),(0,0)$. The second requirement is to define the linguistic quantifier “almost_all” by its membership function. For instance, Fig. 1 illustrates an example of membership function of the considered quantifier. It represents the following function defined on [0, 1]:

$$\mu _{almost\_all}(x)={\left\{ \begin{array}{ll} 0 &{} if\,x\in [0,0.5]\\ \frac{5}{2}(x-\frac{1}{2}) &{} if\, x\in [0.5, 0.9]\\ 1 &{} if\, x \in [0.9, 1] \end{array}\right. }$$

Example 1

We evaluate the considered criterion for the platform “Sakai” according to its symbolic evaluation of Table 7 based on the linguistic quantifier almost_all of Fig. 1. Let F be a fuzzy set of “Sakai” criteria such that:

$$\begin{aligned} S= & {} \{ (c_{1},0.8),(c_{2},0.6),0.8/C_{3},0.8/C_{4},0.6/C_{5},0.8/C_{6},0.2/C_{7},0.8/C_{8}, 0.2/C_{9},\\ {}&1/C_{10}, 0.8/C_{11}, 0.2/C_{12},1/C_{13},0.8/C_{14},0.8/C_{15}, 0.8/C_{16}, 0.2/C_{17}, 0.8/C_{18},\\ {}&0.6/C_{19}, 0/C_{20}, 0.6/C_{21},0.2/C_{22}, 0.6/C_{23}, 1/C_{24}, 0.8/C_{25}\} \end{aligned}$$

Fig. 1.

Example of definition of the quantifier almost_all.

The ascendant ranking of important criteria is:

$C_{10}\ge C_{13}\ge C_{24}\ge C_{1}\ge C_{3}\ge C_{4}\ge C_{6}\ge C_{8}\ge C_{11}\ge C_{14}\ge C_{15}\ge C_{16}\ge C_{18}\ge C_{25}\ge C_{2}\ge C_{5}\ge C_{19}\ge C_{21}\ge C_{23}\ge C_{7}\ge C_{9}\ge C_{12}\ge C_{17}\ge C_{22}\ge C_{20}$

Formula (11) is then applied as follows:

$$\begin{aligned} \theta (Sakai)= & {} max(\\&min(\mu _{Imp}(c_{10}),\mu _{almost\_all}(0.04)), min(\mu _{Imp}(c_{13}),\mu _{almost\_all}(0.08)),\\&min(\mu _{Imp}(c_{24}),\mu _{almost\_all}(0.12)), min(\mu _{Imp}(c_{1}),\mu _{almost\_all}(0.16)),\\&min(\mu _{Imp}(c_{3}),\mu _{almost\_all}(0.2)), min(\mu _{Imp}(c_{4}),\mu _{almost\_all}(0.24)),\\&min(\mu _{Imp}(c_{6}),\mu _{almost\_all}(0.28)), min(\mu _{Imp}(c_{8}),\mu _{almost\_all}(0.32)),\\&min(\mu _{Imp}(c_{11}),\mu _{almost\_all}(0.36)), min(\mu _{Imp}(c_{14}),\mu _{almost\_all}(0.4)),\\&min(\mu _{Imp}(c_{15}),\mu _{almost\_all}(0.44)), min(\mu _{Imp}(c_{16}),\mu _{almost\_all}(0.48)),\\&min(\mu _{Imp}(c_{18}),\mu _{almost\_all}(0.52)), min(\mu _{Imp}(c_{25}),\mu _{almost\_all}(0.56)),\\&min(\mu _{Imp}(c_{2}),\mu _{almost\_all}(0.6)), min(\mu _{Imp}(c_{5}),\mu _{almost\_all}(0.64)),\\&min(\mu _{Imp}(c_{19}),\mu _{almost\_all}(0.68)), (\mu _{Imp}(c_{21}),\mu _{almost\_all}(0.72)),\\&min(\mu _{Imp}(c_{23}),\mu _{almost\_all}(0.76)), min(\mu _{Imp}(c_{7}),\mu _{almost\_all}(0.8)),\\&min(\mu _{Imp}(c_{9}),\mu _{almost\_all}(0.84)), min(\mu _{Imp}(c_{12}),\mu _{almost\_all}(0.88)),\\&min(\mu _{Imp}(c_{17}),\mu _{almost\_all}(0.92)), min(\mu _{Imp}(c_{22}),\mu _{almost\_all}(0.96)),\\&min(\mu _{Imp}(c_{20}),\mu _{almost\_all}(1)))\\&= max(\\&min(1,0),min(1,0),min(1,0), min(0.8,0),min(0.8,0),min(0.8,0),\\&min(0.8,0),min(0.8,0),min(0.8,0),min(0.8,0),min(0.8,0),\\&min(0.8,0),min(0.8,0.05),min(0.8,0.15),min(0.6,0.25),\\&min(0.6,0.35),min(0.6,0.45),min(0.6,0.55),min(0.6,0.65),\\&min(0.2,0.75),min(0.2,0.85),min(0.2,1),min(0.2,1),min(0.2,1),\\&min(0,1) = max(0,0,0,0,0,0,0,0,0,0,0,0,0.05,0.15,0.25,0.35,\\&0.45,0.55,0.6,0.2,0.2,0.2,0.2,0.2,0)=0.6 \end{aligned}$$

By using the same membership function as in Example 1, we compute truth-value ($\theta (P)$) for each considered platform of our set of e-Learning systems. Results are displayed in Table 8.

Table 8. E-Learning platform’s features fuzzy evaluation.

According to these results, we obtain the following ranking for:

Sakai $\tilde{\succ }$ (Claroline $\tilde{\simeq }$ Dokeos $\tilde{\simeq }$ eFront $\tilde{\simeq }$ Olat $\tilde{\simeq }$ Ganesha $\tilde{\simeq }$ AnaXagora) $\tilde{\succ }$ (ILIAS $\tilde{\simeq }$ Open ELMS).

We notice that according to the quantification, we also obtain none distinguishable platforms (Claroline $\tilde{\simeq }$ Dokeos $\tilde{\simeq }$ eFront $\tilde{\simeq }$ Olat $\tilde{\simeq }$ Ganesha $\tilde{\simeq }$ AnaXagora) and (ILIAS $\tilde{\simeq }$ Open ELMS). But, in this case, we can make use of additional information to refine our ranking. Indeed, the result returned by the decomposition based approach can be interpreted also as the supremum satisfaction degree we can obtain for an element x (a platform in our example), along with the minimum subset of criteria which satisfy the fuzzy condition considered in the quantified proposition and its corresponding proportion to the entire set of criteria.

For instance, in Example 1, the degree returned is 0.6 from the $19^{th}$ greatest criteria, and the minimum set of criteria satisfying the quantified proposition is:

$C'=\{C_{10}, C_{13}, C_{24}, C_{1}, C_{3}, C_{4}, C_{6}, C_{8}, C_{11},C_{14}, C_{15}, C_{16}, C_{18}, C_{25}, C_{2}, C_{5}, C_{19},$ $C_{21}, C_{23}$}, which corresponds to $76\%$ of the criteria. We also notice that it is the maximum set of criteria which maximises the satisfaction degree. That means that the addition of any other criterion does not improve the satisfaction degree at all. Then, we define the least satisfaction proportion and greatest satisfaction proportion parameters (denoted by lsp and gsp respectively). Let us first define the following concepts:

• “P : Q C is A” is a fuzzy quantified proposition with Q is a linguistic quantifier, C is a crisp set of m elements, and A a fuzzy predicate,

• $C^{\ge }=[C_1,...,C_m]$ such that $\mu _A(C_1)\ge \mu _A(C_2)\ge ... \ge \mu _A(C_m)$ is ascendantly ordered vector of elements of C regarding the fuzzy predicate A,

Definition 7

(Least Satisfaction Proportion lsp ). The least satisfaction proportion is $lsp=\frac{J}{m}$, where $J=min\{k: k=1,...,m \wedge \theta _{P} = min(\mu _{A}(C^{\ge }_k),Q(\frac{k}{m}))\}$.

Definition 8

(Greatest Satisfaction Proportion gsp ). The greatest satisfaction proportion is $gsp=\frac{\mathcal {J}}{m}$, where $\mathcal {J}=max\{k: k=1,...,m \wedge \theta _{P} = min(\mu _{A}(C^{\ge }_k),Q(\frac{k}{m}))\}$.

Once these parameters are defined, we can make use of them for element ranking as stated by the following proposition.

Proposition 1

Let $c_i$ and $c_k$ two elements of C of fuzzy quantified proposition of the form “P : Q C is A”.

• (a) If $lsp(c_i) < lsp(c_k)$ then $c_k\succ c_i$

• (b) If $lsp(c_i)= lsp(c_k)\wedge gsp(c_i)> gsp(c_k)$ than $c_i\succ c_k$

• (c) If $lsp(c_i)= lsp(c_k)\wedge gsp(c_i) = gsp(c_k)$ than $c_i\simeq c_k$

Proof

The proof of Proposition 1 can be easily obtained based on the monotony (increasing) property of the quantifier Q.

Table 9. lsp and gsp for similar platforms.

Example 2

We apply Proposition 1 on the set of similar platforms of Table 8 obtained in the previous example. The results are displayed in Table 9.

The sub-ranking is then: Claroline $\tilde{\succ }$ Dokeos $\tilde{\succ }$ eFront $\tilde{\succ }$ Ganesha $\tilde{\succ }$ (Olat $\tilde{\simeq }$ AnaXagora) $\tilde{\succ }$ ILIAS $\tilde{\succ }$ Open ELMS.

The obtained ranking is more refined and allow selecting top k e-learning platforms by combining both qualitative and quantitative measures.

6 Conclusion and Future Work

In this paper, we have investigated two approaches for alternative ranking applied on e-Learning platform selection: a preference relation-based approach and flexible approach based on fuzzy quantification. We have illustrated the idea of their combination for ranking refinement which significantly help users make decision according to their needs and objectives. Practically, we have considered categories to describe each e-Learning system; each of which defines some criteria over well-known properties of these systems. We applied our approach on a set of open source e-Learning systems for which we have gathered through short surveys their evaluation on the considered criteria. The first approach assesses the quality of an e-Learning system by considering its maximum possible satisfaction and/or its minimum guaranteed satisfaction. These values are finally used to rank the alternatives from the most to the least satisfactory, and to deliver to the user one or several optimal systems. In the case of several alternatives are delivered as equally satisfactory, we consider the second approach, more flexible from computation point of view, to refine the ranking. The approach makes use of linguistic quantifiers and fuzzy quantified propositions to compute for each alternative an aggregated value corresponding to what extend its criteria are evaluated as important by the user. In addition, we defined two new parameters for ranking refinement called least satisfactory proportion and greatest satisfactory proportion, denoted by lsp and gsp respectively, to assess the quality of the subset of criteria evaluated as important by the user for each alternative.

For future directions, it is still to undertake a larger survey to evaluate the platforms as accurate as possible. It is also worthy to consider user profiles when performing surveys in such a way that we obtain different values for different profiles. A profile can be defined over a population of users based on their interests and training objectives. From theoretical point of the view, the study of the properties of lsp and gsp such as monotony and their relationship with the definition of the linguistic quantifier involved in the fuzzy quantified proposition can reveal the potential of these parameters for proportion quality evaluation in fuzzy quantified propositions.

A Appendix

1.1 Proof of Property 1 Total Order

We only prove hereinafter the property for the preference relation $\succeq$. The proof of the property for the preference relation $\preceq$ is similar to the one of $\succeq$.

Proof 1

($\succeq$ is Total Order). The preference relation $\succeq$ is a total order iff:

• 1. $\succeq$ is reflexive

• 2. $\succeq$ is antisymmetric

• 3. $\succeq$ is transitive

• 1. Relation $\succeq$ is reflexive iff $\forall a \in \mathcal {S}:a\succeq a$. Therefore, a $\succeq$ a iff $pos_\mathcal {S}(a)\le pos_\mathcal {S}(a)$ which is verified for the comparison operator $\le$ since $\le$ is reflexive. Then $\succeq$ is reflexive.

• 2. Relation $\succeq$ is antisymmetric iff $\forall a, b \in \mathcal {S}:a = b$. Then:

  a $\succeq$ b $\wedge$ b $\succeq$ a iff $pos_\mathcal {S}(a) \le pos_\mathcal {S}(b)$ $\wedge$ $pos_\mathcal {S}(b) \le pos_\mathcal {S}(a)$ which is verified for $\le$ since $\le$ is antisymmetric.

  Then $\succeq$ is antisymmetric.

• 3. $\succeq$ is transitive iff $\forall a, b, c \in \mathcal {S}: a \succeq b \wedge b \succeq c \Rightarrow a \succeq c$. As a $\succeq$ b $\wedge$ b $\succeq$ c then $pos_\mathcal {S}(a) \le pos_\mathcal {S}(b)$ $\wedge$ $pos_\mathcal {S}(b)\le pos_\mathcal {S}(c)$. Therefore, $pos_\mathcal {S}(a)\le pos_\mathcal {S}(c)$ since $\le$ is transitive. That means that $a \succeq c$ and $\succeq$ is transitive.

1.2 Proof of prefMax Properties

Proof 2

(prefMax Properties)

• 1. prefMax is associative on $\mathcal {S}$:

  $\forall a, b, c \in \mathcal {S}$, then: $prefMax(prefMax(a,b),c) = prefMax(a,prefMax(b,c))$. We denote by I the left term PrefMax(PrefMax(a, b), c) and by II the right term PrefMax(a, PrefMax(b, c)).

  Table 10 shows results of evaluation of the left and the right terms, which are identical. Therefore prefMax is associative.

• 2. prefMax is commutative iff $\forall a, b \in \mathcal {S}:prefMax(a,b) = prefMax(b,a)$.

  From Table 3, the prefMax matrix is symmetric so prefMax is commutative.

• 3. prefMax is idempotent iff $\forall a \in \mathcal {S}$:

  $prefMax(a,a) = a$.

  From the main diagonal of Table 3, we conclude that prefMax is idempotent.

• 4. prefMax has 0 as neutral element iff $\forall a \in \mathcal {S}:$

  $prefMax(a,0) = a$.

  Table 3 shows that prefMax has 0 as neutral element.

• 5. prefMax has E as absorbent element iff $\forall a \in \mathcal {S}: prefMax(a,E) = E$.

  Table 3 also shows that prefMax has E as absorbent element.

Table 10. Formula results for the property of associativity of prefMax operator.

1.3 Proof of prefMin Properties

Proof 3

(prefMin Properties)

• 1. prefMin is associative on $\mathcal {S}$:

  $\forall a, b, c \in \mathcal {S}$: $prefMin(prefMin(a,b),c) = prefMin(a,prefMin(b,c))$. We denote by I the left term PrefMin(PrefMin(a, b), c) and by II the right term PrefMin(a, PrefMin(b, c)).

  Table 11 shows results of evaluation of the left and the right terms, which are identical. Therefore prefMin is associative.

• 2. prefMin is commutative iff $\forall a, b \in \mathcal {S}: prefMin(a,b) = prefMin(b,a)$.

  From Table 3, the prefMax matrix is symmetric so prefMin is commutative.

• 3. prefMin is idempotent iff $\forall a \in \mathcal {S}: prefMin(a,a) = a$.

  From the main diagonal of Table 3, we conclude that prefMin is idempotent.

• 4. prefMin has E as neutral element iff $\forall a \in \mathcal {S}:$

  $prefMin(a,0) = a$.

  Table 3 shows that prefMin has E as neutral element.

• 5. prefMin has 0 as absorbent element iff $\forall a \in \mathcal {S}:$

  $prefMin(a,E) = E$.

  Table 3 also shows that prefMin has 0 as absorbent element.

Table 11. Formula results for the property of associativity of prefMin operator.