1 Introduction

During the past two decades, recommender systems have played an important role in helping users deal with the problem of information overload, especially in electronic commerce. Recommender systems help users to find their desired items from an enormous mass of available information [1].

There are six main categories of recommender systems [2,3,4]: content-based systems, collaborative filtering systems, demographic recommender systems, knowledge-based recommender systems, community-based recommender systems, and hybrid systems. In the content-based method, items similar to the ones a given user (called an active user) preferred in the past are recommended to him/her, whereas in collaborative filtering the items are proposed based on the interests of the users similar to the active user [3, 5]. In demographic recommender systems, the demographic information of users are applied for recommending the items. Knowledge-based systems propose items to users based on the knowledge about the usefulness of items for users. In community-based recommender systems, the preferences of the user’s friends are considered for recommendation of items to users. In hybrid systems, a combination of the mentioned techniques is applied [6]. Beside the mentioned six categories of recommender systems, deep learning has recently been applied in recommender systems to learn patterns from the users’ profiles [7], items contents [8,9,10], or both [11]. Experiments show that if these sources of data are available, deep learning improves the recommendations quality [7,8,9,10,11]. However, in some applications the data required for deep learning is not available.

Collaborative filtering approach is a widely used technique in many recommender systems and is taken as the basis of many systems [3, 12]. There are two types of collaborative filtering systems: user-based collaborative filtering and item-based collaborative filtering. In the user-based collaborative filtering, users similar to the active user are found and the items which appeal to those users are recommended to the active user. In the item-based approach, the system finds items similar to the items which have been given high ratings by the active user and the similarity of items is measured based on the rating given by the users similar to the active user [13, 14].

In addition to the ratings given by the users, other information resources have also been used to find similar users in collaborative filtering. Users’ personality information is one of those resources and recently has attracted considerable attention. Studies demonstrate a significant relationship between individuals’ personality traits and their interests [15,16,17]. According to these findings, it is expected that recommender systems using these factors gain a better understanding of users and provide them with better services [15]. User’s personality can be identified through implicit or explicit techniques. So far, some studies have been conducted on the use of users’ personality in collaborative filtering recommender systems. These studies will be reviewed in Sect. 3.

However, like any other system, recommender systems face some issues and challenges. The two major challenges they deal with are: ‘new user’ and ‘rating sparsity’. The new user challenge arises due to the lack of enough information about the new-coming user. The data sparsity challenge refers to the insufficiency of data about users’ preferences and priorities [12, 18].

Due to the wide application of collaborative filtering recommender systems during the past decade, researchers have investigated new methods to cope with the drawbacks of these systems [3, 12]. One group of these methods focuses on clustering the items or users or both of them [12, 19]. Some of the studies conducted on the use of clustering in collaborative filtering systems will be reviewed in Sect. 3. By clustering the users, users similar to one another in some aspects, are detected and placed in the same cluster. Next, the blank elements of information about the preferences and priorities of a user is filled by the information extracted from the users similar to him/her. The features applied for clustering the users directly affect the performance of clustering approach; If these features strongly relate to the preferences and priorities of users with the ability to place the users with similar preferences in the same cluster, the clustering approach becomes highly effective in addressing the ‘new user’ and ‘rating sparsity’ problems. Considering the significant correlation between individuals’ personality and their interests [15,16,17], users’ personality factors seem to be suitable features for clustering the users to tackle the ‘new user’ and ‘rating sparsity’ problems. However, to the best of the authors’ knowledge, the users’ personality traits have not yet been applied as the attributes for clustering the users in collaborative filtering.

This study is an attempt to develop a new method in which clustering of users is performed based on their personality traits in order to improve the performance of the recommender systems while facing the new user and data sparsity challenges.

The remainder of the paper is organized as follows: Sect. 2 introduces the big-5 personality model. Section 3 presents the previous works on application of clustering methods in recommender systems as well as the previous studies on application of personality traits in recommender systems. In Sect. 4, the proposed method is explained in detail. In Sect. 5, the proposed method is evaluated and compared to other methods. Finally, in Sect. 6, conclusion and the guidelines for future research are offered.

2 Personality estimation using big-5 personality model

Personality can be considered as a pattern of permanent characteristics followed by every person in his/her behavior [20–25]. There exist several studies on estimating the personality of users in an implicit manner. For example, the user’s personality is estimated by analyzing the contextual information available in the user’s profiles (age, gender, interests, posts, and weblogs) [16,17,18,19,20], the structural information of online social network graph [26,27,28,29,30], or both the contextual and structural information [31,32,33,34]. Considering the fact that implicit methods of identifying users’ personality traits are complex and time-consuming and have unreliable precision, this study, like the majority of studies conducted in this connection, uses the explicit method and questionnaire to identify users’ personality traits. One of the best-known personality models recognized in psychological studies is the big-5 personality model [30]. A great number of empirical studies have employed this model. Based on big-5 model, individuals’ personalities can be studied in five main factors: neuroticism, extroversion, conscientiousness, agreeableness and openness to experience [35, 36]. NEO-PI and NEO-FFI questionnaires are used to estimate the personality according to big-5 model [37]. These questionnaires contain descriptive statements like “I am not a worrier” and “I really enjoy talking to people”, which are rated on a Likert-type scale. The NEO-PI consists of 180 items to compute the scores of the five factors and the 30 subfactors (named facets). The NEO-FFI questionnaire contains 60 items selected from the NEO-PI used only to calculate the scores of the five factors (12 questions per aspect). The NEO-FFI is applied in situations where time is limited but reliable, accurate, and global measures about the five personality factors are required [38, 39]; therefore, in the current study, a dataset based on the NEO-FFI questionnaires [37] has been employed. Some sample items from the NEO-FFI questionnaire are presented in “Appendix”.

3 Literature review

In this section, we first review the previous studies conducted on the use of clustering in the recommender systems. Then we investigate the previous studies related to the application of users’ personality in the recommender systems.

3.1 Studies on the use of clustering in the recommender systems

Clustering is a method for grouping data based on their similarity or proximity. In recommender systems, items or users or both of them could be clustered. Various studies have been conducted on the use of clustering in collaborative filtering systems. Some of those studies are reviewed in this section.

To improve the problem of data sparsity, Xue et al. [40] clustered the users based on their ratings using K-means algorithm. Afterwards, based on this clustering, they performed smoothing and neighbor selection and then, they filled the blank elements of the user-item matrix. Evaluations indicated the high scalability of their method. Besides, under the condition of data sparsity, their method showed less error compared to other methods. Pu Wang [41] clustered the users based on their ratings using the K-means algorithm so as to address the sparsity and scalability problems in collaborative filtering despite the nonlinear growth of number of users and items. Ullah et al. [42] developed a method in which, first, the users were clustered based on their demographic information, and ratings, and then, using this clustering, the unknown elements of the user-item matrix were filled. Afterwards, a temporal aware hybrid model was developed for each user based on which the user’s neighbors were identified. Finally, in the light of the temporal information, the user’s neighbors, and the user-item matrix, some recommendations were offered to the user. Experiments show that their hybrid method has been able to alleviate the new user and data sparsity problems as well as scalability. Li [43] has clustered both the item and users. In Li’s work, the users were clustered using the K-means algorithm on the basis of their ratings, and, afterwards, the unknown ratings for an item in the user-item matrix are filled based on the mean value of rating of the central user for that item. In the next step, the recommendations are offered to users based on the performed clustering. The results demonstrated that this method alleviates the problems of data sparsity and scalability. Altingovde et al. [44] showed that, due to overgeneralization, the cluster-based collaborative filtering methods which only use the information of cluster centroids offer less personalized recommendations and enjoy low precision. For that reason, they suggested that, in order to estimate the ratings, not only the centroids but also all other members within the clusters must be considered. The results demonstrated the high precision and acceptable scalability of their method. To cope with the new user problem Yanxiang et al. [45] clustered users based on characteristics such as age, gender, and occupation. Afterwards, rather than predicting the ratings of each item for each user, they calculated the mean of the ratings given to each item by all the users belonging to each cluster. The experimental results were indicative of the considerable improvement of this method in facing the new user problem in comparison with some former methods. Moradi Dakhel and Mahdavi [46], used K-means clustering and neighbor’s voting in collaborative filtering. To offer recommendations to the active user, they chose n most similar neighbors from the closest cluster to the active user. Afterwards, to predict the ratings of the active user, they used the voting algorithm. The evaluation results demonstrated that this method is faster and more precise compared to some former methods.

As can be seen in the studies reviewed in this section, to the best of the authors’ knowledge no previous study has clustered the users based on their personality traits.

3.2 Studies on the use of personality in the recommender systems

Users with different personality traits prefer items with different features; thus, personality traits of users could be a valuable information to be exploited for personalization of recommendations [17]. So far some researchers have combined users’ personality aspects with recommender systems. The most common objective of these studies are the personalization of items diversity, improving the system’s computational complexity, increasing the user satisfaction, and addressing the cold-start and data sparsity problems [17]. In this section, studies related to this subject are briefly reviewed.

Tintarev et al. [47] investigated the effect of openness personality trait on the user’s preferred level of novelty and diversity in recommendations. They studied a group of participants who acted as the recommender system. The participants recommended items to a fictitious friend, a favorite item and the personality features of whom were known. The results indicate that people who are low on openness to experience benefit more from inter-theme (thematic) rather than inter-genre (categorical) variation. Cantador et al. [48] studied the correlations between users’ personality types and their preferences in movies, TV shows, music, and books contexts. They analyzed 53,226 Facebook users’ profiles composed of their big-5 personality scores and their interests about 16 genres in each of the four mentioned contexts. They extracted some association rules to relate user’s personality types to some of the genres in the considered contexts.

Roshchina et al. [49] offered a personality-based recommender system for recommending hotels to users. They estimated the users’ personality traits based on the big-5 personality model by investigating the comments written by them. In another study, in order to solve the new user problem, Hu and Pu [50] used users’ personality information in collaborative filtering. The three approaches proposed by them were: (1) offering recommendations to users based merely on their personality information, (2) applying a linear combination of personality and ratings of users, and (3) using a cascade approach to cover both the information resources. Afterwards, they compared these methods with the rating-based collaborative filtering and demonstrated their superiority especially that of the cascade approach. In another study, Hu et al. [51] compared the personality-based preference extraction approach with the rating-based approach. Their results demonstrated that there was no difference in the perceived precision in the two systems. However, the required effort, the time spent by the user, and the user’s intention to use the system again were considerably better in the personality-based system than the rating-based recommender system.

Furthermore, Hu and Pu [52] compared the performance of rating-based, personality-based, and the hybrid (rating + personality) similarity measures for addressing the new user problem. Their findings indicate that the personality-based similarity measure exhibits better performance in addressing the new user problem when it is used together with the rating based personality measure in user-based collaborative filtering recommender systems.

To compute the similarity of users in a memory-based collaborative filtering system, Tkalcic et al. [53, 54] used personality information based on the big-5 personality model. In a series of experiments, they compared their method with a system which used only ratings, demonstrating that the use of personality to measure the similarities in a memory-based collaborative filtering decreases the computational complexity and increases the efficiency. Jeong et al. [55] used personality factors to address the problems of data sparsity and scalability in recommender systems. They used MBTI (Myers-Briggs Type Indicator) to estimate the user personality. Following investigations, they chose ‘age’, ‘gender’, and ‘personality’ as the most important factors in movie recommendation. Wu et al. [56] offered a method to improve the challenge of variety reduction in recommendations. They paid attention to the fact that the users’ diversity requirements are related to users’ personality dimensions. Afterwards, they personalized recommendations variety based on the users’ personality. Experimental results have demonstrated that this system is more effective in comparison with the common methods in respect of users’ subjective perceptions. Hu and Pu [57] proposed a recommender system in which the musical preferences of users are identified based on their personality. They showed that users with less information about music were more willing to use the personality-based recommender system. In another study, Lampropoulos et al. [58] developed a cascade-hybrid music recommender system for cell phone users, which used both music genre classification and personality information. When a user sends a music file through his/her cell phone, the system offers music files of the same genre which is consistent with the previous preferences of that user as well as preferences of other users with the same personality. Braunhofer et al. [59] showed that to cope with the new user problem, the use of the users’ personality information is more effective than the use of their demographic information. They also demonstrated that the use of even one factor of the big-5 personality model can improve the quality of recommendations significantly. To address the new user problem, Wang [60] used the linear combination of user-based collaborative filtering and personality-based and item-based collaborative filtering in order to predict the users’ ratings. Their method could be combined with user-based collaborative filtering recommender systems to improve such systems. To tackle the cold-start problem in context-aware recommender systems, Braunhofer et al. [61] combined the demographics-based and content-based context-aware matrix factorization methods; their system switches between these two methods depending on the target recommendation situation. In another study, to address the cold start problem in context aware collaborative filtering, Braunhofer et al. [62] applied the users’ personality traits in an active learning approach based on the matrix factorization model. In their proposed method, the user’s personality is applied together with rating matrix to present items to the user in a continuous manner in order to obtain a high volume of ratings. In their study, the user personality is acquired through a simple questionnaire at the registration time. Fernández-Tobías et al. [63] examined the incorporation of personality in matrix factorization, active learning, and cross-domain approaches to address the new user problem. They revealed that including the users’ personality in cross-domain approach leads to better results, in the condition of availability of a complementary domain.

As reviewed in this section, in a numerous studies the users’ personality factors have been applied in recommender systems. However, to the best of the authors’ knowledge, in none of those studies have the personality factors been used for clustering the users.

4 The proposed method

In this section, a new method is proposed to improve the recommender systems in order to deal with the data sparsity and new user challenges. For this purpose, users’ personality information and users’ ratings are used as two information resources to alleviate the aforementioned problems in the following steps:

  • Upon first entrance into the system, the users fill out the NEO-FFI questionnaire and the users’ five personality factors are identified based on the big-5 personality model.

  • The users are clustered based on their personality factors using K-means algorithm.

  • To alleviate the data sparsity, the item-based collaborative filtering algorithm is applied on each cluster separately and the unknown ratings of users in each cluster are estimated based on the ratings of other users in that cluster.

  • To alleviate the new user problem, first, the closest cluster to the new user is identified. Then, the user-based collaborative filtering is applied on data of that cluster. This is actually an individualistic strategy described in [44]. Therefore, the favorite items of users whose personalities are similar to the new user’s personality are offered to the new user.

  • For an active user who already has rated some items in the system, the closest cluster to him/her has been detected previously. Whenever there is a need for offering recommendations to him/her, the item-based collaborative filtering are applied on the ratings of the users belonging to that closest cluster.

The proposed strategy includes two stages: the pre-processing stage and the online stage. The pre-processing stage is performed prior to the need for recommendation and the online stage is performed whenever there is a need for recommendation. These two stages will be introduced later on. The stages of the proposed method are demonstrated in Fig. 1. The input and output data of each stage is depicted in this diagram.

Fig. 1
figure 1

Stages of the proposed method

Full size image

4.1 Pre-processing stage

In the pre-processing stage of the proposed method, the processes of users clustering, estimation of the unknown ratings, and the estimation of similarities of items are performed. These processes will be explained later on. The pre-processing stage is repeated iteratively in relatively long intervals.

4.1.1 Users clustering

In the proposed method for clustering the users based on their personality, the K-means algorithm is used. The five personality factors of each user obtained from the NEO-FFI questionnaire form a five-dimensional vector. Each dimension corresponds to one factor of the big-5 personality model and is represented by a number between 1 and 7. The vector \({\text{p}}_{\text{u}} = \left( {{\text{p}}_{\text{u}}^{1} ,{\text{ p}}_{\text{u}}^{2} , \ldots ,{\text{p}}_{\text{u}}^{5} } \right)\) describes the normalized personality of user u. The normalized personality vectors of users are considered as the input of the K-means clustering algorithm and the clusters centroids are determined. The studies show that the five factors of big-5 personality model are not independent but they are correlated to each other [64,65,66,67,68]. Therefore, being inspired by the former studies [17, 50, 52], the Pearson’s correlation coefficient [69] is taken as a criterion to measure the similarity between a user and a cluster centroid.

Equation (1) measures the similarity between the personality of a user u and a cluster centroid i (represented by vi).

$${\text{simp}}\left( {{\text{u}},{\text{v}}_{\text{i}} } \right) = \frac{{\mathop \sum \nolimits_{k = 1}^{5} \left( {{\text{p}}_{\text{u}}^{\text{k}} - \overline{{{\text{p}}_{\text{u}} }} } \right)\left( {{\text{p}}_{{{\text{v}}_{\text{i}} }}^{\text{k}} - \overline{{{\text{p}}_{{{\text{v}}_{\text{i}} }} }} } \right)}}{{\sqrt {\mathop \sum \nolimits_{k = 1}^{5} \left( {{\text{p}}_{\text{u}}^{\text{k}} - \overline{{{\text{p}}_{\text{u}} }} } \right)^{2} \mathop \sum \nolimits_{k = 1}^{5} \left( {{\text{p}}_{{{\text{v}}_{\text{i}} }}^{\text{k}} - \overline{{{\text{p}}_{{{\text{v}}_{\text{i}} }} }} } \right)^{2} } }}$$
(1)

In the above equation, the similarity between the new user u and vi is represented by \({\text{simp}}\left( {{\text{u}},{\text{v}}_{\text{i}} } \right)\). \(\overline{{p_{u} }}\) stands for the mean of five personality traits of user u and \(\overline{{p_{{v_{i} }} }}\) stands for the mean of five personality traits of the cluster centroid vi. \({\text{p}}_{\text{u}}^{\text{k}}\) represents the kth trait of the personality vector of user u and \({\text{p}}_{{{\text{v}}i}}^{\text{k}}\) represents the kth trait of cluster centroid vi. The value obtained by this equation falls in the interval [− 1, 1] and the closer it is to 1 the greater the similarity.

Actually the main difference between the proposed methods and the previous clustering-based recommender systems lies in the fact that clustering of users in the proposed method is based on the users’ personality dimensions.

4.1.2 Building a dense user-item matrix for each cluster

Following the clustering, the next step is to fill in the unknown elements in the user-item matrix. For this purpose, the unknown ratings of the users of each cluster are estimated using the weighted mean of the ratings of the other users belonging to that cluster; thus, the sparse matrix becomes a dense matrix of which, all entries are filled with actual or estimated ratings.

Considering that the estimation of unknown ratings of the users is based on the ratings of users with similar personalities, it is expected that the estimation be more precise than the case where estimation is based on the ratings of all the users.

4.1.3 Measuring the similarity of items

In the proposed method, the similarity of items must be measured to be used in the item-based collaborative filtering approach which is adopted to offer recommendations to the existing users. To speed up the proposed method, this step is performed offline. For this to be done, the adjusted cosine similarity measure [70, 71] is applied on the ratings of each cluster members so as to measure the similarity of each pairs of items based on the ratings of that cluster. Therefore, each pair of items may have different similarity degrees from the viewpoint of members of different clusters. These measured similarities will be used to find the neighbors of user’s favorite items in order to offer recommendations.

4.2 Online stage

The online stage is run when there is a need for recommending items to an active user. In this stage, the ratings of the active user to the items which have not been rated previously are predicted. Afterwards, the items with highest ratings are recommended to the user. In the online stage, the process of recommendation to the new user is different from that to other users. These two processes will be explained later on.

4.2.1 Offering recommendations to an already existing user

For an already existing user, the unknown ratings of the items are estimated tentatively in the pre-processing stage to alleviate the data sparsity. In this stage, however, the ratings are estimated more precisely. As mentioned, in the proposed method the item-based approach is adopted to offer recommendations to an already existing user. The closest cluster to the user was identified in advance (see Sect. 4.1.1). This cluster is called c here. In this process, at first, among the items which the active user has not encountered, the items similar to his/her favorite items are identified. For this purpose, the similarities measured based on that cluster in pre-processing stage is used. Afterwards, the active user’s ratings for the similar items are predicted. The predicted ratings assigned by user u to item \(i\), \(R_{u.i}^{ '}\) is calculated through Eq. (2) [70]:

$$R_{u,i}^{ '} = \frac{{\mathop \sum \nolimits_{j \in S} \left( {sim_{c} \left( {i,j} \right) \times R_{u,j} } \right)}}{{\mathop \sum \nolimits_{j \in S} \left( {\left| {sim_{c} \left( {i,j} \right)} \right|} \right)}}$$
(2)

where, S is the set of all the items rated by user u, Ru,j is the rates assigned by user u to item j, and simc(i, j) is the similarity between items i and j calculated based on the ratings of the users belonging to cluster c. As mentioned in Sect. 4.1.3, the similarity of each pair of items is estimated through the adjusted cosine similarity measure [70, 71] on the users’ ratings in the pre-processing stage.

Finally, N items which have received higher ratings based on this prediction are recommended to the active user.

4.2.2 Offering recommendations to a new user

As mentioned before, in the proposed method, the user-based collaborative filtering is used in order to offer recommendations to a new user. For this purpose, first the closest cluster to the new user is identified and then, considering the rates given by the users in that cluster, the ratings of the new user to items are predicted and some recommendations are offered to the new user. In what follows, these two steps are described.

  • Determining the closest cluster to the new user

When the new user fills out the personality questionnaire, his/her 5-dimensional personality vector is determined. The cluster centroids, which have been determined in the pre-processing stage using the K-means algorithm, are representatives of the common personality traits of the users in that cluster. Hence, to find the closest cluster to the new user, the similarity between the normalized personality vector of the new user and each cluster centroid is measured using Pearson’s correlation coefficient [69] [see Eq. (1)]. After measuring the similarity between cluster centroids and the new user, the cluster with the highest similarity is selected. This selected cluster is called \({\text{c}}_{\text{u}}\).

  • Prediction of new user’s ratings

Equation (3) is used to predict the rate given to item i by the new user u. Using this equation, the weighted mean of rates given by the users in cluster \({\text{c}}_{\text{u}}\) (i.e., the closest cluster to the new user u) to item i is calculated.

$${\text{r}}_{{u,{\text{i}}}} = \frac{{\mathop \sum \nolimits_{{{\text{y}} \in {\text{c}}_{\text{u}} }} {\text{w}}_{\text{yi }} \cdot {\text{r}}_{{{\text{y}},{\text{i}}}} }}{{\sum {{\text{w}}_{\text{yi}} } }}$$
(3)

\({\text{r}}_{{{\text{u}},{\text{i}}}}\) is the predicted rate that the new user u will give to item i. \({\text{r}}_{{y,{\text{i}}}}\) is the rate that user y from cluster \({\text{c}}_{\text{u}}\) has given to item i. \(c_{u}\) represents the closest cluster to user u. \({\text{w}}_{\text{yi }}\) is a coefficient which distinguishes between actual and estimated ratings; If item i is actually rated by user y, then \({\text{w}}_{\text{yi }}\) is equal to a pre-defined value α. If the actual rating of user y to item i is unknown but it is estimated in the pre-processing stage for densification of user-item matrix, then \({\text{w}}_{\text{yi }}\) is equal to 1 − α. Parameter α takes value in the interval (0, 1) and its appropriate value is assessed in Sect. 5.2. After predicting the ratings for the items not encountered by the new user, a list of N items with highest predicted ratings, which probably appeal to the new user, is offered to him/her.

5 Evaluation

In this section, first, the dataset used for evaluation is introduced. Afterwards, the effect of control parameters on the performance of the proposed method is investigated, and, finally, the proposed method is compared with some previous methods.

5.1 Dataset

To evaluate the proposed method, a dataset including both personality traits and users’ ratings is required. Therefore, the STS (South Tyrol Suggests) dataset, a dataset collected by a research center at Bozen-Bolzano University in Italy [1, 59, 61, 62], is applied. STS contains the data collected by a context-aware recommender system which offers recommendations to the visitors of South Tyrol region in Italy. It includes 2534 ratings given by 465 users from September 18, 2013 to January 14, 2014 and the number of items here is 249. Many of the ratings accompany the contextual information about the experience. The ratings contextual information and the users’ demographic information are not considered in this study; therefore, they are discarded from the dataset. Among all the users 380 users have completed the personality questionnaire. The ratings given by these 380 users (total of 1723 ratings) supplemented with the users’ 5 personality dimensions constitute the required data for evaluation of this method; therefore, the dataset applied in our study contains two sets of data:

  • The 1723 ratings given by the users to items, within 1–5 range, 1, non-favorability and 5, favorability.

  • The 5 personality dimensions of each one of the 380 users according to the big-5 personality model, within 1–7 range for each personality trait.

To evaluate the methods, the proposed method is implemented in Java and the entire dataset is split into training data (70%) and test data (30%), in a random manner. Considering the fact that the personality factors of only 380 users are available in the STS dataset, we split the dataset in term of users’ ratings, not in term of users. To do this, being inspired by the experimental setup of former studies [50, 70,71,72], 70% of the users’ ratings has been selected in a random manner and applied as the training data, and the remaining ratings of users are applied as the test data. Splitting process repeats for 10 times and the average of results are calculated over the 10 runs. Parameter N, that is the number of recommendations to each user, is considered to be 7.

5.2 The effect of control parameters on the precision of the proposed method

The control parameters of the proposed method consist of α, Ns, and K. K is the number of clusters in K-mean algorithm. Parameter α is the weight of the estimated ratings (the weight of actual ratings is equal to 1 − α) in estimating the ratings through Eq. (3) and Ns is the number of each item’s neighbors in the item-based collaborative filtering and the number of each user’s neighbors in the user-based collaborative filtering. In this section, the effect of values of these parameters on the precision of the proposed methods is investigated and the best values for these parameters are chosen for the mentioned dataset.

First, the proposed method was applied on the dataset for several values of K. The sum of squared errors (SSE) was calculated for each value of K. The most precise results were obtained for 3–7 clusters. Afterwards, the effect of different values of \(\upalpha\) and Ns on the performance of the proposed method was investigated. In this evaluation, the number of clusters was taken as 3 and the value of \(\upalpha\) varied from 0.1 to 0.9. Subsequently, the weight of estimated ratings varied from 0.9 to 0.1. For each value of \(\upalpha\), the values 10, 5, 15, and 20 were assumed for Ns. As illustrated by Fig. 2, the best results were obtained when \(\upalpha\) = 0.4 and Ns = 15. To determine the best values for the control parameters, another evaluation was conducted by changing the value of K in the case where \(\upalpha\) = 0.4 and Ns = 15. Figure 3 displays the results of these experiments.

Fig. 2
figure 2

Mean absolute error of the proposed method for several values of α and Ns

Full size image
Fig. 3
figure 3

Mean absolute error of the proposed method with respect to the number of clusters (K)

Full size image

In the light of these evaluations, K was determined at 3, and \(\upalpha\) at 0.4. Also, Ns was determined at 15 for doing the rest of the experiments.

5.3 Comparison with baseline methods

In this section, we compare our proposed method with some other collaborative filtering methods. To do this, we select two traditional collaborative filtering methods and two state-of-the-art methods for comparison. The applied state-of-the-art methods have similarities with our proposed method; one of them is personality-based and the other one is clustering-based. The mean absolute error and precision criteria are used for comparing the performance of these algorithms in alleviating the data sparsity and the new user problems.

In these evaluations, the new method is referred to as CLPCF and the other methods used for comparison are as follows:

  • The first method is the traditional user-based collaborative filtering (UbCf) where, ratings are predicted based on the entire collection of ratings. Pearson’s correlation coefficient is applied for computing the similarity among users. For handling the new users who have recently joined the system, the average of ratings of all users for any of the items is calculated and used for providing recommendations. For handling the sparsity problem, user-based collaborative filtering algorithm runs in an offline stage and the unknown elements of sparse user-item matrix are filled with the weighted mean of other users’ ratings.

  • The second method is the traditional item-based collaborative filtering (IbCf) where, Pearson’s correlation coefficient is applied as the similarity measure and to provide recommendations for new users, and like as UbCf, the averaged ratings of the items are used. Item-based collaborative filtering algorithm is applied in the offline stage to fill in the sparse user-item matrix.

  • The third method is the personality-based collaborative filtering (PbCf) presented by Hu and Pu [49]. The PbCf follows a cascade approach. At the first stage, the 5 personality dimensions of users (according to the big-5 personality model) are applied for predicting the initial ratings for filling the blank elements of user-item matrix. Pearson’s correlation coefficient is applied on the users’ personality dimensions for computing the similarity among users. Afterwards, the traditional user-based collaborative filtering is applied on the obtained matrix for refining the predations obtained in the first stage. For new users, the users’ personality dimensions are used in finding the similar users and predicting the ratings.

  • The fourth method is a collaborative filtering where, users are clustered based on their ratings through the K-means algorithm (ClbCf) [41, 44, 70]. The unknown ratings of a user are predicted by an equation based on his/her average ratings and the ratings of the other users belonging to his/her related cluster. In this method, adjusted cosine-based similarity is applied for computing the similarity among items and the Euclidean distance is used for computing the similarity among users. For handling the new users, the averages of other users’ ratings are applied.

For comparison, the parameters of the four compared algorithms should be initialized. Hu and Pu [49] evaluated the PbCf method for several values of neighbor size ranging from 5 to 100. They revealed that this method is not significantly sensitive to the number of neighbors. According to their experiments, here, the number of user’s neighbors in both the stages of PbCf method is assumed to be 15. According to the experiences conducted by Wang et al. [70], the best value for the number of item’s neighbors in ClbCf method is 20. Thus, in our evaluation, this parameter in ClbCf is set to 20. In addition, being inspired by the study of Wang et al. [70], the number of clusters in k-means algorithm is set to 30. The number of user’s neighbors in UbCf and the number of item’s neighbors in IbCf are assumed to be 5 and 10, respectively. Our experiments revealed that these values led to higher precision, specially in prediction of ratings for the new users. Parameter N (i.e., the number of recommendations) is a common parameter in all the compared methods. Our experiments showed that assigning values higher than 7 to parameter N decreases the precision metric for the considered dataset, even though it may have positive effect on some of the other metrics (e.g., recall metric). Considering the fact that our goal is to evaluate the methods based on the precision and MAE metrics, we set parameter N to be 7 in all the compared methods, as its best value.

5.3.1 Comparing the methods in the conditions of data sparsity

To evaluate the performance of this proposed method against different levels of data sparsity (50, 75, and 100%), the sparse user-item matrix is prepared as follows, in accordance with Hu and Pu [52, 73]:

  • To have a dataset with the sparsity level of 50%, a dataset is prepared which contains 50% of ratings of all users (except for the recently joined users). In the same manner, to have a dataset with sparsity level of 75%, a dataset is prepared which contains 25% of ratings of all users (except for the recently joined users). The obtained dataset is split into training data (70%) and test data (30%), in a random manner. Dataset preparation process runs 10 times in a random manner, and the results are averaged over the 10 runs.

  • To evaluate the method in the condition of sparsity level of 100%, iteratively, one of the ratings of any of the users is selected in a random manner and applied as the test data, and entire ratings of all of the users are applied as the training data. This evaluation method is named all-but-one [52].

Figures 4 and 5 present the mean absolute error and the precision of the methods in facing different levels of data sparsity. As can be seen, an increase in sparsity level leads to an increase in mean absolute error. However, the results demonstrate that data sparsity has a low effect on the performance of the proposed method.

Fig. 4
figure 4

Mean absolute error of the methods with respect to the density of the user-item matrix

Full size image
Fig. 5
figure 5

Precision of the methods with respect to the density of the user-item matrix

Full size image

5.3.2 Comparing the methods in facing the new user

Users who have rated few items in the system are considered as new users [17, 50,51,52,53, 74]. Here, we define the new users as the users who have given less than 6 ratings. In order to evaluate in the condition of encountering new users, first, the users who have given more than 3 ratings in the system, are selected. Next, among these selected users, some users are chosen in a random manner to play the role of new users (i.e., users who have given few ratings in the system). For these purpose, being inspired by former studies [50, 52, 53], 3, 4, or 5 of their ratings are selected in a random manner as their available ratings (i.e., the training data). The other ratings of these users are considered as test data for which the prediction should be made. This group of users, which constitute about 10% of the entire users, are applied for the evaluation of methods in the condition of encountering new users. The process of preparing the new users dataset repeats for 10 times and the average results are reported. For evaluation of methods in condition of offering recommendations to the existing users, the entire dataset is split into training data (70%) and test data (30%), repeated for 10 times according to the process mentioned in Sect. 5.1.

The mean absolute error of the methods in predicting the ratings of the items for new users and already existing users is presented in Fig. 6 and the precision of the methods is shown in Fig. 7. As illustrated, the precision of the proposed method in facing the new user is higher and its mean absolute error is lower in comparison to the other methods.

Fig. 6
figure 6

Mean absolute error of the methods in prediction of ratings for existing users and new users

Full size image
Fig. 7
figure 7

Precision of the methods in prediction of ratings for existing users and new users

Full size image

6 Conclusion

In this paper, a method was proposed to improve the prediction results of recommender systems in facing the data sparsity and new user challenges. In the proposed method, users’ personality factors are described based on the big-5 personality model and the users are clustered based on their personality factors. Afterwards, the unknown elements of the user-item matrix (in the condition of data sparsity) as well as the rates given to the items by the users (both the new users and the already existing users) are estimated based on the rates given by other users in the related clusters. The evaluations showed that, with regard to reducing mean absolute error and increasing the precision, the proposed method has obtained better results in comparison with the other methods.

Since personality traits are relatively stable, the need to update the clusters is less than in the case where clustering is based on ratings or any other unstable factors. Besides, finding similar users in a cluster requires less processing time and is done faster than finding them among all users. To improve the evaluations, the scalability of the proposed method could be studied accurately in future studies. Furthermore, using lie detectors, statistical methods, etc., the accuracy of the collected personality data could be improved by identifying the inaccurate, dishonest, and fake responses [75, 76]. For example, the sabotage index [77], the fixed individualized chance score [78], and the inter-item standard deviation index [79] may be used for this purpose.

The proposed method can be compared to the techniques other than the mentioned four methods, as well. For example, the proposed method can be compared to the recommender systems that uses deep learning to learn rich features for items or users. In addition, it is suggested that other clustering methods than the K-means algorithm be used for clustering the users based on their personality factors and the results be evaluated. Besides, it is recommended that the explicit method be replaced by implicit methods to identify the users’ personality. Finally, the current emotion of users can be detected and combined with their personality factors [80, 81] to improve the accuracy of the recommendations.