Abstract
Creating and maintaining semantic structures such as ontologies continues to be an important issue. The approach investigated here is to let members of an online community create structured knowledge collaboratively and to use ratings to evaluate the data created. Obviously, such ratings have to be of high quality. Honest rating mechanisms (HRMs) known from literature are a promising means to gain such high-quality ratings. However, the design of such mechanisms for collaborative knowledge creation and their effectiveness have not been studied so far. To evaluate the effects of an HRM on rating quality in this context, we have conducted several experiments with online communities. We find that an HRM increases rating quality and “punishes” rating errors. We also find that rating-based rewards increase the quality of the structured knowledge created.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Albers, A., et al.: A new perspective on product engineering overcoming sequential process models. In: The Future of Design Methodology. Springer London (2011)
Auer, S., Dietzold, S.: OntoWiki—a tool for social, semantic collaboration. In: Proceedings of the 5th International Semantic Web Conference ISWC (2006)
Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P., Kraut, R.E.: Using social psychology to motivate contributions to online communities. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, CSCW ’04, pp. 212–221. ACM, New York, NY, USA (2004)
Benkler, Y.: The wealth of networks: how social production transforms markets and freedom. Yale University Press, New Haven, CT, USA (2006)
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., Taylor, J.: Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, pp. 1247–1250. ACM, New York, NY, USA (2008)
Braun, S., Schmidt, A., Walter, A., Nagypal, G., Zacharias, V.: Ontology maturing: a collaborative Web 2.0 approach to ontology engineering. In: Proceedings of the Workshop on Social and Collaborative Construction of Structured Knowledge at the 16th International World Wide Web Conference (WWW 07), Banff, Canada (2007)
Buitelaar, P., Cimiano, P.: Ontology Learning and Population: Bridging the Gap Between Text and Knowledge. Frontiers in Artificial Intelligence and Applications. Ios Press (2008)
Campanini, S.E., Castagna, P., Tazzoli, R.: Platypus wiki: a semantic wiki wiki Web. In: 1st Italian Semantic Web Workshop (2004)
Cosley, D., Frankowski, D., Kiesler, S., Terveen, L., Riedl, J.: How oversight improves member-maintained communities. In: Proceedings of the SIGCHI conference on Human Factors in Computing Systems, CHI ’05, pp. 11–20. ACM, New York, NY, USA (2005)
Cosley, D., et al.: Is seeing believing?: how recommender system interfaces affect users’ opinions. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2003)
Douceur, J.R.: The sybil attack. In: IPTPS (2002)
Eckert, K., et al.: Crowdsourcing the assembly of concept hierarchies. In: Proceedings of the 10th Annual Joint Conference on Digital Libraries - JCDL ’10 (2010)
Efron, B.: Forcing a sequential experiment to be balanced. Biometrika 58(3), 403–417 (1971)
Fleiss, J.L.: Statistical Methods for Rates and Proportions, 2nd edn. Wiley, New York (1981)
Holsapple, C.W., Joshi, K.D.: A collaborative approach to ontology design. ACM Commun. 45(2), 42–47 (2002)
Hütter, C., Kühne, C., Böhm, K.: Peer production of structured knowledge—an empirical study of ratings and incentive mechanisms. In: Proceeding of the 17th ACM Conference on Information and Knowledge Mining - CIKM ’08 (2008)
Jäschke, R., et al.: Organizing publications and bookmarks in bibsonomy. In: CKC (2007)
Jurca, R., Faltings, B.: Minimum payments that reward honest reputation feedback. In: Proceedings of the 7th ACM Conference on Electronic Commerce - EC ’06 (2006)
Jurca, R., Faltings, B.: Collusion-resistant, incentive-compatible feedback payments. In: Proceedings of the 8th ACM Conference on Electronic Commerce, EC ’07, pp. 200–209. ACM, New York, NY, USA (2007)
Kittur, A., Kraut, R.E.: Harnessing the wisdom of crowds in Wikipedia: quality through coordination. In: Proceedings of the 2008 ACM Conference on Computer Supported Cooperative work, CSCW ’08, pp. 37–46. ACM, New York, NY, USA (2008)
Kochhar, S., Mazzocchi, S., Paritosh, P.: The anatomy of a large-scale human computation engine. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP ’10, pp. 10–17. ACM, New York, NY, USA (2010)
Kotis, K., Vouros, A.: Human-centered ontology engineering: The hcome methodology. Knowl. Inf. Syst. 10(1), 109–131 (2006)
Miller, N., Resnick, P., Zeckhauser, R.: Eliciting informative feedback: the peer-prediction method. Manag. Sci. 51(9), 1359–1373 (2005)
Moor, A.D., Leenheer, P.D., Meersman, R., Starlab, V.: Dogma-mess: a meaning evolution support system for interorganizational ontology engineering. In: In Proc. of the 14th International Conference on Conceptual Structures, ICCS 2006, pp. 189–203. Springer-Verlag (2006)
Noy, N., Chugh, A., Alani, H.: The ckc challenge: exploring tools for collaborative knowledge construction. IEEE Intell. Syst. 23(1), 64–68 (2008)
Pepper, S., Architect, S.I., Infotek, S.: The TAO of Topic Maps—Finding the Way in the Age of Infoglut. In: Proceedings of XML Europe (2000)
Prelec, D.: A Bayesian truth serum for subjective data. Science (New York, N.Y.) 306, 462–466 (2004)
Schaffert, S.: IkeWiki: a semantic wiki for collaborative knowledge management. In: Tolksdorf, R., Simperl, E.P.B., Schild, K. (eds.) In: Proceedings of the First International Workshop on Semantic Technologies in Collaborative Applications STICA 06, pp. 388–396 (2006)
Shaw, A.D., Horton, J.J., Chen, D.L.: Designing incentives for inexpert human raters. In: Proceedings of the ACM 2011 Conference on Computer Supported Cooperative work, CSCW ’11, pp. 275–284. ACM, New York, NY, USA (2011)
Siorpaes, K., Hepp, M.: Games with a purpose for the semantic Web. IEEE Intell. Syst. 23(3), 50–60 (2008)
Siorpaes, K., Simperl, E.: Human intelligence in the process of semantic content creation. World Wide Web 13(1–2), 33–59 (2010)
Souzis, A.: Building a semantic wiki. IEEE Intell. Syst. 20, 87–91 (2005)
Sunagawa, E., Kozaki, K., Kitamura, Y., Mizoguchi, R.: An environment for distributed ontology development based on dependency management. In: International Semantic Web Conference, pp. 453–468 (2003)
Tijerino, Y.A., Embley, D.W., Lonsdale, D.W., Ding, Y., Nagy, G.: Towards ontology generation from tables. World Wide Web 8(3), 261–285 (2005)
Tudorache, T., Noy, N.F.: Collaborative protege. In: CKC (2007)
Viegas, F.B., Wattenberg, M., Kriss, J., van Ham, F.: Talk before you type: coordination in Wikipedia. In: Proceedings of the 40th Annual Hawaii International Conference on System Sciences, HICSS ’07, p. 78. IEEE Computer Society, Washington, DC, USA (2007)
Völkel, M., Krötzsch, M., Vrandecic, D., Haller, H., Studer, R.: Semantic Wikipedia. In: WWW, pp. 585–594 (2006)
von Ahn, L.: Games with a purpose. IEEE Comput. 39(6), 92–94 (2006)
Vrandecic, D., Pinto, H.S., Tempich, C., Sure, Y.: The diligent knowledge processes. J. Knowl. Manag. 9(5), 85–96 (2005)
Wilkinson, D.M., Huberman, B.A.: Assessing the value of cooperation in Wikipedia. First Monday 12(4) (2007)
de Winter, J.C., Dodou, D.: Five-point Likert items: t test versus Mann-Whitney-Wilcoxon. Pract. Assess., Res. Eval. 15(11) (2010)
Xu, Z., Ni, Y., He, W., Lin, L., Yan, Q.: Automatic extraction of OWL ontologies from UML class diagrams: a semantics-preserving approach. World Wide Web 15(5–6), 517–545 (2011)
Zacharias, V., Braun, S.: Soboleo—social bookmarking and lighweight engineering of ontologies. In: CKC (2007)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kühne, C., Böhm, K. Assessing the suitability of an honest rating mechanism for the collaborative creation of structured knowledge. World Wide Web 17, 85–104 (2014). https://doi.org/10.1007/s11280-012-0193-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11280-012-0193-1