Abstract
Crowdsourcing has been an emerging machine learning paradigm. It collects labels from human crowds as inputs typically through the Internet. Due to limitations on knowledge, social-economic status, and other factors, participants may often have ambiguity in labeling some instances in practice. In this work, we propose interval-valued labels (IVLs), instead of commonly used binary-valued ones, to manage such kind of uncertainty in crowdsourcing. IVLs possess interval specific statistic and probabilistic properties. With them, this work presents an algorithm that is able to make an inference with a favorable matching probability as a main result. The algorithm also implies an index, which measures the overall uncertainty of collected IVLs quantitatively. Reported computational experiments further evidence that we may better manage uncertainty in crowdsourcing with IVLs than without.
This work is partially supported by the US National Science Foundation through the grant award NSF/OIA-1946391.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
Let interval
be an estimation of another interval 
, then the accuracy ratio of the estimation is defined as 
, where w( ) returns the width of an interval, and 
returns the convex hull of 
.
be an estimation of another interval 
, then the accuracy ratio of the estimation is defined as 
, where w( ) returns the width of an interval, and 
returns the convex hull of 
.References
Barbosa, N., Chen, M.: Rehumanized crowdsourcing: a labeling framework addressing bias and ethics in machine learning. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–12 (2019). https://doi.org/10.1145/3290605.3300773
Bi, W., Wang, L., Kwok, J., Tu, Z.: Learning to predict from crowdsourced data. In: Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI 2014, pp. 82–91 (2014)
Dai, J., Wang, W., Mi, J.: Uncertainty Measurement for Interval-valued Information Systems, Information Sciences. Elsevier (2013)
He, L., Hu, C.: Midpoint method and accuracy of variability forecasting. J. Empirical Econ. 38, 705–715 (2009). https://doi.org/10.1007/s00181-009-0286-6
He, L., Hu, C.: Impacts of interval computing on stock market forecasting. J. Comput. Econ. 33(3), 263–276 (2009). https://doi.org/10.1007/s10614-008-9159-x
Hu, C., et al.: Knowledge Processing with Interval and Soft Computing. Springer, London (2008). https://doi.org/10.1007/978-1-84800-326-2
Hu, C., He, L.: An application of interval methods to stock market forecasting. J. Reliab. Comput. 13, 423–434 (2007). https://doi.org/10.1007/s11155-007-9039-4
Hu, C.: Interval function and its linear least-squares approximation. In: Proceedings of the 2011 International Workshop on Symbolic-Numeric Computation, SNC 2011, pp. 16–23. ACM (2012). https://doi.org/10.1145/2331684.2331689
Hu, C., Hu, Z.H.: On statistics, probability, and entropy of interval-valued datasets. In: Lesot, M.J., et al. (eds.) Information Processing and Management of Uncertainty in Knowledge-Based Systems, IPMU 2020. Communications in Computer and Information Science, vol. 1239, pp. 422–435. Springer, Cham. (2020). https://doi.org/10.1007/978-3-030-50153-2_31
Hu, C., Hu, Z.H.: A computational study on the entropy of interval-valued datasets from the stock market. In: Lesot, M.J., et al. (eds.) Information Processing and Management of Uncertainty in Knowledge-Based Systems, IPMU 2020. Communications in Computer and Information Science, vol. 1239, pp. 407–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50153-2_32
Huynh, V., et al.: On decision making under interval uncertainty: a new justification of Hurwicz optimism-pessimism approach and its use in group decision making. In: The 39th International Symposium on Multiple-Valued Logic, pp. 214–220 (2009)
Korvin, A., Hu, C., Chen, P.: Generating and applying rules for interval valued fuzzy observations. Lecture Notes in Computer Science, vol. 3177, pp. 279–284, Springer, Heidelberg (2004)
Marupally, P., Paruchuri, V., Hu, C.: Bandwidth variability prediction with rolling interval least squares (RILS). In: Proceedings of the 50th ACM SE Conference, Tuscaloosa, AL, USA, 29–31 March 2012, pp. 209–213. ACM (2012). https://doi.org/10.1145/2184512.2184562
Nordin, B., Hu, C., Chen, B., Sheng, V.S.: Interval-valued centroids in K-means algorithms. In: Proceedings of the 11th IEEE International Conference on Machine Learning and Applications (ICMLA), Boca Raton, FL, USA, pp. 478–481. IEEE (2012). https://doi.org/10.1109/ICMLA.2012.87
Parer, J., Hamilton, E.: Comparison of 5 experts and computer analysis in rule-based fetal heart rate interpretation. Am J. Obstetrics Gynecol. 203(5), 451.E1–451.E7 (2010). https://doi.org/10.1016/j.ajog.2010.05.037
Qiu, L., et al.: CrowdSelect: increasing accuracy of crowdsourcing tasks through behavior prediction and user selection. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 539–548 (2016). https://doi.org/10.1145/2983323.2983830
Rhodes, C., Lemon, J., Hu, C.: An interval-radial algorithm for hierarchical clustering analysis. In: 14th IEEE International Conference on Machine Learning and Applications (ICMLA), Miami, FL, USA, pp. 849–856. IEEE (2015). https://doi.org/10.1109/ICMLA.2015.118
Sheng, V.S., Provost, F., Ipeirotis, P.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, 24–27 August, pp. 614–622 (2008). https://doi.org/10.1145/1401890.1401965
Sheng, V.S., Zhang, J.: Machine learning with crowdsourcing: a brief summary of the past research and future directions. In: Proceedings of the 33rd Conference on Artificial Intelligence, AAAI 2019, pp. 9837–9843 (2019). https://doi.org/10.1609/aaai.v33i01.33019837
Sheng, V.S., Zhang, J., Bin, G., Wu, X.: Majority voting and pairing with multiple noisy labeling. IEEE Trans. Knowl. Data Eng. 31(7), 1355–1368 (2019)
Smyth, P.: Learning with probabilistic supervision. In: Petsche, T. (ed.) Computational Learning Theory and Natural Learning Systems, vol. III: Selecting Good Models. MIT Press (1995)
Wang, G., Wang, T., Zheng, H., Zhao, B.: Man vs. machine: practical adversarial detection of malicious crowdsourcing workers. In: Proceedings of the 23rd USENIX Security Symposium, San Diego, CA, pp. 239–254. USENIX Association (2014)
Zhang, J., Wu, X., Sheng, V.S.: Learning from crowdsourced labeled data: a survey. Artif. Intell. Rev. 46, 543–576 (2016). https://doi.org/10.1007/s10462-016-9491-9
Zhang, X., Pan, X., Wang, S.: Label quality improvement in crowdsourcing with ensemble TSK fuzzy classifier. In: 2019 IEEE 14th International Conference on Intelligent Systems and Knowledge Engineering (ISKE), Dalian, China, pp. 290–296 (2019). https://doi.org/10.1109/ISKE47853.2019.9170348
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hu, C., Sheng, V.S., Wu, N., Wu, X. (2022). Managing Uncertainty in Crowdsourcing with Interval-Valued Labels. In: Rayz, J., Raskin, V., Dick, S., Kreinovich, V. (eds) Explainable AI and Other Applications of Fuzzy Techniques. NAFIPS 2021. Lecture Notes in Networks and Systems, vol 258. Springer, Cham. https://doi.org/10.1007/978-3-030-82099-2_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-82099-2_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-82098-5
Online ISBN: 978-3-030-82099-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)