Abstract
Learning based on facial features for detection and recognition of people’s identities, emotions and image aesthetics has been widely explored in computer vision and biometrics. However, automatic discovery of users’ preferences to certain of faces (i.e., style), to the best of our knowledge, has never been studied, due to the subjective, implicative, and uncertain characteristic of psychological preference. Therefore, in this paper, we contribute to an answer to whether users’ psychological preference can be modeled and computed after observing several faces. To this end, we first propose an efficient approach for discovering the personality preference related facial features from only a very few anchors selected by each user, and make accurate predictions and recommendations for users. Specifically, we propose to discover the style of faces (DiscoStyle) for human’s psychological preference inference towards personalized face recommendation system/application. There are four merits of our DiscoStyle: 1) Transfer learning is exploited from identity related facial feature representation to personality preference related facial feature. 2) Appearance and geometric landmark feature are exploited for preference related feature augmentation. 3) A multi-level logistic ranking model with on-line negative sample selection is proposed for online modeling and score prediction, which reflects the users’ preference degree to gallery faces. 4) A large dataset with different facial styles for human’s psychological preference inference is developed for the first time. Experiments show that our proposed DiscoStyle can well achieve users’ preference reasoning and recommendation of preferred facial styles in different genders and races.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
F. Schroff, D. Kalenichenko, J. Philbin. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Boston, USA, pp. 815–823, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7298682.
Y. Taigman, M. Yang, M. A. Ranzato, L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 1701–1708, 2014. DOI: https://doi.org/10.1109/CVPR.2014.220.
Y. Sun, X. G. Wang, X. O. Tang. Deep learning face representation from predicting 10, 000 classes. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 1891–1898, 2014. DOI: https://doi.org/10.1109/CVPR.2014.244.
J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, Y. Ma. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 2, pp. 210–227, 2009. DOI: https://doi.org/10.1109/TPAMI.2008.79.
J. K. Chen, Z. H. Chen, Z. R. Chi, H. Fu. Facial expression recognition in video with multiple feature fusion. IEEE Transactions on Affective Computing, vol. 9, no. 1, pp. 38–50, 2018. DOI: https://doi.org/10.1109/TAFFC.2016.2593719.
L. Zhang, D. Zhang, M. M. Sun, F. M. Chen. Facial beauty analysis based on geometric feature: Toward attractiveness assessment application. Expert Systems with Applications, vol. 82, pp. 252–265, 2017. DOI: https://doi.org/10.1016/j.eswa.2017.04.021.
Y. Fu, G. D. Guo, T. S. Huang. Age synthesis and estimation via faces: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 11, pp. 1955–1976, 2010. DOI: https://doi.org/10.1109/TPAMI.2010.36.
E. Eidinger, R. Enbar, T. Hassner. Age and gender estimation of unfiltered faces. IEEE Transactions on Information Forensics and Security, vol. 9, no. 12, pp. 2170–2179, 2014. DOI: https://doi.org/10.1109/TIFS.2014.2359646.
Z. Lian, Y. Li, J. H. Tao, J. Huang, M. Y. Niu. Expression analysis based on face regions in real-world conditions. International Journal of Automation and Computing, vol. 17, no. 1, pp. 96–107, 2020. DOI: https://doi.org/10.1007/s11633-019-1176-9.
H. S. Du, Q. P. Hu, D. F. Qiao, I. Pitas. Robust face recognition via low-rank sparse representation-based classification. International Journal of Automation and Computing, vol. 12, no. 6, pp. 579–587, 2015. DOI: https://doi.org/10.1007/s11633-015-0901-2.
H. Wu, Z. W. Chen, G. H. Tian, Q. Ma, M. L. Jiao. Item ownership relationship semantic learning strategy for personalized service robot. International Journal of Automation and Computing, vol. 17, no. 3, pp. 390–402, 2020. DOI: https://doi.org/10.1007/s11633-019-1206-7.
D. Zhang, Q. J. Zhao, F. M. Chen. Quantitative analysis of human facial beauty using geometric features. Pattern Recognition, vol. 44, no. 4, pp. 940–950, 2011. DOI: https://doi.org/10.1016/j.patcog.2010.10.013.
F. M. Chen, X. H. Xiao, D. Zhang. Data-driven facial beauty analysis: Prediction, retrieval and manipulation. IEEE Transactions on Affective Computing, vol. 9, no. 2, pp. 205–216, 2018. DOI: https://doi.org/10.1109/TAFFC.2016.2599534.
A. Krizhevsky, I. Sutskever, G. E. Hinton. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, USA, PP.1097–1105, 2012.
K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, USA, 2015. https://arxiv.org/abs/1409.1556.
W. Y. Liu, Y. D. Wen, Z. D. Yu, M. Yang. Large-margin softmax loss for convolutional neural networks. In Proceedings of the 33rd International Conference on Machine Learning, New York, USA, 2016.
S. J. Pan, Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–1359, 2010. DOI: https://doi.org/10.1109/TKDE.2009.191.
K. Saenko, B. Kulis, M. Fritz, T. Darrell. Adapting visual category models to new domains. In Proceedings of the 11th European Conference on Computer Vision, Springer, Heraklion, Greece, 2010. DOI: https://doi.org/10.1007/978-3-642-15561-1_16.
M. S. Long, H. Zhu, J. M. Wang, M. I. Jordan. Unsupervised domain adaptation with residual transfer networks. In Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 136–144, 2016.
L. Zhang, W. M. Zuo, D. Zhang. LSDT: Latent sparse domain transfer learning for visual adaptation. IEEE Transactions on Image Processing, vol. 25, no. 3, pp. 1177–1191, 2016. DOI: https://doi.org/10.1109/TIP.2016.2516952.
L. Zhang, S. S. Wang, G. B. Huang, W. M. Zuo, J. Yang, D. Zhang. Manifold criterion guided transfer learning via intermediate domain generation. IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 12, pp. 3759–3773, 2019. DOI: https://doi.org/10.1109/TNNLS.2019.2899037.
N. Tajbakhsh, J. Y. Shin, S. R. Gurudu, R. T. Hurst, C. B. Kendall, M. B. Gotway, J. M. Liang. Convolutional neural networks for medical image analysis: Full training or fine unning? IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1299–1312, 2016. DOI: https://doi.org/10.1109/TMI.2016.2535302.
H. C. Shin, H. R. Roth, M. C. Gao, L. Lu, Z. Y. Xu, I. Nogues, J. H. Yao, D. Mollura, R. M. Summers. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1285–1298, 2016. DOI: https://doi.org/10.1109/TMI.2016.2528162.
D. Marmanis, M. Datcu, T. Esch, U. Stilla. Deep learning earth observation classification using imagenet pretrained networks. IEEE Geoscience and Remote Sensing Letters, vol. 13, no. 1, pp. 105–109, 2016. DOI: https://doi.org/10.1109/LGRS.2015.2499239.
X. W. Yao, J. W. Han, G. Cheng, X. M. Qian, L. Guo. Semantic annotation of high-resolution satellite images via weakly supervised learning. IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 6, pp. 3660–3671, 2016. DOI: https://doi.org/10.1109/TGRS.2016.2523563.
M. Xie, N. Jean, M. Burke, D. Lobell, S. Ermon. Transfer learning from deep features for remote sensing and poverty mapping. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, AAAI, Phoenix, USA, 2015.
N. Jean, M. Burke, M. Xie, W. M. Davis, D. B. Lobell, S. Ermon. Combining satellite imagery and machine learning to predict poverty. Science, vol. 353, no. 6301, pp. 790–794, 2016. DOI: https://doi.org/10.1126/science.aaf7894.
Q. Y. Duan, L. Zhang, W. M. Zuo. From face recognition to kinship verification: An adaptation approach. In Proceedings of IEEE International Conference on Computer Vision Workshops, IEEE, Venice, Italy, pp. 1590–1598, 2017. DOI: https://doi.org/10.1109/ICCVW.2017.187.
L. Zhang, Q. Y. Duan, D. Zhang, W. Jia, X. Z. Wang. Advkin: Adversarial convolutional network for kinship verification. IEEE Transactions on Cybernetics, published online, 2020. DOI: https://doi.org/10.1109/TCYB.2019.2959403.
C. Q. Hong, J. Yu, J. Zhang, X. N. Jin, K. H. Lee. Multimodal face-pose estimation with multitask manifold deep learning. IEEE Transactions on Industrial Informatics, vol. 15, no. 7, pp. 3952–3961, 2019. DOI: https://doi.org/10.1109/TII.2018.2884211.
Q. C. Zhu, Z. H. Chen, Y. C. Soh. A novel semisupervised deep learning method for human activity recognition. IEEE Transactions on Industrial Informatics, vol. 15, no. 7, pp. 3821–3830, 2019. DOI: https://doi.org/10.1109/TII.2018.2889315.
Y. D. Yang, W. Li, T. A. Gulliver, S. F. Li. Bayesian deep learning-based probabilistic load forecasting in smart grids. IEEE Transactions on Industrial Informatics, vol. 16, no. 7, pp. 4703–4713, 2020. DOI: https://doi.org/10.1109/TII.2019.2942353.
L. Zhang, D. Zhang. Efficient solutions for discreteness, drift, and disturbance (3D) in electronic olfaction. IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 48, no. 2, pp. 242–254, 2018. DOI: https://doi.org/10.1109/TSMC.2016.2597800.
L. Zhang, P. L. Deng. Abnormal odor detection in electronic nose via self-expression inspired extreme learning machine. IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 49, no. 10, pp. 1922–1932, 2019. DOI: https://doi.org/10.1109/TSMC.2017.2691909.
T. Serre, G. Kreiman, M. Kouh, C. Cadieu, U. Knoblich, T. Poggio. A quantitative theory of immediate visual recognition. Progress in Brain Research, vol. 165, pp. 33–56, 2007. DOI: https://doi.org/10.1016/S0079-6123(06)65004-8.
D. Cheng, Y. H. Gong, S. P. Zhou, J. J. Wang, N. N. Zheng. Person re-identification by multi-channel parts-based CNN with improved triplet loss function. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 1335–1344, 2016. DOI: https://doi.org/10.1109/CVPR.2016.149.
E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, B. Schiele. DepprrCut: A depper, stronger, and faster multi-person pose estimation model. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, 2016. DOI: https://doi.org/10.1007/978-3-319-46466-43.
Y. Li, H. Z. Qi, J. F. Dai, X. Y. Ji, Y. C. Wei. Fully convolutional instance-aware semantic segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp. 4438–4446, 2017. DOI: https://doi.org/10.1109/CVPR.2017.472.
C. Dong, C. C. Loy, K. M. He, X. O. Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016. DOI: https://doi.org/10.1109/TPAMI.2015.2439281
K. M. He, X. Y. Zhang, S. Q. Ren, J. Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp.770–778, 2016. DOI: https://doi.org/10.1109/CVPR.2016.90.
G. Huang, Z. Liu, L. van der Maaten, K. Q. Weinberger. Densely connected convolutional networks, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp.2261–2269, 2017. DOI: https://doi.org/10.1109/CVPR.2017.243.
S. Q. Ren, K. M. He, R Girshick, J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of Advances in Neural Information Processing Systems 28, Montreal, Canada, 2015.
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, A. C. Berg. SSD: Single shot multibox detector. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherland, pp. 21–37, 2016. DOI: https://doi.org/10.1007/978-3-319-46448-0_2.
J. Redmon, A. Farhadi. Yolo9000: Better, faster, stronger. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp. 6517–6525, 2017. DOI: https://doi.org/10.1109/CVPR.2017.690.
Z. Cao, T. Simon, S. E. Wei, Y. Sheikh. Realtime multi-person 2D pose estimation using part affinity fields. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp.1302–1310, 2017. DOI: https://doi.org/10.1109/CVPR.2017.143.
X. L. Wang, T. T. Xiao, Y. N. Jiang, S. Shao, J. Sun, C. H. Shen. Repulsion loss: Detecting pedestrians in a crowd. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp.7774–7783, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00811.
Z. X. Feng, J. H. Lai, X. H. Xie. Learning view-specific deep networks for person re-identification. IEEE Transactions on Image Processing, vol. 27, no. 7, pp. 3472–3483, 2018. DOI: https://doi.org/10.1109/TIP.2018.2818438.
L. Q. Liu, C. Xiong, H. W. Zhang, Z. H. Niu, M. Wang, S. C. Yan. Deep aging face verification with large gaps. IEEE Transactions on Multimedia, vol. 18, no. 1, pp. 64–75, 2016. DOI: https://doi.org/10.1109/TMM.2015.2500730.
Z. F. Li, D. H. Gong, X. L. Li, D. C. Tao. Aging face recognition: A hierarchical learning model based on local patterns selection. IEEE Transactions on Image Processing, vol. 25, no. 5, pp. 2146–2154, 2016. DOI: https://doi.org/10.1109/TIP.2016.2535284.
U. Park, Y. Y. Tong, A. K. Jain. Age-invariant face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 5, pp. 947–954, 2010. DOI: https://doi.org/10.1109/TPAMI.2010.14.
H. Dibeklioglu, A. A. Salah, T. Gevers. Like father, like son: Facial expression dynamics for kinship verification. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Sydney, Australia, pp.1497–1504, 2013. DOI: https://doi.org/10.1109/ICCV.2013.189.
R. G. Fang, K. D. Tang, N. Snavely, T. Chen. Towards computational models of kinship verification. In Proceedings of IEEE International Conference on Image Processing, IEEE, Hong Kong, China, pp. 1577–1580, 2010. DOI: https://doi.org/10.1109/ICIP.2010.5652590.
H. B. Yan, J. W. Lu, X. Z. Zhou. Prototype-based discriminative feature learning for kinship verification. IEEE Transactions on Cybernetics, vol. 45, no. 11, pp. 2535–2545, 2015. DOI: https://doi.org/10.1109/TCYB.2014.2376934.
D. I. Perrett, K. A. May, S. Yoshikawa. Facial shape and judgements of female attractiveness. Nature, vol. 368, no. 6468, pp. 239–242, 1994. DOI: https://doi.org/10.1038/368239a0.
K. P. Zhang, Z. P. Zhang, Z. F. Li, Y. Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, vol. 23, no. 10, pp. 1499–1503, 2016. DOI: https://doi.org/10.1109/LSP.2016.2603342.
D. Yi, Z. Lei, S. C. Liao, S. Z. Li. Learning face representation from scratch. https://arxiv.org/abs/1411.7923, 2014.
V. Kazemi, J. Sullivan. One millisecond face alignment with an ensemble of regression trees. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 1867–1874, 2014. DOI: https://doi.org/10.1109/CVPR.2014.241.
G. B. Huang, M. Ramesh, T. Berg, E. Learned-Miller. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments, Technical Report, 07–49, Department of Computer Science, University of Massachusetts, USA, 2007.
N. Dalal, B. Triggs. Histograms of oriented gradients for human detection. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, San Diego, USA, pp. 886–893, 2005. DOI: https://doi.org/10.1109/CVPR.2005.177.
B. Schölkopf, R Williamson, A. Smola, J. Shawe-Taylor, J. Platt. Support vector method for novelty detection. In Proceedings of the 12th International Conference on Neural Information Processing Systems, Denver, USA, pp. 582–588, 1999.
C. C. Chang, C. J. Lin. Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, vol. 2, no. 3, Article number 27, 2011. DOI: https://doi.org/10.1145/1961189.1961199.
G. B. Huang, H. M. Zhou, X. J. Ding, R. Zhang. Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics — Part B: Cybernetics, vol. 42, no. 2, pp. 513–529, 2012. DOI: https://doi.org/10.1109/TSMCB.2011.2168604.
L. van der Maaten, G. Hinton. Visualizing data using T-SNE. Journal of Machine Learning Research, vol. 9, pp. 2579–2605, 2008.
Acknowledgments
This work was supported by National Natural Science Fund of China (No. 61771079), Chongqing Natural Science Fund (No. cstc2018jcyjAX0250) and Chongqing Youth Talent Program. The authors would like to thank the volunteers for their contribution in labeling the StyleFace for preferences modeling.
Author information
Authors and Affiliations
Corresponding author
Additional information
Recommended by Associate Editor Bin Luo
Zhen-Wei He received B. Eng. degree in information engineering from Tianjin University, China in 2014. From July 2014 to June 2016, he worked in Chongqing Cable Network Inc., China. Now, he is a Ph. D degree candidate in Chongqing University, China.
His research interests include deep learning and computer vision.
Lei Zhang received the Ph. D degree in circuits and systems from the College of Communication Engineering, Chongqing University, China in 2013. He worked as a Post-Doctoral Fellow with Hong Kong Polytechnic University, China from 2013 to 2015. He is currently a professor/distinguished research fellow with Chongqing University, China. He has authored more than 90 scientific papers in top journals and top conferences. He serves as associate editors for IEEE Transactions on Instrumentation and Measurement, Neural Networks, etc. He is a senior member of IEEE.
His research interests include machine learning, pattern recognition, computer vision and intelligent systems.
Fang-Yi Liu received the B. Eng. degree in communication engineering from Guangxi University, China in 2017. Since September 2017, he is a master student in information and communication engineering in Chongqing University, China.
His research interests include person re-identification and deep learning.
Rights and permissions
About this article
Cite this article
He, ZW., Zhang, L. & Liu, FY. DiscoStyle: Multi-level Logistic Ranking for Personalized Image Style Preference Inference. Int. J. Autom. Comput. 17, 637–651 (2020). https://doi.org/10.1007/s11633-020-1244-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11633-020-1244-1