Abstract
Colorization is a process that converts a grayscale image into a colored one that looks as natural as possible. Over the years this task has received a lot of attention. Existing colorization methods rely on different color spaces: RGB, YUV, Lab, etc. In this chapter, we aim to study their influence on the results obtained by training a deep neural network, to answer the following question: “Is it crucial to correctly choose the right color space in deep learning-based colorization?” First, we briefly summarize the literature and, in particular, deep learning-based methods. We then compare the results obtained with the same deep neural network architecture with RGB, YUV, and Lab color spaces. Qualitative and quantitative analysis do not conclude similarly on which color space is better. We then show the importance of carefully designing the architecture and evaluation protocols depending on the types of images that are being processed and their specificities: strong/small contours, few/many objects, recent/archive images.
Similar content being viewed by others
References
Afifi, M., Brubaker, M.A., Brown, M.S.: HistoGAN: controlling colors of gan-generated and real images via color histograms. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7941–7950 (2021)
Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In: Conference on Computer Vision and Pattern Recognition Workshops, pp. 126–135 (2017)
Antic, J.: Deoldify (2019). https://github.com/jantic/DeOldify
Anwar, S., Tahir, M., Li, C., Mian, A., Khan, F.S., Muzaffar, A.W.: Image colorization: a survey and dataset (2020). arXiv preprint arXiv:2008.10774
Arbelot, B., Vergne, R., Hurtut, T., Thollot, J.: Automatic texture guided color transfer and colorization. In: Expressive, Elsevier, pp. 21–32 (2016)
Arbelot, B., Vergne, R., Hurtut, T., Thollot, J.: Local texture-based color transfer and colorization. Comput. Graph. 62, 15–27 (2017)
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International Conference on Machine Learning, vol. 70, pp. 214–223 (2017)
Bugeau, A., Ta, V.-T.: Patch-based image colorization. In: International Conference on Pattern Recognition, pp. 3058–3061 (2012)
Bugeau, A., Ta, V.-T., Papadakis, N.: Variational exemplar-based image colorization. IEEE Trans. Image Process. 23(1), 298–307 (2014)
Cao, Y., Zhou, Z., Zhang, W., Yu, Y.: Unsupervised diverse colorization via generative adversarial networks. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 151–166 (2017)
Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: European Conference on Computer Vision, pp. 126–139 (2008)
Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: IEEE International Conference on Computer Vision, pp. 415–423 (2015)
Chia, A.Y.-S., Zhuo, S., Gupta, R.K., Tai, Y.-W., Cho, S.-Y., Tan, P., Lin, S.: Semantic colorization with internet images. In: ACM SIGGRAPH ASIA (2011)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
Deshpande, A., Rock, J., Forsyth, D.: Learning large-scale automatic image colorization. In: IEEE International Conference on Computer Vision (2015)
Deshpande, A., Lu, J., Yeh, M.-C., Jin Chong, M., Forsyth, D.: Learning diverse image colorization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6837–6845 (2017)
Di Blasi, G., Reforgiato, D.: Fast colorization of gray images. In: Eurographics Italian, Eurographics Association (2003)
Ding, X., Xu, Y., Deng, L., Yang, X.: Colorization using quaternion algebra with automatic scribble generation. In: Advances in Multimedia Modeling, pp. 103–114 (2012)
Ding, K., Ma, K., Wang, S., Simoncelli, E.P.: Comparison of full-reference image quality models for optimization of image processing systems. Int. J. Comput. Vis. 129(4), 1258–1281 (2021)
Dowson, D., Landau, B.: The Fréchet distance between multivariate normal distributions. J. Multivar. Anal. 12(3), 450–455 (1982)
Drew, M.S., Finlayson, G.D.: Improvement of colorization realism via the structure tensor. Int. J. Image Graph. 11(4), 589–609 (2011)
Ebner, M.: Color Constancy, vol. 7. Wiley, Hoboken (2007)
Efros, A., Leung, T.: Texture synthesis by non-parametric sampling. In: IEEE International Conference on Computer Vision, pp. 1033–1038 (1999)
Fairchild, M.D. : Color Appearance Models. Wiley, Hoboken (2013)
Fang, F., Wang, T., Zeng, T., Zhang, G.: A superpixel-based variational model for image colorization. IEEE Trans. Vis. Comput. Graph. 26(10), 2931–2943 (2019)
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016a)
Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. J. Vis. 16(12), 326 (2016b)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (2014)
Guadarrama, S., Dahl, R., Bieber, D., Norouzi, M., Shlens, J., Murphy, K.: Pixcolor: pixel recursive colorization. In: British Machine Vision Conference (2017)
Gu, S., Timofte, R., Zhang, R.: Ntire 2019 challenge on image colorization: report. In: Conference on Computer Vision and Pattern Recognition Workshops (2019)
Gupta, R.K., Chia, A.Y.-S., Rajan, D., Ng, E.S., Zhiyong, H.: Image colorization using similar images. In: ACM International Conference on Multimedia, pp. 369–378 (2012)
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)
He, M., Chen, D., Liao, J., Sander, P.V., Yuan, L.: Deep exemplar-based colorization. ACM Trans. Graph. 37(4), 1–16 (2018)
Heu, J., Hyun, D.-Y., Kim, C.-S., Lee, S.-U.: Image and video colorization based on prioritized source propagation. In: IEEE International Conference on Image Processing, pp. 465–468 (2009)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Ho, J., Kalchbrenner, N., Weissenborn, D., Salimans, T.: Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180 (2019)
Huang, Y.-C., Tung, Y.-S., Chen, J.-C., Wang, S.-W., Wu, J.-L.: An adaptive edge detection based colorization algorithm and its applications. In: ACM International Conference on Multimedia, pp. 351–354 (2005)
Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst (2007)
Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. 35(4), 1–11 (2016)
Irony, R., Cohen-Or, D., Lischinski, D.: Colorization by example. In: Eurographics Conference on Rendering Techniques (2005)
Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711 (2016)
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 8110–8119 (2020)
Kawulok, M., Kawulok, J., Smolka, B.: Discriminative textural features for image and video colorization. IEICE Trans. Inf. Syst. 95-D(7), 1722–1730 (2012)
Kong, G., Tian, H., Duan, X., Long, H.: Adversarial edge-aware image colorization with semantic segmentation. IEEE Access 9, 28194–28203 (2021)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2009)
Kumar, M., Weissenborn, D., Kalchbrenner, N.: Colorization transformer. arXiv preprint arXiv:2102.04432 (2021)
Lagodzinski, P., Smolka, B.: Digital image colorization based on probabilistic distance transformation. In: 50th International Symposium ELMAR, vol. 2, pp. 495–498 (2008)
Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: European Conference on Computer Vision, pp. 577–593 (2016)
Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Trans. Graph. 23(3), 689–694 (2004)
Lézoray, O., Ta, V.-T., Elmoataz, A.: Nonlocal graph regularization for image colorization. In: International Conference on Pattern Recognition, pp. 1–4 (2008)
Li, B., Lai, Y.-K., Rosin, P.L.: Example-based image colorization via automatic feature selection and fusion. Neurocomputing 266, 687–698 (2017a)
Li, B., Zhao, F., Su, Z., Liang, X., Lai, Y.-K., Rosin, P.L.: Example-based image colorization using locality consistent sparse representation. IEEE Trans. Image Process. 26(11), 5188–5202 (2017b)
Li, B., Lai, Y.-K., John, M., Rosin, P.L.: Automatic example-based image colorization using location-aware cross-scale matching. IEEE Trans. Image Process. 28(9), 4606–4619 (2019)
Li, B., Lai, Y.-K., Rosin, P.L.: A review of image colourisation. In: Handbook of Pattern Recognition and Computer Vision, p. 139. World Scientific, Singapore (2020)
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: European Conference on Computer Vision, pp. 740–755 (2014)
Ling, Y., Au, O.C., Pang, J., Zeng, J., Yuan, Y., Zheng, A.: Image colorization via color propagation and rank minimization. In: IEEE International Conference on Image Processing, pp. 4228–4232 (2015)
Liu, S., Zhang, X.: Automatic grayscale image colorization using histogram regression. Pattern Recogn. Lett. 33(13), 1673–1681 (2012)
Luan, Q., Wen, F., Cohen-Or, D., Liang, L., Xu, Y.-Q., Shum, H.-Y.: Natural image colorization. In: Eurographics Conference on Rendering Techniques, pp. 309–320 (2007)
Luo, X., Zhang, X., Yoo, P., Martin-Brualla, R., Lawrence, J., Seitz, S.M.: Time-travel rephotography. arXiv preprint arXiv:2012.12261 (2020)
Mouzon, T., Pierre, F., Berger, M.-O.: Joint CNN and variational model for fully-automatic image colorization. In: Scale Space and Variational Methods in Computer Vision, pp. 535–546 (2019)
Nazeri, K., Ng, E., Ebrahimi, M.: Image colorization using generative adversarial networks. In: International Conference on Articulated Motion and Deformable Objects, pp. 85–94 (2018)
Oord, A.V.D., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., Kavukcuoglu, K.: Conditional image generation with PixelCNN decoders. In: Advances in Neural Information Processing Systems (2016)
Pang, J., Au, O.C., Tang, K., Guo, Y.: Image colorization using sparse representation. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 1578–1582 (2013)
Pierre, F., Aujol, J.-F.: Recent approaches for image colorization. In: Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging, springer (2020)
Pierre, F., Aujol, J.-F., Bugeau, A., Ta, V.-T.: A unified model for image colorization. In: European Conference on Computer Vision Workshops, pp. 297–308 (2014)
Pierre, F., Aujol, J.-F., Bugeau, A., Papadakis, N., Ta, V.-T.: Luminance-chrominance model for image colorization. SIAM J. Imaging Sci. 8(1), 536–563 (2015a)
Pierre, F., Aujol, J.-F., Bugeau, A., Ta, V.-T.: Luminance-Hue Specification in the RGB Space. In: Scale Space and Variational Methods in Computer Vision, pp. 413–424. Springer, Cham (2015b)
Pucci, R., Micheloni, C., Martinel, N.: Collaborative image and object level features for image colourisation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2160–2169 (2021)
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: International Conference on Learning Representations (2016)
Ren, X., Malik, J.: Learning a classification model for segmentation. In: IEEE International Conference on Computer Vision, pp. 10–17 (2003)
Riba, E., Mishkin, D., Ponsa, D., Rublee, E., Bradski, G.: Kornia: an open source differentiable computer vision library for PyTorch. In: Winter Conference on Applications of Computer Vision, pp. 3674–3683 (2020)
Royer, A., Kolesnikov, A., Lampert, C.H.: Probabilistic image colorization. In: British Machine Vision Conference (2017)
Salimans, T., Karpathy, A., Chen, X., Kingma, D.P.: PixelCNN++: improving the PixelCNN with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 (2017)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
Su, J.-W., Chu, H.-K., Huang, J.-B.: Instance-aware image colorization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7968–7977 (2020)
Tai, Y.-W., Jia, J., Tang, C.-K.: Local color transfer via probabilistic segmentation by expectation-maximization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 747–754 (2005)
Vitoria, P., Raad, L., Ballester, C.: ChromaGAN: adversarial picture colorization with semantic class distribution. In: Winter Conference on Applications of Computer Vision, pp. 2445–2454 (2020)
Wan, S., Xia, Y., Qi, L., Yang, Y.-H., Atiquzzaman, M.: Automated colorization of a grayscale image with seed points propagation. IEEE Trans. Multimedia 22(7), 1756–1768 (2020a)
Wan, Z., Zhang, B., Chen, D., Zhang, P., Chen, D., Liao, J., Wen, F.: Bringing old photos back to life. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2747–2757 (2020b)
Wang, S., Zhang, Z.: Colorization by matrix completion. In: AAAI Conference on Artificial Intelligence (2012)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Welsh, T., Ashikhmin, M., Mueller, K. Transferring color to greyscale images. ACM Trans. Graph. 21(3), 277–280 (2002)
Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: large-scale scene recognition from abbey to zoo. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3485–3492 (2010)
Yao, Q., James, T.K.: Colorization by patch-based local low-rank matrix completion. In: AAAI Conference on Artificial Intelligence (2015)
Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 15(5), 1120–1129 (2006)
Yoo, S., Bahng, H., Chung, S., Lee, J., Chang, J., Choo, J.: Coloring with limited data: few-shot colorization via memory augmented networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., Xiao, J.: LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 (2015)
Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: European Conference on Computer Vision, pp. 649–666 (2016)
Zhang, R., Zhu, J.-Y., Isola, P., Geng, X., Lin, A.S., Yu, T., Efros, A.A.: Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph. 36, 1–11 (2017)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1452–1464 (2017)
Acknowledgements
This study has been carried out with financial support from the French Research Agency through the PostProdLEAP project (ANR-19-CE23-0027-01) and from the EU Horizon 2020 research and innovation program NoMADS (Marie Skodowska-Curie grant agreement No 777826). This chapter was written together with another chapter of the current handbook, titled “Analysis of Different Losses for Deep Learning Image Colorization”.All authors have contributed to both chapters.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Section Editor information
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this entry
Cite this entry
Bugeau, A., Giraud, R., Raad, L. (2022). Influence of Color Spaces for Deep Learning Image Colorization. In: Chen, K., Schönlieb, CB., Tai, XC., Younces, L. (eds) Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging. Springer, Cham. https://doi.org/10.1007/978-3-030-03009-4_125-1
Download citation
DOI: https://doi.org/10.1007/978-3-030-03009-4_125-1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03009-4
Online ISBN: 978-3-030-03009-4
eBook Packages: Springer Reference MathematicsReference Module Computer Science and Engineering