Abstract
Material perception is a current hot topic. Recently a basic research on SHITSUKAN (material perception) has advanced under MEXT (Ministry of Education, Culture, Sports, Science and Technology) in Japan. It is expected to bring innovation for not only traditional craft, ceramic or plastic arts but also more realistic picture displays on 4K/8K HD TVs and VR/CG world. The material perception is said as a phenomenon that our brain feels from retinal images. Now, the analysis is progressing what features of optical images are more strongly related to the stimulus inside the visual cortex of V1–V5.
BRDF model describes the Specular and Diffusion components of optical surface reflection which carry “gloss” and “texture” appearance and used to adjust or modify material appearances.
Different from BRDF or other models, this paper tries to transfer a material color appearance from one to another images. First, the retinal image is converted to visual cortex image based on LPT (Log-Polar Transform). Since LPT samples the retinal image higher rate at the fovea but lower rate at the peripherals, the color information gathers to central areas in the visual cortex. After the LPT, PCM (Principal Component Matching) is applied to make the color matching between source and target images. Using the joint LPT-PCM model, a material color appearance of target image is transferred to source image without any priori informations on the target.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Background
Human observers can recognize material property at a glance through our sensory organ. Without touching materials, we can tell whether they would feel hard or soft, rough or smooth, wet or dry.
The material perception is said as a perceptual phenomenon of feeling or sensation that our brain perceives from optical image projected onto retina. Though, it’s hard to untangle what information of the retinal image stimulates the visual cortex and how it induces the material feeling in our brain. The mechanism of INNER VSION in brain is still a black box at present [1].
As a framework for material perception, Tsumura initiated the skin color appearance and proposed the concept of appearance delivering system [2].
In Brain Information Science research on SHITSUKAN by MEXT in Japan, the first stage (2010–2014, led by Dr. H. Komatsu) has just finished and the second stage (2015–2019 led by Dr. S. Nishida) stepped forward into “multi-dimensional” material perception and now is approaching to the final goal.
In spite of the complexity in material appearance mechanism, human sensations such as “gloss/mat”, “transparent/translucent”, “metal/cloth” are controllable by an intuitive but a smart technique.
For instance, Motoyoshi and Nishida et al. [3] noticed the “gloss” perception appears when the luminance histogram is skewed. If it’s stretched smoothly to the higher luminance, the object looks “glossy” but looks “mat”, if compressed to the lower.
Sawayama and Nishida [4] developed “wet” filter by a combination of exponent-shaped TRC and boosted color saturation. It’s very interesting any “skew” in the image features induces a sensational material perception. The finding of “skew” effect seems heuristic and intuitive. However, the mechanism why and how such sensations as “gloss” or “wet” are activated by the “skew” effect in INNER VISION is not still untangled yet.
On the other hand, many R&D for practical applications are making steady progresses in private enterprises. As a typical successful example, a specular reflection control algorithm based on BRDF (Bidirectional Reflectance Distribution Function) is implemented in LSI chip and mounted on next generation 4K HD TV “REGZA” [5].
2 Color Transfer Model Between Images
Since the material perceptions such as gloss or clarity are related to a variety of factors [6], it’s hard to specify the cause of perceptual feeling to a single factor. Nevertheless, trials on material or textual appearances transfer between CG images [7] or 3D objects [8] are reported. Especially, color appearance plays an important role in the material perception. The color transfer model [9] tried to change the color atmosphere of source scene A into that of target scene B, where the clustered color distribution of A is roughly matched with that of B. There, the use of vision-based lαβ color space [10] attracted interest.
2.1 lαβ Color Transfer Model
The lαβ is known as an orthogonal luminance-chrominance color space simply transformed from RGB by the following Step1 and Step2 and the color distribution of source image is changed to match with that of target (reference) image by the scaling process in Step3 and the color atmosphere of target is transferred to the source via the inverse transform in Step4 as follows
-
Step1: RGB to LMS cone response transform
$$ \left[ {\begin{array}{*{20}l} L \hfill \\ M \hfill \\ S \hfill \\ \end{array} } \right] = \left[ {\begin{array}{*{20}l} { 0. 3 8 1} \hfill & { 0. 5 7 8} \hfill & { 0. 0 4 0} \hfill \\ { 0. 1 9 7} \hfill & { 0. 7 2 4} \hfill & { 0. 0 7 8} \hfill \\ { 0. 0 2 4} \hfill & { 0. 1 2 9} \hfill & { 0. 8 4 4} \hfill \\ \end{array} } \right]\left[ {\begin{array}{*{20}l} R \hfill \\ G \hfill \\ B \hfill \\ \end{array} } \right] $$(1) -
Step2: LMS to lαβ transform with orthogonal luminance l and chrominance αβ
$$ \left[ {\begin{array}{*{20}l} l \hfill \\ \alpha \hfill \\ \beta \hfill \\ \end{array} } \right] = \left[ {\begin{array}{*{20}l} {{1 \mathord{\left/ {\vphantom {1 {\sqrt 3 }}} \right. \kern-0pt} {\sqrt 3 }}} \hfill & 0 \hfill & 0 \hfill \\ 0 \hfill & {{1 \mathord{\left/ {\vphantom {1 {\sqrt 6 }}} \right. \kern-0pt} {\sqrt 6 }}} \hfill & 0 \hfill \\ 0 \hfill & 0 \hfill & {{1 \mathord{\left/ {\vphantom {1 {\sqrt 2 }}} \right. \kern-0pt} {\sqrt 2 }}} \hfill \\ \end{array} } \right]\left[ {\begin{array}{*{20}l} 1 \hfill & 1 \hfill & 1 \hfill \\ 1 \hfill & 1 \hfill & { - 2} \hfill \\ 1 \hfill & { - 1} \hfill & 0 \hfill \\ \end{array} } \right]\left[ {\begin{array}{*{20}l} {log\,L} \hfill \\ {log\,M} \hfill \\ {log\,S} \hfill \\ \end{array} } \right] $$(2) -
Step3: Scaling of lαβ around the mean values {\( \bar{l}\,\bar{\alpha }\,\bar{\beta } \)} by the ratio of standarddeviation to make match the color distributions between source and target images.
$$ \begin{aligned} & l^{{\prime }} = ({{\sigma_{DST}^{l} } \mathord{\left/ {\vphantom {{\sigma_{DST}^{l} } {\sigma_{ORG}^{l} }}} \right. \kern-0pt} {\sigma_{ORG}^{l} }} )\left( {l - \bar{l}} \right) \, \\ & \alpha^{{\prime }} = ({{\sigma_{DST}^{\alpha } } \mathord{\left/ {\vphantom {{\sigma_{DST}^{\alpha } } {\sigma_{ORG}^{\alpha } }}} \right. \kern-0pt} {\sigma_{ORG}^{\alpha } }} )\left( {\alpha - \bar{\alpha }} \right) \\ & \beta^{{\prime }} = ({{\sigma_{DST}^{\beta } } \mathord{\left/ {\vphantom {{\sigma_{DST}^{\beta } } {\sigma_{ORG}^{\beta } }}} \right. \kern-0pt} {\sigma_{ORG}^{\beta } }} )\left( {\beta - \bar{\beta }} \right) \\ \end{aligned} $$(3)Where, \( \sigma_{ORG}^{l} \) and \( \sigma_{DST}^{\alpha } \) denote the standard deviation of luminance l for the source image and that of chrominance α for the target image, and so on.
-
Step4: Inverse transform \( \left[ {l^{{\prime }} \alpha^{{\prime }} \beta^{{\prime }} } \right] \Rightarrow \left[ {L^{{\prime }} M^{{\prime }} S^{{\prime }} } \right] \Rightarrow \left[ {R^{{\prime }} G^{{\prime }} B^{{\prime }} } \right] \).
Finally, the scaled \( l^{{\prime }} \alpha^{{\prime }} \beta^{{\prime }} \) source image with the color distribution matched to the target image is displayed on sRGB monitor.
2.2 PCM Color Transfer Model
Prior to lαβ model, the author et al. developed PCM (Principal Component Matchng) method [11, 12] for transferring the color atmosphere from one scene to another as illustrated in Fig. 1. The lαβ model works well between the scenes with color similarity but not for the scenes with color dissimilarity and often fails. While, PCM model works almost stable between the scenes with color dissimilarities and advanced toward automatic scene color interchange [13,14,15].
In our basic object-to-object PCM model a vector X in a color cluster is projected onto a vector Y in PC space by Hotelling Transform as
Where, \( \varvec{\mu} \) denotes the mean vector and the matrix A is formed by the set of eigen vectors {\( \varvec{e}_{1} \,\varvec{e}_{2} \,\varvec{e}_{3} \)} of covariance matrix ΣX as
The covariance matrix ΣY of {Y} is diagonalized in terms of A and ΣX with the elements composed of the eigen values {\( \lambda_{1} \,\lambda_{2} \,\lambda_{3} \)} of ΣX as
Thus the color vectors in source and target images are mapped to the same PC space and the following equations are formed to make match a source vector YORG to a target vector YDST through the scaling matrix S as follows.
Solving (7) and (8), we get the following relation between a source color XORG and a target color XDST to be transferred and matched.
The matching matrix MPCM is given by
Where, AORG and ADST denote the eigen matrices for the source color cluster and the target color cluster. In the scaling matrix S, λ1ORG means the 1st eigenvalue of the source and λ2DST the 2nd eigenvalue of the target, etc. These are obtained from each covariance matrix.
In general, the PCM model works better than lαβ even for the scenes with color dissimilarities, because of using the statistical characteristics of covariance matrix.
Figure 2 shows a successful example in both lαβ and PCM models for the images with color similarity. While, in case of Fig. 3, lα failes to change the color atmosphere of A into that of B due to their color dissimilarities, but works well in PCM.
3 Color Transfer by Spectral Decomposition of Covariance
Following the lαβ model, a variety of improved or alternative color transfer models have been reported. As a basic drawback in lαβ model, Pitié et al. [16] pointed out that it’s not based on the statistical covariance but only on the mean values and variances in the major lαβ axes. Hence PCM model is better than lαβ because of using the statistical covariance matrix ΣX with the Hotelling transform onto the PC space. At the same time, Pitie suggested to make use of orthogonal spectral decomposition paying the attention to the Hermitian (Self adjoint) property of symmetric matrix ΣX with real eigenvalues.
3.1 Eigen Value Decomposition (EVD) of Covariance
In general, the covariance matrix Σ in a clustered color distribution of image is a real symmetric matrix. The square root of Σ for source and target images is decomposed by eigenvalues as
AORG and ADST denote the eigen matrices for source and target images. DORG and DDST are given by the diagonal matrices with the entries of their eigen values respectively.
Now, the color matching matrix MEigen corresponding to Eq. (11) is given by
3.2 Singular Value Decomposition (SVD)
A m × n Matrix Σ is decomposed by SVD as the product of matrices U, V, and W
Where, U and V are m × m and n × n orthogonal matrices. If Σ is a m × n rectangular matrix of rank-r, matrix W is composed of r × r diagonal matrix with the singular values as its entries and the remaining small null matrices.
Because the covariance Σ is a 3 × 3 real symmetric matrix, the singular values equal to the eigenvalues and SVD equals EVD in Eq. (12).
3.3 Cholesky Decomposition
Cholesky, a compact spectral decomposition method, decomposes the covariance Σ as a simple product of lower triangular matrix and its transpose as follows.
Where, Chol[*] denotes the Cholesky decomposition. The lower triangular matrix L is obtained by the iteration just like as Gaussian elimination method (details omitted).
The color matching matrix MChol to transfer the color atmosphere of target image into the source is given by
4 Color Transfer by PCM After Mapping to Visual Cortex
4.1 Retina to Visual Cortex Mapping by Log Polar Transform
The PCM model works well to transfer the color atmosphere between the images even with color dissimilarities. However, any human visual characteristic has not been taken into account. In this paper, a striking feature in the spatial color distributions in our visual cortex image is introduced to improve the performance in PCM.
The mapping to visual cortex from retina is mathematically described by Schwartz’s complex Logarithmic Polar Transform (LPT) [17].
The complex vector z pointing a pixel located at (x, y) in the retina is transformed to a new vector log (z) by LPT as follows.
The retinal image is sampled at spatially-variant resolution on the polar coordinate (ρ, θ), that is, in the radial direction, fine in the fovea but coarser towards peripheral according to the logarithm of ρ, while in the angle direction, at a constant pitch Δθ and stored to the coordinate (u, v) in the striate cortex V1. Figure 4 illustrates a sketch how the retinal image is sampled, stored in the striate cortex, and played back to retina.
4.2 Discrete Log Polar Transform
In the discrete LPT system, (ρ, θ) is digitized to R number of rings and S number of sectors. The striate cortex image is stored in the new Cartesian coordinates (u, v) as
ρ0 denotes the radius of blind spot and ρ ≥ ρ0 prevents for the points near origin not to be mapped to the negative infinite-point. This regulation is called CBS (Central Blind Spot) model. Figure 5 illustrates how the image “sunflower” is sampled in LPT lattice and transformed to striate cortex image, then stored in the coordinates (u, v).
The height h(u) and width w(u) of an unit cell between u + 1 and u are given by the following equations. Hence the area α(u) of unit cell increases exponentially with u.
As sensed in Fig. 5, the color is sampled finer in the center but coarser towards peripheral. The pixels in the yellow petals occupy larger area than peripheral. This spatially-variant characteristics to collect the color information on the viewpoint must be reflected in the population density in the color distribution of striate cortex image.
Figure 6 is another example for a pink rose “cherry shell”. It shows how the color distribution is concentrated on the pinkish petal area around at the central viewpoint in the striate cortex image. Hence it’ll be better for applying PCM not on the original but on the striate cortex image after LPT to perform the color matching more effective for the object of attention.
Now the basic PCM matrix MPCM in Eq. (11) is applied to the covariance after LPT and we get newly the following color transfer matrix.
Figure 7 illustrates the color transfer process in LPTPCM model. In this sample, both the source image A and target image B are first transformed to the visual cortex images by LPT, then the clustered color distribution in cortex image A is transformed to match with that of cortex image by PCM. As a result, the material appearance of greenish transparent wine glass B looks to be transferred to that of gold mask image A.
Since the original images A and B have color dissimilarity, it’s a hard to make the color matching only by the single use of basic PCM. While, by just placing LPT before PCM, the feeling of greenish wine glass B is well conveyed to that of gold mask A.
5 Experimental Results and Discussions
The performance of proposed LPTPCM model is compared with the other methods mentioned in Sect. 3. Figure 8 shows the results for the same images used in Fig. 7. The lαβ model fails for such images with color dissimilarity. The source image colors remain almost unchanged. Eigenvalue and Cholesky decomposition methods reflect the greenish target colors a little bit, but look unnatural. In the basic PCM model, the black in eyes and the green in mask face seem to have replaced unnatural. Any mismatches in the directions of PC axes might occur. While, LPTPCM model worked successful for transferring the color atmosphere of wine glass to that of gold mask.
Figure 9 shows another example for color transfer between three glass vases with different patterns. As well, lαβ was hardly function remaining the source colors almost unchanged. Though Eigenvalue and Cholesky decomposition methods showed certain effects, a partial color mixing happened between the source B and target A as shown in B to A color matching. PCM and LPTPCM looks like a neck and neck. But looking carefully, LPTPCM gives a little bit better impression than PCM due to conveying the clean textures in the target.
Figure 10 is a comparison in PCM and LPTPCM for handcraft pots. Both achieved the expected results. It’s hard to tell which is better. How to make a quantitative evaluation is left behind as a future challenge.
On the other hand, Fig. 11 shows a result for color transfer between the images with heterogeneous textures. (a) tried to transfer the color atmosphere of “greenish wine glass” to that of “reddish Porsche”, where only LPTPCM was successful.
Figure 12 shows the performance between PCM vs. LPTPCM in case of changing the target image B to the gold mask or handcraft pot B. In the upper case of gold mask target, LPTPCM clearly reflects the feeling of the target, but in the lower case of green pot target, it’s hard to tell which is better, maybe, depending on personal preference.
For the sake of simplicity, the basic PCM is applied assuming a single clustered image. In the case of multi-clustered image, any segmentation is needed for separeting the colored objects to each cluster then the oblect-to-object PCM is performed. But, it’s hard to find the corresponding pair of objects particularly in the case of dissimilar color images [12,13,14]. Hence, the proposed model is not universal but limited to the images handled as a single cluster. Also, it should be noted on the margin of image background. Figure 13 shows how the results in PCM differes by the margin of background, because the white margins influence on the image color clusters. As clearly seen, LPTPCM is insensitive to the margins and robust than PCM. The reason why comes from that LPT mimics the retina to/from cortex imaging called Foveation.
6 Conclusions
This paper challenged to apply the scene color transfer methods to the material appearance transfer. The proposed LPTPCM model is a joint LPT-PCM algorithm. Prior to PCM (Principal Component Matching), the source A and Target B retinal images are transformed to striate cortex images by LPT (Log-Polar-Transform). The key is to make use of color concentration characteristics on the central viewpoint of striate cortex by LPT. The performance of conventional PCM is significantly enhanced by the cooperation with LPT. The proposed model transfers the color atmosphere of target image B to that of source image A without any a priori information or optical measurement for the material properties. The question is how to evaluate the transformed image is perceptually acceptable or not. Any quantitative quality measure is hoped to be developed and is left behind as a future work.
References
Zeki, S.: Inner Vision: An Exploration of Art and the Brain. Oxford University Press, Oxford (1999)
Tsumura, N., et al.: Estimating reflectance property from refocused images and its application to auto material appearance balancing. JIST 59(3), 30501-1–30501-6 (2015)
Motoyoshi, I., Nishida, S., Sharan, L., Adelson, E.H.: Image statistics and the perception of surface qualities. Nature 447, 206–209 (2007)
Sawayama, M., Nishida, S.: Visual perception of surface wetness. J. Vis. 15(12), 937 (2015)
Kobiki, H., et al.: Specular reflection control technology to increase glossiness of images. Toshiba Rev. 68(9), 38–41 (2013)
Fleming, R.W.: Visual perception of materials and their properties. Vis. Res. 94, 62–75 (2014)
Mihálik, A., Durikovi£, R.: Material appearance transfer between images. In: SCCG 2009, Proceedings of the 2009 Spring Conference, CG, pp. 55–58 (2009)
Nguyen, C.H., et al.: 3D material style transfer. In: Proceedings of the EUROGRAPHICS, vol. 31, no. 2 (2012)
Reinhard, E., et al.: Color transfer between images. IEEE CG Appl. 21, 34–40 (2001)
Ruderman, D.L., et al.: Statistics of cone responses to natural images: implications for visual coding. JOSA A-15(8), 2036–2045 (1998)
Kotera, H., et al.: Object-oriented color matching by image clustering. In: Proceedings of the CIC6, pp. 154–158 (1998)
Kotera, H., et al.: Object-to-object color mapping by image segmentation. J. Electron. Imaging 10(4)/1, 977–987 (2001)
Kotera, H., Horiuchi, T.: Automatic interchange in scene colors by image segmentation. In: Proceedings of the CIC12, pp. 93–99 (2004)
Kotera, H., et al.: Automatic color interchange between images. In: Proceedings of the AIC 2005, pp. 1019–1022 (2015)
Kotera, H.: Intelligent image processing. J. SID 14(9), 745–754 (2006)
Pitié, F., Kokaram, A.: The linear Monge-Kantorovitch colour mapping for example-based colour transfer. In: Proceedings of the IET CVMP, pp. 23–31 (2007)
Schwartz, E.L.: Spatial mapping in the primate sensory projection: analytic structure and relevance to perception. Biol. Cybern. 25, 181–194 (1977)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Kotera, H. (2019). Material Appearance Transfer with Visual Cortex Image. In: Tominaga, S., Schettini, R., Trémeau, A., Horiuchi, T. (eds) Computational Color Imaging. CCIW 2019. Lecture Notes in Computer Science(), vol 11418. Springer, Cham. https://doi.org/10.1007/978-3-030-13940-7_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-13940-7_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-13939-1
Online ISBN: 978-3-030-13940-7
eBook Packages: Computer ScienceComputer Science (R0)