Keywords

1 Introduction

As time progresses, diseases like color blindness (CB) have become more prevalent. Dichromatopsia and achromatopsia are the two most common types of CB. Dichromatic blindness is further categorized into protanopia, deuteranopia, and tritanopia. CB individuals are impacted to varying degrees in their daily lives. For instance, CB can affect a person’s ability to perform certain tasks that rely on color recognition, such as reading charts and graphs, identifying traffic lights, and choosing clothing that matches. In some cases, it can also impact a person’s ability to perceive depth and contrast, making it more difficult to navigate certain environments. CB can also have social implications, as color is often used as a means of communication and expression in our culture. For example, CB may not be able to appreciate the beauty of a sunset or a colorful painting in the same way as someone with normal color vision. They may also feel left out of certain social activities, such as viewing fireworks or participating in color-coded events.

Despite these challenges, CB individuals can still lead fulfilling lives. With the help of assistive technologies, such as color filters and apps that assist with color recognition, they can overcome many of the obstacles presented by their condition. Additionally, increased awareness and understanding of CB can help to create a more inclusive and accommodating society for those who live with this condition. The image recoloring technology for people with CB can be used to enhance the readability and aesthetics of images and help people with CB better understand and appreciate images. However, there are some problems with the existing methods, such as color inauthenticity and color disharmony. For this scenario, we have explored a method for recoloring images that ensures the resulting image is natural, harmonious, and distinguishable in color.

In this article, we will explore the technology of image recoloring for people with CB and introduce a method we have developed for achieving this goal. First, we will briefly review the background and current research status of CB, followed by a detailed presentation of our proposed method. We will then conduct experimental validation to demonstrate the effectiveness of our approach. Finally, we will summarize our findings.

2 Related Work

Palette-based color manipulation provides a versatile approach to image editing, utilizing palettes that can be categorized into two main types. The first type [1] involves the utilization of both the original and target palettes. In this method, the key colors of the original image are extracted through clustering techniques, forming the original palette. Simultaneously, the target palette is either predefined or trained in advance. By mapping the colors from the original palette to the corresponding colors in the target palette, effective color manipulation and transformation can be achieved. This approach is commonly employed in tasks such as image recoloration and color theme enhancement. On the other hand, the second type [2] solely relies on the target palette. This approach proves particularly useful for image recoloration and color enhancement, as it eliminates the need for the original palette. By directly mapping the colors in the image to the colors within the target palette, the image’s overall color scheme can be effectively modified and transformed.

Colorblindness image recoloring can be categorized into four distinct approaches. The first approach involves grayscale conversion [3, 4], where the image is transformed into grayscale using an objective function that considers the differences in pixel values. However, this method is often considered too simplistic, as it discards the valuable color information embedded within the image. The second approach is based on image segmentation [5, 6], where the image is initially segmented to identify areas that may be indistinguishable from individuals with color vision deficiencies. These identified areas are then replaced with colors that exhibit high contrast, ensuring better visibility for CB individuals. However, this method may be susceptible to external noise and is limited by the accuracy and reliability of the segmentation process. The third approach involves color conversion [7,8,9], which aims to preserve the color information while adapting it to be more distinguishable for CB individuals. This is achieved by applying linear operations within various color spaces, such as LMS, LAB, or HSV. However, the quality and speed of the results obtained through this approach heavily depend on the specific linear operation algorithms employed. The fourth approach leverages the power of neural networks. X. Zhang [10] proposed the use of generative adversarial networks (GANs) to facilitate image recoloring while incorporating constraints to exert better control over the direction and outcome of the recoloring process. By training the GAN on a dataset of color-corrected images, the neural network can learn to generate visually appealing recolored versions of input images while considering the constraints imposed during the training process. These various approaches to color manipulation and colorblindness image recoloring provide a range of techniques and methodologies, each with its strengths and limitations. By understanding and leveraging these approaches, we can effectively address the challenges associated with color manipulation and ensure that visual content is accessible and inclusive for individuals with color vision deficiencies.

To simulate CB, researchers have proposed mathematical models for CB simulation. The research progress of simulation models provides robust support in our endeavor to better understand and address CB issues. By simulating the visual perception of CB individuals, we can delve deeper into the impact of CB on daily life and design improved assistive tools and educational training programs. In this paper, we utilized J.-B. Bao’s method [11] for CB simulation. By adjusting the weights in the formula, different degrees of CB can be simulated.

The field of CB assistive devices [18] has witnessed significant research advancements, aiming to provide individuals with color vision deficiencies with enhanced color perception and improved visual experiences. Researchers have explored various technological solutions to develop innovative assistive devices. One approach involves the use of augmented reality (AR) and virtual reality (VR) technologies, where wearable devices or headsets are equipped with specialized algorithms to enhance color discrimination. These devices can modify color representations in real-time, allowing users to perceive colors more accurately. Another line of research focuses on developing specialized filters or lenses that can be integrated into eyewear, such as glasses or contact lenses. These filters selectively alter the wavelengths of light to enhance color perception for specific types of CB. Additionally, smartphone applications and digital tools have been developed to assist colorblind individuals in distinguishing colors through real-time image processing and color correction algorithms. These applications provide on-the-go assistance and enable color identification in various contexts. As research progresses, there is a growing emphasis on personalized and customizable approaches, tailoring assistive devices to individual color vision deficiencies. These advancements hold immense potential in improving the quality of life and accessibility for individuals with color vision deficiencies, empowering them to navigate the colorful world more effectively.

3 Approach

Our approach consists of two main parts: recommended palette generation and image recoloring. The color palette generation part is the color palette of different categories of objects produced by statistical data sets of object colors, and the image recoloring part is the use of a series of operations to generate recoloring images for CB. The flowchart of our approach is presented in Fig. 1.

Fig. 1.
figure 1

Flowchart of our method.

Full size image

3.1 Recommended Palette Generation

We use the clustering method to generate the palette. We choose the Oxford flower dataset [12] as our dataset. According to the size of the dataset, we define the number of object hue clusters as five. We preprocess this dataset and then use tools to batch-remove image backgrounds. We use K-Means to perform two rounds of clustering on the hue distribution of images. The first clustering obtains the hue center value array (hue color palette) of each image arranged in descending order of frequency. The second clustering performs another round of clustering on all the hue color palettes of each image type to obtain five representative hue color palettes.

3.2 Image Recoloring

At present, we have undertaken a comprehensive exploration of the entire process to assess its feasibility. To facilitate the interactive segmentation of the image’s foreground and background, we have employed the widely used GrabCut method [13]. This method offers a user-friendly approach, as it requires only the manual drawing of a bounding box around the target object. By leveraging user interaction, the segmentation boundary can be fine-tuned, ensuring precise and accurate results.

In our research, we have placed significant emphasis on achieving accurate and detailed classification through fine-grained training. To accomplish this, we have adopted the transfer learning methodology, a powerful technique that leverages pre-existing knowledge from a pre-trained model. For the fine-grained model training, we have selected ResNet-152 [14], a state-of-the-art deep neural network renowned for its exceptional performance. Our training process begins by utilizing the pre-trained model to train the weights of the fully connected layers. This initial step allows the model to grasp the underlying patterns and features specific to our fine-grained classification task. Subsequently, we proceed to retrain and update the weights of all layers within the network. This comprehensive training approach ensures that the model becomes well-adapted to our specific dataset, enabling it to accurately recognize and categorize objects at a fine-grained level.

In order to enhance the realism of recolored objects, we implemented a sophisticated approach that involved the utilization of pre-calculated recommended color palettes and conducted color transfer operations in the image foreground. To accomplish this, we employed the method proposed by E. Reinhard [15], which focuses on adjusting the H (hue) channel of the image’s HSV (hue, saturation, value) color space while incorporating the information from the recommended palette[19,20,21]. This meticulous approach enabled us to achieve highly accurate color transfer while ensuring that the natural appearance of the image was preserved. By leveraging pre-calculated recommended color palettes, we were able to tap into a vast collection of carefully curated color combinations that are known to produce visually pleasing and realistic results. These color palettes serve as a valuable resource, providing guidance and inspiration for the recoloring process. The core of our methodology lies in the adjustment of the H channel within the HSV color space. By aligning the hue values of the original image with those recommended in the palette, we ensure that the recolored objects exhibit a harmonious and authentic appearance. This adjustment process takes into account the specific color characteristics of the recommended palette, allowing for seamless integration of the new colors while preserving the overall naturalness and visual coherence of the image.

$$ I = \frac{\sigma_t }{{\sigma_s }} \left( {S^H - mean(T^H )} \right) + mean(T^H ) $$
(1)

It conforms to the color characteristics of the palette and satisfies the naturalness (see examples in Fig. 2).

Fig. 2.
figure 2

Several sets of color transfer examples.

Full size image

In order to solve the problem of disharmony between the image background and adjusted foreground color, we adjust the image background harmoniously. In our image harmonization process, we employed the method proposed by D. Cohen-Or [16], which offers effective techniques for achieving visual harmony in images. To facilitate this harmonization, we carefully selected reference templates provided by D. Cohen-Or, each representing a specific harmonization scheme. These templates consist of color circles, and we considered the shaded area within each circle as the valid region for the template. Furthermore, the templates can be rotated to adapt to different image compositions and requirements. To ensure a harmonious visual composition, we calculated a hue harmony template that best matched the foreground of the image. By analyzing the colors present in the foreground, we determined the optimal hue harmony template that would complement and enhance its visual appeal. Subsequently, we adjusted the hue of the image’s background to align with the selected foreground harmony template. This adjustment process aimed to create a coherent and visually pleasing blend between the foreground and background elements. It is worth noting that we introduced an additional judgment criterion in this process. If the optimal palette for the foreground corresponds to a single shadow area template, we adjusted the rotation angle by 180 degrees. This adjustment was made to ensure that the foreground and background elements were sufficiently distinguishable and visually balanced. By rotating the template, we could achieve a better composition and enhance the overall harmony of the image. Several examples of harmonization groups can be seen in Fig. 3.

Fig. 3.
figure 3

Color harmonization example.

Full size image

After recoloring the images, we conducted a CB simulation to ensure their accessibility for individuals. This simulation involved generating a simulation map that represents how the recolored images would appear to individuals with different types of CB. To quantify the hue distinction between the foreground and background in each simulated image, we employed the Bhattacharyya distance metric [17]. This distance measure allowed us to calculate the degree of color differentiation between the foreground and the background. By analyzing the simulated images using the Bhattacharyya distance, we were able to identify the original image from the simulated set that exhibited the highest degree of distinction between the foreground and the background. This image was then selected as the final result, ensuring that it provided the most noticeable and discernible color contrast.

Through the CB simulation and the Bhattacharyya distance calculation, we prioritized the visual clarity and distinguishability of the recolored images for individuals with color vision deficiencies. By selecting the image with the maximum distinction, we aimed to provide an inclusive and accessible viewing experience for individuals with different types of CB. This approach demonstrates our commitment to enhancing the accessibility of visual content, ensuring that individuals with color vision deficiencies can perceive and distinguish important visual elements within the images. By incorporating CB simulation and the Bhattacharyya distance metric, we contribute to creating a more inclusive and accommodating environment for individuals, allowing them to fully engage with and appreciate visual content.

4 Evaluation

Our method belongs to color conversion and is compared to the methods of W. Woods [7], S. Choudhry [8], and Y. Wang [9]. The comparison methods include both objective and subjective evaluations, and the images used for evaluation are all from the Oxford flower dataset [12].

4.1 Objective Evaluation

We utilized four metrics to quantitatively measure the effectiveness of our approach: foreground-background distinction, harmonization, SSIM, and PSNR. Mean measurement results for 15 randomly selected images from the dataset are presented in Tables 1 and 2. Figure 4 presents several sample sets from the objective evaluation.

Fig. 4.
figure 4

Examples of objective evaluation. A total of four sets of images are included.

Full size image
Table 1. Statistics of the difference between our method and others.
Full size table
Table 2. Ablation experiment.
Full size table

Table 1 shows that our method has improved compared to others’ methods, mainly in the indicators of differentiation and harmony. Table 2 proves that our method is effective in combining various modules after ablation experiments.

4.2 Subjective Evaluation

Since there is no accurate way to classify CB in the medical field and it is a non-trivial task to find CB patients, most current related works use simulation programs in this field. We conduct a subjective experiment with 20 participants on multiple groups of CB simulation images that were randomly ordered. The mean scores are shown in Fig. 5. It turns out that our approach has improved to some extent.

Fig. 5.
figure 5

Mean scores for each method.

Full size image

5 Conclusion

Given the shortcomings of the existing colorblind recoloration technology, we have explored a set of methods to achieve image recoloration. Good results are obtained through the modules of foreground naturalness adjustment, harmonization adjustment, and distinction evaluation of foreground and background. Our approach focuses on illustrating the workability of the entire process. The advantages and disadvantages of each branching module technique are not the focus. Of course, there are still some shortcomings in our method. Next, we intend to improve the segmentation part first, so that it can automatically recognize multiple objects and adjust the color of each of them. Then, optimize the algorithm to increase the processing speed. Finally, we wanted to create an electronic device (CB aid) that would find a balance between speed and quality to enable real-time recoloring for CB patients.