CMMF-Net: a generative network based on CLIP-guided multi-modal feature fusion for thermal infrared image colorization

2.1. Visible image colorization

Image colorization involves transforming a monochromatic TIR image into a colorized image with three channels. Traditional grayscale image colorization techniques often necessitate user participation. For instance, the Scribble-based method relies on prior color annotations provided by users, which are then extended to the entire grayscale image^[18]. Another approach, the Example-based method, utilizes color data from reference images to assist in the colorization process^[19].

In recent years, the field of image colorization has witnessed a significant shift towards deep learning-based approaches, which have demonstrated superior performance compared to traditional methods. Larsson et al. introduced a fully automated technique utilizing CNNs^[20]. Building upon this, Zhao et al. integrated image segmentation information to further enhance colorization accuracy^[21]. The integration of generative adversarial network (GAN) models^[22] has also become prevalent in numerous colorization tasks^[14,23]. Moreover, recent advancements in denoising diffusion probabilistic models (DDPM)^[24,25] have introduced a novel colorization approach based on visible image diffusion models. This method utilizes visible images as conditions for diffusion models, resulting in color images through denoising processes. Impressively, this technique has shown promising results in the realm of visible image colorization^[26]. Collectively, these advancements underscore the transformative potential of deep learning techniques in revolutionizing image colorization processes.

2.2. TIR image colorization

With the continuous development of deep learning, various popular network frameworks have been applied in TIR image colorization tasks^[27–31]. However, due to the substantial differences in data distribution between TIR and visible images, alongside the abundance of TIR datasets, CNN architectures often struggle to adapt effectively to TIR image colorization tasks. TIR stands out as the pioneering CNN-based colorization network, leveraging the U-Net architecture for TIR image colorization^[27]. While TIR boasts lightweight design, its colorization performance remains less than satisfactory.

In contrast, the inherent capability of GAN-based architectures in image generation has led to the predominance of GAN-based networks in image colorization. Leveraging the advantages of GAN structures, these networks demonstrate enhanced performance and effectiveness in generating realistic colorized images. Advancements in TIR image colorization have predominantly been driven by GAN-based approaches, notably exemplified by Pix2Pix^[23], which, while enhancing image details and colors to some extent, still exhibits a degree of unnaturalness compared to true images. Subsequent refinements, such as those introduced by Kuang et al., have significantly bolstered colorization quality through the integration of composite loss functions^[28]. Innovations such as the ToDayGAN-based network proposed by Lou et al. have further pushed the boundaries by effectively translating nighttime TIR images into visually satisfying daytime colorized counterparts^[32]. Additionally, efforts by Liao et al. with the mixed-skipping (MS) U-Net architecture have notably improved colorization quality through advanced attention mechanisms^[30]. He et al. proposed a large kernel network and a two-branch structure network incorporating a Transformer module to fully process TIR image features^[31]. However, due to inherent defects in the image, the color effect remains unsatisfactory. Recently, several networks have been proposed for the colorization of TIR videos, including Recycle-GAN^[33] and unpaired infrared-to-visible video translation (I2VGAN)^[34]. These approaches primarily focus on enhancing the realism of video content.

2.3. CLIP net

In tasks involving vision and language, a profound grasp of semantics is pivotal for achieving enhanced collaborative representation. In the realm of comprehending visual content, diverse backbone models^[35,36] have emerged, aimed at pure visual comprehension. These models have convincingly demonstrated their efficacy across substantial datasets^[37]. In recent cross-modality research, numerous methods^[38–40] have been introduced. These approaches primarily emphasize understanding the intricate connection between visual and textual elements across diverse modalities. CLIP^[41] is a type of neural network trained on various pairs of images and texts. It can be directed using natural language to predict the most relevant textual snippet given a specific image, without the need for direct optimization for the task, similar to the zero-shot capability of GPT-2 and GPT-3. CLIP consists of two models: text encoder and image encoder. The text encoder is utilized to extract features from text and can employ commonly used text transformer models from NLP. On the other hand, the image encoder is employed to extract features from images, and it can use conventional CNN models or ViTs. In this paper, we will utilize its pretrained text-encoder module to extract text information.

3.1. Overview

Figure 1 presents the architecture of the model. The proposed method integrates both image feature information and textual information, enabling a more comprehensive and accurate understanding and processing of the relevant features of TIR images. We utilize the pretrained CLIP model to capture and process text-based feature information. In this approach, the Text_Encoder module refers to the one used in the original CLIP method. Meanwhile, we utilize the Encoder module of the ViT model^[42] to extract image feature information from TIR images. The ViT model is an image processing model based on the Transformer architecture, which has achieved significant success in the field of image processing. It can effectively extract feature information from images and encode it into vector representations. Next, the text and image feature information obtained through the CI module is processed. Through a multi-head attention mechanism, relationships between text and images are established, ultimately resulting in enhanced image feature representations based on text. The cross-modality design presented in this paper consists of two transformer blocks. These blocks are implemented based on the ViT. Unlike the conventional ViT, we integrate the generation of the attention mechanism's Key (K), Query (Q), and Value (V) with the modality interaction between images and text. Specifically, we generate K from the text and Q and V from image embeddings. Additionally, we employ a multi-head attention mechanism to enable the model to capture more global information. The detailed processing flow is illustrated in Figure 2. Next, drawing from the successful practice of the LK_U-Net module in the first large kernel U-Net and attention_U-Net-transformer (ViT-Based)-based generative adversarial network (LKAT-GAN) method^[31] for TIR image colorization task, we employ this U-Net module to process and decode the text-image features obtained from the previous stage to generate the final colorized image. To achieve higher-quality results, The total loss comprises content loss, perceptual loss, total variation loss, and SSIM loss. Details will be provided in the following sections.

Figure 1. The overall framework. Including a ViT Image_Encoder module, a CLIP Text_Encoder module, a cross-modality alignment module and a U-net module. CMMF-Net takes image-sentence pairs as input, and outputs the colorized image. ViT: Vision transformer; CLIP: contrastive language-image pretraining; CMMF-Net: a generative network based on clip-guided multi-modal feature fusion for thermal infrared image colorization.

Figure 2. The CI module. The text feature is Q, and the image feature or the current layer output feature is K and V for multi-head attention operation, and the corresponding relationship is established. CI: Cross-modal interaction.

3.2. Text_Encoder and Image_Encoder

In the context of the large-scale dataset WebImageText^[41] collected and curated by the OpenAI team, the emergence of the CLIP model signifies a significant breakthrough in the field of text-image prediction matching. The key innovation of the CLIP method lies in its contrastive learning framework, which matches natural language descriptions with images, prompting the model to learn a shared embedding space. In this space, relevant language descriptions and images are mapped to nearby points, while irrelevant language descriptions and images are pushed apart, enabling the model to better understand the semantic relationships between images and language.

Through pretraining on large-scale image-text pairs, the CLIP model has learned a universal cross-modal representation. This representation not only captures rich relationships between images and language but also exhibits strong generalization capabilities. In practice, this means that when faced with new tasks or categories, the CLIP model possesses powerful zero-shot learning abilities. Even in unseen scenarios, this model can perform inference and generalization because the universal representation it has learned demonstrates strong generalization properties.

In this paper, we designed a general module that introduces text guidance for image colorization. The text input for our method is based on the observation of typical object information in the dataset, and the text features are extracted using the CLIP Text Encoder model. The Text Encoder module of CLIP is essentially a basic Bert, fundamentally composed of Transformer modules, so the structures of Text Encoder and Image Encoder are quite similar. In CLIP, the Text_Encoder consists of 12 layers of Transformer Encoder. Due to its pretraining phase, where it has learned rich semantic information representations through large-scale datasets, by transferring the parameters of this module and combining them with the Image_Encoder module of the ViT model and the CI module, we extend these advantages to the TIR image colorization task. In our approach, the depth of the Image_Encoder module is set to nine layers. The Text_Encoder module, pretrained on massive datasets, provides a strong foundation, enabling our approach to accurately extract relevant features. These features capture the rich associations between text descriptions and image features, providing robust support for the colorization task.

3.3. CI

To better adapt the text features obtained by the transferred Text_Encoder module and the image features obtained by the Image_Encoder module to TIR image colorization, we designed the CI module to process the TIR image feature information and corresponding text information. The specific structure is depicted in Figure 2.

The overall structure is similar to the Transformer module, where we use the token sequence representation obtained by the Text_Encoder module as Query, and the image feature information obtained by the Image_Encoder module as Key and Value, which are jointly inputted into the CI module for multi-head attention operations. Through attention mechanisms, the relationships between text and images are established, and then the text-enhanced image representation is obtained through the multi-layer perceptron (MLP) layer output. The output of the current layer is then fed along with the text token into the next layer to repeat the above operation. Finally, the OutputProj layer is used to adjust the feature size and dimensions, outputting the image features with text feature information. Since the input image features and text features already contain positional information, no Embedding operation is performed in the CI module; instead, the features are directly inputted into our Transformer_Block.

For the Text_Encoder module with fixed parameters, the input text is mapped to a predefined feature space, which mainly contains categorical classification information of text tokens. In the CI module, the fixed text feature information can effectively highlight and emphasize image features, enhancing the model's understanding. As for the Image_Encoder module, with the involvement of text features, through network training and optimization, it can better retrieve and process image features. The enhanced image features ultimately ameliorate the problem of inaccuracies in directly extracting image features from TIR images, resulting in significant enhancements in both image details and color information in the final colorized images.

3.4. U-Net

We introduced a U-Net network module as a critical component in the colorization task. This module plays a vital role in the network, effectively integrating text and image information, and is crucial in generating colorized images. The design inspiration for the U-Net module comes from the innovation in the LKAT-GAN^[31] work, particularly the U-Net module with large kernel convolutions. This structure better handles feature extraction and fusion in colorization tasks. Specifically, during the downsampling process, we employ a network with large kernel convolutions to effectively extract important feature information from the original TIR images while ensuring that these features maintain the same size as the features with text information. Next, the downsampled features and the comprehensive output features of text-image are concatenated for final upsampling. This approach fully utilizes the information from both types of features while maintaining coherence and consistency between them. Similar to most U-Net networks, skip connections and concatenation are applied between the downsampling and upsampling modules, preserving more detailed information and enhancing the performance of network and robustness. Finally, through a convolutional layer with a 3 × 3 kernel, the number of channels is adjusted to 3, outputting the final colorized result. This step ensures that the generated color images have good visual effects and quality. In summary, the U-Net network module in our approach is trainable, meaning it has the capability to adjust and optimize parameters, allowing for flexible tuning based on tasks and data to meet the demands of different scenarios, thereby enhancing the quality and accuracy of colorization results. Additionally, by combining the downsampling and upsampling processes, this module can effectively fuse text and image information without losing feature details. During the downsampling stage, the network with large kernel convolutions can better capture global features and structural information of the image. In the upsampling stage, skip connections and concatenation operations preserve detailed information and seamlessly integrate text and image information. This efficient fusion mechanism provides reliable support for the colorization task, achieving more accurate and clearer colorization effects while maintaining image quality.

3.5. Loss function

The loss function of our model comprises four components: Content loss $$L_{content}$$, Perceptual loss $$L_{perceptual}$$, Total Variant loss $$L_{tv}$$, and SSIM loss $$L_{ssim}$$.

Content loss can ensure the minimization of differences in chrominance and luminance between the generated colorized image and ground truth image. $$L_{content}$$ is defined as follows:

where y represents he actual colorized image, x indicates the input infrared image, z denotes the text vector, and G(x, z) stands for our colorization model.

Perceptual loss can regulate the perceptual details between the colorized image and the ground truth image. Based on the experience of existing research, the VGG-19^[43] network pretrained on the Image-Net dataset is utilized as the perceptual feature extractor. $$L_{perceptual}$$ is defined as follows:

where $${\mathrm{\Phi }}_i$$ represents the $$\text{i}$$-th layer features extracted by the pretrained network.

Total variant loss can significantly enhance the spatial smoothness of the images generated by the model while simultaneously reducing image noise. First, the pixel differences of the generated image along the height and width dimensions are calculated. Then, the total variation value along the width direction is obtained. Finally, the total variation values are summed and normalized by dividing by the total number of pixels in the image, yielding the average total variation. $$L_{tv}$$ is formulated as follows:

where W and H denote the height and width of the image, respectively.

SSIM loss can minimize the difference between generated images and ground truth, we introduce SSIM loss which can measure the similarity between two images. $$L_{ssim}$$ is defined as follows:

where $${\mu }_x$$ and $${\mu }_y$$ represent the mean luminance values of image x and image y, C is a small constant introduced to prevent numerical instability, and $${\sigma }_x$$ and $${\sigma }_y$$ represent the contrast values of image x and image y, $${\sigma }_{xy}$$ represents the structural correlation between the two images.

Thus, the overall loss function for our network is formulated as follows:

where $${\lambda }_{content}$$, $${\lambda }_{adv}$$, $${\lambda }_{perceptua}$$, and $${\lambda }_{tv}$$ are parameters that control the weight of each loss function.

4.1. Dataset and implementation details

4.2. Experiments on KAIST dataset

We select six comparative TIR image colorization methods: Pix2pix^[23], thermal infrared image colorization using mixed-skipping UNet and generative adversarial network (MUGAN)^[30], LKAT-GAN^[31], PealGAN^[32], CycleGAN^[33], saliency map-guided colorization with generative adversarial network (SCGAN)^[29], and thermal infrared colorization via conditional generative adversarial network (TIC-CGAN)^[28]. The best performance metrics of the aforementioned methods are shown in Table 1. To illustrate the objective evaluation of each model, line graphs showing the scores from 16 to 20 epochs are provided in Figure 4. Across all epochs, the two metrics of our model consistently surpass those of other methods, clearly demonstrating the potential and performance of our method. Moreover, the outstanding SSIM values provide additional evidence that the results produced by our method are highly consistent with the ground truth. In quantitative comparisons, CMMF Net exhibits the best overall performance, excelling in both detail reconstruction and color. The qualitative evaluation metrics of the aforementioned methods are illustrated in Figure 5. Overall, SCGAN exhibits the weakest performance, with inadequate color and detail reconstruction. Similarly, TIC-CGAN demonstrates shortcomings in reconstructing fine image details. While PealGAN, as an unsupervised model, excels in handling unpaired datasets, its performance on paired datasets is inferior to that of supervised models. MUGAN, although yielding better results than TIC-CGAN and SCGAN, still exhibits unsatisfactory detail reconstruction, with minor noise present in certain cases. LKAT-GAN produces more realistic colors and richer details compared to the other methods, but the object recognition is not clear and accurate enough, and the image noise is significant. In terms of image object details, our proposed method is significantly better than other comparison methods, and the colorized images generated by our model are smoother and less noisy. However, it is worth noting that the results generated by our method, as a whole, do not appear as bright compared to other methods.

Figure 3. Colorized images using different image colorization methods on IRVI. (A) TIR images; (B) I2VGAN^[34]; (C) TICC-GAN^[28]; (D) SCGAN^[29]; (E) PealGAN^[32]; (F) MUGAN^[30]; (G) LKAT-GAN^[31]; (H) CMMF-Net; (I) True RGB images. The original infrared images (A) and RGB images (J) were obtained from https://github.com/BIT-DA/I2V-GAN, while the other images were reproduced using methods from other papers and generated with our laboratory's equipment. The specific methods can be found in the corresponding references listed in the bibliography. IRVI: Dataset for infrared-to-visible video translation; TIR: thermal infrared; I2VGAN: unpaired infrared-to-visible video translation; TICC-GAN: thermal infrared colorization via conditional generative adversarial network; SCGAN: saliency map-guided colorization with generative adversarial network; MUGAN: thermal infrared image colorization using xixed-skipping UNet and generative adversarial network; LKAT-GAN: a GAN for thermal infrared image colorization based on large kernel and attentionUNet-transformer; CMMF-Net: a generative network based on clip-guided multi-modal feature fusion for thermal infrared image colorization.

Figure 4. Results of different image colorization methods on KAIST. X-axis is the epoch and Y-axis is the index. KAIST: A multispectral pedestrian dataset, proposed by the Korea Advanced Institute of Science and Technology.

Figure 5. Colorized images using different image colorization methods on KAIST. (A) TIR images; (B) CycleGAN^[33]; (C) Pix2pix^[23]; (D) SCGAN^[29]; (E) TIC-CGAN^[28]; (F) PealGAN^[32]; (G) MUGAN^[30]; (H) LKAT-GAN^[31]; (I) CMMF-Net; (J) True RGB images. The original infrared images (A) and RGB images (J) were obtained from https://soonminhwang.github.io/rgbt-ped-detection/data/, while the other images were reproduced using methods from other papers and generated with our laboratory's equipment. The specific methods can be found in the corresponding references listed in the bibliography. KAIST: A multispectral pedestrian dataset, proposed by the Korea Advanced Institute of Science and Technology; TIR: thermal infrared; SCGAN: saliency map-guided colorization with generative adversarial network; TIC-CGAN: thermal infrared colorization via conditional generative adversarial network; MUGAN: thermal infrared image colorization using xixed-skipping UNet and generative adversarial network; LKAT-GAN: a GAN for thermal infrared image colorization based on large kernel and attentionUNet-transformer; CMMF-Net: a generative network based on clip-guided multi-modal feature fusion for thermal infrared image colorization.

4.3. Experiments on IRVI dataset

On the IRVI dataset, we conducted complementary experiments with I2VGAN^[34], SCGAN^[29], TICC-GAN^[28], PealGAN^[32], MUGAN^[30] and LKAT-GAN^[31]. Due to the limited size of the dataset, we combined the data from two scenarios to form the training set, while selecting data from other scenarios for the test set. The IRVI dataset encompasses a wide range of scenes and diverse image content, including low-light, haze, and rain/snow weather conditions. The image content includes vehicles, pedestrians, buildings, and natural landscapes. The results show that our method consistently achieves the best overall performance across various lighting conditions and diverse image content, further confirming the superior generalization capability of our model. From the perspective of objective indicators, our method achieves the best performance compared with other comparison methods in traffic scenarios, and also performs well in monitoring scenarios, second only to LKAT-GAN method, as shown in Table 2. From the perspective of objective indicators, our method achieves optimal performance compared with other comparison methods in traffic scenes. Moreover, as shown in Figure 3, compared with other comparison methods, our proposed method has better performance in image detail and less noise, but the overall image is darker. In general, in the IRVI dataset, our method achieves good results in the objective indicators, but there are some shortcomings in subjective evaluation. For example, the detailed texture is better than that of other methods, but we lose the edges. The edges of other methods show better visual results, but with more pixel-level loss compared with our method, and the metrics reveal this phenomenon.

4.4. Ablation study

To evaluate the contribution of each module in CMMF-Net, we conducted an ablation study. Two experimental settings were designed to verify the effect of the module by excluding some parts of the original structure. The first experiment involved removing the Text_Encoder module experiment, which excluded text information in the model. The second experiment removed the CI module, which did not perform the text and image feature interaction processing but directly added the two features for output. Through these experiments, we were able to analyze the impact of each module on the colorization task and determine which modules were most critical for the final performance improvement.

This study will demonstrate the impact of each innovative module on the colorization results through ablation experiments. The colorization results of each ablation method are shown in Figure 6. For the subjective evaluation, the color results generated by CMMF-Net have been slightly improved in terms of detail and overall, and no obvious difference can be seen in the color results of each ablation method, which also proves that the performance of our overall method is excellent. The objective evaluation results are shown in Table 3. Finally, the overall CMMF-Net model had the highest SSIM and PSNR indicators among all ablation methods. Through the ablation experiment without adding text information, it is found that the introduction of text features can effectively assist the model to better extract and process image feature information, and improve the network color performance. This proves the superiority of our proposed multi-modal model. In the ablation experiment without CI module, it is found that the effect of the model is improved when text information is directly added to the network without text and image related feature processing, which once again proves the auxiliary role of text information to the model. At the same time, the performance of the model is further improved after the addition of CI module, which also proves the importance of this module for multi-modal models in colorization tasks. In addition, the results of two ablation experiments show that our overall network model is significantly better than other comparison models.

Figure 6. Colorization images of ablation studies on KAIST. (A) TIR images; (B) Without Text_Encoder block; (C) Without CI block; (D) CMMF-Net; (E) True RGB images. The original infrared images (A) were obtained from https://soonminhwang.github.io/rgbt-ped-detection/data/, while the other images were generated through our own experiments. KAIST: A multispectral pedestrian dataset, proposed by the Korea Advanced Institute of Science and Technology; TIR: thermal infrared; CI: cross-modal interaction; CMMF-Net: a generative network based on clip-guided multi-modal feature fusion for thermal infrared image colorization.

Authors' contributions

Made substantial contributions to the research process and wrote the original draft: Jiang, Q., Zhou, T., Jin, X.

Performed data acquisition: He, Y., Ma, W.

Provided guidance and support: Hou, J., Abdul Ghan, A.S., Miao, S.

Availability of data and materials

The original infrared images in Figures 4-6 were sourced from publicly accessible datasets, which are available for open access and use. The other images used in the comparative experiments were generated using our laboratory's experimental equipment. The usage of these images complies with the usage guidelines of the platforms and datasets. Additionally, these two datasets are widely utilized by other researchers in the same field, and the method we employed to generate the comparative experimental images is consistent with common practices in the field. Specifically, the datasets used are: the KAIST Dataset and the IRVI Dataset.

Financial support and sponsorship

The research was supported by Yunnan Fundamental Research Projects (202301AW070007, 202301AU070210, 202201AT070112), Yunnan Province Expert Workstations (202305AF150078), Major Scientific and Technological Project of Yunnan Province (202202AD080002, 202302AD080006), National Key Research and Development Program of China under Grant No. 2024YF03014300 and National Natural Science Foundation of China (62261060).

Conflicts of interest

Jin, X. is a Junior Editorial Board Member of the journal Intelligence & Robotics. Together with Jiang, Q., he serves as a Guest Editor for the Special Issue titled "Applications of Generative Adversarial Networks in Computer Vision and Image Processing". They are not involved in any steps of the editorial process, including reviewer selection, manuscript handling, or decision-making. The other authors declared that there are no conflicts of interest.

Ethical approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Copyright

© The Author(s) 2025.