Low-light Image Enhancement Method Combining Gated Transformation Mechanism and GAN

doi:10.19678/j.issn.1000-3428.0067252

Abstract

Abstract:

To address the problems of paired image data dependence, serious detail loss, and noise amplification in low-light image enhancement, a low-light image enhancement method that combines a gated channel transformation mechanism with Generative Adversarial Network(GAN) is proposed. This method can be trained without low/normal-light image pairs. First, a feature extraction network composed of multi-scale convolution residual modules and a gated channel transformation convolution and residual module is designed. The Gated Channel Transformation(GCT) unit extracts global context features and multi-scale local feature information of the input images. In the feature fusion network, the convolution residual structure is used to fully fuse the extracted deep and shallow features, and a horizontal jump connection structure is introduced to retain the detailed feature information to the maximum extent in obtaining the final enhanced image. Finally, the joint loss function is introduced to guide the network training process, restraining image noise and enhancing image color more naturally and symmetrically. The experimental results show that this method can effectively improve the brightness and contrast of the low-light image and reduce image noise, generating a clearer enhanced image and a more realistic color. In practice, average Peak Signal-to-Noise Ratio(PSNR), Structural Similarity Index Measure(SSIM), and Natural Image Quality Evaluator(NIQE) reaches 16.48 dB, 0.93, 3.37, respectively. Compared with other algorithms, the proposed method provides significant advantages in subjective visual analysis and objective index evaluation.

Key words: low-light image enhancement, convolution residual structure, Gated Channel Transformation(GCT) unit, unsupervised learning, Generative Adversarial Network(GAN)

摘要：

针对低光照图像增强过程中存在的配对图像数据依赖、细节损失严重和噪声放大问题，提出结合门控通道变换机制和生成对抗网络（GAN）的低光照图像增强方法AGR-GAN，该方法可以在没有低/正常光图像对的情况下进行训练。首先，设计特征提取网络，该网络由多个基于门控通道变换单元的多尺度卷积残差模块构成，以提取输入图像的全局上下文特征和多尺度局部特征信息；然后，在特征融合网络中，采用卷积残差结构将提取的深浅层特征进行充分融合，再引入横向跳跃连接结构，最大程度保留细节特征信息，获得最终的增强图像；最后，引入联合损失函数指导网络训练过程，抑制图像噪声，使增强图像色彩更自然匀称。实验结果表明，该方法在主观视觉分析和客观指标评价方面相较其他算法均具有显著优势，其能有效提高低光照图像的亮度和对比度，减弱图像噪声，增强后的图像更清晰且色彩更真实，峰值信噪比、结构相似度和无参考图像质量评价指标平均可达16.48 dB、0.93和3.37。

关键词: 低光照图像增强, 卷积残差结构, 门控通道变换单元, 无监督学习, 生成对抗网络

Yinyin HE, Jing HU, Zhibo CHEN, Rongguo ZHANG. Low-light Image Enhancement Method Combining Gated Transformation Mechanism and GAN[J]. Computer Engineering, 2024, 50(2): 247-255.

何银银, 胡静, 陈志泊, 张荣国. 融合门控变换机制和GAN的低光照图像增强方法[J]. 计算机工程, 2024, 50(2): 247-255.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0067252

http://www.ecice06.com/EN/Y2024/V50/I2/247

Figures/Tables 9

Fig.1 Overall structure of AGR-GAN

Fig.2 Structure of generate network

Fig.3 Structure of double discriminator

Fig.4 Comparison of enhancement effects of different methods on five low-light datasets

Fig.5 Comparison of enhancement effects of different methods on Test-ImageSet dataset

Fig.6 Visual comparison of ablation results

References 28

1	马龙, 马腾宇, 刘日升. 低光照图像增强算法综述. 中国图象图形学报, 2022, 27(5): 1392- 1409.
	MA L, MA T Y, LIU R S. The review of low-light image enhancement. Journal of Image and Graphics, 2022, 27(5): 1392- 1409.
2	彭大鑫, 甄彤, 李智慧. 低光照图像增强研究方法综述. 计算机工程与应用, 2023, 59(18): 14- 27.
	PENG D X, ZHEN T, LI Z H. Review of research methods on low-light image enhancement. Computer Engineering and Applications, 2023, 59(18): 14- 27.
3	REZA A M. Realization of the Contrast Limited Adaptive Histogram Equalization(CLAHE) for real-time image enhancement. Journal of VLSI Signal Processing Systems for Signal, Image and Video Technology, 2004, 38(1): 35- 44. doi: 10.1023/B:VLSI.0000028532.53893.82
4	GUO X J, LI Y, LING H B. LIME: low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing, 2017, 26(2): 982- 993. doi: 10.1109/TIP.2016.2639450
5	WEI C, WANG W J, YANG W H, et al. Deep Retinex decomposition for low-light enhancement[EB/OL]. [2023-03-17]. https://arxiv.org/abs/1808.04560.pdf.
6	ZHANG Y H, ZHANG J W, GUO X J. Kindling the darkness: a practical low-light image enhancer[C]//Proceedings of the 27th ACM International Conference on Multimedia. New York, USA: ACM Press, 2019: 1632-1640.
7	LV F, LU F, WU J, et al. MBLLEN: low-light image/video enhancement using CNNs[EB/OL]. [2023-03-17]. https://phi-ai.buaa.edu.cn/publications/papers/MBLLEN.pdf.
8	REN W Q, LIU S F, MA L, et al. Low-light image enhancement via a deep hybrid network. IEEE Transactions on Image Processing, 2019, 28(9): 4364- 4375. doi: 10.1109/TIP.2019.2910412
9	ZHANG F, LI Y, YOU S D, et al. Learning temporal consistency for low light video enhancement from single images[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C. , USA: IEEE Press, 2021: 4967-4976.
10	LIU S G, ZHANG Y. Detail-preserving underexposed image enhancement via optimal weighted multi-exposure fusion. IEEE Transactions on Consumer Electronics, 2019, 65(3): 303- 311. doi: 10.1109/TCE.2019.2893644
11	GUO C L, LI C Y, GUO J C, et al. Zero-reference deep curve estimation for low-light image enhancement[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C. , USA: IEEE Press, 2020: 1780-1789.
12	JIANG Y F, GONG X Y, LIU D, et al. EnlightenGAN: deep light enhancement without paired supervision. IEEE Transactions on Image Processing, 2021, 30, 2340- 2349. doi: 10.1109/TIP.2021.3051462
13	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks. Communications of the ACM, 2020, 63(11): 139- 144. doi: 10.1145/3422622
14	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of MICCAI 2015. Berlin, Germany: Springer, 2015: 234-241.
15	YANG Z X, ZHU L C, WU Y, et al. Gated channel transformation for visual recognition[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C. , USA: IEEE Press, 2020: 11794-11803.
16	ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C. , USA: IEEE Press, 2017: 1125-1134.
17	JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of ECCV 2016. Berlin, Germany: Springer, 2016: 694-711.
18	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2023-03-17]. https://arxiv.org/abs/1409.1556.pdf.
19	ALY H A, DUBOIS E. Image up-sampling using total-variation regularization with a new observation model. IEEE Transactions on Image Processing, 2005, 14(10): 1647- 1659. doi: 10.1109/TIP.2005.851684
20	MAO X, LI Q, XIE H, et al. Least squares generative adversarial networks[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D. C. , USA: IEEE Press, 2017: 2794-2802.
21	JOLICOEUR-MARTINEAU A. The relativistic discriminator: a key element missing from standard GAN[EB/OL]. [2023-03-17]. https://arxiv.org/abs/1807.00734.pdf.
22	LEE C, LEE C, KIM C S. Contrast enhancement based on layered difference representation[C]//Proceedings of the 19th IEEE International Conference on Image Processing. Washington D. C. , USA: IEEE Press, 2012: 965-968.
23	MA K D, ZENG K, WANG Z. Perceptual quality assessment for multi-exposure image fusion. IEEE Transactions on Image Processing, 2015, 24(11): 3345- 3356. doi: 10.1109/TIP.2015.2442920
24	WANG S H, ZHENG J, HU H M, et al. Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2013, 22(9): 3538- 3548. doi: 10.1109/TIP.2013.2261309
25	VONIKAKIS V, ANDREADIS I, GASTERATOS A. Fast centre-surround contrast modification. IET Image Processing, 2008, 2(1): 19. doi: 10.1049/iet-ipr:20070012
26	LOH Y P, CHAN C S. Getting to know low-light images with the Exclusively Dark dataset. Computer Vision and Image Understanding, 2019, 178, 30- 42. doi: 10.1016/j.cviu.2018.10.010
27	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600- 612. doi: 10.1109/TIP.2003.819861
28	MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 2013, 20(3): 209- 212. doi: 10.1109/LSP.2012.2227726

[1]	Shupeng WANG, Yindi HE. Uneven Illumination Image Enhancement Algorithm Fusing Feature Attention Mechanism [J]. Computer Engineering, 2023, 49(8): 232-239.
[2]	SHEN Mengqiang, YU Wennian, YI Li, SONG Nan. Full-Time Scale Speech Enhancement Method Based on GAN [J]. Computer Engineering, 2023, 49(6): 115-122,130.
[3]	ZHAI Sheping, ZHANG Yuhang, BAI Xiaoxia. Knowledge Graph Embedding Negative Sampling Method Fused with Entity Neighborhood Information [J]. Computer Engineering, 2023, 49(3): 95-104.
[4]	HE Yue, CHEN Guangsheng, JING Weipeng, XU Zekun. Remote Sensing Image Retrieval Based on Deep Multi-Similarity Hashing Method [J]. Computer Engineering, 2023, 49(2): 206-212.
[5]	Ning YAN, Yueyang LI, Haichi LUO. Unsupervised Anomaly Detection Based on Block Pyramid Memory Module [J]. Computer Engineering, 2023, 49(12): 304-310.
[6]	Jinsheng CHEN, Wenzhen MA, Shaofeng FANG, Ziming ZOU. Object Recognition of Atmospheric Gravity Wave Based on Foundation Airglow Images [J]. Computer Engineering, 2023, 49(11): 13-23.
[7]	XU Chaoyue, YU Ying, HE Penghao, LI Miao, MA Yuhui. Multi-Scale Low-Light Image Enhancement Network Based on U-Net [J]. Computer Engineering, 2022, 48(8): 215-223.
[8]	LI Chen, HOU Jin, LI Jinbiao, CHEN Zirui. Infrared and Visible Image Fusion Method Based on Attention and Residual Concatenation [J]. Computer Engineering, 2022, 48(7): 234-240.
[9]	DING Yongjun, HUANG Shan. An Improved Generative Adversarial Network for Image Dehazing [J]. Computer Engineering, 2022, 48(6): 207-212.
[10]	XU Runhao, CHENG Jixiang, LI Zhidan, FU Xiaolong. Face Recognition with Occlusion Based on Cyclic Generative Adversarial Networks [J]. Computer Engineering, 2022, 48(5): 289-296,305.
[11]	TANG Jiamin, HAN Hua, HUANG Li. Coarse-grained and Fine-grained Features Extraction Based on Unsupervised Learning in Pedestrian Re-identification [J]. Computer Engineering, 2022, 48(4): 269-275,283.
[12]	HE Tao, YU Shuman, XU He. Single Image Dehazing Method Based on Conditional Generative Adversarial Network and Knowledge Distillation [J]. Computer Engineering, 2022, 48(4): 165-172.
[13]	LIN Hong, CHEN Zhuangyuan, REN Shuo, LI Lin, LI Yuqiang. Facial Attribute Migration Across Multiple Image Domains Using Selective Transfer and Hinge Adversary [J]. Computer Engineering, 2022, 48(4): 179-190.
[14]	YIN Xin, ZHANG Zhancheng. Person Image Synthesis Based on Posture Guidance and Attribute Decomposition [J]. Computer Engineering, 2022, 48(11): 224-230,239.
[15]	GONG Faming, XU Chenxi, LI Juejin. Adversarial Deep Learning-based Unauthorized Construction Site Recognition Using UAV-assisted Aerial Photography [J]. Computer Engineering, 2022, 48(1): 275-280,287.

Please choose a citation manager

Content to export