基于生成对抗网络的自然场景低照度增强模型

doi:10.19678/j.issn.1000-3428.0067136

摘要/Abstract

摘要：

当前大多数低照度图像增强模型在自然环境中难以兼顾照度增强和原图特征保留两方面的效果，而且不能很好地适应多种不同的低照度场景，为此，提出一种基于生成对抗网络的改进模型。首先通过标准卷积提取浅层特征，然后利用光照一致性损失构建全局-局部照明学习（GLIE）模块。在GLIE内部设计基于全局-局部的特征提取结构，通过移位窗口自注意力机制和多尺度空洞卷积分别实现场景级特征提取和光照增强平滑。通过原图特征保留块对GLIE的输出特征和浅层特征进行拼接融合和通道注意力加强，实现对原图特征的保留和噪声抑制。在此基础上，通过改进的损失函数在模型训练过程中同时实现对照度增强和原图特征保留的有效监督。实验结果表明，该模型主观效果真实自然，与Retinex-Net、EnlightenGAN等主流模型相比，其对原图色彩纹理细节保留和噪声抑制的效果更好，在整体测试数据集上自然图像质量评估指标和亮度顺序误差分别达到3.88和199.4，在不同测试数据集中2个指标都取得了前3名的结果，整体性能良好。

关键词: 低照度增强, 自注意力机制, 空洞卷积, 特征融合, 图像降噪

Abstract:

In the most current low-light image enhancement models, both illumination enhancement and original image feature preservation are difficult to achieve and hard to adapt to a variety of different low-light conditions in natural scenes.To address these issues, an improved model based on Generative Adversarial Network(GAN)is proposed. The model first extracts shallow features through normal convolution, thereby constructing a Global-Local Illumination Estimation(GLIE) module with illumination consistency loss. A global-local feature extraction structure is designed inside the GLIE module, and scene-level feature learning and smoothness of lighting enhancement are simultaneously realized through the Swin Transformer and multi-scale dilated convolution.Subsequently, the Original Feature Retention-Block(OFR-Block) is used to splice and fuse the output with shallow features of the lighting learning module. Channel attention is further strengthened to realize the preservation of the original image features and noise suppression.In addition, effective supervision of illumination enhancement and original image feature preservation during model training is achieved through an improved loss function. The experimental results demonstrate that the subjective effect of this model is real and natural, with improved preservation of color texture details of the original image and noise suppression compared with mainstream models such as Retinex-Net and EnlightenGAN. The Natural Image Quality Evaluation(NIQE) and Lightness-Order-Error(LOE) reached 3.88 and 199.4 on test data, respectively, achieving the top three results on different test datasets, with better overall performance.

Key words: low-light enhancement, self attention mechanism, dilated convolution, future fusion, image noise suppression

杨瑞君, 秦晋京, 程燕. 基于生成对抗网络的自然场景低照度增强模型[J]. 计算机工程, 2024, 50(1): 279-288.

Ruijun YANG, Jinjing QIN, Yan CHENG. Low-Light Enhancement Model in Natural Scenes Based on Generative Adversarial Network[J]. Computer Engineering, 2024, 50(1): 279-288.

http://www.ecice06.com/CN/Y2024/V50/I1/279

图/表 14

图1 移位窗口自注意力块的基本结构

Fig.1 Basic structure of Swin Transformer block

图2 生成器网络结构

Fig.2 Generator network structure

图3 GIE-Net网络结构

Fig.3 GIE-Net network structure

图4 双线性上采样结构

Fig.4 Bilinear upsampling structure

图5 LIE-Block网络结构

Fig.5 LIE-Block network structure

图6 OFR-Block网络结构

Fig.6 OFR-Block network structure

图7 逆光场景下的实验效果

Fig.7 Experimental effects under backlit scenes

图8 夜景下的实验效果

Fig.8 Experimental effects under night scenes

图9 高频信息丰富场景下的实验效果

Fig.9 Experimental effects under scenarios with rich high-frequency information

图10 细节对比实验结果

Fig.10 Details comparison experiment results

图11 消融实验结果

Fig.11 Results of ablation experiment

参考文献 28

1	LORE K G, AKINTAYO A, SARKAR S. LLNet: a deep autoencoder approach to natural low-light image enhancement. Pattern Recognition, 2017, 61, 650- 662. doi: 10.1016/j.patcog.2016.06.008
2	LV F, LU F, WU J, et al. MBLLEN: low-light image/video enhancement using CNNs[EB/OL]. [2023-02-05]. https://bmva-archive.org.uk/bmvc/2018/contents/papers/0700.pdf.
3	WEI C, WANG W, YANG W, et al. Deep Retinex decomposition for low-light enhancement[EB/OL]. [2023-02-05]. https://arxiv.org/abs/1808.04560.pdf.
4	LAND E H, MCCANN J J. Lightness and Retinex theory. Royal Society Open Science, 1971, 61(1): 1- 11.
5	ZHANG Y H, ZHANG J W, GUO X J. Kindling the darkness: a practical low-light image enhancer[C]//Proceedings of the 27th ACM International Conference on Multimedia. New York, USA: ACM Press, 2019: 1632-1640.
6	JIANG Y F, GONG X Y, LIU D, et al. EnlightenGAN: deep light enhancement without paired supervision. IEEE Transactions on Image Processing, 2021, 30, 2340- 2349. doi: 10.1109/TIP.2021.3051462
7	GUO C L, LI C Y, GUO J C, et al. Zero-reference deep curve estimation for low-light image enhancement[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 1780-1789.
8	LI C Y, GUO C L, CHEN C L. Learning to enhance low-light image via Zero-reference deep curve estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(8): 4225- 4238.
9	LIU R S, MA L, ZHANG J A, et al. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 10561-10570.
10	WU W H, WENG J, ZHANG P P, et al. URetinex-Net: retinex-based deep unfolding network for low-light image enhancement[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 5901-5910.
11	YANG W H, WANG S Q, FANG Y M, et al. From fidelity to perceptual quality: a semi-supervised approach for low-light image enhancement[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 3063-3072.
12	CRESWELL A, WHITE T, DUMOULIN V, et al. Generative adversarial networks: an overview. IEEE Signal Processing Magazine, 2018, 35(1): 53- 65. doi: 10.1109/MSP.2017.2765202
13	江泽涛, 钱艺, 伍旭, 等. 一种基于ARD⁃GAN的低照度图像增强方法. 电子学报, 2021, 49(11): 2160- 2165. doi: 10.12263/DZXB.20200822
	JIANG Z T, QIAN Y, WU X, et al. Low light image enhancement method based on ARD-GAN. Acta Electronica Sinica, 2021, 49(11): 2160- 2165. doi: 10.12263/DZXB.20200822
14	刘文, 杨梅芳, 聂江天, 等. 基于局部生成对抗网络的水上低照度图像增强. 计算机工程, 2021, 47(5): 16- 23. URL
	LIU W, YANG M F, NIE J T, et al. Low-light maritime image enhancement based on local generative adversarial network. Computer Engineering, 2021, 47(5): 16- 23. URL
15	LIU Z, LIN Y T, CAO Y, et al. Swin Transformer: hierarchical vision Transformer using shifted windows[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 10012-10022.
16	REN Y X, WU J, XIAO X F, et al. Online multi-granularity distillation for GAN compression[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 6793-6803.
17	CAO H, WANG Y Y, CHEN J, et al. Swin-Unet: Unet-like pure transformer for medical image segmentation[EB/OL]. [2023-02-05]. https://arxiv.org/abs/2105.05537.
18	LI J C, FANG F M, MEI K F, et al. Multi-scale residual network for image super-resolution[EB/OL]. [2023-02-05]. https://link.springer.com/chapter/10.1007/978-3-030-01237-3_32.
19	CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[EB/OL]. [2023-02-05]. https://arxiv.org/abs/1706.05587.pdf.
20	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7132-7141.
21	MAO X D, LI Q, XIE H R, et al. Least squares generative adversarial networks[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 2794-2802.
22	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2023-02-05]. https://arxiv.org/abs/1409.1556.pdf.
23	LEE C, LEE C, KIM C S. Contrast enhancement based on layered difference representation[C]//Proceedings of 2012 IEEE International Conference on Image Processing. Washington D. C., USA: IEEE Press, 2012: 965-968.
24	GUO X J, LI Y, LING H B. LIME: low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing, 2017, 26(2): 982- 993. doi: 10.1109/TIP.2016.2639450
25	MA K D, ZENG K, WANG Z. Perceptual quality assessment for multi-exposure image fusion. IEEE Transactions on Image Processing, 2015, 24(11): 3345- 3356. doi: 10.1109/TIP.2015.2442920
26	LIU J Y, XU D J, YANG W H, et al. Benchmarking low-light image enhancement and beyond. International Journal of Computer Vision, 2021, 129(4): 1153- 1184.
27	MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 2013, 20(3): 209- 212. doi: 10.1109/LSP.2012.2227726
28	WANG S H, ZHENG J, HU H M, et al. Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Transactions on Image Processing, 2013, 22(9): 3538- 3548. doi: 10.1109/TIP.2013.2261309

[1]	范文卓, 吴涛, 许俊平, 李庆庆, 张建林, 李美惠, 魏宇星. 基于多分辨率特征融合的任意尺度图像超分辨率重建[J]. 计算机工程, 2023, 49(9): 217-225.
[2]	孙龙, 张荣芬, 刘宇红, 饶庭漓. 监控视角下密集人群口罩佩戴检测算法[J]. 计算机工程, 2023, 49(9): 313-320.
[3]	李嘉新, 侯进, 盛博莹, 周宇航. 基于改进YOLOv5的遥感小目标检测网络[J]. 计算机工程, 2023, 49(9): 256-264.
[4]	韩璐, 霍纬纲, 张永会, 刘涛. 基于多尺度特征融合与双注意力机制的多元时间序列预测[J]. 计算机工程, 2023, 49(9): 99-108.
[5]	刘志浩, 孟凡云, 王金鹤, 张楠. 基于空洞卷积与注意力模块的立体匹配算法[J]. 计算机工程, 2023, 49(8): 223-231.
[6]	宋志娜, 李莎, 杨建明, 徐川. 基于特征与区域定位增强的遥感舰船目标检测[J]. 计算机工程, 2023, 49(8): 257-264.
[7]	张欣怡, 张飞, 郝斌, 高鹭, 任晓颖. 基于改进YOLOv5的口罩佩戴检测算法[J]. 计算机工程, 2023, 49(8): 265-274.
[8]	杨长沛, 廖列法. 基于门控空洞卷积特征融合的中文命名实体识别[J]. 计算机工程, 2023, 49(8): 85-95.
[9]	杨祖赫, 黎智辉, 唐云祁, 晏于文, 宋华青. 结合语义与图像信息的行人属性识别算法[J]. 计算机工程, 2023, 49(8): 215-222, 231.
[10]	陈露萌, 曹彦彦, 黄民, 谢鑫钢. 基于改进YOLOv5的火焰检测方法[J]. 计算机工程, 2023, 49(8): 291-301, 309.
[11]	刘豪, 吴红兰, 房宇轩. 结合全局上下文信息的高效人体姿态估计[J]. 计算机工程, 2023, 49(7): 102-109.
[12]	吴珊, 周凤. 基于改进SSD算法的小目标检测[J]. 计算机工程, 2023, 49(7): 179-188.
[13]	谢虹, 姜文刚. RRA-InceptionV3结合鲁棒稀疏表示的表情识别方法[J]. 计算机工程, 2023, 49(7): 196-203.
[14]	齐咏生, 杜晓旭, 朱俊峰, 高胜利, 刘利强. 基于增强型轻量深度网络的牧区牲畜高效检测[J]. 计算机工程, 2023, 49(7): 278-287.
[15]	殷文君, 黄建华, 纪元法. 基于改进密集卷积网络的皮肤肿瘤分类方法[J]. 计算机工程, 2023, 49(7): 288-294.

选择文件类型/文献管理软件名称

选择包含的内容