Lightweight Photorealistic Image Style Transfer with Collaborative Optimization of Shuffle Gate Attention and Channel Alignment Whitening and Coloring Transform

doi:10.19678/j.issn.1000-3428.0069830

Abstract

Abstract: Existing photorealistic image style transfer algorithms typically do not fully consider the issues of algorithm model size and computational efficiency while pursuing improvements in image realism and stylization intensity. Therefore, applying these methods to low computing power devices is difficult. To address this issue, this study proposes a lightweight real image style transfer algorithm. The VGG19 is replaced with the ShuffleNet V2 lightweight network as the feature extractor, with block-wise training and skip-connection techniques introduced to significantly reduce the number of parameters and improve the speed of image style transfer. To better balance the content and style of the transferred images, the study also proposes a Shuffle Gated Channel Attention Mechanism (SGCAM) and Channel Alignment Whitening and Coloring Transform (CAWCT). SGCAM combines channel shuffling with gating mechanisms efficiently, which not only enhances the realism of generated images but also further maintains the advantage of the lightweight algorithm. CAWCT significantly boosts the stylization intensity of the generated images by introducing binary operations to match the whitened content features and style features for similarity. Experimental results show that the parameter size of the proposed algorithm is only 14.8% of that of PhotoWCT². It takes only 4.22 s to transfer an image with a resolution of 1 000×750 pixel, which is 0.79 s faster than that achieved by PhotoWCT². Simultaneously, the quality and stylization strength of the generated images are significantly improved. In performance evaluations, the Structural Similarity Index Measure (SSIM) and Peak Signal-to-Noise Ratio (PSNR) indicators increase by 0.031 dB and 0.066 dB, respectively, while the Content loss, Gram loss, and Style loss metrics decrease by 0.227, 0.138×10^-5, and 0.116, respectively.

Key words: style transfer, lightweight, ShuffleNet V2, channel attention, similarity match

摘要： 现有的真实图像风格迁移算法在追求提升图像的真实感和风格化强度的同时,通常未充分考虑算法模型尺寸和计算效率问题,因此很难适用于低算力设备。为解决这一问题,提出一种轻量级真实图像风格迁移算法。使用ShuffleNet V2轻量级网络替代VGG19作为特征提取器,并引入块式训练和跳跃连接技术,旨在大幅度减少参数量,提高图像的风格迁移速度。同时,为了更好地平衡迁移图像的内容和风格,设计混洗门控通道注意力机制(SGCAM)和通道对齐策略(CAWCT)。SGCAM将通道混洗和门控机制巧妙结合,不仅增强了生成图像的真实感,还进一步保持了算法轻量化的优势。CAWCT通过引入二值化操作对白化后的内容特征和风格特征进行相似性匹配,显著提升了生成图像的风格化强度。实验结果表明,所提算法的参数量仅为PhotoWCT²的14.8%,迁移一张1 000×750像素的图像只需4.22 s,比PhotoWCT²少0.79 s,同时生成图像的质量和风格化强度均得到明显提升,结构相似性(SSIM)和峰值信噪比(PSNR)指标分别提高0.031 dB和0.066 dB,内容损失(Content loss)、Gram损失(Gram loss)和风格损失(Style loss)指标分别降低0.227、0.138×10^-5和0.116。

关键词: 风格迁移, 轻量级, ShuffleNet V2, 通道注意力, 相似性匹配

CLC Number:

TP391

LIU Huilin, FANG Qiong, WANG Yansi, ZHANG Shunxiang, SU Shuzhi. Lightweight Photorealistic Image Style Transfer with Collaborative Optimization of Shuffle Gate Attention and Channel Alignment Whitening and Coloring Transform[J]. Computer Engineering, 2026, 52(2): 197-208.

刘惠临, 方琼, 王燕思, 张顺香, 苏树智. 混洗门控注意力与通道对齐策略协同优化的轻量级真实图像风格迁移[J]. 计算机工程, 2026, 52(2): 197-208.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069830

https://www.ecice06.com/EN/Y2026/V52/I2/197

References

[1] GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE Press, 2016: 2414-2423.
[2] DENG Y Y, TANG F, DONG W M, et al. StyTr2: image style transfer with transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 11316-11326.
[3] 毛琳, 王萌, 杨大伟. 内容特征一致性风格迁移网络[J]. 计算机辅助设计与图形学学报, 2022, 34(6): 892-900. MAO L, WANG M, YANG D W. Content consistency preserving style transfer network[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(6): 892-900. (in Chinese)
[4] CHIANG P-Z, TSAI M S, TSENG H Y, et al. Stylizing 3D scene via implicit representation and HyperNetwork[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Waikoloa, USA: IEEE Press, 2022: 215-224.
[5] WU J Y, HOU L F, LI Z J, et al. Preserving structural consistency in arbitrary artist and artwork style transfer[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(3): 2830-2838.
[6] LUAN F J, PARIS S, SHECHTMAN E, et al. Deep photo style transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE Press, 2017: 6997-7005.
[7] LI Y J, LIU M Y, LI X T, et al. A closed-form solution to photorealistic image stylization[C]//Proceedings of ECCV 2018. Berlin, Germany: Springer, 2018: 468-483.
[8] 张颖涛, 张杰, 张睿, 等. 全局信息引导的真实图像风格迁移[J]. 计算机科学, 2022, 49(7): 100-105. ZHANG Y T ZHANG J, ZHANG R, et al. Photorealistic style transfer guided by global information[J]. Computer Science, 2022, 49(7): 100-105. (in Chinese)
[9] GUNAWAN A, KIM S Y, SIM H, et al. Modernizing old photos using multiple references via photorealistic style transfer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver, Canada: IEEE Press, 2023: 12460-12469.
[10] WU Z J, ZHU Z, DU J P, et al. CCPL: contrastive coherence preserving loss for versatile style transfer[C]//Proceedings of ECCV 2022. Berlin, Germany: Springer, 2022: 189-206.
[11] CHIU T Y, GURARI D. Line search-based feature transformation for fast, stable, and tunable content-style control in photorealistic style transfer[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Waikoloa, USA: IEEE Press, 2023: 249-258.
[12] YOO J, UH Y, CHUN S, et al. Photorealistic style transfer via wavelet transforms[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea: IEEE Press, 2019: 9035-9044.
[13] AN J, XIONG H Y, HUAN J, et al. Ultrafast photorealistic style transfer via neural architecture search[C]//Proceedings of the AAAI Conference on Artificial Intelligence.[S. l.]: AAAI Press, 2020: 10443-10450.
[14] CHIU T Y, GURARI D. PhotoWCT²: compact autoencoder for photorealistic style transfer resulting from blockwise training and skip connections of high-frequency residuals[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Waikoloa, USA: IEEE Press, 2022: 2978-2987.
[15] AN J, LI T, HUANG H Z, et al. Is bigger always better? An empirical study on efficient architectures for style transfer and beyond[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Waikoloa, USA: IEEE Press, 2023: 4073-4083.
[16] 吕文锐, 普园媛, 赵征鹏, 等. 基于字形约束和注意力的艺术字体风格迁移[J]. 计算机工程, 2024, 50(12): 306-317. LV W R, PU Y Y, ZHAO Z P, et al. Artistic font style transfer based on glyph constraints and attention[J]. Computer Engineering, 2024, 50(12): 306-317. (in Chinese)
[17] PARK D Y, LEE K H. Arbitrary style transfer with style-attentional networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE Press, 2019: 5873-5881.
[18] LIU S H, LIN T W, HE D L, et al. AdaAttN: revisit attention mechanism in arbitrary neural style transfer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, Canada: IEEE Press, 2021: 6629-6638.
[19] ZHU M R, HE X, WANG N N, et al. All-to-key attention for arbitrary style transfer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Paris, France: IEEE Press, 2023: 23052-23062.
[20] HONG K, JEON S, LEE J, et al. AesPA-Net: aesthetic pattern-aware style transfer networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Paris, France: IEEE Press, 2023: 22701-22710.
[21] GE B, HU Z S, XIA C X, et al. Arbitrary style transfer method with attentional feature distribution matching[J]. Multimedia Systems, 2024, 30(2): 96.
[22] MA N N, ZHANG X Y, ZHENG H T, et al. ShuffleNet V2: practical guidelines for efficient CNN architecture design[C]//Proceedings of ECCV 2018. Berlin, Germany: Springer, 2018: 122-138.
[23] HU J, SHEN L, SUN G. Squeeze-and-Excitation networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE Press, 2018: 7132-7141.
[24] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of ECCV 2014. Berlin, Germany: Springer, 2014: 740-755.
[25] MA Y N, ZHAO C Q, BASU A, et al. RAST: restorable arbitrary style transfer via multi-restoration[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Waikoloa, USA: IEEE Press, 2023: 331-340.

Please choose a citation manager

Content to export