Controllable Underwater Image Generation Method Based on Improved ControlNet

doi:10.19678/j.issn.1000-3428.0252345

Abstract

Abstract: Underwater image generation technology is crucial for filling data gaps in marine exploration. The authenticity and diversity of the generated images directly affect the reliability of the subsequent analytical studies. Existing models typically possess enormous parameter quantities, with prolonged training and inference processes. Additionally, the generated underwater images suffer from insufficient clarity, and distortions exist in the structures and edges of image subjects. Moreover, the inference process does not adequately consider the unique optical properties of underwater environments. Therefore, the authenticity of the generated images is not optimized. To resolve these issues, this study proposes UW-ControlNet, a novel network built using the ControlNet architecture that performs parameter fine-tuning on a pretrained Stable Diffusion model. It combines structural constraints from conditional images with semantic guidance from textual prompts to achieve a cross-modal controllable generation of underwater images. A Lightweight Feature Extraction Network （LFEN） is introduced to optimize the feature extraction process of conditional images, thereby enhancing the convergence and inference speeds of the model. A Correlation Matrix-based Channel Attention (CMCA) model is designed to decouple and couple global channel features corresponding to the background with local channel features corresponding to the subject, optimize the text-image multimodal alignment in the generation process, and enhance the credibility of the generation results. A Structure-Semantics Constraint Enhancement Module (SSCEM) is constructed to prevent constraint information loss caused by downsampling, ensuring structural consistency between the generated images and conditional images. The experimental results confirm that UW-ControlNet surpasses existing methods in both quantitative and qualitative metrics, demonstrating significant application potential.

Key words: underwater image generation, cross-modal generation, ControlNet, Lightweight Feature Extraction Network (LFEN), channel attention mechanism, feature enhancement

摘要： 水下图像生成技术作为填补海洋探索领域数据缺口的重要途径，生成图像的真实性和多样性将直接影响后续分析研究的可靠性。现有模型通常参数量庞大，训练与推理过程耗时较长，生成的水下图像清晰度不足，图像主体的结构和边缘存在畸变现象，推理过程未能充分考虑水下环境独特的光学特性，生成图像的真实性有待优化。为此，基于ControlNet模型提出UW-ControlNet水下图像生成框架，对预训练的Stable Diffusion模型进行参数微调，将条件图像的结构约束与文本提示的语义约束相结合，实现水下图像的跨模态可控生成。引入轻量化特征提取网络（LFEN），改进条件图像的特征提取过程，提高模型的收敛速度和推理速度。设计基于关联矩阵的通道注意力（CMCA）模块，将背景对应的全局通道特征与主体对应的局部通道特征进行解耦与耦合，在优化生成过程中的文本-图像多模态对齐，增强生成结果的可信度。构建结构-语义约束增强模块（SSCEM），避免下采样过程导致的约束信息丢失，确保生成图像与条件图像的结构一致。实验结果表明，通过UW-ControlNet生成的水下图像在定量指标和定性对比上均优于现有方法，展现出良好的应用价值。

关键词: 水下图像生成, 跨模态生成, ControlNet, 轻量化特征提取网络, 通道注意力机制, 特征增强

CLC Number:

TP391

LIU Tianquan, LU Cunyue, WANG Xiaolong, LUO Runshu. Controllable Underwater Image Generation Method Based on Improved ControlNet[J]. Computer Engineering, 2026, 52(11): 374-386.

刘天权, 鹿存跃, 王晓龙, 罗润书. 基于改进ControlNet的可控水下图像生成方法[J]. 计算机工程, 2026, 52(11): 374-386.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0252345

https://www.ecice06.com/EN/Y2026/V52/I11/374

References

[1] 罗旭东, 袁笛, 常晓军,等. 基于不确定性启发图像增强的水下目标跟踪[J]. 计算机工程, 2025, 51(1): 11-19.LUO X D, YUAN D, CHANG X J, et al. Underwater target tracking based on uncertainty-inspired image enhancement[J]. Computer Engineering, 2025, 51(1): 11-19. (in Chinese)
[2] ALABA S Y, NABI M M, SHAH C, et al. Class-aware fish species recognition using deep learning for an imbalanced dataset[J]. Sensors, 2022, 22(21): 8268.
[3] 刘子健, 王兴梅, 陈伟京, 等. 基于硬负样本对比学习的水下图像生成方法[J]. 模式识别与人工智能, 2024, 37(10):887-909.LIU Z J, WANG X M, CHEN W J, et al. Underwater image generation method based on contrastive learning with hard negative samples[J]. Pattern Recognition and Artificial Intelligence, 2024,37(10): 887-909. (in Chinese)
[4] HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. New York, USA: MIT Press, 2020: 6840-6851.
[5] 赵宏, 宋馥荣, 李文改. 基于SE-AdvGAN的图像对抗样本生成方法研究[J]. 计算机工程, 2025, 51(2): 300-311. ZHAO H, SONG F R, LI W G. Research on image adversarial example generation method based on SE-AdvGAN[J]. Computer Engineering, 2025, 51(2): 300-311. (in Chinese)
[6] WANG N, ZHOU Y B,HAN F L, et al. UWGAN: underwater GAN for real-world underwater color restoration and dehazing [EB/OL]. [2025-03-15]. https://arxiv.org/pdf/1912.10269.
[7] LI J, SKINNER K A, EUSTICE R M, et al. WaterGAN: unsupervised generative network to enable real-time color correction of monocular underwater images[J]. IEEE Robotics and Automation Letters, 2018, 3(1): 387-394.
[8] KWON G, YE J C. Diffusion-based image translation using disentangled style and content representation[C]//Proceedings of the 11th International Conference on Learning Representations. New York, USA: [s.n], 2015:1123-1126.
[9] ZHANG H Y, YAO F H, GONG Y F, et al. Marine biology image generation based on diffusion-stylegan2[J]. IEEE Access, 2024, 12: 37310-37325.
[10] ZHANG L M, RAO A Y, AGRAWALA M. Adding conditional control to text-to-image diffusion models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2023: 3813-3824.
[11] ROMBACH R, BLATTMANN A, LORENZ D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 10674-10685.
[12] PENG B H, WANG J, ZHANG Y C, et al. ControlNeXt: powerful and efficient control for image and video generation [EB/OL]. [2025-03-15]. https://arxiv.org/abs/2408.06070.
[13] LI M, YANG T J N, KUANG H F, et al. ControlNet++: improving conditional controls with efficient consistency feedback[EB/OL].[2025-03-15]. https://arxiv.org/abs/2404.07987.
[14] QIN C, ZHANG S, YU N, et al. UniControl: a unified diffusion model for controllable visual generation in the wild [EB/OL]. [2025-03-15]. https://arxiv.org/pdf/2305.11147.
[15] ZHAO S H, CHEN D D, CHEN Y C. Uni-ControlNet: all-in-one control to text-to-image diffusion models[C]//Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, USA: MIT Press, 2023: 11124-11150.
[16] MOU C, WANG X T, XIE L B, et al. T2I-Adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models [EB/OL]. [2025-03-15]. https://arxiv.org/pdf/2302.08453.
[17] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 7132-7141.
[18] LI X, WANG W H, HU X L, et al. Selective kernel networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2019: 510-519.
[19] WANG Q L, WU B G, ZHU P F, et al. ECA-Net: efficient channel attention for deep convolutional neural networks[C] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2020: 11531-11539.
[20] SANDLER M, HOWARD A, ZHU M L, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 4510-4520.
[21] OUYANG D L, HE S, ZHANG G Z, et al. Efficient multi-scale attention module with cross-spatial learning[C] //Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D.C., USA: IEEE Press, 2023: 1-5.
[22] LI X, HU X L, YANG J. Spatial group-wise enhance: improving semantic feature learning in convolutional networks[EB/OL]. [2025-03-15]. https://arxiv.org/abs/1905.09646.
[23] HU J, SHEN L, SUN G, et al. Gather-Excite: exploiting feature context in convolutional neural networks[C]//Proceedings of International Conference on Neural Information Processing Systems. Montréal, Canada: MIT Press, 2018: 9423-9433.
[24] WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 3-19.
[25] LIU H J, LIU F Q, FAN X Y, et al. Polarized self-attention: towards high-quality pixel-wise mapping[J]. Neurocomputing, 2022, 506: 158-167.
[26] HONG L, WANG X, ZHANG G, et al. USOD10K: a new benchmark dataset for underwater salient object detection[J]. IEEE Transactions on Image Processing, 2025, 34: 1602-1615.
[27] LI J N, LI D X, XION C M, et al. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation[C]//Proceedings of the 39th International Conference on Machine Learning. Baltimore, USA: PMLR, 2022: 12888-12900.
[28] RANFTL R, LASINGER K, HAFNER D, et al. Towards robust monocular depth estimation: mixing datasets for zero-shot cross-dataset transfer[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44: 1623-1637.
[29] YANG M, SOWMYA A. An underwater color image quality evaluation metric[J]. IEEE Transactions on Image Processing, 2015, 24(12): 6062-6071.
[30] PANETTA K, GAO C, AGAIAN S. Human-visual-system-inspired underwater image quality measures[J]. IEEE Journal of Oceanic Engineering, 2016, 41(3): 541-551.
[31] HESSEL J, HOLTZMAN A, FORBES M, et al. CLIPScore: a reference-free evaluation metric for image captioning[C]//Proceedings of Conference on Empirical Methods in Natural Language Processing. San Diego, USA: Association for Computational Linguistics, 2021: 7514-7528.
[32] YANG L H, KANG B Y, HUANG Z L, et al. Depth anything: unleashing the power of large-scale unlabeled data[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2024: 10371-10381.

Please choose a citation manager

Content to export