基于梯度频率多阶段引导的深度图超分辨率方法

doi:10.19678/j.issn.1000-3428.0070478

计算机工程 ›› 2026, Vol. 52 ›› Issue (5): 192-202. doi: 10.19678/j.issn.1000-3428.0070478

• 计算机视觉与图形图像处理 • 上一篇下一篇

基于梯度频率多阶段引导的深度图超分辨率方法

蔡江河¹, 陈飞¹^,*(), 姜凡², 程航³, 王美清³

1. 福州大学计算机与大数据学院, 福建福州 350108
2. 北不列颠哥伦比亚大学计算机科学系, 加拿大乔治王子城 V2N 4Z9
3. 福州大学数学与统计学院, 福建福州 350108

收稿日期:2024-10-12 修回日期:2024-11-26 出版日期:2026-05-15 发布日期:2026-05-12
通讯作者: 陈飞
作者简介:
蔡江河, 男, 硕士研究生, 主研方向为图像处理与重建
陈飞(通信作者), 教授
姜凡, 教授
程航, 教授
王美清, 教授
基金资助:
国家自然科学基金(62471141)

Depth Map Super-Resolution Method Based on Gradient—Frequency Multi-Stage Guidance

CAI Jianghe¹, CHEN Fei¹^,*(), JIANG Fan², CHEN Hang³, WANG Meiqing³

1. College of Computer and Data Science, Fuzhou University, Fuzhou 350108, Fujian, China
2. Department of Computer Science, University of Northern British Columbia, Prince George V2N 4Z9, Canada
3. School of Mathematics and Statistics, Fuzhou University, Fuzhou 350108, Fujian, China

Received:2024-10-12 Revised:2024-11-26 Online:2026-05-15 Published:2026-05-12
Contact: CHEN Fei

摘要/Abstract

摘要：

颜色图像引导的深度图超分辨率(GDSR)旨在利用同一场景下高分辨率(HR)彩色图像提供的引导信息, 将低分辨率(LR)深度图重建为HR深度图。尽管基于空间域的学习方法可以有效提高深度图的整体重建质量, 但在从LR深度图进行重建时, 仍然面临边缘结构模糊的问题。为了解决该问题, 提出梯度频率引导的多阶段整合深度图重建网络(GFNet), 利用颜色图像的梯度先验和频率信息来增强深度边缘结构细节的重建。首先, 设计一个梯度特征提取(GFE)模块, 结合RGB图像的梯度先验知识来优化LR深度图的梯度结构。其次, 设计一个空间频率双路引导融合(SFI)模块, 将RGB图像中精确的高频成分传递到LR深度图中, 从而引导深度图丢失的高频信息重建。最后, 采用一种新颖的隐式神经函数来提高深度图的分辨率。实验结果表明, 在8倍缩放因子的情况下, GFNet在数据集NYUv2、Middlebury以及真实场景数据集RGB-D-D上均方根误差(RMSE)指标分别达到了2.48、1.62、2.57 cm, 相比于结构更复杂的模型GeoDSR, 分别降低了0.14、0.06、0.12 cm, 并在边缘结构细节方面优于对比方法, 证明了所提方法的有效性。

关键词: 深度图超分辨率, 边缘结构, 梯度特征, 频率引导, 隐式神经函数

Abstract:

Color image Guided Depth Super-Resolution (GDSR) aims to reconstruct a High-Resolution (HR) depth map from its Low-Resolution (LR) version with guidance from HR color images of the same scene. Although learning-based methods used in the spatial domain can effectively enhance the overall reconstruction quality of depth maps, they exhibit edge structure blurring during reconstruction from LR depth maps. To address this issue, this paper proposes a Gradient—Frequency guided multi-stage integration Network (GFNet). This network utilizes gradient priors and frequency information from color images to enhance the reconstruction of edge structure details in depth maps. First, a gradient feature extraction module is designed to incorporate the gradient prior knowledge of RGB images and thus optimize the gradient structures of LR depth maps. Subsequently, a spatial—frequency dual-path guidance module is developed to map precise high-frequency components from RGB images onto LR depth maps, thereby guiding the reconstruction of high-frequency information lost in the depth maps. Finally, a novel implicit neural function is employed to improve the resolution of depth maps. Experimental results show that, under an 8× scaling factor, GFNet achieves RMSE values of 2.48, 1.62, and 2.57 cm on the NYUv2, Middlebury, and real-world RGB-D-D datasets, respectively, outperforming the more complex GeoDSR model by 0.14, 0.06, and 0.12 cm, respectively. Additionally, it surpasses comparison methods in terms of edge structure detail reconstruction, demonstrating its effectiveness.

Key words: depth map Super-Resolution (SR), edge structure, gradient feature, frequency guidance, implicit neural function

蔡江河, 陈飞, 姜凡, 程航, 王美清. 基于梯度频率多阶段引导的深度图超分辨率方法[J]. 计算机工程, 2026, 52(5): 192-202.

CAI Jianghe, CHEN Fei, JIANG Fan, CHEN Hang, WANG Meiqing. Depth Map Super-Resolution Method Based on Gradient—Frequency Multi-Stage Guidance[J]. Computer Engineering, 2026, 52(5): 192-202.

https://www.ecice06.com/CN/Y2026/V52/I5/192

图/表 14

图1 不同算法的重建结果(8倍缩放因子)

Fig.1 Reconstruction results of different algorithms (8× scaling factor)

图2 RGB和深度图的梯度/频谱可视化

Fig.2 Visualization of the gradient/spectrum of RGB and depth maps

图3 GFNet结构

Fig.3 Structure of GFNet

图4 SFIB结构

Fig.4 Structure of SFIB

图5 在NYUv2、Middlebury、RGB-D-D数据集上深度图重建的可视化结果(8倍缩放因子)

Fig.5 Visualization of depth map reconstruction results on the NYUv2, Middlebury and RGB-D-D datasets (8× scaling factor)

图6 真实场景下RGB-D-D数据集上深度图重建可视化

Fig.6 Visualization of depth map reconstruction results on RGB-D-D dataset

图7 模块消融实验中间特征层可视化

Fig.7 Visualization of intermediate depth features in module ablation experiments

图8 不同上采样方法的RMSE对比(16倍缩放因子)

Fig.8 Comparison of RMSE for different upsampling methods (16× scaling factor)

图9 不同上损失函数的RMSE对比(8倍缩放因子)

Fig.9 Comparison of RMSE for different loss functions (8× scaling factor)

参考文献 36

1	PENG W L, PAN H, LIU H, et al. IDA-3D: instance-depth-aware 3D object detection from stereo vision for autonomous driving[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 13012-13021.
2	SHOTTON J, FITZGIBBON A, COOK M, et al. Real-time human pose recognition in parts from single depth images[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2011: 1297-1304.
3	谭巨全, 王然. 特征融合下田径录像3D人体动作DTW捕捉算法. 计算机工程, 2024, 50 (7): 71- 78. doi: 10.19678/j.issn.1000-3428.0069491
	TAN J Q , WANG R . Dynamic time warping capture algorithm for 3D human body movements in track and field video recording under feature fusion. Computer Engineering, 2024, 50 (7): 71- 78. doi: 10.19678/j.issn.1000-3428.0069491
4	CHOE J, IM S, RAMEAU F, et al. VolumeFusion: deep depth fusion for 3D scene reconstruction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 16066-16075.
5	RIEGLER G, RVTHER M, BISCHOF H. ATGV-Net: accurate depth super-resolution[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 268-284.
6	YANG Q X, YANG R G, DAVIS J, et al. Spatial-depth super resolution for range images[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2007: 1-8.
7	HE K M , SUN J , TANG X O . Guided image filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35 (6): 1397- 1409. doi: 10.1109/TPAMI.2012.213
8	LUTIO R, BECKER A, DARONCO S, et al. Learning graph regularisation for guided super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 1969-1978.
9	WANG X H , CHEN X H , NI B B , et al. Learning continuous depth representation via geometric spatial aggregator. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37 (3): 2698- 2706. doi: 10.1609/aaai.v37i3.25369
10	MA C, RAO Y M, CHENG Y A, et al. Structure-preserving super resolution with gradient guidance[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 7766-7775.
11	SHI W Z, CABALLERO J, HUSZAR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1874-1883.
12	DUMOULIN V, VISIN F. A guide to convolution arithmetic for deep learning[EB/OL]. [2024-08-01]. https://arxiv.org/pdf/1603.07285.
13	FUOLI D, VAN GOOL L, TIMOFTE R. Fourier space losses for efficient perceptual image super-resolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 2340-2349.
14	CHEN Y B, LIU S F, WANG X L. Learning continuous image representation with local implicit image function[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 8624-8634.
15	KOPF J , COHEN M F , LISCHINSKI D , et al. Joint bilateral upsampling. ACM Transactions on Graphics, 2007, 26 (3): 96. doi: 10.1145/1276377.1276497
16	HAM B , CHO M , PONCE J . Robust guided image filtering using nonconvex potentials. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40 (1): 192- 207. doi: 10.1109/TPAMI.2017.2669034
17	HE L Z, ZHU H G, LI F, et al. Towards fast and accurate real-world depth super-resolution: benchmark dataset and baseline[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 9225-9234.
18	ZHAO Z X, ZHANG J S, XU S, et al. Discrete cosine transform network for guided depth map super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 5687-5697.
19	METZGER N, DAUDT R C, SCHINDLER K. Guided depth super-resolution by deep anisotropic diffusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2023: 18237-18246.
20	QIAO X , GE C Y , ZHANG Y M , et al. Depth super-resolution from explicit and implicit high-frequency features. Computer Vision and Image Understanding, 2023, 237, 103841. doi: 10.1016/j.cviu.2023.103841
21	YAO Z S , FAN G D , FAN J F , et al. Spatial-frequency dual-domain feature fusion network for low-light remote sensing image enhancement. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62, 1- 16.
22	TANCIK M , SRINIVASAN P , MILDENHALL B , et al. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 2020, 33, 7537- 7547.
23	谭明旺, 张选德. 基于隐式神经表示的图像超分辨率. 软件工程, 2024, 27 (11): 20- 24.
	TAN M M , ZHANG X D . Image super-resolution based on implicit neural representation. Software Engineering, 2024, 27 (11): 20- 24.
24	LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C., USA: IEEE Press, 2017: 1132-1140.
25	ZHANG Y L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 294-310.
26	HARIS M, SHAKHNAROVICH G, UKITA N. Deep back-projection networks for super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 1664-1673.
27	ZHOU M , HUANG J , FANG Y C , et al. RETRACTED: pan-sharpening with customized transformer and invertible neural network. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36 (3): 3553- 3561. doi: 10.1609/aaai.v36i3.20267
28	FRIGO M, JOHNSON S G. FFTW: an adaptive software architecture for the FFT[C]//Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D. C., USA: IEEE Press, 1998: 1381-1384.
29	SILBERMAN N, HOIEM D, KOHLI P, et al. Indoor segmentation and support inference from RGBD images[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2012: 746-760.
30	HIRSCHMULLER H, SCHARSTEIN D. Evaluation of cost functions for stereo matching[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2007: 1-8.
31	LU S, REN X F, LIU F. Depth enhancement via low-rank matrix completion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2014: 3390-3397.
32	LI Y, HUANG J B, AHUJA N, et al. Deep joint image filtering[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 154-169.
33	KIM B, PONCE J, HAM B. Deformable kernel networks for guided depth map upsampling[EB/OL]. [2024-08-01]. https://arxiv.org/pdf/1903.11286.
34	TANG J X, CHEN X K, ZENG G. Joint implicit image function for guided depth super-resolution[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York, USA: ACM Press, 2021: 4390-4399.
35	METZGER N, DAUDT R C, SCHINDLER K. Guided depth super-resolution by deep anisotropic diffusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2023: 18237-18246.
36	JIANG X, KUANG Z, GUO C, et al. The devil is in the details: boosting guided depth super-resolution via rethinking cross-modal alignment and aggregation[EB/OL]. [2024-08-01]. https://arxiv.org/pdf/2401.08123.

[1]	耿荣, 孙钦东, 曹晗, 王艳. 空频域联合优化的通用对抗扰动生成方法[J]. 计算机工程, 2026, 52(1): 176-187.
[2]	屈雁秋,陈锋. 基于随机丛林的人体部件分类方法[J]. 计算机工程, 2018, 44(1): 247-251.
[3]	林鹏, 张超, 李竹良, 赵宇明. 基于深度图像学习的人体部位识别[J]. 计算机工程, 2012, 38(16): 185-188.
[4]	廖斌;陈岩. 双尺度边缘结构相似度的图像质量评估[J]. 计算机工程, 2007, 33(19): 199-200,.

选择文件类型/文献管理软件名称

选择包含的内容

基于梯度频率多阶段引导的深度图超分辨率方法

Depth Map Super-Resolution Method Based on Gradient—Frequency Multi-Stage Guidance

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 36

相关文章 4

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于梯度频率多阶段引导的深度图超分辨率方法

Depth Map Super-Resolution Method Based on Gradient—Frequency Multi-Stage Guidance

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 36

相关文章 4

编辑推荐

Metrics

本文评价