具有跨尺度Transformer的高效多视图立体网络

doi:10.19678/j.issn.1000-3428.0068947

摘要/Abstract

摘要：

现有深度多视图立体(MVS)方法将Transformer引入级联网络, 以实现高分辨率深度估计, 从而实现高精确度和完整度的三维重建结果。然而, 基于Transformer的方法受计算成本的限制, 无法扩展到更精细的阶段。为此, 提出一种新颖的跨尺度Transformer的MVS网络, 在不增加额外计算的情况下处理不同阶段的特征表示。引入一种自适应匹配感知Transformer(AMT), 在多个尺度上使用不同的交互式注意力组合。这种组合策略使所提网络能够捕捉图像内部的上下文信息, 并增强图像之间的特征关系。此外, 设计双特征引导聚合(DFGA), 将粗糙的全局语义信息嵌入到更精细的代价体构建中, 以进一步增强全局和局部特征的感知。同时, 通过设计一种特征度量损失, 用于评估变换前后的特征偏差, 以减少特征错误匹配对深度估计的影响。实验结果表明, 在DTU数据集中, 所提网络的完整度和整体度量达到0.264、0.302, 在Tanks and temples 2个大场景的重建平均值分别达到64.28、38.03。

关键词: 多视图立体, 特征匹配, Transformer网络, 注意力机制, 三维重建

Abstract:

At present, deep Multi-View Stereo (MVS) methods widely introduce Transformers into cascade networks to achieve high-resolution depth estimation, thereby ensuring highly accurate and complete 3D reconstruction results. However, Transformer-based methods are limited by their computational costs and cannot be extended to more refined stages. To solve this problem, this paper proposes a novel cross-scale Transformer-based MVS network that can manage feature representations at different stages without incurring additional computation. In particular, this study introduces an Adaptive Matching-aware Transformer (AMT), which uses different interactive attention combinations on multiple scales, enabling the proposed network to capture contextual information within images and enhance the feature relationships between images. In addition, this study proposes Dual Feature Guided Aggregation(DFGA) to embed coarse global semantic information into finer cost body construction, further enhancing the perception of global and local features. Simultaneously, a feature metric loss is designed to evaluate feature deviation before and after the Transformation and thereby reduce the impact of feature mismatch on depth estimation. Experimental results show that the integrity and overall measurements of the proposed network are 0.264 and 0.302 on the DTU dataset, respectively. The average reconstruction values for Tank and temples scenarios are 64.28 and 38.03, respectively.

Key words: Multi-View Stereo(MVS), feature matching, Transformer network, attention mechanism, 3D reconstruction

王思成, 江浩, 陈晓. 具有跨尺度Transformer的高效多视图立体网络[J]. 计算机工程, 2024, 50(11): 266-275.

WANG Sicheng, JIANG Hao, CHEN Xiao. Efficient Multi-View Stereo Network with Cross-Scale Transformer[J]. Computer Engineering, 2024, 50(11): 266-275.

https://www.ecice06.com/CN/Y2024/V50/I11/266

图/表 13

图1 本文所提的多视图立体匹配网络结构

Fig.1 Structure of the multi-view stereo matching network proposed in this article

图2 AMT模块的结构

Fig.2 Structure of the AMT module

图3 DFGA模块的结构

Fig.3 Structure of DFGA module

图4 在DTU数据集上不同方法的点云误差比较

Fig.4 Comparison of point cloud errors of different methods on DTU dataset

图5 在DTU数据集上各物体的重建展示结果

Fig.5 Reconstruction display results of each object on DTU dataset

图6 在Tanks and temples数据集上最先进方法的点云误差比较

Fig.6 Comparison of point cloud errors of state-of-the-art methods on the Tanks and temples dataset

图7 在Tanks and temples数据集上各物体的重建展示结果

Fig.7 Reconstruction display results of each object on Tanks and temples dataset

参考文献 39

1	CAMPBELL N D F, VOGIATZIS G, HERNANDEZ C, et al. Using multiple hypotheses to improve depth-maps for multi-view stereo[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2008: 766-779.
2	FURUKAWA Y, PONCE J. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 32 (8): 1362- 1376. doi: 10.1109/TPAMI.2009.161
3	GALLIANI S, LASINGER K, SCHINDLER K. Massively parallel multiview stereopsis by surface normal diffusion[C]//Proceedings of the IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2015: 873-881.
4	SCHONBERGER J L, FRAHM J M. Structure-from-motion revisited[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 4104-4113.
5	GU X D, FAN Z W, ZHU S Y, et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 2495-2504.
6	YAO Y, LUO Z X, LI S W, et al. MVSNet: depth inference for unstructured multi-view stereo[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 785-801.
7	WEI Z Z, ZHU Q T, MIN C, et al. AA-RMVSNet: adaptive aggregation recurrent multi-view stereo network[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 6187-6196.
8	YAO Y, LUO Z X, LI S W, et al. Recurrent MVSNet for high-resolution multi-view stereo depth inference[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 8626-8634.
9	尹晨阳, 职恒辉, 李慧斌. 基于深度学习的双目立体匹配方法综述. 计算机工程, 2022, 48 (10): 1- 12. URL
	YIN C Y, ZHI H H, LI H B. Survey of binocular stereo-matching methods based on deep learning. Computer Engineering, 2022, 48 (10): 1- 12. URL
10	刘会杰, 柏正尧, 程威, 等. 融合注意力机制和多层U-Net的多视图立体重建. 中国图象图形学报, 2022, 27 (2): 475- 485.
	LIU H J, BAI Z Y, CHENG W, et al. Fusion attention mechanism and multilayer U-Net for multiview stereo. Journal of Image and Graphics, 2022, 27 (2): 475- 485.
11	朱秋明, 倪浩然, 华博宇, 等. 无人机毫米波信道测量与建模研究综述. 移动通信, 2022, 46 (12): 2- 11. URL
	ZHU Q M, NI H R, HUA B Y, et al. A survey of UAV millimeter-wave channel measurement and modeling. Mobile Communications, 2022, 46 (12): 2- 11. URL
12	张在琛, 江浩. 智能超表面使能无人机高能效通信信道建模与传输机理分析. 电子学报, 2023, 51 (10): 2623- 2634. doi: 10.12263/DZXB.20221352
	ZHANG Z C, JIANG H. Channel modeling and characteristics analysis for high energy-efficient RIS-assisted UAV communications. Acta Electonica Sinica, 2023, 51 (10): 2623- 2634. doi: 10.12263/DZXB.20221352
13	ZBONTAR J, LECUN Y. Computing the stereo matching cost with a convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 1592-1599.
14	KENDALL A, MARTIROSYAN H, DASGUPTA S, et al. End-to-end learning of geometry and context for deep stereo regression[C]//Proceedings of the IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 66-75.
15	JI M Q, GALL J, ZHENG H T, et al. SurfaceNet: an end-to-end 3D neural network for multiview stereopsis[C]//Proceedings of the IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 2307-2315.
16	KAR A, HÄNE C, MALIK J. Learning a multi-view stereo machine[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 30.
17	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 5998-6008.
18	DING Y K, YUAN W T, ZHU Q T, et al. TransMVSNet: global context-aware multi-view stereo network with transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 8575-8584.
19	LIAO J, DING Y, SHAVIT Y, et al. Wt-MVSNet: window-based transformers for multi-view stereo[C]//Proceedings of Advances in Neural Information Processing Systems. New York, USA: ACM Press, 2022: 8564-8576.
20	SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue: learning feature matching with graph neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 4938-4947.
21	SUN J, SHEN Z, WANG Y, et al. LoFTR: detector-free local feature matching with transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 8922-8931.
22	WANG Q, ZHANG J M, YANG K L, et al. MatchFormer: interleaving attention in Transformers for feature matching[EB/OL]. [2023-11-01]. https://arxiv.org/abs/2203.09645.
23	WANG X F, ZHU Z, HUANG G, et al. MVSTER: epipolar transformer for efficient multi-view stereo[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2022: 573-591.
24	YANG Z P, REN Z L, SHAN Q, et al. MVS2D: efficient multi-view stereo via attention-driven 2D convolutions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 8574-8584.
25	LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 2117-2125.
26	AANAES H, JENSEN R R, VOGIATZIS G, et al. Large-scale data for multiple-view stereopsis. International Journal of Computer Vision, 2016, 120 (6): 153- 168.
27	KNAPITSCH A, PARK J, ZHOU Q Y, et al. Tanks and temples: benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 2017, 36 (4): 1- 13. doi: 10.1145/3072959.3073599
28	YAO, Y, LUO Z X, LI S W, et al. BlendedMVS: a large-scale dataset for generalized multi-view stereo networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 1790-1799.
29	KATHAROPOULOS A. VYAS A, PAPPAS N, et al. Transformers are RNNs: fast autoregressive Transformers with linear attention[EB/OL]. [2023-11-01]. https://arxiv.org/abs/2006.16236v3.
30	YANG J Y, MAO W, ALVAREZ J M. Cost volume pyramid based depth inference for multi-view stereo[EB/OL]. [2023-11-01]. https://arxiv.org/abs/1912.08329.
31	MA X J, GONG Y, WANG Q R, et al. EPP-MVSNet: epipolar-assembling based depth prediction for multi-view stereo[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 5732-5740.
32	WANG F J H, GALLIANI S, VOGEL C, et al. PatchmatchNet: learned multi-view patchmatch stereo[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 14194-14203.
33	XI J F, SHI Y H, WANG Y J, et al. RayMVSNet: learning ray-based 1D implicit fields for accurate multi-view stereo[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 8595-8605.
34	WANG S, LI B, DAI Y. Efficient multi-view stereo by iterative dynamic cost volume[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 8655-8664.
35	PENG R, WANG R J, WANG Z Y, et al. Rethinking depth estimation for multi-view stereo: a unified representation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 8645-8654.
36	ZHANG Z, HU Y, GAO H, et al. Bi-ClueMVSNet: learning bidirectional occlusion clues for multi-view stereo[C]//Proceedings of International Joint Conference on Neural Networks (IJCNN). Washington D. C., USA: IEEE Press, 2023: 1-8.
37	YAN, Q, WANG, Q, ZHAO, K, et al. Rethinking disparity: a depth range free multi-view stereo based on disparity[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l.]: AAAI Press, 2023: 3091-3099.
38	XU G W, WANG X Q, DING X H, et al. Iterative geometry encoding volume for stereo matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2023: 21919-21928.
39	MI Z, DI C, XU D. Generalized binary search network for highly-efficient multiview stereo[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 12991-13000.

[1]	李俊俊, 董建刚, 李坤. 基于Kubernetes的集群节能策略研究[J]. 计算机工程, 2024, 50(9): 82-91.
[2]	林畅, 郭伟, 任哲聪, 金海波. 基于Transformer的目标跟踪与分割统一算法[J]. 计算机工程, 2024, 50(9): 130-141.
[3]	李泽霖, 吕兆峰, 陈富强, 李克. 基于多跳信息融合的实体对齐模型[J]. 计算机工程, 2024, 50(9): 142-152.
[4]	王汝英, 马嘉骏, 董建强, 刘万龙, 张海涛, 尹凯, 赵博超. 基于MTS-BiGRU-DMHSA的工业负荷预测方法[J]. 计算机工程, 2024, 50(9): 169-178.
[5]	朱凯, 李理, 张彤, 江晟, 别一鸣. 基于Transformer的多阶段运动模糊图像修复网络[J]. 计算机工程, 2024, 50(9): 276-285.
[6]	张天鹏, 韩晶, 吕学强. 基于多任务学习的超分辨率辅助小目标检测[J]. 计算机工程, 2024, 50(9): 304-312.
[7]	郭敏, 张熙涵, 李阳. 融合注意力的教师互一致性半监督医学图像分割[J]. 计算机工程, 2024, 50(9): 313-323.
[8]	曾钰琦, 刘博, 钟柏昌, 钟瑾. 智慧教育下基于改进YOLOv8的学生课堂行为检测算法[J]. 计算机工程, 2024, 50(9): 344-355.
[9]	饶日昕, 王怡文, 曾砺志, 童心恬, 赵海涛. 面向废旧电缆检测的轻量化网络模型[J]. 计算机工程, 2024, 50(8): 22-30.
[10]	李华昱, 张智康, 闫阳, 岳阳. 基于知识图谱增强的领域多模态实体识别[J]. 计算机工程, 2024, 50(8): 31-39.
[11]	王蕾, 党时鹏, 潘丰. 基于卷积神经网络的隐匿性旁路预测模型[J]. 计算机工程, 2024, 50(8): 40-49.
[12]	陈瀚, 赵春蕾, 蒋昊达, 王春东. 基于融合模型与语义网络的App用户意图识别研究[J]. 计算机工程, 2024, 50(8): 50-63.
[13]	徐晓滨, 张云硕, 施凡, 常雷雷, 陶志刚. 基于特征匹配度与异类子模型融合的安全性评估方法[J]. 计算机工程, 2024, 50(8): 113-122.
[14]	王夙喆, 张雪英, 陈晓玉, 李凤莲, 吴泽林. 基于有效注意力和GAN结合的脑卒中EEG增强算法[J]. 计算机工程, 2024, 50(8): 336-344.
[15]	王宇, 祁琦, 王纯, 许才. 储能变流器信号高精度故障诊断方法[J]. 计算机工程, 2024, 50(8): 389-396.

选择文件类型/文献管理软件名称

选择包含的内容