基于改进Vision Transformer的局部光照一致性估计

doi:10.19678/j.issn.1000-3428.0068905

摘要/Abstract

摘要：

光照一致性是增强现实(AR)系统中实现虚实有机融合的关键因素之一。由于拍摄视角的局限性和场景光照的复杂性, 开发者在估计全景照明信息时通常忽略局部光照一致性, 从而影响最终的渲染效果。为解决这一问题, 提出一种基于改进视觉Transformer(ViT)结构的局部光照一致性估计框架(ViTLight)。首先利用ViT编码器提取特征向量并计算回归球面谐波(SH)系数, 进而恢复光照信息; 其次改进ViT编码器结构, 引入多头自注意力交互机制, 采用卷积运算引导注意力头之间相互联系, 在此基础上增加局部感知模块, 扫描每个图像分块并对局部像素进行加权求和, 捕捉区域内的特定特征, 有助于平衡全局上下文特征和局部光照信息, 提高光照估计的精度。在公开数据集上对比主流特征提取网络和4种经典光照估计框架, 实验和分析结果表明, ViTLight在图像渲染准确率方面高于现有框架, 其均方根误差(RMSE)和结构相异性(DSSIM)指标分别为0.129 6和0.042 6, 验证了该框架的有效性与正确性。

关键词: 增强现实, 光照估计, 球面谐波系数, 视觉Transformer, 多头自注意力

Abstract:

Illumination consistency is a key factor in achieving the organic fusion of virtual and real elements in Augmented Reality (AR) systems. Owing to the constraints of capture perspectives and the complexity of scene illumination, developers often overlook local illumination consistency when estimating panoramic lighting information, thereby affecting the final rendering quality. To address this issue, this study proposes a local illumination consistency estimation framework, ViTLight, based on an improved Vision Transformer (ViT) structure. First, the framework uses a ViT encoder to extract feature vectors and calculate regression Spherical Harmonic (SH) coefficients, then recovers illumination information. Second, the ViT encoder structure is enhanced by introducing a multi-head self-attention interaction mechanism. Convolution operation guides the interplay between attention heads. Additionally, a local perception module is integrated to actively scan each image block and perform weighted summation on local pixels to capture specific features within regions. This proactive approach balances global contextual features and local illumination information, ultimately improving the precision of illumination estimation. The mainstream feature extraction network and four classical illumination estimation frameworks are compared on public datasets. The experimental results and analysis indicate that ViTLight is superior to existing frameworks in terms of image rendering accuracy, and its Root Mean Square Error (RMSE) and Structural Dissimilarity (DSSIM) index reach 0.129 6 and 0.042 6, respectively, which verifies its effectiveness and correctness.

Key words: Augmented Reality (AR), illumination estimation, Spherical Harmonics (SH) coefficient, Vision Transformer (ViT), multi-head self-attention

王杨, 宋世佳, 王鹤琴, 袁振羽, 赵立军, 吴其林. 基于改进Vision Transformer的局部光照一致性估计[J]. 计算机工程, 2025, 51(2): 312-321.

WANG Yang, SONG Shijia, WANG Heqin, YUAN Zhenyu, ZHAO Lijun, WU Qilin. Estimation of Local Illumination Consistency Based on Improved Vision Transformer[J]. Computer Engineering, 2025, 51(2): 312-321.

https://www.ecice06.com/CN/Y2025/V51/I2/312

图/表 16

图1 ViTLight光照估计框架

Fig.1 ViTLight illumination estimation framework

图2 改进ViT的模型结构

Fig.2 Structure of improved ViT model

图3 多头自注意力交互机制

Fig.3 Multi-head self-attention interaction mechanism

图4 局部特征编码器结构

Fig.4 Structure of local feature encoder

图5 不同注意力头个数下的指标变化

Fig.5 Changes of indicators under different numbers of attention heads

图6 模型改进前后效果

Fig.6 Effect before and after model improvement

图7 LPConv位置对模型性能的影响

Fig.7 The impact of LPConv location on model performance

图8 网络消融可视化分析

Fig.8 Visualization analysis of network ablation

图9 不同网络模型的光照估计效果

Fig.9 The effectiveness of illumination estimation using different network models

图10 不同网络的评价指标变化

Fig.10 Changes in evaluation indicators of different networks

参考文献 33

1	刘万奎, 刘越. 用于增强现实的光照估计研究综述. 计算机辅助设计与图形学学报, 2016, 28 (2): 197- 207. doi: 10.3969/j.issn.1003-9775.2016.02.001
	LIU W K , LIU Y . A review of illumination estimation for augmented reality. Journal of Computer-Aided Design & Computer Graphics, 2016, 28 (2): 197- 207. doi: 10.3969/j.issn.1003-9775.2016.02.001
2	MOHAMMADKHORASANI A , MALEK K , MOJIDRA R , et al. Augmented reality-computer vision combination for automatic fatigue crack detection and localization. Computers in Industry, 2023, 149, 103936. doi: 10.1016/j.compind.2023.103936
3	CAO J , LAM K Y , LEE L H , et al. Mobile augmented reality: user interfaces, frameworks, and intelligence. ACM Computing Surveys, 2023, 55 (9): 1- 36.
4	滕嘉玮, 赵岩, 张艾嘉, 等. 基于二维仿射变换的几何一致性虚实融合. 光学精密工程, 2022, 30 (11): 1374- 1382. doi: 10.37188/OPE.20223011.1374
	TENG J W , ZHAO Y , ZHANG A J , et al. Virtual-real fusion with geometric consistency based on two-dimensional affine transformation. Optics and Precision Engineering, 2022, 30 (11): 1374- 1382. doi: 10.37188/OPE.20223011.1374
5	LEGENDRE C, MA W C, FYFFE G, et al. DeepLight: learning illumination for unconstrained mobile mixed reality[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2019: 5918-5928.
6	BHATT D , PATEL C , TALSANIA H , et al. CNN variants for computer vision: history, architecture, application, challenges and future scope. Electronics, 2021, 10 (20): 2470. doi: 10.3390/electronics10202470
7	GARON M, SUNKAVALLI K, HADAP S, et al. Fast spatially-varying indoor lighting estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2019: 6908-6917.
8	ZHU Y J, ZHANG Y D, LI S, et al. Spatially-varying outdoor lighting estimation from intrinsics[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2021: 12834-12842.
9	DONG S , WANG P , ABBAS K . A survey on deep learning and its applications. Computer Science Review, 2021, 40, 100379. doi: 10.1016/j.cosrev.2021.100379
10	SONG S R, FUNKHOUSER T. Neural illumination: lighting prediction for indoor environments[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2019: 6918-6926.
11	WANG G C, YANG Y N, LOY C C, et al. StyleLight: HDR panorama generation for lighting estimation and editing[EB/OL]. [2023-10-05]. https://arxiv.org/abs/2207.14811.
12	GARDNER M A , SUNKAVALLI K , YUMER E , et al. Learning to predict indoor illumination from a single image. ACM Transactions on Graphics, 2017, 36 (6): 1- 14.
13	曹天池, 李秀实, 李丹, 等. 基于图像分解的光照估计算法. 计算机工程与科学, 2021, 43 (8): 1422- 1428. doi: 10.3969/j.issn.1007-130X.2021.08.011
	CAO T C , LI X S , LI D , et al. Illumination estimation based on image decomposition. Computer Engineering & Science, 2021, 43 (8): 1422- 1428. doi: 10.3969/j.issn.1007-130X.2021.08.011
14	SOMANATH G, KURZ D. HDR environment map estimation for real-time augmented reality[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2021: 11298-11306.
15	HOLD-GEOFFROY Y, SUNKAVALLI K, HADAP S, et al. Deep outdoor illumination estimation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2017: 7312-7321.
16	ZHAO J , CHALMERS A , RHEE T . Adaptive light estimation using dynamic filtering for diverse lighting conditions. IEEE Transactions on Visualization and Computer Graphics, 2021, 27 (11): 4097- 4106. doi: 10.1109/TVCG.2021.3106497
17	CHENG H , XU C , WANG J , et al. Fast and accurate illumination estimation using LDR panoramic images for realistic rendering. IEEE Transactions on Visualization and Computer Graphics, 2023, 29 (12): 5235- 5249. doi: 10.1109/TVCG.2022.3205614
18	吴广运, 周治平. 基于阴影检测的增强现实光照一致性实现. 激光与光电子学进展, 2022, 59 (2): 350- 355.
	WU G Y , ZHOU Z P . Realizing illumination consistency in augmented reality based on shadow detection. Laser & Optoelectronics Progress, 2022, 59 (2): 350- 355.
19	CHENG D C , SHI J , CHEN Y Y , et al. Learning scene illumination by pairwise photos from rear and front mobile cameras. Computer Graphics Forum, 2018, 37 (7): 213- 221. doi: 10.1111/cgf.13561
20	SUN Y K, LI D, LIU S, et al. Learning illumination from a limited field-of-view image[C]//Proceedings of the IEEE International Conference on Multimedia & Expo Workshops. Washington D.C., USA: IEEE Press, 2020: 1-6.
21	LI Z Q , YU L , OKUNEV M , et al. Spatiotemporally consistent HDR indoor lighting estimation. ACM Transactions on Graphics, 2023, 42 (3): 1- 15.
22	LIU C L , WANG L Y , LI Z , et al. Real-time lighting estimation for augmented reality via differentiable screen-space rendering. IEEE Transactions on Visualization and Computer Graphics, 2022, 29 (4): 2132- 2145.
23	ZHANG A J , ZHAO Y , WANG S G . An improved augmented-reality framework for differential rendering beyond the lambertian-world assumption. IEEE Transactions on Visualization and Computer Graphics, 2020, 27 (12): 4374- 4386.
24	LI M T, GUO J, CUI X F, et al. Deep spherical Gaussian illumination estimation for indoor scene[C]//Proceedings of the ACM Multimedia Asia. New York, USA: ACM Press, 2019: 1-6.
25	ZHAN F N , ZHANG C G , YU Y C , et al. EMLight: lighting estimation via spherical distribution approximation. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35 (4): 3287- 3295. doi: 10.1609/aaai.v35i4.16440
26	赵宏, 陈志文, 郭岚, 等. 基于ViT与语义引导的视频内容描述生成. 计算机工程, 2023, 49 (5): 247- 254. URL
	ZHAO H , CHEN Z W , GUO L , et al. Video content caption generation based on ViT and semantic guidance. Computer Engineering, 2023, 49 (5): 247- 254. URL
27	HAN K , WANG Y H , CHEN H T , et al. A survey on Vision Transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45 (1): 87- 110. doi: 10.1109/TPAMI.2022.3152247
28	CHAUDHARI S, MITHAL V, POLATKAN G, et al. An attentive survey of attention models[EB/OL]. [2023-10-05]. http://arxiv.org/abs/1904.02874v3.
29	ZHANG Q , YANG Y B . ResT: an efficient transformer for visual recognition. Advances in Neural Information Processing Systems, 2021, 34, 15475- 15485.
30	GREEN R. Spherical harmonic lighting: the gritty details[EB/OL]. [2023-10-05]. https://www.cse.chalmers.se/~uffe/xjobb/Readings/GlobalIllumination/Spherical%20Harmonic%20Lighting%20-%20the%20gritty%20details.pdf.
31	HOWARD A, SANDLER M, CHEN B, et al. Searching for MobileNetV3[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2019: 1314-1324.
32	ZHANG X Y, ZHOU X Y, LIN M X, et al. ShuffleNet: an extremely efficient convolutional neural network for mobile devices[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 6848-6856.
33	HAN K, WANG Y H, TIAN Q, et al. GhostNet: more features from cheap operations[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2020: 1580-1589.

[1]	蔡俊民, 梁正友, 孙宇, 陈子奥. 基于可变形三维图卷积的轻量级点云分类研究[J]. 计算机工程, 2024, 50(9): 255-265.
[2]	陈瀚, 赵春蕾, 蒋昊达, 王春东. 基于融合模型与语义网络的App用户意图识别研究[J]. 计算机工程, 2024, 50(8): 50-63.
[3]	刘瑞康, 刘伟铭, 段梦飞, 谢玮, 戴愿. 基于双通道Transformer的地铁站台异物检测[J]. 计算机工程, 2024, 50(4): 197-207.
[4]	曹广硕, 黄瑞章, 陈艳平, 秦永彬. 基于多模态学习的乳腺癌生存预测研究[J]. 计算机工程, 2024, 50(1): 296-305.
[5]	李现国, 李滨. 基于Transformer和多尺度CNN的图像去模糊[J]. 计算机工程, 2023, 49(9): 226-233, 245.
[6]	宋煜, 张帅, 严永辉, 钱柱中. 基于冗余任务消减的边缘应用性能优化[J]. 计算机工程, 2021, 47(3): 209-217,226.
[7]	赵亚南, 刘渊, 宋设. 融合多头自注意力机制的金融新闻极性分析[J]. 计算机工程, 2020, 46(8): 85-92.
[8]	李一, 冯楠, 谭顺成. 基于KCF与改进ORB的增强现实方法[J]. 计算机工程, 2019, 45(8): 230-235.
[9]	赵越,李晶皎,王爱侠,杨丹. 基于IEKF-SLAM的未知场景增强现实跟踪注册算法[J]. 计算机工程, 2016, 42(1): 272-277.
[10]	孙博文，邱子鉴，沈斌，张艳鹏. 基于方向信息的随机蕨特征匹配算法[J]. 计算机工程, 2014, 40(5): 192-195,202.
[11]	叶聪丽，陈一民，黄晨，马德宜，李启明，陆壬淼. 视频透视增强现实系统中的运动模糊研究[J]. 计算机工程, 2013, 39(9): 267-270.
[12]	张皓，王涌天，陈靖，刘越. 基于LBS的移动增强实境浏览器[J]. 计算机工程, 2013, 39(9): 311-316.
[13]	罗斌, 葛双全. 支持装配引导的单手指手势交互方法[J]. 计算机工程, 2013, 39(8): 243-248.
[14]	黄晨, 陈一民, 徐升, 王曦晨, 李芸. 增强现实系统中大范围磁跟踪注册问题研究[J]. 计算机工程, 2012, 38(14): 290-292.
[15]	翟亮亮, 王涌天, 杨健, 刘越. 基于ARcode的移动增强现实系统研究[J]. 计算机工程, 2012, 38(10): 247-249.

选择文件类型/文献管理软件名称

选择包含的内容