基于漫反射的被动非视域成像

doi:10.19678/j.issn.1000-3428.0067904

摘要/Abstract

摘要： 非视域(NLOS)成像是一种综合成像和计算重构的技术,指在不直接拍摄场景的情况下通过获取介质上隐藏场景的散射或反射信息对其进行重建。目前的NLOS成像还处于早期发展阶段,场景模型、目标信息重建等尚无系统研究方法。为此,提出一种针对无遮挡、非自发光场景的NLOS成像方法。基于光辐射理论,分析该场景下漫反射面的成像与隐藏物体形状的关系,确定NLOS成像模型与重建目标。使用渲染软件结合运动图像专家组7(MPEG7)数据集,生成符合实际物理意义的漫反射被动非视域全影(DS-NLOS)数据集。构建被动非视域重建网络模型(Re-NLOS),采用视觉Transformer(ViT)结构结合生成式对抗网络(GAN)提取采集的漫反射面图像的全局特征,并恢复隐藏物体形状。在DS-NLOS数据集上的实验结果表明,该方法能够从漫反射面上恢复隐藏物体的形状信息,在测试集20个类别的物体上的峰值信噪比(PSNR)和结构相似性(SSIM)相比漫反射面全影图像平均提高了5.85 dB和0.038 1,对真实室内场景也具有一定的恢复能力。

关键词: 被动非视域成像, 漫反射, 全影图像, 生成式对抗网络, 亮度传输

Abstract: Non-Line-of-Sight (NLOS) imaging, which combines imaging and computational reconstruction, describes the reconstruction of hidden scenes in a medium by capturing scattered or reflected information without directly imaging the scene. NLOS imaging is still in the early stages of its development, and systematic research methods for scene modeling and target information reconstruction are lacking. To address these issues, an NLOS imaging method for unobstructed and non-self-luminous scenes is proposed. Based on optical radiation theory, the relationship between the imaging of diffuse reflection surfaces in the scene and the shape of hidden objects is analyzed to determine the NLOS imaging model and reconstruction targets. A Diffuse reflection full-Shadow passive NLOS (DS-NLOS) dataset that resembles physical reality is generated by combining a rendering software with the Motion Picture Experts Group 7 (MPEG7) dataset . A passive NLOS Reconstruction network model (Re-NLOS) is constructed using a Visual Transformer (ViT) structure in combination with a Generative Adversarial Network (GAN) to extract global features from captured diffuse reflection surface images and recover the shape of hidden objects. Experimental results on the DS-NLOS dataset demonstrate that this method can recover the shape information of hidden objects from diffusely reflected surfaces. In comparison with the diffuse reflection full-shadow images, the average Peak Signal-to-Noise Ratio (PSNR) for 20 object categories in the present test set is increased by 5.85 dB, and the average Structural SIMilarity (SSIM ) is increased by 0.038 1. This method also demonstrates restore capabilities in real indoor scenes.

Key words: passive Non-Line-of-Sight(NLOS) imaging, diffuse reflection, full-shadow image, Generative Adversarial Network(GAN), brightness transfer

中图分类号:

TN911

吴翠翠, 王维东. 基于漫反射的被动非视域成像[J]. 计算机工程, 2024, 50(5): 26-32.

WU Cuicui, WANG Weidong. Passive Non-Line-of-Sight Imaging Based on Diffuse Reflection[J]. Computer Engineering, 2024, 50(5): 26-32.

https://www.ecice06.com/CN/Y2024/V50/I5/26

参考文献

[1] LAURENZIS M, CHRISTNACHER F, VELTEN A. Study of a dual mode SWIR active imaging system for direct imaging and non-line-of-sight vision[EB/OL].[2023-05-11]. https://www.spiedigitallibrary.org/conference-proceedings-of-spie/9465/1/Study-of-a-dual-mode-SWIR-active-imaging-system-for/10.1117/12.2175857.short.
[2] PANDHARKAR R, VELTEN A, BARDAGJY A, et al. Estimating motion and size of moving non-line-of-sight objects in cluttered environments[C]//Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Washington D.C., USA:IEEE Press, 2011:265-272.
[3] GARIEPY G, TONOLINI F, HENDERSON R, et al. Detection and tracking of moving objects hidden from view[J]. Nature Photonics, 2016, 10:23-26.
[4] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words:Transformers for image recognition at scale[EB/OL].[2023-05-11]. https://arxiv.org/abs/2010.11929.
[5] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11):139-144.
[6] METZLER C A, LINDELL D B, WETZSTEIN G. Keyhole imaging:non-line-of-sight imaging and tracking of moving objects along a single optical path[J]. IEEE Transactions on Computational Imaging, 2021, 7:1-12.
[7] TORRALBA A, FREEMAN W T. Accidental pinhole and pinspeck cameras:revealing the scene outside the picture[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA:IEEE Press, 2012:374-381.
[8] 苏晴. 被动非视域半影成像方法的研究[D]. 哈尔滨:哈尔滨工业大学, 2020. SU Q. Research on passive non-line-of-sight penumbra imaging method[D].Harbin:Harbin Institute of Technology, 2020. (in Chinese)
[9] BOUMAN K L, YE V, YEDIDIA A B, et al. Turning corners into cameras:principles and methods[C]//Proceedings of IEEE International Conference on Computer Vision(ICCV). Washington D.C., USA:IEEE Press, 2017:2270-2278.
[10] SEIDEL S W, MURRAY-BRUCE J, MA Y, et al. Two-dimensional non-line-of-sight scene estimation from a single edge occluder[J]. IEEE Transactions on Computational Imaging, 2020, 7:58-72.
[11] TANAKA K, MUKAIGAWA Y, KADAMBI A. Polarized non-line-of-sight imaging[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Washington D.C., USA:IEEE Press, 2020:2136-2145.
[12] BARADAD M, YE V, YEDIDIA A B, et al. Inferring light fields from shadows[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA:IEEE Press, 2018:6267-6275.
[13] SWEDISH T, HENLEY C, RASKAR R. Objects as cameras:estimating high-frequency illumination from shadows[C]//Proceedings of IEEE/CVF International Conference on Computer Vision(ICCV). Washington D.C., USA:IEEE Press, 2021:2593-2602.
[14] DENG R J, JIN X, DU D Y. 3D location and trajectory reconstruction of a moving object behind scattering media[J]. IEEE Transactions on Computational Imaging, 2022, 8:371-384.
[15] SASAKI T, LEGER J R. Non-line-of-sight object location estimation from scattered light using plenoptic data[J]. Journal of the Optical Society of America A:Optics, Image Science, and Vision, 2021, 38(2):211-228.
[16] GENG R X, HU Y, LU Z, et al. Passive non-line-of-sight imaging using optimal transport[J]. IEEE Transactions on Image Processing, 2022, 31:110-124.
[17] ZHENG S S, LIAO M H, WANG F, et al. Non-line-of-sight imaging under white-light illumination:a two-step deep learning approach[J]. Optics Express, 2021, 29(24):40091-40105.
[18] CHEN W Z, DANEAU S, BROSSEAU C, et al. Steady-state non-line-of-sight imaging[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Washington D.C., USA:IEEE Press, 2019:6790-6799.
[19] TANCIK M, SATAT G, RASKAR R. Flash photography for data-driven hidden scene recovery[EB/OL].[2023-05-11]. https://arxiv.org/abs/1810.11710.
[20] 吴术孔. 基于漫反射信息的非视域成像方法的研究[D]. 南京:东南大学, 2021. WU S K. Research on non-line-of-sight imaging method based on diffuse reflection information[D]. Nanjing:Southeast University, 2021. (in Chinese)
[21] 于亭义, 乔木, 刘红林, 等. 基于深度学习的非视域成像[J]. 光学学报, 2019, 39(7):79-85. YU T Y, QIAO M, LIU H L, et al. Non-line-of-sight imaging through deep learning[J]. Acta Optica Sinica, 2019, 39(7):79-85.(in Chinese)
[22] KRSKA W, SEIDEL S W, SAUNDERS C, et al. Double your corners, double your fun:the doorway camera[C]//Proceedings of IEEE International Conference on Computational Photography(ICCP). Washington D.C., USA:IEEE Press, 2022:1-12.
[23] TANCIK M, SWEDISH T, SATAT G, et al. Data-driven non-line-of-sight imaging with a traditional camera[EB/OL].[2023-05-11]. https://opg.optica.org/abstract.cfm?URI=ISA-2018-IW2B.6.
[24] AITTALA M, SHARMA P, MURMANN L, et al. Computational mirrors:blind inverse light transport by deep matrix factorization[EB/OL].[2023-05-11]. https://arxiv.org/abs/1912.02314.
[25] TANCIK M. Non-line-of-sight imaging using data-driven approaches[D]. Cambridge, USA:Massachusetts Institute of Technology, 2018.
[26] 李现国, 李滨. 基于Transformer和多尺度CNN的图像去模糊[J]. 计算机工程, 2023, 49(9):226-233, 245. LI X G, LI B. Image deblurring based on Transformer and multi-scale CNN[J]. Computer Engineering, 2023, 49(9):226-233, 245.(in Chinese)
[27] 缪斯, 祝永新. 针对图像盲去模糊的可微分神经网络架构搜索方法[J]. 计算机工程, 2021, 47(9):313-320. MIAO S, ZHU Y X. Differentiable neural architecture search method for blind image deblurring[J]. Computer Engineering, 2021, 47(9):313-320.(in Chinese)
[28] GULRAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of Wasserstein GANs[EB/OL].[2023-05-11]. https://arxiv.org/abs/1704.00028.
[29] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].[2023-05-11]. https://arxiv.org/abs/1409.1556.

选择文件类型/文献管理软件名称

选择包含的内容