Enhanced Desktop Display System Supporting Multi-Modal Interaction

doi:10.19678/j.issn.1000-3428.0061698

Abstract

Abstract: Traditional display devices are limited in their ability to present a large amount of complex content to users with in a given physical space.In contrast, Augmented Reality(AR) Head Mounted Display(HMD) devices can enhance users' perception of the real world by overlaying a three-dimensional visualization on their field of view to display richer content in several forms without occupying additional physical space.This study presents the design of an enhanced desktop display system that integrates virtual content in AR space with real screen.Through spatial positioning technology based on QR code recognition, real computer screens are mapped to aunified virtual interactive space.By establishing a window layout calculation model, the system can automatically generate windows and set their layouts according to user-defined parameters.The system supports multi-modal gesture interaction modes including keyboard and mouse input as well as voice control through technologies such as Bluetooth communication, network transmission, an underlying mapping of the operating system, and voice recognition based on gesture detection.The mouse movement strategy of the system is designed to expand the various operation modes of standard mouse devices into a three-dimensional space.The results of experiments show that users are able to complete common tasks with the proposed system in about 10%~30% less time on average compared with traditional input methods such as air gesture-based and mouse interaction.Moreover, the proposed approach exhibited a high interaction efficiency, and is able to display the cursor position correctly when moving across windows continuously and jumping instantaneously.

Key words: Head Mounted Display(HMD) device, enhanced display, integration of virtuality and reality, multi-modal interaction, human-computer interaction

摘要： 传统的显示设备在受限的物理空间内难以向用户呈现大量画面和复杂内容，而AR头戴式显示设备通过将三维的可视化内容悬浮显示在用户眼前，在不占用额外物理空间的条件下可增强真实世界的画面显示，呈现形式更为丰富的内容。设计AR虚拟空间与真实电脑画面虚实融合的桌面增强显示系统。通过基于二维码识别的空间定位技术将真实电脑画面映射至虚拟空间内，实现交互空间的统一，同时构建窗口布局计算模型使得系统可以根据用户自定义参数自动生成窗口并设置其布局。在此基础上，利用蓝牙通信、网络传输、操作系统底层映射和结合视线检测的语音识别等技术支持手势、键鼠和语音的多模态交互方式，设计鼠标移动策略以扩展鼠标在三维空间下的多种操作模式。实验结果表明，与隔空手势交互、鼠标交互等传统交互方式相比，该系统在处理常见电脑任务时平均耗时节省10%~30%，具有较高的交互效率，且在跨窗口连续移动和瞬间跳转时能够正确显示鼠标位置。

关键词: 头戴式显示设备, 增强显示, 虚实融合, 多模态交互, 人机交互

CLC Number:

TP311

WANG Ziheng, JIANG Zhongding. Enhanced Desktop Display System Supporting Multi-Modal Interaction[J]. Computer Engineering, 2022, 48(7): 177-188.

王子珩, 姜忠鼎. 支持多模态交互的桌面增强显示系统[J]. 计算机工程, 2022, 48(7): 177-188.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0061698

http://www.ecice06.com/EN/Y2022/V48/I7/177

Figures/Tables 24

References

[1] BI X J, BALAKRISHNAN R.Comparing usage of a large high-resolution display to single or dual desktop displays for daily work[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.New York, USA:ACM Press, 2009:1005-1014.
[2] ENS B M, FINNEGAN R, IRANI P P.The Personal Cockpit:a spatial interface for effective task switching on head-worn displays[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.New York, USA:ACM Press, 2014:3171-3180.
[3] ENS B, IRANI P.Spatial analytic interfaces:spatial user interfaces for in situ visual analytics[J].IEEE Computer Graphics and Applications, 2017, 37(2):66-79.
[4] BELLGARDT M, PICK S, ZIELASKO D, et al.Utilizing immersive virtual reality in everydaywork[C]//Proceedings of the 3rd Workshop on Everyday Virtual Reality.Washington D.C., USA:IEEE Press, 2017:1-4.
[5] FEREYDOONI N, WALKER B N.Virtual reality as aremote workspace platform:opportunities and challenges[C]//Proceedings of Microsoft New Future of Work Virtual Symposium.New York, USA:[s.n.], 2020:1-8.
[6] OFEK E, GRUBERT J, PAHUD M, et al.Towards a practical virtual office for mobile knowledge workers[EB/OL].[2021-04-15].https://arxiv.org/abs/2009. 02947.
[7] GRUBERT J, OFEK E, PAHUD M, et al.The office of the future:virtual, portable, and global[J].IEEE Computer Graphics and Applications, 2018, 38(6):125-133.
[8] Facebook Company.Facebook Infinite Office[EB/OL].[2021-04-15].https://www.oculus.com/blog/facebook-connect-oculus-quest-2-aaa-gaming-partnerships-and-more/.
[9] REKIMOTO J, SAITOH M.Augmented surfaces:a spatially continuous work space for hybrid computing environments[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.New York, USA:ACM Press, 1999:378-385.
[10] Microsoft Company.Mirage:Virtual Monitors[EB/OL].[2021-04-15].https://www.mirage-app.com/.
[11] REIPSCHLAGER P, FLEMISCH T, DACHSELT R.Personal augmented reality for information visualization on large interactive displays[J].IEEE Transactions on Visualization and Computer Graphics, 2020, 27(2):1182-1192.
[12] BUTSCHER S, HUBENSCHMID S, MÜLLER J, et al.Clusters, trends, and outliers:how immersive technologies can facilitate the collaborative analysis of multidimensional data[C]//Proceedings of CHI Conference on Human Factors in Computing Systems.New York, USA:ACM Press, 2018:1-12.
[13] LI Z, ANNETT M, HINCKLEY K, et al.HoloDoc:enabling mixed reality workspaces that harness physical and digital content[C]//Proceedings of CHI Conference on Human Factors in Computing Systems.New York, USA:ACM Press, 2019:1-14.
[14] REN J, WENG Y T, ZHOU C C, et al.Understanding window management interactions in AR headset+ smartphone interface[C]//Proceedings of Extended Abstracts of CHI Conference on Human Factors in Computing Systems.New York, USA:ACM Press, 2020:1-8.
[15] ZHU F Y, GROSSMAN T.BISHARE:exploring bidirectional interactions between smartphones and head-mounted augmented reality[C]//Proceedings of CHI Conference on Human Factors in Computing Systems.New York, USA:ACM Press, 2020:1-14.
[16] NORMAND E, MCGUFFIN M J.Enlarging a smartphone with AR to create a handheld VESAD (virtually extended screen-aligned display)[C]//Proceedings of IEEE International Symposium on Mixed and Augmented Reality.Washington D.C., USA:IEEE Press, 2018:123-133.
[17] BIENER V, SCHNEIDER D, GESSLEIN T, et al.Breaking the screen:interaction across touchscreen boundaries in virtual reality for mobile knowledge workers[J].IEEE Transactions on Visualization and Computer Graphics, 2020, 26(12):3490-3502.
[18] SERRANO M, ENS B, YANG X D, et al.Gluey:developing a head-worn display interface to unify the interaction experience in distributed display environments[C]//Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services.New York, USA:ACM Press, 2015:161-171.
[19] PAVANATTO L, NORTH C, BOWMAN D A, et al.Do we still need physical monitors?an evaluation of the usability of AR virtual monitors for productivity work[C]//Proceedings of IEEE Virtual Reality and 3D User Interfaces.Washington D.C., USA:IEEE Press, 2021:759-767.
[20] Microsoft Company.MRTK[EB/OL].[2021-04-15].https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/?view=mrtk unity-2021-04.
[21] Microsoft Company.QR code tracking[EB/OL].[2021-04-15].https://docs.microsoft.com/en-us/windows/mixed-reality/develop/platform-capabilities-and-apis/qr-code-tracking.
[22] ENS B, HINCAPIÉ-RAMOS J D, IRANI P.Ethereal planes:a design framework for 2D information space in 3D mixed reality environments[C]//Proceedings of the 2nd ACM Symposium on Spatial User Interaction.New York, USA:ACM Press, 2014:2-12.
[23] BENKO H, FEINER S.Multi-monitor mouse[C]//Proceedings of CHI'05 Extended Abstracts on Human Factors in Computing Systems.New York, USA:ACM Press, 2005:1208-1211.

Please choose a citation manager

Content to export