作者投稿和查稿 主编审稿 专家审稿 编委审稿 远程编辑

计算机工程 ›› 2022, Vol. 48 ›› Issue (7): 177-188. doi: 10.19678/j.issn.1000-3428.0061698

• 体系结构与软件技术 • 上一篇    下一篇

支持多模态交互的桌面增强显示系统

王子珩1,2, 姜忠鼎1,2   

  1. 1. 复旦大学 软件学院, 上海 200438;
    2. 复旦大学 上海市数据科学重点实验室, 上海 200438
  • 收稿日期:2021-05-20 修回日期:2021-07-14 出版日期:2022-07-15 发布日期:2021-07-21
  • 作者简介:王子珩(1996—),男,硕士,主研方向为计算机图形学、人机交互、增强现实;姜忠鼎(通信作者),副教授、博士。
  • 基金资助:
    教育部第二批“新工程”研究与实践项目(E-XTYR20200621)。

Enhanced Desktop Display System Supporting Multi-Modal Interaction

WANG Ziheng1,2, JIANG Zhongding1,2   

  1. 1. Software School, Fudan University, Shanghai 200438, China;
    2. Shanghai Key Laboratory of Data Science, Fudan University, Shanghai 200438, China
  • Received:2021-05-20 Revised:2021-07-14 Online:2022-07-15 Published:2021-07-21

摘要: 传统的显示设备在受限的物理空间内难以向用户呈现大量画面和复杂内容,而AR头戴式显示设备通过将三维的可视化内容悬浮显示在用户眼前,在不占用额外物理空间的条件下可增强真实世界的画面显示,呈现形式更为丰富的内容。设计AR虚拟空间与真实电脑画面虚实融合的桌面增强显示系统。通过基于二维码识别的空间定位技术将真实电脑画面映射至虚拟空间内,实现交互空间的统一,同时构建窗口布局计算模型使得系统可以根据用户自定义参数自动生成窗口并设置其布局。在此基础上,利用蓝牙通信、网络传输、操作系统底层映射和结合视线检测的语音识别等技术支持手势、键鼠和语音的多模态交互方式,设计鼠标移动策略以扩展鼠标在三维空间下的多种操作模式。实验结果表明,与隔空手势交互、鼠标交互等传统交互方式相比,该系统在处理常见电脑任务时平均耗时节省10%~30%,具有较高的交互效率,且在跨窗口连续移动和瞬间跳转时能够正确显示鼠标位置。

关键词: 头戴式显示设备, 增强显示, 虚实融合, 多模态交互, 人机交互

Abstract: Traditional display devices are limited in their ability to present a large amount of complex content to users with in a given physical space.In contrast, Augmented Reality(AR) Head Mounted Display(HMD) devices can enhance users' perception of the real world by overlaying a three-dimensional visualization on their field of view to display richer content in several forms without occupying additional physical space.This study presents the design of an enhanced desktop display system that integrates virtual content in AR space with real screen.Through spatial positioning technology based on QR code recognition, real computer screens are mapped to aunified virtual interactive space.By establishing a window layout calculation model, the system can automatically generate windows and set their layouts according to user-defined parameters.The system supports multi-modal gesture interaction modes including keyboard and mouse input as well as voice control through technologies such as Bluetooth communication, network transmission, an underlying mapping of the operating system, and voice recognition based on gesture detection.The mouse movement strategy of the system is designed to expand the various operation modes of standard mouse devices into a three-dimensional space.The results of experiments show that users are able to complete common tasks with the proposed system in about 10%~30% less time on average compared with traditional input methods such as air gesture-based and mouse interaction.Moreover, the proposed approach exhibited a high interaction efficiency, and is able to display the cursor position correctly when moving across windows continuously and jumping instantaneously.

Key words: Head Mounted Display(HMD) device, enhanced display, integration of virtuality and reality, multi-modal interaction, human-computer interaction

中图分类号: