Author Login Chief Editor Login Reviewer Login Editor Login Remote Office

Computer Engineering ›› 2026, Vol. 52 ›› Issue (10): 407-417. doi: 10.19678/j.issn.1000-3428.0252069

• Large Language Models and Generative Artificial Intelligence • Previous Articles     Next Articles

Vision-and-Language Navigation in Outdoor Scenes Based on Large Language Model and Exploration Module

LIU Ziyi1,2,3,4, SHA Ying1,2,3,4,*()   

  1. 1. College of Informatics, Huazhong Agricultural University, Wuhan 430070, Hubei, China
    2. Engineering Research Center of Intelligent Technology for Agriculture, Ministry of Education, Wuhan 430070, Hubei, China
    3. Hubei Engineering Technology Research Center of Agricultural Big Data, Wuhan 430070, Hubei, China
    4. Key Laboratory of Smart Farming Technology for Agricultural Animals, Ministry of Agriculture and Rural Affairs, Wuhan 430070, Hubei, China
  • Received:2025-01-20 Revised:2025-04-08 Online:2026-10-15 Published:2025-05-15
  • Contact: SHA Ying

基于大语言模型和探索模块的户外场景视觉语言导航

刘梓熠1,2,3,4, 沙灜1,2,3,4,*()   

  1. 1. 华中农业大学信息学院, 湖北 武汉 430070
    2. 农业智能技术教育部工程研究中心, 湖北 武汉 430070
    3. 湖北省农业大数据工程技术研究中心, 湖北 武汉 430070
    4. 农业农村部智慧养殖技术重点实验室, 湖北 武汉 430070
  • 通讯作者: 沙灜
  • 作者简介:

    刘梓熠, 男, 硕士研究生, 主研方向为人工智能、视觉语言导航

    沙灜(CCF高级会员、通信作者), 教授、博士

  • 基金资助:
    国家自然科学基金面上项目(62272188)

Abstract:

Vision-and-Language Navigation (VLN) tasks aim to guide an agent to move to a target location in a 3D or real-world environment based on language instructions. However, the traditional end-to-end VLN algorithms have certain limitations. When an erroneous action occurs during navigation planning, the agent tends to enter incorrect paths, resulting in an inability to continue following the instructions or exploring unnecessary areas. To address this issue, an agent called Nav-Explore is proposed, which is based on a Large Language Model (LLM) and an exploration module. The proposed agent leverages the reasoning capabilities of the LLM to predict the next action according to the language instructions and current visual information and uses an exploration module to balance exploration and utilization. The exploration module employs an ε-greedy strategy for toggling between the normal navigation and exploration modes. When the random probability is below ε, the agent explores possible future paths to assess the feasibility of the next actions, thus avoiding incorrect decisions. If the probability exceeds ε, it directly uses the output of the LLM for navigation. This modular design enables Nav-Explore to effectively enhance navigation success rates and improve the generalization ability of the agent in unseen environments. The experimental results demonstrate that Nav-Explore achieves superior performance on two outdoor VLN benchmark datasets, Touchdown and Map2seq, significantly increasing navigation success rates. Furthermore, Nav-Explore exhibits strong generalization capabilities, effectively completing navigation tasks in different environments.

Key words: Vision-and-Language Navigation (VLN), Large Language Model (LLM), exploration, multimodal, agent

摘要:

视觉语言导航(VLN) 任务旨在引导智能体根据语言指令在3D或真实环境中移动到目标位置。然而, 传统端到端深度学习VLN算法存在不足, 智能体在导航规划中一旦出现错误动作, 就容易进入错误路径, 导致无法继续遵循指令或探索不必要的区域。为了解决这一问题, 提出一种基于大语言模型(LLM)和探索模块的智能体Nav-Explore。该智能体利用LLM强大的推理能力, 结合语言指令和当前视觉信息预测下一步动作, 并引入探索模块以平衡探索与利用。探索模块通过ε-greedy策略决定智能体在正常导航和探索模式间切换, 当随机概率小于ε时进入探索模式, 智能体通过探索可能的未来路径, 提前评估下一步行动的可行性, 从而有效避免做出错误决策。而当随机概率大于ε时, 智能体直接采用LLM输出的动作进行导航。这种模块化设计使得Nav-Explore方法能够有效地提升导航成功率, 并增强智能体在未见环境中的泛化能力。实验结果表明, Nav-Explore在Touchdown和Map2seq 2个户外环境VLN基准数据集上取得了优异的性能, 显著提升了导航成功率。此外, Nav-Explore也展现出较强的泛化能力, 能够在不同的环境中有效地完成导航任务。

关键词: 视觉语言导航, 大语言模型, 探索, 多模态, 智能体