Extraction of Typical Scene Elements from Street View Images Based on Large Multimodal Models

doi:10.19678/j.issn.1000-3428.0070469

Abstract

Abstract:

Scene elements are fundamental for understanding urban geographic information, and their accurate extraction is essential for smart city development and geographic information systems. To address the complexity of street-view images, the limitations of existing deep learning models in interpreting complex scenes, and challenges in associating visual data with context, a method based on large multimodal models for extracting typical scene elements from street-view images is proposed. First, the approach extends the Large Language and Vision Assistant (LLaVA) by integrating a multilayer perceptron and a high-resolution visual encoder to create GeoLLaVA. Second, a Street View Visual-Instruction-Following Dataset is constructed for scene element extraction tasks to provide multidimensional instructions. The model is fine-tuned using visual instructions to enhance contextual understanding. Low-Rank Adaptation (LoRA) is used to optimize the computational efficiency. Finally, GeoLLaVA generates multidimensional scene descriptions from street-view images and extracts key element keywords for effective scene-element extraction. In comparative experiments using semantic segmentation, object detection, and other multimodal models, GeoLLaVA demonstrates significant advantages, achieving F1 scores of 0.938, 0.842, and 0.829 for the extraction of traffic signals, intersections, and parking lots, respectively. A comparison between the model before and after fine-tuning clearly demonstrates the effectiveness of the fine-tuning process. Ablation studies further validate the performance improvements achieved by the modified GeoLLaVA architecture, and LoRA effectively reduces computational resource consumption. Regional application experiments using batch inference on street view images with geographic coordinates and a comparison with OpenStreetMap (OSM) data not only confirm the accuracy of the model but also highlight the limitations of OSM data in providing comprehensive element information.

Key words: Large Multimodal Model (LMM), typical scene elements, Large Language and Vision Assistant (LLaVA), street view images, Low-Rank Adaptation (LoRA)

摘要：

场景要素是理解城市地理信息的核心, 准确提取场景要素对于智慧城市建设和地理信息系统发展至关重要。为应对街景图像场景的复杂性、现有视觉深度学习模型在理解复杂场景和要素方面的局限性, 以及视觉信息与上下文关联的挑战, 提出了一种基于大型多模态模型(LMM)的典型街景场景要素提取方法。首先, 基于LLaVA(Large Language and Vision Assistant)模型引入多层感知机(MLP)和高分辨率视觉编码器, 构建GeoLLaVA模型; 其次, 针对街景场景要素提取任务构建街景视觉-指令跟随数据集, 提供多维度指令, 通过视觉指令微调模型, 增强其对复杂街景场景的上下文理解, 同时, 引入低秩自适应(LoRA)技术降低计算资源需求; 最后, 通过GeoLLaVA模型生成街景图像的多维度场景描述, 并提取关键词以获得典型场景要素。在与语义分割、目标检测及其他多模态模型的对比实验中, GeoLLaVA表现出了显著优势, 在交通信号灯、交叉路口和停车场要素提取任务中分别取得了0.938、0.842和0.829的F1值。模型微调前后的对比展现了微调的有效性。消融实验进一步验证了GeoLLaVA改进结构对性能提升的贡献以及LoRA在降低计算资源消耗方面的有效性。区域应用实验通过批量推理特定区域的街景图像, 提取要素并结合地理位置进行可视化展示, 与开放街景地图(OSM)数据对比, 验证了模型的准确性并揭示了OSM在提供要素信息方面的不足。

关键词: 大型多模态模型, 典型场景要素, LLaVA模型, 街景图像, 低秩自适应

PAN Keyue, GUO Wei, CHENG Xiang, LIU Yi. Extraction of Typical Scene Elements from Street View Images Based on Large Multimodal Models[J]. Computer Engineering, 2026, 52(7): 331-344.

潘可悦, 呙维, 程湘, 刘异. 基于大型多模态模型的街景图像典型场景要素提取[J]. 计算机工程, 2026, 52(7): 331-344.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0070469

https://www.ecice06.com/EN/Y2026/V52/I7/331

Figures/Tables 11

Fig.1 GeoLLaVA architecture diagram

Fig.2 Street view — instruction following dataset

Fig.3 LoRA method training process

Fig.4 Pre-processing results for street view panorama data

Fig.5 Street view image description information generated by GeoLLaVA

Fig.6 Distribution of elements extracted by GeoLLaVA and description of a specific location

Fig.7 Visualization of scene elements distribution and comparison with OSM results

References 51

1	HJELM R D, FEDOROV A, LAVOIE-MARCHILDON S, et al. Learning deep representations by mutual information estimation and maximization[EB/OL]. [2024-12-19]. https://arxiv.org/abs/1808.06670.
2	张丽英, 裴韬, 陈宜金, 等. 基于街景图像的城市环境评价研究综述. 地球信息科学学报, 2019, 21 (1): 46- 58.
	ZHANG L Y , PEI T , CHEN Y J , et al. A review of urban environmental assessment based on street view images. Journal of Geo-Information Science, 2019, 21 (1): 46- 58.
3	DOERSCH C , SINGH S , GUPTA A , et al. What makes Paris look like Paris?. Communications of the ACM, 2015, 58 (12): 103- 110. doi: 10.1145/2830541
4	NGUYEN Q C , SAJJADI M , MCCULLOUGH M , et al. Neighbourhood looking glass: 360° automated characterisation of the built environment for neighbourhood effects research. Journal of Epidemiology and Community Health, 2018, 72 (3): 260- 266. doi: 10.1136/jech-2017-209456
5	ZHANG F , ZHANG D , LIU Y , et al. Representing place locales using scene elements. Computers, Environment and Urban Systems, 2018, 71, 153- 164. doi: 10.1016/j.compenvurbsys.2018.05.005
6	DEWI C , CHEN R C , ZHUANG Y C , et al. Image enhancement method utilizing YOLO models to recognize road markings at night. IEEE Access, 2024, 12, 131065- 131081. doi: 10.1109/ACCESS.2024.3440253
7	ZHAO H S, SHI J P, QI X J, et al. Pyramid scene parsing network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE Press, 2017: 6230-6239.
8	LIANG X C , ZHAO T H , BILJECKI F . Revealing spatio-temporal evolution of urban visual environments with street view imagery. Landscape and Urban Planning, 2023, 237, 104802. doi: 10.1016/j.landurbplan.2023.104802
9	XU S X, ZHANG C H, FAN L B, et al. AddressCLIP: empowering vision-language models for city-wide image address localization[C]//Proceedings of ECCV 2024. Berlin, Germany: Springer, 2025: 76-92.
10	NGIAM J, KHOSLA A, KIM M, et al. Multimodal deep learning[C]//Proceedings of the International Conference on Machine Learning (ICML). [S. l. ]: PMLR, 2011: 689-696.
11	HE W T , MA H J , LI S H , et al. Using augmented small multimodal models to guide large language models for multimodal relation extraction. Applied Sciences, 2023, 13 (22): 12208. doi: 10.3390/app132212208
12	OUYANG T J, ZHANG X, HAN Z Y, et al. Health CLIP: depression rate prediction using health related features in satellite and street view images[C]//Proceedings of the ACM Web Conference 2024. New York, USA: ACM Press, 2024: 1142-1145.
13	OPENAI, ACHIAM J, ADLER S, et al. GPT-4 technical report[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2303.08774.
14	LEE Y J , LI C Y , LIU H T , et al. Visual instruction tuning. Advances in Neural Information Processing Systems, 2023, 23, 34892- 34916.
15	ZHANG Y , ZHANG F , CHEN N C . Migratable urban street scene sensing method based on vision language pre-trained model. International Journal of Applied Earth Observation and Geoinformation, 2022, 113, 102989. doi: 10.1016/j.jag.2022.102989
16	JI Y H, GAO S. Evaluating the effectiveness of large language models in representing textual descriptions of geometry and spatial relations[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2307.03678https://arxiv.org/abs/2307.03678.
17	CHIANG W L, LI Z, LIN Z, et al. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality[EB/OL]. [2024-12-19]. https://lmsys.org/blog/2023-03-30-vicuna.
18	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of International Conference on Machine Learning. [S. l. ]: PMLR, 2021: 8748-8763.
19	HU E J, SHEN Y L, WALLIS P, et al. LoRA: low-rank adaptation of large language models[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2106.09685.
20	BAI Y , ZHAO Y , SHAO Y J , et al. Deep learning in different remote sensing image categories and applications: status and prospects. International Journal of Remote Sensing, 2022, 43 (5): 1800- 1847. doi: 10.1080/01431161.2022.2048319
21	徐永智. 基于街景影像的建筑物底部轮廓提取[D]. 北京: 北京建筑大学, 2017.
	XU Y Z. Extracting building footprints from digital measurable images[D]. Beijing: Beijing University of Civil Engineering and Architecture, 2017. (in Chinese)
22	LU J S, YANG J W, BATRA D, et al. Neural baby talk[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE Press, 2018: 7219-7228.
23	CAMPBELL A , BOTH A , SUN Q . Detecting and mapping traffic signs from Google Street View images using deep learning and GIS. Computers, Environment and Urban Systems, 2019, 77, 101350. doi: 10.1016/j.compenvurbsys.2019.101350
24	QIU W S , ZHANG Z Y , LIU X , et al. Subjective or objective measures of street environment, which are more effective in explaining housing prices?. Landscape and Urban Planning, 2022, 221, 104358. doi: 10.1016/j.landurbplan.2022.104358
25	CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[EB/OL]. [2024-12-19]. https://arxiv.org/abs/1706.05587.
26	SHIHAB I F, BHAGAT S R, SHARMA A. Precise and robust sidewalk detection: leveraging ensemble learning to surpass LLM limitations in urban environments[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2405.14876.
27	REDMON J, FARHADI A. YOLO9000: better, faster, stronger[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE Press, 2017: 6517-6525.
28	CORDTS M, OMRAN M, RAMOS S, et al. The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE Press, 2016: 3213-3223.
29	ZHOU B L , ZHAO H , PUIG X , et al. Semantic understanding of scenes through the ADE20K dataset. International Journal of Computer Vision, 2019, 127 (3): 302- 321. doi: 10.1007/s11263-018-1140-0
30	ZHANG Y , LIU P Y , BILJECKI F . Knowledge and topology: a two layer spatially dependent graph neural networks to identify urban functions with time-series street view image. ISPRS Journal of Photogrammetry and Remote Sensing, 2023, 198, 153- 168. doi: 10.1016/j.isprsjprs.2023.03.008
31	WU M L , HUANG Q Y , GAO S , et al. Mixed land use measurement and mapping with street view images and spatial context-aware prompts via zero-shot multimodal learning. International Journal of Applied Earth Observation and Geoinformation, 2023, 125, 103591. doi: 10.1016/j.jag.2023.103591
32	ZHAO Y H, ZHONG E H, YUAN C Y, et al. TG-LMM: enhancing medical image segmentation accuracy through text-guided large multi-modal model[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2409.03412.
33	PICARD C, EDWARDS K M, DORIS A C, et al. From concept to manufacturing: evaluating vision-language models for engineering design[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2311.12668.
34	YANG Y F , WANG S Q , LI D Y , et al. GeoLocator: a location-integrated Large Multimodal Model (LMM) for inferring geo-privacy. Applied Sciences, 2024, 14 (16): 7091. doi: 10.3390/app14167091
35	JAYATI S, CHOI E, BURTON H, et al. Leveraging large multimodal models to augment image-based building damage assessment[C]//Proceedings of the 7th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery. New York, USA: ACM Press, 2024: 79-85.
36	HAO X X, CHEN W, YAN Y B, et al. UrbanVLP: multi-granularity vision-language pretraining for urban socioeconomic indicator prediction[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2403.16831.
37	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2010.11929.
38	CHEN X L, FAN H Q, GIRSHICK R, et al. Improved baselines with momentum contrastive learning[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2003.04297.
39	TOLSTIKHIN I O , HOUISBY N , KOLESNIKOV A , et al. MLP-Mixer: an all-MLP architecture for vision. Advances in Neural Information Processing Systems, 2021, 34, 24261- 24272.
40	AYUPOV S, CHIRKOVA N. Parameter-efficient finetuning of transformers for source code[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2212.05901.
41	ZHANG Z, SABUNCU M. Generalized cross-entropy loss for training deep neural networks with noisy labels[EB/OL]. [2024-12-19]. https://arxiv.org/abs/1805.07836?context=cs.
42	杨冬菊, 黄俊涛. 基于大语言模型的中文科技文献标注方法. 计算机工程, 2024, 50 (9): 113- 120. doi: 10.19678/j.issn.1000-3428.0068400
	YANG D J , HUANG J T . Chinese scientific literature annotation method based on large language model. Computer Engineering, 2024, 50 (9): 113- 120. doi: 10.19678/j.issn.1000-3428.0068400
43	YASEEN M. What is YOLOv9: an in-depth exploration of the internal features of the next-generation object detector[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2409.07813.
44	KHANAM R, HUSSAIN M. YOLOv11: an overview of the key architectural enhancements[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2410.17725.
45	WU Z, LIU X, GILITSCHENSKI I. EventCLIP: adapting CLIP for event-based object recognition[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2306.06354.
46	CHEN Z, WANG W Y, CAO Y, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2412.05271.
47	GRATTAFIORI A, DUBEY A, JAUHRI A, et al. The llama 3 herd of models[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2407.21783.
48	WANG P, BAI S, TAN S N, et al. Qwen2-VL: enhancing vision-language model's perception of the world at any resolution[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2409.12191.
49	AGRAWAL P, ANTONIAK S, BOU HANNA E, et al. Pixtral 12B[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2410.07073.
50	TEAM G, GEORGIEV P, LEI V I, et al. Gemini 1.5: unlocking multimodal understanding across millions of tokens of context[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2403.05530.
51	CHIANG W L, ZHENG L M, SHENG Y, et al. Chatbot arena: an open platform for evaluating LLMs by human preference[EB/OL]. [2024-12-19]. https://arxiv.org/abs/2403.04132.

Please choose a citation manager

Content to export