大规模开放遥感影像地图渲染与缓存优化

doi:10.19678/j.issn.1000-3428.0068142

摘要/Abstract

摘要：

遥感数据规模庞大且增长迅速, 目前可公开访问的遥感影像数据已接近EB级别, 然而类型多样、结构复杂、存储文件大等特点给大规模开放遥感数据的发现、共享与使用带来诸多不便。在线地图可使用户无须下载便可对海量云端遥感数据执行可视化分析, 是一种高效的数据服务方式。针对传统地图技术方案存在的瓦片渲染效率低、遥感数据适配性差等问题, 从遥感数据时空属性特征及用户访问行为特征出发, 依托遥感数据云平台GSCloud, 设计并实现面向海量遥感数据的高效地图服务平台TiMap。TiMap由分布式地图瓦片渲染引擎TiRender与分布式地图瓦片缓存TiCache构成。TiRender通过将地图瓦片渲染操作转换为分布式环境下的同步实时渲染任务与异步批量预渲染任务, 充分利用多节点并行计算的优势, 快速响应客户端的地图瓦片请求。TiCache负责缓存TiRender产生的地图瓦片, 以提升后续重复地图瓦片请求的响应速度, TiCache中的地图瓦片缓存分配算法基于疏远度实现, 可以保证多节点的负载均衡。实验结果表明, TiRender与TiCache均比同类技术方案的性能更好, 两者协同工作可使TiMap在100 ms内快速响应大规模地图瓦片请求。

关键词: 遥感大数据, 遥感地图服务, 地图瓦片渲染, 地图瓦片缓存, 数据疏远度

Abstract:

Remote sensing data are witnessing a rapid growth, with the volume of publicly available remote sensing images expected to reach the Exabyte (EB) scale. However, diverse product types, complex internal structures, and large file sizes pose many challenges to the discovery and sharing of large-scale open remote sensing data. Online maps provide an efficient and effortless method for users to perform online visualization analyses of massive cloud remote sensing images without downloading. Given the low rendering efficiency of map tiles and the poor adaptability of remote sensing data in traditional map technologies, this paper introduces TiMap, an efficient big web map platform for large-scale, multi-source remote sensing data. By leveraging the spatiotemporal attributes and user access behavior characteristics of remote sensing data on the GSCloud platform, TiMap enhances the performance and adaptability of remote sensing data rendering. TiMap comprises a distributed map tile rendering module (TiRender) and a distributed map tile cache module (TiCache). TiRender transforms map tile rendering operations into synchronous real-time rendering and asynchronous batch pre-rendering tasks, leveraging multi-node parallel computing for the rapid response of map tiles. The map tiles generated by TiRender are then cached and managed by TiCache. TiCache distributes map tiles over multi-cache nodes based on layer diversity, which maintains the workload balance of all cache nodes. Experiments demonstrate that TiRender and TiCache outperform other similar technologies. Their collaborative functionality enables TiMap to respond quickly to large-scale map tile requests within 100 ms.

Key words: remote sensing big data, remote sensing map service, map tile rendering, map tile cache, data diversity

周小华, 周园春, 孟珍, 王学志. 大规模开放遥感影像地图渲染与缓存优化[J]. 计算机工程, 2024, 50(7): 227-239.

Xiaohua ZHOU, Yuanchun ZHOU, Zhen MENG, Xuezhi WANG. Large-Scale Open Remote Sensing Images Map Rendering and Cache Optimization[J]. Computer Engineering, 2024, 50(7): 227-239.

https://www.ecice06.com/CN/Y2024/V50/I7/227

图/表 22

图1 遥感地图服务基本过程

Fig.1 Basic process of remote sensing map service

图2 TiMap图存储结构

Fig.2 Graph storage structure of TiMap

图3 TiRender地图瓦片分布式渲染任务调度流程

Fig.3 TiRender map tile distributed rendering task scheduling process

图4 TiCache中的地图瓦片缓存动态分配流程

Fig.4 Dynamic allocation process of map tile cache in TiCache

图5 地图瓦片大小分布

Fig.5 Distribution of map tiles size

图6 遥感影像图谱距离示意图

Fig.6 Schematic diagram of remote sensing image map distance

图7 计算疏远度并确定缓存节点的流程

Fig.7 The process of calculating distance and determining cache nodes

图8 基于Hilbert编码的地图瓦片缓存映射

Fig.8 Map tiles cache mapping based on Hilbert encoding

图9 Hilbert递归编码示意图

Fig.9 Schematic diagram of Hilbert recursive encoding

图10 TiMap测试环境部署拓扑

Fig.10 Deployment topology of TiMap test environment

图11 同/异步渲染进程比例对应的地图瓦片响应时间

Fig.11 Map tile response time corresponding to the ratio of synchronous/asynchronous rendering processes

图12 不同级别的地图瓦片响应时间对比

Fig.12 Comparison of response time for map tiles of different levels

图13 TiRender与Spark响应时间对比

Fig.13 Response time comparison between TiRender and Spark

图14 R7与C1~C16的疏远度

Fig.14 The diversity between R7 and C1~C16

图15 R5、R1、R2与C1~C16的疏远度

Fig.15 The diversity between R5, R1, R2, and C1~C16

图16 R1~R8影像的地图瓦片缓存分配情况

Fig.16 Map tile cache allocation for R1~R8 images

图17 R9、RM与C1~C16的疏远度

Fig.17 The diversity between R9, RM, and C1~C16

图18 不同用户的平均加载时间对比

Fig.18 Average loading time comparison among different users

图19 TiCache与MapCache缓存瓦片数量对比

Fig.19 Comparison of the number of tiles cached in TiCache and MapCache

图20 TiCache与MapCache地图加载时间对比

Fig.20 Comparison of map loading time between TiCache and MapCache

参考文献 25

1	XU C, DU X P, FAN X T, et al. Cloud-based storage and computing for remote sensing big data: a technical review. International Journal of Digital Earth, 2022, 15(1): 1417- 1445. doi: 10.1080/17538947.2022.2115567
2	VEENENDAAL B, BROVELLI M A, LI S N. Review of Web mapping: eras, trends and directions. ISPRS International Journal of Geo-Information, 2017, 6(10): 317. doi: 10.3390/ijgi6100317
3	GUI Z P, CAO J, LIU X J, et al. Global-scale resource survey and performance monitoring of public OGC web map services. ISPRS International Journal of Geo-Information, 2016, 5(6): 88. doi: 10.3390/ijgi5060088
4	CAPOLUPO A, MONTERISI C, SAPONIERI A, et al. An interactive WebGIS framework for coastal erosion risk management. Journal of Marine Science and Engineering, 2021, 9(6): 567. doi: 10.3390/jmse9060567
5	康宁, 徐青, 周杨, 等. 一种基于图形硬件的海量地形实时可视化算法. 系统仿真学报, 2007, 19(17): 3988- 3992. doi: 10.3969/j.issn.1004-731X.2007.17.033
	KANG N, XU Q, ZHOU Y, et al. Graphic hardware-based algorithm for real-time visualization of massive terrain dataset. Journal of System Simulation, 2007, 19(17): 3988- 3992. doi: 10.3969/j.issn.1004-731X.2007.17.033
6	HE G J, XIONG W, LUO C, et al. A MPI-based parallel pyramid building algorithm for large-scale remote sensing images[EB/OL]. [2023-06-05]. https://ieeexplore.ieee.org/document/7378567.
7	JEONG I K, HONG M G, HAHN K S, et al. Performance study of satellite image processing on graphics processors unit using CUDA. Korean Journal of Remote Sensing, 2012, 28(6): 683- 691. doi: 10.7780/kjrs.2012.28.6.8
8	ZUO X Y, ZHANG Z, QIAO B J, et al. A dynamic acceleration method for remote sensing image processing based on CUDA. Wireless Networks, 2021, 27(6): 3995- 4007. doi: 10.1007/s11276-021-02715-x
9	WANG X, YANG R Z, LIU X Z. Research on general system of remote sensing satellite data preprocessing based on MPI + CUDA[C]//Proceedings of the 3rd International Conference on Geology, Mapping and Remote Sensing. Washington D. C., USA: IEEE Press, 2022: 607-612.
10	MISRA C, BHATTACHARYA S, GHOSH S K. A fast scalable distributed Kriging algorithm using Spark framework. International Journal of Data Science and Analytics, 2020, 10(3): 249- 264. doi: 10.1007/s41060-020-00215-3
11	WANG S H, ZHONG Y, LU H, et al. Geospatial big data analytics engine for Spark[C]//Proceedings of the 6th ACM SIGSPATIAL Workshop on Analytics for Big Geospatial Data. New York, USA: ACM Press, 2017: 42-45.
12	GUO J F, HUANG C L, HOU J L. A scalable computing resources system for remote sensing big data processing using GeoPySpark based on Spark on K8s. Remote Sensing, 2022, 14(3): 521. doi: 10.3390/rs14030521
13	ELDAWY A, MOKBEL M F. SpatialHadoop: a MapReduce framework for spatial data[C]//Proceedings of the 31st IEEE International Conference on Data Engineering. Washington D. C., USA: IEEE Press, 2015: 1352-1363.
14	YU J, WU J X, SARWAT M. GeoSpark: a cluster computing framework for processing large-scale spatial data[C]//Proceedings of the 23rd SIGSPATIAL International Conference on Advances in Geographic Information Systems. Washington D. C., USA: IEEE Press, 2015: 1-4.
15	成洪权. 基于GeoTrellis的遥感影像大数据管理与服务技术研究[D]. 武汉: 武汉大学, 2019.
	CHENG H Q. Research on management and service technology of remote sensing image big data based on GeoTrellis[D]. Wuhan: Wuhan University, 2019. (in Chinese)
16	ELDAWY A, MOKBEL M F, JONATHAN C. HadoopViz: a MapReduce framework for extensible visualization of big spatial data[C]//Proceedings of the 32nd IEEE International Conference on Data Engineering. Washington D. C., USA: IEEE Press, 2016: 601-612.
17	YU J, ZHANG Z S, SARWAT M. GeoSparkViz: a scalable geospatial data visualization framework in the apache Spark ecosystem[C]//Proceedings of the 30th International Conference on Scientific and Statistical Database Management. Washington D. C., USA: IEEE Press, 2018: 1-12.
18	孟祥海, 王学志, 赵江华, 等. 轻量级遥感数据分布式调度框架DataboxMR. 数据与计算发展前沿, 2021, 3(3): 95- 110. URL
	MENG X H, WANG X Z, ZHAO J H, et al. DataboxMR: a lightweight remote sensing data distributed scheduling framework. Frontiers of Data & Computing, 2021, 3(3): 95- 110. URL
19	LIU H L, NIE Y F. Tile-based map service GeoWebCache middleware[C]//Proceedings of IEEE International Conference on Intelligent Computing and Intelligent Systems. Washington D. C., USA: IEEE Press, 2010: 692-697.
20	GARSIDE A J, SALMAN M, TRIPATHI V Y. Tile Cache[EB/OL]. [2023-06-05]. https://patentimages.storage.googleapis.com/51/e8/0b/c6405c3f688f72/US8922575.pdf.
21	YANG J H, PING W Y, LIU L, et al. Memcache and MongoDB based GIS Web service[C]//Proceedings of the 2nd International Conference on Cloud and Green Computing. Washington D. C., USA: IEEE Press, 2012: 126-129.
22	WAN L, HUANG Z, PENG X. An effective NoSQL-based vector map tile management approach. ISPRS International Journal of Geo-Information, 2016, 5(11): 215. doi: 10.3390/ijgi5110215
23	曹晓裴. 面向分布式存储系统Ceph的遥感影像瓦片存储及其关键技术[D]. 杭州: 浙江大学, 2020.
	CAO X P. Remote sensing image tile storage and its key technology for distributed storage system Ceph[D]. Hangzhou: Zhejiang University, 2020. (in Chinese)
24	MASÓ J, POMAKIS K, JULIÀ N. OpenGIS Web Map Tile Service (WMTS) implementation standard version 1.0.0[EB/OL]. [2023-06-05]. https://www.ogc.org/standard/wmts/.
25	ZHOU X H, WANG X Z, ZHOU Y C, et al. RSIMS: large-scale heterogeneous remote sensing images management system. Remote Sensing, 2021, 13(9): 1815. doi: 10.3390/rs13091815

选择文件类型/文献管理软件名称

选择包含的内容