Access Method for Traffic Infrastructure Monitoring Point Data Based on Virtual Gateway

doi:10.19678/j.issn.1000-3428.0069225

Abstract

Abstract:

To address the challenges posed by the diversity of sources, structures, and protocols in transportation infrastructure monitoring point data, this study analyzes the requirements for data access and proposes a virtual gateway for the access of monitoring point data within a transportation infrastructure based on the Netty architecture. The study describe in detail the configuration methods and cluster allocation strategies for integrating monitoring points through the gateway, and it defines the data transmission formats under protocols such as the HyperText Transfer Protocol (HTTP), Transmission Control Protocol (TCP), and User Datagram Protocol (UDP). The study also designs an encoding and verification mechanism for data transmission messages. Through enhanced real monitoring data samples, the performance of the virtual gateway is evaluated using distributed message simulation tools. The findings reveal that the virtual gateway facilitates unified access to multipoint and multiprotocol transportation infrastructure monitoring data, achieving data access and storage time of 8.14 s and 9.75 s per hundred million records, respectively. The average time for data traceability is 2.96 s, thus showing the gateway's capacity to handle monitoring data of points at the billion-scale level. This capability significantly supports the research and application of digital monitoring and transportation infrastructure analysis.

Key words: Netty framework, Kafka message queue, data access, platformization, Internet of Things(IoT) data

摘要：

针对交通基础设施点位监测数据来源多样、结构各异、协议不一的特点, 分析具体接入需求, 提出一种基于Netty架构的交通基础设施监测点位数据接入虚拟网关。阐述网关接入监测点位的配置方法及集群分配策略, 定义超文本传输协议(HTTP)、传输控制协议(TCP)、用户数据报协议(UDP)下的数据传输格式, 设计一种数据传输消息的编码和校验机制。在真实监测数据样本增强的基础上, 利用分布式消息模拟工具对虚拟网关性能进行测试, 结果表明, 该虚拟网关实现了多点位多协议的交通基础设施监测数据统一接入, 每亿条数据接入时间、存储时间分别达到8.14 s和9.75 s, 平均数据溯源时间为2.96 s, 具有亿级规模点位监测数据的接入能力, 可为交通基础设施数字化监测的研究和应用提供理论支撑。

关键词: Netty框架, Kafka消息队列, 数据接入, 平台化, 物联网数据

Liang HUANG, Peng ZOU, Jingjing CAO, Jian HU, Zexin YAN, Xiaodie HUANG. Access Method for Traffic Infrastructure Monitoring Point Data Based on Virtual Gateway[J]. Computer Engineering, 2024, 50(4): 50-59.

黄亮, 邹鹏, 曹菁菁, 胡健, 颜泽锌, 黄小蝶. 基于虚拟网关的交通基础设施监测点位数据接入方法[J]. 计算机工程, 2024, 50(4): 50-59.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069225

http://www.ecice06.com/EN/Y2024/V50/I4/50

Figures/Tables 17

Fig.1 Traffic infrastructure data access intelligent virtual gateway structure

Fig.2 Virtual gateway data access configuration page

Fig.3 Data access process

Fig.4 Message format reference under TCP protocol

Fig.5 Message format

Fig.6 UDP data access frame

Fig.7 Packet concatenation and half-packet verification process

Fig.8 Testing process and real data transmission diagram

Fig.9 Data storage test results

Fig.10 Execution results of traceability scenario 4

Fig.11 Buoy light monitoring data visualization

References 26

1	CHEUNG W F, LIN T H, LIN Y C. A real-time construction safety monitoring system for hazardous gas integrating wireless sensor network and building information modeling technologies. Sensors, 2018, 18 (2): 436. doi: 10.3390/s18020436
2	李琦, 姚龙. 基于REST架构的湖泊环境监测物联网平台. 计算机工程, 2016, 42 (11): 27-31, 37. doi: 10.3969/j.issn.1000-3428.2016.11.005
	LI Q, YAO L. Internet of Things platform for lake environmental monitoring based on REST architecture. Computer Engineering, 2016, 42 (11): 27-31, 37. doi: 10.3969/j.issn.1000-3428.2016.11.005
3	王庆贺, 刘瑞鑫, 孙立晔, 等. BIM与云、物联网技术在桥梁全生命周期中的研究及应用. 铁道标准设计, 2022, 66 (12): 78- 83. URL
	WANG Q H, LIU R X, SUN L Y, et al. Applications of BIM, cloud computing and Internet of Things technology in full life cycle of bridges. Railway Standard Design, 2022, 66 (12): 78- 83. URL
4	王建伟, 高超, 董是, 等. 道路基础设施数字化研究进展与展望. 中国公路学报, 2020, 33 (11): 101- 124. doi: 10.3969/j.issn.1001-7372.2020.11.011
	WANG J W, GAO C, DONG S, et al. Current status and future prospects of existing research on digitalization of highway infrastructure. China Journal of Highway and Transport, 2020, 33 (11): 101- 124. doi: 10.3969/j.issn.1001-7372.2020.11.011
5	夏烨, 龚丰宗, 尚志强, 等. 基于监测数据衰减响应模式提取的斜拉桥阻尼比特性分析. 中国公路学报, 2023, 36 (7): 90- 101. URL
	XIA Y, GONG F Z, SHANG Z Q, et al. Damping ratio analysis of cable-stayed bridges based on attenuation response pattern extraction from monitoring data. China Journal of Highway and Transport, 2023, 36 (7): 90- 101. URL
6	SHANG Z Q, XIA Y, CHEN L, et al. Damping ratio identification using attenuation responses extracted by time series semantic segmentation. Mechanical Systems and Signal Processing, 2022, 180, 109287. doi: 10.1016/j.ymssp.2022.109287
7	HOU Y, LI Q H, ZHANG C, et al. The state-of-the-art review on applications of intrusive sensing, image processing techniques, and machine learning methods in pavement monitoring and analysis. Engineering, 2021, 7 (6): 845- 856. doi: 10.1016/j.eng.2020.07.030
8	米春, 李思颖, 牟佳祎, 等. 基于多传感器的城市隧道环境监测数据协同融合方法研究. 现代隧道技术, 2023, 60 (5): 177- 185. URL
	MI C, LI S Y, MOU J Y, et al. Study on collaborative fusion method for urban tunnel environmental monitoring data based on multi-sensor. Modern Tunnelling Technology, 2023, 60 (5): 177- 185. URL
9	FAN X P, LIU Y F. Time-variant reliability prediction of bridge system based on BDGCM and SHM data. Structural Control and Health Monitoring, 2018, 25 (7): e2185. doi: 10.1002/stc.2185
10	黄观文, 陈孜, 徐永福. 膨胀土边坡GNSS实时监测技术. 测绘学报, 2023, 52 (11): 1873- 1882. URL
	HUANG G W, CHEN Z, XU Y F. GNSS real-time monitoring technology of expansive soil slope. Acta Geodaetica et Cartographica Sinica, 2023, 52 (11): 1873- 1882. URL
11	钟玉琪, 张红, 张舸, 等. 基于组合荷载响应特征融合的桥梁结构智能损伤识别方法. 实验力学, 2023, 38 (2): 151- 164. URL
	ZHONG Y Q, ZHANG H, ZHANG G, et al. An intelligent damage identification method based on response feature fusion under combined loadings for bridge structures. Journal of Experimental Mechanics, 2023, 38 (2): 151- 164. URL
12	龚小飞, 高志华, 郭颜斌. 基于熵权模型的引航风险评价研究. 中国水运, 2023, (11): 120- 123. URL
	GONG X F, GAO Z H, GUO Y B. Research on pilotage risk assessment based on entropy weight model. China Water Transport, 2023, (11): 120- 123. URL
13	周涛, 邹进贵, 郭际明. 隧道变形监测与智能预警方法研究. 测绘通报, 2022, (S2): 91- 94. URL
	ZHOU T, ZOU J G, GUO J M. Research on tunnel deformation monitoring and intelligent early warning method. Bulletin of Surveying and Mapping, 2022, (S2): 91- 94. URL
14	王代君, 傅俊敏, 周天图, 等. 区域交通基础设施群监测系统设计及应用关键技术研究. 公路交通科技, 2022, 39 (6): 124- 135. URL
	WANG D J, FU J M, ZHOU T T, et al. Study on key technology for design and application of regional transport infrastructure group monitoring system. Journal of Highway and Transportation Research and Development, 2022, 39 (6): 124- 135. URL
15	LAN L N, SHI R S, WANG B, et al. An IoT unified access platform for heterogeneity sensing devices based on edge computing. IEEE Access, 2019, 7, 44199- 44211. doi: 10.1109/ACCESS.2019.2908684
16	顾振德, 刘子辰, 龙隆, 等. 基于Netty的IoT终端通信服务系统设计. 计算机应用与软件, 2019, 36 (4): 135- 139. URL
	GU Z D, LIU Z C, LONG L, et al. Design of IoT terminal communication service system based on Netty. Computer Applications and Software, 2019, 36 (4): 135- 139. URL
17	林俊晖. 物联网通用数据接入平台的设计与实现[D]. 重庆: 重庆邮电大学, 2018.
	LIN J H. Design and implementation of a general data access platform for Internet of Things[D]. Chongqing: Chongqing University of Posts and Telecommunications, 2018. (in Chinese)
18	甄凯成. 面向物联网应用的数据接入和存储系统的设计与实现[D]. 合肥: 中国科学技术大学, 2019.
	ZHEN K C. Design and implementation of data ingestion and storage system for IoT applications[D]. Hefei: University of Science and Technology of China, 2019. (in Chinese)
19	王圣博, 王泽昊, 高梓桐, 等. 一种物联网设备的数据接入方法. 物联网技术, 2023, 13 (3): 64- 66. URL
	WANG S B, WANG Z H, GAO Z T, et al. Data access method for Internet of Things equipment. Internet of Things Technologies, 2023, 13 (3): 64- 66. URL
20	赵润发, 娄渊胜, 叶枫, 等. 基于Flink的工业大数据平台研究与应用. 计算机工程与设计, 2022, 43 (3): 886- 894. URL
	ZHAO R F, LOU Y S, YE F, et al. Research and application of industrial big data platform based on Flink. Computer Engineering and Design, 2022, 43 (3): 886- 894. URL
21	甄凯成, 黄河, 宋良图. 基于Netty和Kafka的物联网数据接入系统. 计算机工程与应用, 2020, 56 (5): 135- 140. URL
	ZHEN K C, HUANG H, SONG L T. Internet of Things data injection system based on Netty and Kafka. Computer Engineering and Applications, 2020, 56 (5): 135- 140. URL
22	THEIN K M M. Apache Kafka: next generation distributed messaging system. International Journal of Scientific Engineering and Technology Research, 2014, 47 (3): 9478- 9483. URL
23	刘美佳, 张箐. 基于分布式集群架构的遥感数据传输机制. 计算机工程, 2021, 47 (10): 180- 185. doi: 10.19678/j.issn.1000-3428.0059668
	LIU M J, ZHANG Q. Remote sensing data transmission mechanism based on distributed cluster architecture. Computer Engineering, 2021, 47 (10): 180- 185. doi: 10.19678/j.issn.1000-3428.0059668
24	SOBIN C C. A survey on architecture, protocols and challenges in IoT. Wireless Personal Communications, 2020, 112 (3): 1383- 1429. URL
25	BENIWAL G, SINGHROVA A. A systematic literature review on IoT gateways. Journal of King Saud University-Computer and Information Sciences, 2022, 34 (10): 9541- 9563.
26	VAN D G, VAN D P D. Influencing factors in the scalability of distributed stream processing jobs. IEEE Access, 2021, 9, 109413- 109431. doi: 10.1109/ACCESS.2021.3102645

[1]	LIU Meijia, ZHANG Qing. Remote Sensing Data Transmission Mechanism Based on Distributed Cluster Architecture [J]. Computer Engineering, 2021, 47(10): 180-185.
[2]	ZHONG Qing, JUE Hua-Kun, CHEN Dui-Min, XIAO Yong, DANG San-Lei. Application of Internet of Things Technology in Equipment All Life Cycle Management [J]. Computer Engineering, 2012, 38(5): 247-250,254.
[3]	MENG Chao, LIU Gao, HU Hong-Yang. Energy Efficient Data Access Technique Based on Readahead Strategy [J]. Computer Engineering, 2012, 38(08): 44-46.
[4]	CHEN Ji-Rong. Data Grid Access Model Based on Tuscany Das [J]. Computer Engineering, 2011, 37(5): 86-87,90.
[5]	LI Yan-Yan, YAN Xue-Feng. Design and Implementation of Data Access Mechanism in Combat System [J]. Computer Engineering, 2011, 37(19): 230-232.
[6]	TIAN Ming-yan; ZHANG Shui-ping; LIU Lu-ping. Multi-resource Data Access Based on Grid [J]. Computer Engineering, 2010, 36(7): 84-86.
[7]	JIN Qi-bing; WANG Yan-yan; LONG Ping. Research on OPC Cross Platform Data Access Based on FDT [J]. Computer Engineering, 2010, 36(4): 265-266.
[8]	XIONG Feng-Guang, HAN Xie. Design of Database Middleware in NATS [J]. Computer Engineering, 2010, 36(17): 74-76.
[9]	SONG Hong-Bin, XIAO Xiao-Jiang, XU Meng, JIN Ning. Improvement of Data Caching DGA Algorithm in Wireless Ad-hoc Network [J]. Computer Engineering, 2010, 36(14): 82-83.
[10]	WANG Hai-Yan. Unified Access Interface of Campus Grid Heterogeneous Data [J]. Computer Engineering, 2010, 36(11): 73-74,77.
[11]	LIU Bin; ZHANG Hong-jun; HAO Wen-ning. General Simulation Data Service Based on HLA [J]. Computer Engineering, 2010, 36(06): 281-283.
[12]	LI Xiao-hang; HU Xiao-peng; LI Gang. Improvement and Implementation of Heterogeneous Database Replication Approach [J]. Computer Engineering, 2009, 35(8): 79-81.
[13]	穆运峰;申利民. Modeling and Implementation for Flexible Information System Based on Meta-data [J]. Computer Engineering, 2008, 34(16): 37-40.
[14]	WANG Sheng-ping; LI Qing. Implementation of Shared Buffer Data Flow in Data Replication Management [J]. Computer Engineering, 2007, 33(20): 83-85.
[15]	WU Kai-chao; LI Jia-sheng ; XIAO Yun; ZHOU Yuan-chun; YAN Bao-ping. Event-driven Based Asynchronous Message Pass Model for Scientific Data Grid [J]. Computer Engineering, 2007, 33(15): 76-78.

Please choose a citation manager

Content to export