作者投稿和查稿 主编审稿 专家审稿 编委审稿 远程编辑

计算机工程 ›› 2021, Vol. 47 ›› Issue (10): 180-185. doi: 10.19678/j.issn.1000-3428.0059668

• 移动互联与通信技术 • 上一篇    下一篇

基于分布式集群架构的遥感数据传输机制

刘美佳1,2, 张箐1   

  1. 1. 中国科学院空天信息创新研究院, 北京 100094;
    2. 中国科学院大学 电子电气与通信工程学院, 北京 100049
  • 收稿日期:2020-10-09 修回日期:2020-11-10 发布日期:2020-11-18
  • 作者简介:刘美佳(1993-),女,硕士研究生,主研方向为传输系统优化、IPv6网络评估;张箐(通信作者),正高级工程师。
  • 基金资助:
    国家民用空间基础设施“十二五”陆地观测卫星地面系统建设项目(HK-KJSEW-CS-XCD-01)。

Remote Sensing Data Transmission Mechanism Based on Distributed Cluster Architecture

LIU Meijia1,2, ZHANG Qing1   

  1. 1. Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China;
    2. School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100049, China
  • Received:2020-10-09 Revised:2020-11-10 Published:2020-11-18

摘要: 对地观测遥感卫星数据传输系统采用Linux虚拟服务器集群架构传输遥感卫星数据,而该集群中承载传输任务的新旧服务器在实现断点续传的过程中由于控制信息不同步会造成数据片重复。设计一种基于分布式系统架构的遥感卫星数据处理机制DPM,采用Kafka消息队列快速存储数据包,利用代码模块使Spark Streaming精准提交消息偏移量,并通过数据传输速率与进度统计实时记录DPM运行状态信息。测试结果表明,DPM机制可精准记录与提交消息偏移量并实现数据续传功能,从而确保遥感卫星数据的准确稳定传输。

关键词: 数据传输系统, 遥感卫星数据, 分布式系统架构, 数据处理机制, Kafka消息队列

Abstract: The data transmission systems of remote sensing satellites for earth observation generally adopt Linux Virtual Server Cluster (LVS)architecture to transmit remote sensing data.However, when the LVSs for data transmission in the cluster try to resume transmission at breakpoints, the control information of the old servers is not synchronized with that of new servers, leading to duplicated data slices.To address the problem, a data processing mechanism based on distributed system architecture for remote sensing satellites, called DPM, is designed.The mechanism employs the Kafka message queue to quickly store packets, and uses a new code module to enable Spark Streaming to submit message offset value accurately.Then efficient data transmission rate and progress statistics method are used to record the running state of the DPM mechanism in real time.Experimental results show that the proposed mechanism can accurately record and submit the offset value of messages, and resume data transmission.It can ensure the accurate and stable transmission of remote sensing satellite data.

Key words: data transmission system, remote sensing satellite data, distributed system architecture, data processing mechanism, Kafka message queue

中图分类号: