Multivariate Time Series Forecasting Based on Multi-Scale Feature Fusion and Dual-Attention Mechanism

doi:10.19678/j.issn.1000-3428.0065846

Abstract

Abstract:

Each subsequence of the Multivariate Time Series(MTS) contains multi-scale characteristics of different time spans, comprising information such as development process, direction, and trend. However, existing time series prediction models cannot effectively capture multi-scale features and evaluate their importance. In this study, a MTS prediction network, FFANet, is proposed based on multi-scale temporal feature fusion and a Dual-Attention Mechanism(DAM).FFANet effectively integrates multi-scale features and focuses on important parts.Utilizing the parallel temporal dilation convolution layer in the multi-scale temporal feature fusion module endows the model with multiple receptive domains to extract features of temporal data at different scales and adaptively fuse them based on their importance. Using a DAM to recalibrate the fused temporal features, FFANet focuses on features that make significant contributions to prediction by assigning temporal and channel attention weights and weighting them to the corresponding temporal features. The experimental results show that compared with AR, VARMLP, RNN-GRU, LSTNet-skip, TPA-LSTM, MTGNN, and AttnAR time series prediction models, FFANet achieves average reduction of 0.152 3、0.120 0、0.074 3、0.035 4、0.021 5、0.012 1、0.020 0 in RRSE prediction error on Traffic, Solar Energy, and Electricity datasets, respectively.

Key words: Multivariate Time Series(MTS) forecasting, Convolutional Neural Network(CNN), multi-scale feature, feature fusion, attention mechanism

摘要：

多元时间序列的各子序列包含不同时间跨度的多尺度特征，现有时间序列预测模型不能有效地捕获多尺度特征以及评估其重要程度。提出一种基于多尺度时序特征融合与双注意力机制的多元时间序列预测网络FFANet，有效融合多尺度特征并关注其中重要部分。通过多尺度时序特征融合模块中并行的时序膨胀卷积层，使模型具有多种感受域，从而提取时序数据在不同尺度上的特征，并根据重要性对其进行自适应融合。利用双注意力模块对融合的时序特征进行重新标定，通过分配时序和通道注意力权重并加权至对应的时序特征，使FFANet聚焦对预测有重要贡献的特征。实验结果表明，相比AR、VARMLP、RNN-GRU、LSTNet-skip、TPA-LSTM、MTGNN和AttnAR时间序列预测模型，FFANet在Traffic、Solar Energy和Electricity数据集上的RRSE预测误差分别平均降低0.152 3、0.120 0、0.074 3、0.035 4、0.021 5、0.012 1、0.020 0。

关键词: 多元时间序列预测, 卷积神经网络, 多尺度特征, 特征融合, 注意力机制

Lu HAN, Weigang HUO, Yonghui ZHANG, Tao LIU. Multivariate Time Series Forecasting Based on Multi-Scale Feature Fusion and Dual-Attention Mechanism[J]. Computer Engineering, 2023, 49(9): 99-108.

韩璐, 霍纬纲, 张永会, 刘涛. 基于多尺度特征融合与双注意力机制的多元时间序列预测[J]. 计算机工程, 2023, 49(9): 99-108.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0065846

https://www.ecice06.com/EN/Y2023/V49/I9/99

Figures/Tables 9

Fig.1 Overall structure of the FFANet

Fig.2 Structure of the dual-attention module

Fig.3 Visualization results among different datasets

Fig.4 Ablation experiment results

Fig.5 Influence of node embedding dimension e on CORR

Fig.6 Influence of characteristic channel number C on CORR on Electricity dataset

References 25

1	原继东, 王志海. 时间序列的表示与分类算法综述. 计算机科学, 2015, 42(3): 1- 7. URL
	YUAN J D, WANG Z H. Review of time series representation and classification techniques. Computer Science, 2015, 42(3): 1- 7. URL
2	汤强, 谢明中, 罗元盛. 基于SVR的用电负荷特征三维回归模型. 计算机工程, 2017, 43(9): 300-303, 309. doi: 10.3969/j.issn.1000-3428.2017.09.051
	TANG Q, XIE M Z, LUO Y S. SVR based three dimensional regression model of power load characteristics. Computer Engineering, 2017, 43(9): 300-303, 309. doi: 10.3969/j.issn.1000-3428.2017.09.051
3	朱原媛, 杨有龙, 张恒伟. 基于贝叶斯网络的混沌时间序列预测. 计算机工程与应用, 2012, 48(13): 100- 104. URL
	ZHU Y Y, YANG Y L, ZHANG H W. Chaotic time series prediction based on Bayesian network. Computer Engineering and Applications, 2012, 48(13): 100- 104. URL
4	HWANG Y, TONG A, CHOI J. Automatic construction of nonparametric relational regression models for multiple time series[C]//Proceedings of the 33rd International Conference on International Conference on Machine Learning. New York, USA: ACM Press, 2016: 3030-3039.
5	李晓, 卢先领. 基于双重注意力机制和GRU网络的短期负荷预测模型. 计算机工程, 2022, 48(2): 291-296, 305. doi: 10.3969/j.issn.1007-130X.2022.02.014
	LI X, LU X L. Method for forecasting short-term power load based on dual-stage attention mechanism and gated recurrent unit network. Computer Engineering, 2022, 48(2): 291-296, 305. doi: 10.3969/j.issn.1007-130X.2022.02.014
6	FARNOOSH A, AZARI B, OSTADABBAS S. Deep switching auto-regressive factorization: application to time series forecasting[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l. ]: AAAI Press, 2021: 7394-7403.
7	RASUL K, SEWARD C, SCHUSTER I, et al. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting[EB/OL]. [2022-08-23]. https://arxiv.org/abs/2101.12072v2.
8	CHEN Z, MA Q, LIN Z. Time-aware multi-scale RNNs for time series modeling[C]//Proceedings of International Joint Conference on Artificial Intelligence. Barcelona, Spain: [s. n], 2021: 2285-2291.
9	ELMAN J L. Finding structure in time. Cognitive Science, 1990, 14(2): 179- 211. doi: 10.1207/s15516709cog1402_1
10	SIAMI-NAMINI S, TAVAKOLI N, SIAMI NAMIN A. A comparison of ARIMA and LSTM in forecasting time series[C]//Proceedings of the 17th International Conference on Machine Learning and Applications. Washington D. C., USA: IEEE Press, 2018: 1394-1401.
11	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60(6): 84- 90. doi: 10.1145/3065386
12	YE J C, LIU Z H, DU B W, et al. Learning the evolutionary and multi-scale graph structure for multivariate time series forecasting[C]//Proceedings of the 28th Conference on Knowledge Discovery and Data Mining. New York, USA: ACM Press, 2022: 2296-2306.
13	WANG Y Y, SMOLA A, MADDIX D, et al. Deep factors for forecasting[EB/OL]. [2022-08-23]. https://arxiv.org/abs/1905.12417v1.
14	SEN R, YU H F, DHILLON I. Think globally, act locally: a deep neural network approach to high-dimensional time series forecasting[EB/OL]. [2022-08-23]. https://arxiv.org/abs/1905.03806v2.
15	LAI G K, CHANG W C, YANG Y M, et al. Modeling long- and short-term temporal patterns with deep neural networks[C]//Proceedings of the 41st International Conference on Research & Development in Information Retrieval. New York, USA: ACM Press, 2018: 95-104.
16	GONG G J, AN X N, MAHATO N K, et al. Research on short-term load prediction based on Seq2seq model. Energies, 2019, 12(16): 3199. doi: 10.3390/en12163199
17	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 1-9.
18	WU Z H, PAN S R, LONG G D, et al. Connecting the dots: multivariate time series forecasting with graph neural networks[C]//Proceedings of the 26th International Conference on Knowledge Discovery & Data Mining. New York, USA: ACM Press, 2020: 753-763.
19	YU F, KOLTUN V. Multi-scale context aggregation by dilated convolutions[EB/OL]. [2022-08-23]. https://arxiv.org/pdf/1511.07122.pdf.
20	CHEN B Y, LI P X, SUN C, et al. Multi attention module for visual tracking. Pattern Recognition, 2019, 87, 80- 93. doi: 10.1016/j.patcog.2018.10.005
21	SHIH S Y, SUN F K, LEE H Y. Temporal pattern attention for multivariate time series forecasting. Machine Learning, 2019, 108(8/9): 1421- 1441.
22	鹿天柱, 钱晓超, 何舒, 等. 一种基于深度学习的时间序列预测方法. 控制与决策, 2021, 36(3): 645- 652. URL
	LU T Z, QIAN X C, HE S, et al. A time series prediction method based on deep learning. Control and Decision, 2021, 36(3): 645- 652. URL
23	YOO J, KANG U. Attention-based autoregression for accurate and efficient multivariate time series forecasting[C]//Proceedings of International Conference on Data Mining. Philadelphia, USA: Society for Industrial and Applied Mathematics, 2021: 531-539.
24	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[EB/OL]. [2022-08-23]. https://arxiv.org/pdf/1807.06521.pdf.
25	ZHANG G P. Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing, 2003, 50, 159- 175.

[1]	ZHANG Tianpeng, HAN Jing, LÜ Xueqiang. Super-Resolution-Aided Small-Target Detection Based on Multi-Task Learning [J]. Computer Engineering, 2024, 50(9): 304-312.
[2]	GUO Min, ZHANG Xihan, LI Yang. Integrated Attentional Teacher Mutual Consistency Semi-Supervised Medical Image Segmentation [J]. Computer Engineering, 2024, 50(9): 313-323.
[3]	GAO Yubao, WEN Zhicheng. Dual Decoder Image Denoising Method Based on Attention Mechanism [J]. Computer Engineering, 2024, 50(9): 324-332.
[4]	ZENG Yuqi, LIU Bo, ZHONG Baichang, ZHONG Jin. Student Classroom Behavior Detection Algorithm Based on Improved YOLOv8 in Smart Education [J]. Computer Engineering, 2024, 50(9): 344-355.
[5]	WANG Zhihao, QIAN Yuntao. Super-Resolution Reconstruction of Spatiotemporal Fusion for Dual-Stream Remote Sensing Images Based on Swin Transformer [J]. Computer Engineering, 2024, 50(9): 33-45.
[6]	LI Junjun, DONG Jiangang, LI Kun. Research on Kubernetes-based Cluster Energy-Saving Strategy [J]. Computer Engineering, 2024, 50(9): 82-91.
[7]	LIN Chang, GUO Wei, REN Zhecong, JIN Haibo. Unification Algorithm for Object Tracking and Segmentation Based on Transformer [J]. Computer Engineering, 2024, 50(9): 130-141.
[8]	LI Zelin, LÜ Zhaofeng, CHEN Fuqiang, LI Ke. Entity Alignment Model Based on Multi-Hop Information Fusion [J]. Computer Engineering, 2024, 50(9): 142-152.
[9]	WANG Ruying, MA Jiajun, DONG Jianqiang, LIU Wanlong, ZHANG Haitao, YIN Kai, ZHAO Bochao. Industrial Load Forecasting Method Based on MTS-BiGRU-DMHSA [J]. Computer Engineering, 2024, 50(9): 169-178.
[10]	LI Junyi, LI Xiangyang, LONG Chaoxun, LI Haiyan, LI Hongsong, YU Pengfei. Wild Mushroom Classification Based on Multi-level Region Selection and Cross-layer Feature Fusion [J]. Computer Engineering, 2024, 50(9): 179-188.
[11]	ZHANG Lu, TIAN Chunwei, SONG Huansheng, LIU Shigang. Multi-Level Dual-Tree Complex Wavelet Network for Low-Dose CT Image Denoising [J]. Computer Engineering, 2024, 50(9): 266-275.
[12]	ZHU Kai, LI Li, ZHANG Tong, JIANG Sheng, BIE Yiming. Multi-Stage Motion Blur Image Restoration Network Based on Transformer [J]. Computer Engineering, 2024, 50(9): 276-285.
[13]	Wanqiu ZHAO, Junhu ZHANG, Haitao LI. Feature Fusion Network with Parallel Structure for Building Segmentation [J]. Computer Engineering, 2024, 50(8): 239-248.
[14]	Hong ZHAO, Xiao WANG. Study on Lesion Segmentation of Melanoma Images Based on Swin-Transformer [J]. Computer Engineering, 2024, 50(8): 249-258.
[15]	Li MIN, Bingjie DONG, Dong AN. Semantic Segmentation Algorithm Based on Multi-Attention Mechanism and Cross-Feature Fusion [J]. Computer Engineering, 2024, 50(8): 282-289.

Please choose a citation manager

Content to export