基于预训练递归Transformer-Mixer的多维时间序列分类研究

doi:10.19678/j.issn.1000-3428.0069143

摘要/Abstract

摘要： 多维时间序列分类在工业、医疗、金融等领域有着广泛应用,在工业产品质量控制、疾病预测、金融风险控制等方面发挥着重要作用。多维时间序列时间依赖关系和空间依赖关系同等重要,传统多维时间序列模型只对时间或空间某一维度重点关注。为此,提出一种基于预训练递归Transformer-Mixer的多维时间序列分类模型PRTMMTSC。模型基于Transformer-Mixer模块充分学习多维时间序列时间和空间的关联关系。为进一步提升分类模型的性能,受异常检测模型的启发,将预训练后的隐藏层特征和残差特征进行融合,并采用PolyLoss损失函数对模型进行训练。为减少模型训练参数量,模型中Transformer-Mixer模块采用递归方式构建,使多层可训练参数量仅为单层Transformer-Mixer参数量。在UEA多维时间数据集上的实验结果表明,所提模型的性能优于对比模型,相较于TARNet模型和RLPAM模型的准确率分别提升3.03%和4.69%。在UEA及IF钢夹渣缺陷分类的消融实验验证预训练方式、Transformer-Mixer模块、残差信息及PolyLoss损失函数的有效性。

关键词: 多维时间序列分类, Transformer-Mixer模块, 机器学习, 预训练, IF钢夹渣缺陷预报

Abstract: Multi-dimensional time series classification is widely used in industry, medical treatment, finance and other fields; it plays an important role in industrial product quality control, disease prediction, financial risk control and so on. Aiming at the problem that time dependence and spatial dependence of multi-dimensional time series are equally important, and that traditional multi-dimensional time series models only focus on a certain dimension of time or space, this paper proposes a multi-dimensional time series classification model based on the pre-trained recursive Transformer-Mixer PRTMMTSC. The model is based on a Transformer-Mixer module that can fully learn the temporal and spatial correlations of multi-dimensional time series. To further improve the classification performance, inspired by the anomaly detection model, the proposed model combines the pre-trained hidden layer features and the residual features, and uses the PolyLoss loss function for training. To reduce the number of model training parameters, the Transformer-Mixer module in the model is constructed recursively,so that the number of multi-layer trainable parameters is only the number of single-layer Transformer-Mixer parameters. The experimental results on the UEA datasets show that the performance of the proposed model is better than that of the contrast models. Compared with the TARNet model and the RLPAM model, the accuracy of proposed model has increased by 3.03% and 4.69%, respectively. Ablation experiments on the UEA and the IF steel inclusions defect classification further illustrate the effectiveness of the proposed pre-trained method, Transformer-Mixer module, residual information, and the PolyLoss loss function.

Key words: multi-dimensional time series classification, Transformer-Mixer module, machine learning, pre-training, IF steel inclusions defect prediction

中图分类号:

TP391

邓泽先, 张云贵, 张琳. 基于预训练递归Transformer-Mixer的多维时间序列分类研究[J]. 计算机工程, 2025, 51(5): 154-165.

DENG Zexian, ZHANG Yungui, ZHANG Lin. Research on Multi-Dimensional Time Series Classification Based on the Pre-Trained Recursive Transformer-Mixer[J]. Computer Engineering, 2025, 51(5): 154-165.

https://www.ecice06.com/CN/Y2025/V51/I5/154

参考文献

[1] MEHDIYEV N, LAHANN J, EMRICH A, et al. Time series classification using deep learning for process planning:a case from the process industry[J]. Procedia Computer Science, 2017, 114:242-249.
[2] DONG F, CHEN S, DEMACHI K, et al. Attention-based time series analysis for data-driven anomaly detection in nuclear power plants[J]. Nuclear Engineering and Design, 2023, 404:112161.
[3] ROCHETEAU E, LIÒ P, HYLAND S. Predicting length of stay in the intensive care unit with temporal pointwise convolutional networks[EB/OL].[2023-11-25] . https://arxiv.org/abs/2006.16109v1.
[4] 田红丽,崔姚,闫会强.融合图卷积和卷积自注意力的股票预测方法[J].计算机工程与应用, 2024, 60(4):192-199. TIAN H L, CUI Y, YAN H Q. Stock prediction method combining graph convolution and convolution self-attention[J]. Computer Engineering and Applications, 2024, 60(4):192-199.(in Chinese)
[5] GROSSI E, VALBUSA G, BUSCEMA M. Detection of an autism EEG signature from only two EEG channels through features extraction and advanced machine learning analysis[J]. Clinical EEG and Neuroscience, 2021, 52(5):330-337.
[6] 詹熙,黎维,潘志松. Multi-shapelet:一种基于shapelet的多变量时间序列分类方法[J].数据采集与处理, 2023, 38(2):386-400. ZHAN X, LI W, PAN Z S. Multi-shapelet:a multivariate time series classification method based on shapelet[J]. Journal of Data Acquisition and Processing, 2023, 38(2):386-400.(in Chinese)
[7] SCHÄFER P, LESER U. Multivariate time series classification with WEASEL+MUSE[EB/OL].[2023-11-25] . https://arxiv.org/pdf/1711.11343.
[8] LI G, CHOI B, XU J, et al. ShapeNet:a shapelet-neural network approach for multivariate time series classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence. California, USA:AAAI Press, 2021:8375-8383.
[9] BAGNALL A, DAU H A, LINES J, et al. The UEA multivariate time series classification archive, 2018[EB/OL].[2023-11-25] . https://arxiv.org/pdf/1811.00075.pdf.
[10] SHOKOOHI-YEKTA M, WANG J, KEOGH E. On the non-trivial generalization of dynamic time warping to the multi-dimensional case[C]//Proceedings of the 2015 SIAM International Conference on Data Mining (SDM). Philadelphia, USA:Society for Industrial and Applied Mathematics, 2015:289-297.
[11] SHOKOOHI-YEKTA M, HU B, JIN H X, et al. Generalizing DTW to the multi-dimensional case requires an adaptive approach[J]. Data Mining and Knowledge Discovery, 2017, 31(1):1-31.
[12] 沈彧,陈庆奎.面向时间序列相似性的疾病风险评估模型[J].小型微型计算机系统, 2022, 43(9):1869-1876. SHEN Y, CHEN Q K. Disease risk assessment model based on similarity measurement of multivariate time series[J]. Journal of Chinese Computer Systems, 2022, 43(9):1869-1876.(in Chinese)
[13] ZHENG Y, LIU Q, CHEN E H, et al. Exploiting multi-channels deep convolutional neural networks for multivariate time series classification[J]. Frontiers of Computer Science, 2016, 10(1):96-112.
[14] KARIM F, MAJUMDAR S, DARABI H, et al. Multivariate LSTM-FCNs for time series classification[J]. Neural Networks, 2019, 116:237-245.
[15] ISMAIL FAWAZ H, LUCAS B, FORESTIER G, et al. Inceptiontime:finding AlexNet for time series classification[EB/OL].[2023-11-25] . https://arxiv.org/pdf/1909.04939.pdf.
[16] ZHANG X C, GAO Y F, LIN J, et al. TapNet:multivariate time series classification with attentional prototypical network[C]//Proceedings of the AAAI Conference on Artificial Intelligence. California, USA:AAAI Press, 2020:6845-6852.
[17] 霍纬纲,侯振环.基于多尺度卷积自注意力的多维时间序列预测[J].计算机工程与设计, 2023, 44(4):1250-1258. HUO W G, HOU Z H. Multivariate time series forecasting using multi-scale convolutional self-attention[J]. Computer Engineering and Design, 2023, 44(4):1250-1258.(in Chinese)
[18] 刘杭,殷歆,陈杰,等.基于混合网络模型的多维时间序列预测[J].计算机工程, 2023, 49(1):121-129. LIU H, YIN X, CHEN J, et al. Multi-dimensional time-series prediction based on hybrid network models[J]. Computer Engineering, 2023, 49(1):121-129.(in Chinese)
[19] 王慧强,陈楚皓,吕宏武,等.基于双向稀疏Transformer的多变量时序分类模型[J].小型微型计算机系统, 2024, 45(3):555-561. WANG H Q, CHEN C H, LV H W, et al. Multivariate time series classification model based on bidirectional sparse Transformer[J]. Journal of Chinese Computer Systems, 2024, 45(3):555-561.(in Chinese)
[20] WU H X, WU J L, XU J H, et al. Flowformer:linearizing Transformers with conservation flows[EB/OL].[2023-11-25] . https://arxiv.org/abs/2202.06258?context=cs.AI.
[21] DEMPSTER A, PETITJEAN F, WEBB G I. ROCKET:exceptionally fast and accurate time series classification using random convolutional kernels[J]. Data Mining and Knowledge Discovery, 2020, 34(5):1454-1495.
[22] MUNIR M, SIDDIQUI S A, DENGEL A, et al. DeepAnT:a deep learning approach for unsupervised anomaly detection in time series[J]. IEEE Access, 2019, 7:1991-2005.
[23] ZHOU Y J, XU K, HE F, et al. Application of time series data anomaly detection based on deep learning in continuous casting process[J]. ISIJ International, 2022, 62(4):689-698.
[24] MA M, SUN C, CHEN X F. Deep coupling autoencoder for fault diagnosis with multimodal sensory data[J]. IEEE Transactions on Industrial Informatics, 2018, 14(3):1137-1145.
[25] CHOWDHURY R R, ZHANG X, SHANG J, et al. TARNet:task-aware reconstruction for time-series Transformer[C]//Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York, USA:ACM Press, 2022:212-220.
[26] 邓昕,刘朝晖,欧阳燕,等.基于CNN CBAM-BiGRU Attention的加密恶意流量识别[J].计算机工程, 2023, 49(11):178-186. DENG X, LIU Z H, OU Y Y, et al. Encrypted malicious traffic identification based on CNN CBAM-BiGRU attention[J]. Computer Engineering, 2023, 49(11):178-186.(in Chinese)
[27] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA:ACM Press, 2007:6000-66010.
[28] TOLSTIKHIN I, HOULSBY N, KOLESNIKOV A, et al. MLP-Mixer:an all-MLP architecture for vision[EB/OL].[2023-11-25] . https://arxiv.org/abs/2105.01601.
[29] ZONG B, SONG Q, MIN M R, et al. Deep autoencoding gaussian mixture model for unsupervised anomaly detection[EB/OL].[2023-11-25] . https://openreview.net/pdf?id=BJJLHbb0-.
[30] LENG Z Q, TAN M X, LIU C X, et al. PolyLoss:a polynomial expansion perspective of classification loss functions[EB/OL].[2023-11-25] . https://arxiv.org/abs/2204.12511.
[31] DEMPSTER A, SCHMIDT D F, WEBB G I. MiniRocket:a very fast (almost) deterministic transform for time series classification[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York, USA:ACM Press, 2021:248-257.
[32] GAO G, GAO Q, YANG X, et al. A reinforcement learning-informed pattern mining framework for multivariate time series classification[C]//Proceedings of the 31st International Joint Conference on Artificial Intelligence. California, USA:IJCAI, 2022:2994-3000.
[33] 李守华,李俊.汽车用高强度IF钢的研究进展[J].上海金属, 2007, 5:66-70. LI S H, LI J. Progress in research of high strength IF steel for automotive applications[J]. Shanghai Metals, 2007, 5:66-70.(in Chinese)

选择文件类型/文献管理软件名称

选择包含的内容