Research on Multi-Dimensional Time Series Classification Based on the Pre-Trained Recursive Transformer-Mixer

doi:10.19678/j.issn.1000-3428.0069143

Abstract

Abstract:

Multi-dimensional time series classification is widely used in industry, medical treatment, finance and other fields; it plays an important role in industrial product quality control, disease prediction, financial risk control and so on. Aiming at the problem that time dependence and spatial dependence of multi-dimensional time series are equally important, and that traditional multi-dimensional time series models only focus on a certain dimension of time or space, this paper proposes a multi-dimensional time series classification model based on the pre-trained recursive Transformer-Mixer PRTMMTSC. The model is based on a Transformer-Mixer module that can fully learn the temporal and spatial correlations of multi-dimensional time series. To further improve the classification performance, inspired by the anomaly detection model, the proposed model combines the pre-trained hidden layer features and the residual features, and uses the PolyLoss loss function for training. To reduce the number of model training parameters, the Transformer-Mixer module in the model is constructed recursively, so that the number of multi-layer trainable parameters is only the number of single-layer Transformer-Mixer parameters. The experimental results on the UEA datasets show that the performance of the proposed model is better than that of the contrast models. Compared with the TARNet model and the RLPAM model, the accuracy of proposed model has increased by 3.03% and 4.69%, respectively. Ablation experiments on the UEA and the IF steel inclusions defect classification further illustrate the effectiveness of the proposed pre-trained method, Transformer-Mixer module, residual information, and the PolyLoss loss function.

Key words: multi-dimensional time series classification, Transformer-Mixer module, machine learning, pre-training, IF steel inclusions defect prediction

摘要：

多维时间序列分类在工业、医疗、金融等领域有着广泛应用, 在工业产品质量控制、疾病预测、金融风险控制等方面发挥着重要作用。多维时间序列时间依赖关系和空间依赖关系同等重要, 传统多维时间序列模型只对时间或空间某一维度重点关注。为此, 提出一种基于预训练递归Transformer-Mixer的多维时间序列分类模型PRTMMTSC。模型基于Transformer-Mixer模块充分学习多维时间序列时间和空间的关联关系。为进一步提升分类模型的性能, 受异常检测模型的启发, 将预训练后的隐藏层特征和残差特征进行融合, 并采用PolyLoss损失函数对模型进行训练。为减少模型训练参数量, 模型中Transformer-Mixer模块采用递归方式构建, 使多层可训练参数量仅为单层Transformer-Mixer参数量。在UEA多维时间数据集上的实验结果表明, 所提模型的性能优于对比模型, 相较于TARNet模型和RLPAM模型的准确率分别提升3.03%和4.69%。在UEA及IF钢夹渣缺陷分类的消融实验验证预训练方式、Transformer-Mixer模块、残差信息及PolyLoss损失函数的有效性。

关键词: 多维时间序列分类, Transformer-Mixer模块, 机器学习, 预训练, IF钢夹渣缺陷预报

DENG Zexian, ZHANG Yungui, ZHANG Lin. Research on Multi-Dimensional Time Series Classification Based on the Pre-Trained Recursive Transformer-Mixer[J]. Computer Engineering, 2025, 51(5): 154-165.

邓泽先, 张云贵, 张琳. 基于预训练递归Transformer-Mixer的多维时间序列分类研究[J]. 计算机工程, 2025, 51(5): 154-165.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069143

https://www.ecice06.com/EN/Y2025/V51/I5/154

Figures/Tables 12

Fig.1 The structure of the Transformer module

Fig.2 The overall structure of the PRTMMTSC model

Fig.3 The structure of the recursive Transformer-Mixer module

Fig.4 The structure of the classification head

Fig.5 Recovery results of a certain sample channel signal on FD、FM、NATOPS and CR datasets

Fig.6 Transformer-Mixer for spatial and temporal attention maps of a certain sample on FM dataset

References 33

1	MEHDIYEV N , LAHANN J , EMRICH A , et al. Time series classification using deep learning for process planning: a case from the process industry. Procedia Computer Science, 2017, 114, 242- 249. doi: 10.1016/j.procs.2017.09.066
2	DONG F , CHEN S , DEMACHI K , et al. Attention-based time series analysis for data-driven anomaly detection in nuclear power plants. Nuclear Engineering and Design, 2023, 404, 112161. doi: 10.1016/j.nucengdes.2023.112161
3	ROCHETEAU E, LIÒ P, HYLAND S. Predicting length of stay in the intensive care unit with temporal pointwise convolutional networks[EB/OL]. [2023-11-25]. https://arxiv.org/abs/2006.16109v1.
4	田红丽, 崔姚, 闫会强. 融合图卷积和卷积自注意力的股票预测方法. 计算机工程与应用, 2024, 60 (4): 192- 199.
	TIAN H L , CUI Y , YAN H Q . Stock prediction method combining graph convolution and convolution self-attention. Computer Engineering and Applications, 2024, 60 (4): 192- 199.
5	GROSSI E , VALBUSA G , BUSCEMA M . Detection of an autism EEG signature from only two EEG channels through features extraction and advanced machine learning analysis. Clinical EEG and Neuroscience, 2021, 52 (5): 330- 337. doi: 10.1177/1550059420982424
6	詹熙, 黎维, 潘志松. Multi-shapelet: 一种基于shapelet的多变量时间序列分类方法. 数据采集与处理, 2023, 38 (2): 386- 400.
	ZHAN X , LI W , PAN Z S . Multi-shapelet: a multivariate time series classification method based on shapelet. Journal of Data Acquisition and Processing, 2023, 38 (2): 386- 400.
7	SCHÄFER P, LESER U. Multivariate time series classification with WEASEL+MUSE[EB/OL]. [2023-11-25]. https://arxiv.org/pdf/1711.11343.
8	LI G, CHOI B, XU J, et al. ShapeNet: a shapelet-neural network approach for multivariate time series classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence. California, USA: AAAI Press, 2021: 8375-8383.
9	BAGNALL A, DAU H A, LINES J, et al. The UEA multivariate time series classification archive, 2018[EB/OL]. [2023-11-25]. https://arxiv.org/pdf/1811.00075.pdf.
10	SHOKOOHI-YEKTA M, WANG J, KEOGH E. On the non-trivial generalization of dynamic time warping to the multi-dimensional case[C]//Proceedings of the 2015 SIAM International Conference on Data Mining (SDM). Philadelphia, USA: Society for Industrial and Applied Mathematics, 2015: 289-297.
11	SHOKOOHI-YEKTA M , HU B , JIN H X , et al. Generalizing DTW to the multi-dimensional case requires an adaptive approach. Data Mining and Knowledge Discovery, 2017, 31 (1): 1- 31. doi: 10.1007/s10618-016-0455-0
12	沈彧, 陈庆奎. 面向时间序列相似性的疾病风险评估模型. 小型微型计算机系统, 2022, 43 (9): 1869- 1876.
	SHEN Y , CHEN Q K . Disease risk assessment model based on similarity measurement of multivariate time series. Journal of Chinese Computer Systems, 2022, 43 (9): 1869- 1876.
13	ZHENG Y , LIU Q , CHEN E H , et al. Exploiting multi-channels deep convolutional neural networks for multivariate time series classification. Frontiers of Computer Science, 2016, 10 (1): 96- 112. doi: 10.1007/s11704-015-4478-2
14	KARIM F , MAJUMDAR S , DARABI H , et al. Multivariate LSTM-FCNs for time series classification. Neural Networks, 2019, 116, 237- 245. doi: 10.1016/j.neunet.2019.04.014
15	ISMAIL FAWAZ H, LUCAS B, FORESTIER G, et al. Inceptiontime: finding AlexNet for time series classification[EB/OL]. [2023-11-25]. https://arxiv.org/pdf/1909.04939.pdf.
16	ZHANG X C, GAO Y F, LIN J, et al. TapNet: multivariate time series classification with attentional prototypical network[C]//Proceedings of the AAAI Conference on Artificial Intelligence. California, USA: AAAI Press, 2020: 6845-6852.
17	霍纬纲, 侯振环. 基于多尺度卷积自注意力的多维时间序列预测. 计算机工程与设计, 2023, 44 (4): 1250- 1258.
	HUO W G , HOU Z H . Multivariate time series forecasting using multi-scale convolutional self-attention. Computer Engineering and Design, 2023, 44 (4): 1250- 1258.
18	刘杭, 殷歆, 陈杰, 等. 基于混合网络模型的多维时间序列预测. 计算机工程, 2023, 49 (1): 121- 129. doi: 10.19678/j.issn.1000-3428.0063718
	LIU H , YIN X , CHEN J , et al. Multi-dimensional time-series prediction based on hybrid network models. Computer Engineering, 2023, 49 (1): 121- 129. doi: 10.19678/j.issn.1000-3428.0063718
19	王慧强, 陈楚皓, 吕宏武, 等. 基于双向稀疏Transformer的多变量时序分类模型. 小型微型计算机系统, 2024, 45 (3): 555- 561.
	WANG H Q , CHEN C H , LV H W , et al. Multivariate time series classification model based on bidirectional sparse Transformer. Journal of Chinese Computer Systems, 2024, 45 (3): 555- 561.
20	WU H X, WU J L, XU J H, et al. Flowformer: linearizing Transformers with conservation flows[EB/OL]. [2023-11-25]. https://arxiv.org/abs/2202.06258?context=cs.AI.
21	DEMPSTER A , PETITJEAN F , WEBB G I . ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 2020, 34 (5): 1454- 1495. doi: 10.1007/s10618-020-00701-z
22	MUNIR M , SIDDIQUI S A , DENGEL A , et al. DeepAnT: a deep learning approach for unsupervised anomaly detection in time series. IEEE Access, 2019, 7, 1991- 2005. doi: 10.1109/ACCESS.2018.2886457
23	ZHOU Y J , XU K , HE F , et al. Application of time series data anomaly detection based on deep learning in continuous casting process. ISIJ International, 2022, 62 (4): 689- 698. URL
24	MA M , SUN C , CHEN X F . Deep coupling autoencoder for fault diagnosis with multimodal sensory data. IEEE Transactions on Industrial Informatics, 2018, 14 (3): 1137- 1145. URL
25	CHOWDHURY R R, ZHANG X, SHANG J, et al. TARNet: task-aware reconstruction for time-series Transformer[C]//Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York, USA: ACM Press, 2022: 212-220.
26	邓昕, 刘朝晖, 欧阳燕, 等. 基于CNN CBAM-BiGRU Attention的加密恶意流量识别. 计算机工程, 2023, 49 (11): 178- 186. doi: 10.19678/j.issn.1000-3428.0066558
	DENG X , LIU Z H , OU Y Y , et al. Encrypted malicious traffic identification based on CNN CBAM-BiGRU attention. Computer Engineering, 2023, 49 (11): 178- 186. doi: 10.19678/j.issn.1000-3428.0066558
27	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2007: 6000-66010.
28	TOLSTIKHIN I, HOULSBY N, KOLESNIKOV A, et al. MLP-Mixer: an all-MLP architecture for vision[EB/OL]. [2023-11-25]. https://arxiv.org/abs/2105.01601.
29	ZONG B, SONG Q, MIN M R, et al. Deep autoencoding gaussian mixture model for unsupervised anomaly detection[EB/OL]. [2023-11-25]. https://openreview.net/pdf?id=BJJLHbb0-.
30	LENG Z Q, TAN M X, LIU C X, et al. PolyLoss: a polynomial expansion perspective of classification loss functions[EB/OL]. [2023-11-25]. https://arxiv.org/abs/2204.12511.
31	DEMPSTER A, SCHMIDT D F, WEBB G I. MiniRocket: a very fast (almost) deterministic transform for time series classification[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York, USA: ACM Press, 2021: 248-257.
32	GAO G, GAO Q, YANG X, et al. A reinforcement learning-informed pattern mining framework for multivariate time series classification[C]//Proceedings of the 31st International Joint Conference on Artificial Intelligence. California, USA: IJCAI, 2022: 2994-3000.
33	李守华, 李俊. 汽车用高强度IF钢的研究进展. 上海金属, 2007, 5, 66- 70.
	LI S H , LI J . Progress in research of high strength IF steel for automotive applications. Shanghai Metals, 2007, 5, 66- 70.

[1]	ZHUANG Ziwei, ZHU Junguo. Vietnamese Text Error Detection Method for Multi-source Text [J]. Computer Engineering, 2025, 51(5): 93-102.
[2]	GENG Xia, WANG Yao. Cloth-Changing Person Re-Identification Method Based on CLIP Enhanced Fine-Grained Features [J]. Computer Engineering, 2025, 51(4): 293-302.
[3]	ZHU Hong, WANG Kuoran, ZHU Tong. Entity Similarity Metrics and Alignment Method Based on the Union of Multi-Side Information Representations [J]. Computer Engineering, 2025, 51(3): 64-75.
[4]	RAO Dongning, XU Zhenghui, LIANG Ruishi. Research on Answer Generation Based on Knowledge Base Question Answering [J]. Computer Engineering, 2025, 51(2): 94-101.
[5]	YAO Lifeng, CAI Manchun, ZHU Yi, CHEN Yonghao, ZHANG Yiwen. Encrypted Traffic Classification Model Based on Byte Coding and Pre-Training Tasks [J]. Computer Engineering, 2025, 51(2): 188-201.
[6]	FEI Tao, Aishan Wumaier, DU Wenxu, ZHU Cuicui. Multi-Granularity and Multi-Aspect Pronunciation Quality Evaluation Method Based on Squeezeformer [J]. Computer Engineering, 2025, 51(1): 81-87.
[7]	Yuhang CHEN, Yong YANG, Xianmusiya·Maimaitiming, Palidan·Tuerxun, Xiaochao FAN, Ge REN, Yufeng DIAO. Automatic Essay Scoring Method Based on Topic Perception and Semantic Enhancement [J]. Computer Engineering, 2024, 50(8): 363-371.
[8]	LI Yongfei, LI Mingyang, CHANG Xin, CAO Kexin. Anomaly Detection of IoT Water Quality Monitoring Data Based on Explainable Deep Learning [J]. Computer Engineering, 2024, 50(6): 179-187.
[9]	XU Mingliang, LI Fangyuan, MA Haoran, HE Fei. Spike Sorting Algorithms for Large-Scale Neural Recording [J]. Computer Engineering, 2024, 50(6): 1-34.
[10]	LI Tianfang, PU Yuanyuan, ZHAO Zhengpeng, XU Dan, QIAN Wenhua. Image-to-Image Translation Based on CLIP and Dual-Spatially Adaptive Normalization [J]. Computer Engineering, 2024, 50(5): 229-240.
[11]	HOU Yutao, Abudukelimu Abulizi, SHI Yaqing, Mayilamu Musideke, Halidanmu Abudukelimu. Research on Low-Resource Language Machine Translation for the ″Belt and Road″ [J]. Computer Engineering, 2024, 50(4): 332-341.
[12]	Mingcheng YU, Yagu DANG, Qilin WU, Xu JI, Kexin BI. Research on Automatic Scoring for English Essay Based on Multi-Scale Context [J]. Computer Engineering, 2024, 50(3): 259-266.
[13]	Yi SUN, Huimei WANG, Ming XIAN, Hang XIANG. Research on Heterogeneous Computing Scheduling Strategy for Kubeflow [J]. Computer Engineering, 2024, 50(2): 25-32.
[14]	ZHANG Cai, MA Ziqiang, YAN Bo. Design of a Machine Learning-Based Sentiment Analysis Model for Government Weibo [J]. Computer Engineering, 2024, 50(12): 386-395.
[15]	Haowei ZHOU, Yong LIU, Ping XUAN. Fake News Detection Based on Pre-Training and Multi-Modal Fusion [J]. Computer Engineering, 2024, 50(1): 289-295.

Please choose a citation manager

Content to export