Field-Aware Click-Through Rate Prediction Model Based on Attention Mechanism

doi:10.19678/j.issn.1000-3428.0064134

Abstract

Abstract: Click-Through Rate(CTR) prediction is one of the most important tools for ad placement.Predicting the CTR of an ad and making recommendations to users can increase ad revenue.Field-aware click-through rate prediction models are superior to other click-through rate prediction models because they consider the field information; however, they generate a large amount of redundant information during feature interaction, which results in a low prediction accuracy.A Field-aware Attention Embedding Neural Network(FAENN) model is herein proposed.This model uses a Self-Attentive Mechanism(SAM) to distribute weights to the input vectors of the embedding layer.This helps to clearly identify the importance of the field-aware embedded features, speeding up the training process.The lower-order feature interaction layer focuses on the explicit first-order information of the features and the second-order interaction feature information and outputs the effective features to the higher-order interaction layer.The higher-order feature interaction layer combines the learned interaction vectors with the deep neural network to capture higher-order feature interactions to improve prediction accuracy.The experimental results show that the FAENN model has a higher prediction accuracy than the FM, FFM, and AFM models.

Key words: Click-Through Rate(CTR) prediction, represents learning, embedding technology, Self-Attention Mechanism(SAM), Feature Interaction(FI)

摘要： 点击率预测是广告投放的重要手段之一，通过预测广告点击率对用户进行效推荐，能够提高广告收益。在点击率预测任务中，场感知点击率预测模型由于考虑了场信息，表现出一定优越性，但在进行特征交互时会产生大量冗余信息，导致预测准确率较低。提出一种场感知注意嵌入神经网络（FAENN）模型，通过自注意力机制对嵌入层的输入向量进行权重分配，以较好地区分场感知嵌入特征的重要程度，加快模型训练速度。同时使用低阶特征交互层关注特征的一阶显性信息和二阶交互特征信息，并将有效特征输出到高阶交互层，利用高阶特征交互层将学习到的相互作用向量与深度神经网络相结合，捕捉更高阶的特征交互作用，以提高预测准确率。实验结果表明，FAENN模型相比于FM、FFM、AFM等模型有较高的预测准确率。

关键词: 点击率预测, 表示学习, 嵌入技术, 自注意力机制, 特征交互

CLC Number:

TP393

SHEN Xueli, HAN Qianwen. Field-Aware Click-Through Rate Prediction Model Based on Attention Mechanism[J]. Computer Engineering, 2023, 49(3): 80-86,94.

沈学利, 韩倩雯. 基于注意力机制的场感知点击率预测模型[J]. 计算机工程, 2023, 49(3): 80-86,94.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0064134

http://www.ecice06.com/EN/Y2023/V49/I3/80

Figures/Tables 12

References

[1] MCMAHAN H, HOLT G, SCULLEY D, et al.Ad click prediction:a view from the trenches[C]//Proceedings of the 19th International Conference on Knowledge Discovery and Data Mining.New York, USA:ACM Press, 2013:1222-1230.
[2] 邓路佳, 刘平山.基于GMM-FMs的广告点击率预测研究[J].计算机工程, 2019, 45(5):122-126. DENG L J, LIU P S.Research on click-through rate prediction of advertisement based on GMM-FMs[J].Computer Engineering, 2019, 45(5):122-126.(in Chinese)
[3] GUO H F, TANG R M, YE Y M, et al.DeepFM:a factorization-machine based neural network for CTR prediction[EB/OL].[2022-02-08].https://arxiv.org/abs/1703.04247.
[4] ZHOU G R, MOU N, FAN Y, et al.Deep interest evolution network for click-through rate prediction[C]//Proceedings of AAAI Conference on Artificial Intelligence.Menlo Park, USA:AAAI Press, 2019:5941-5948.
[5] CHENG H T, KOC L, HARMSEN J, et al.Wide & deep learning for recommender systems[C]//Proceedings of the 1st Workshop on Deep Learning for Recommender Systems.New York, USA:ACM Press, 2016:7-10.
[6] GUO H F, CHEN B, TANG R M, et al.An embedding learning framework for numerical features in CTR prediction[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.New York, USA:ACM Press, 2021:2910-2918.
[7] ZHANG W N, DU T M, WANG J.Deep learning over multi-field categorical data[M]//FERRO N, CRESTANI F, MOENS M F, et al.Advances in Information Retrieval.Berlin, Germany:Springer, 2016:45-57.
[8] JOGLEKAR M R, LI C, ADAMS J K, et al.Neural input search for large scale recommendation models[EB/OL].[2022-02-08].https://arxiv.org/abs/1907.04471.
[9] QU Y R, CAI H, REN K, et al.Product-based neural networks for user response prediction[C]//Proceedings of the 16th International Conference on Data Mining.New York, USA:ACM Press, 2017:1149-1154.
[10] YANG Y, XU B L, SHEN S F, et al.Operation-aware neural networks for user response prediction[J].Neural Networks, 2020, 121:161-168.
[11] XIAO J, YE H, HE X N, et al.Attentional factorization machines:learning the weight of feature interactions via attention networks[EB/OL].[2022-02-08].https://arxiv.org/abs/1708.04617.
[12] SONG W P, SHI C C, XIAO Z P, et al.AutoInt:automatic feature interaction learning via self-attentive neural networks[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management.New York, USA:ACM Press, 2019:1161-1170.
[13] CHENG W Y, SHEN Y Y, HUANG L P.Adaptive factorization network:learning adaptive-order feature interactions[EB/OL].[2022-02-08].https://arxiv.org/abs/1909.03276.
[14] 蒋兴渝, 黄贤英, 陈雨晶, 等.基于特征增强聚合的融合广告点击率预测模型[J].计算机工程, 2022, 48(1):312-320. JIANG X Y, HUANG X Y, CHEN Y J, et al.Hybrid advertising click-through rate prediction model based on feature enhancement combination[J].Computer Engineering, 2022, 48(1):312-320.(in Chinese)
[15] JUAN Y, ZHUANG Y, CHIN W S, et al.Field-aware factorization machines for CTR prediction[C]//Proceedings of the 10th ACM Conference on Recommender Systems.New York, USA:ACM Press, 2016:43-50.
[16] 许王昊, 肖秦琨.基于注意力机制的兴趣网络点击率预估模型[J].计算机工程, 2021, 47(1):101-108. XU W H, XIAO Q K.Click-through rate prediction model of interest network based on attention mechanism[J].Computer Engineering, 2021, 47(1):101-108.(in Chinese)
[17] HASSANIN M, ANWAR S, RADWAN I, et al.Visual attention methods in deep learning:an In-Depth survey[EB/OL].[2022-02-08].https://arxiv.org/pdf/2204. 07756v1.pdf.
[18] VASWANI, ASHISH, et al.Attention is all you need[EB/OL].[2022-02-08].https://arxiv.org/abs/1706.03762.
[19] HUANG T W, ZHANG Z Q, ZHANG J L.FiBiNET:combining feature importance and bilinear feature interaction for click-through rate prediction[C]//Proceedings of the 13th ACM Conference on Recommender Systems.New York, USA:ACM Press, 2019:169-177.
[20] ZHOU G R, SONG C R, ZHU X Q, et al.Deep interest network for click-through rate prediction[EB/OL].[2022-02-08].https://arxiv.org/abs/1706.06978.
[21] FENG Y F, LV F Y, SHEN W C, et al.Deep session interest network for click-through rate prediction[EB/OL].[2022-02-08].https://arxiv.org/abs/1905.06482.
[22] ZHANG J L, HUANG T W, ZHANG Z Q.FAT-DeepFFM:field attentive deep field-aware factorization machine[EB/OL].[2022-02-08].https://arxiv.org/abs/1905.06336.
[23] LONGSTAFF I D, CROSS J F.A pattern recognition approach to understanding the multi-layer perception[J].Pattern Recognition Letters, 1987, 5(5):315-319.
[24] Criteo Dateset[EB/OL].[2022-02-08].https://kaggle.com/c/criteo-display-ad-challenge, 2014.
[25] Avazu Dateset[EB/OL].[2022-02-08].https://www.kaggle.com/c/avazu-ctr-prediction, 2015.

Please choose a citation manager

Content to export