基于多特征融合的食品图像分类

doi:10.19678/j.issn.1000-3428.0068731

摘要/Abstract

摘要：

随着生活水平的提升, 人们对健康饮食的需求与日俱增, 食品图像识别成为热门研究课题之一。食品加工和烹饪过程的不同造成了同类食品的形状和颜色存在差异, 不同类别的食品也可能会呈现相似的视觉特征, 因此食品图像的识别较一般图像识别难度更大。为了解决上述问题, 提出基于多特征融合的食品图像分类网络MTFNet。首先, 将图像的RGB彩色通道数据与局部二值模式(LBP)对应的纹理特征相融合作为骨干挤压和激励网络(SENet)的输入。接着, 利用细节注意力模块挖掘不同位置上各通道的权重, 进而对各层特征图进行局部增强, 提升特征图局部表征能力。然后, 利用自注意力机制计算特征图各通道之间的自注意力权重, 挖掘特征图间的相关性, 提取图像的全局特征。最后, 将局部增强特征和全局特征拼接融合后进行图像分类。实验结果表明, 在食品图像数据集ETH Food101、ChineseFoodNet和ISIA Food-500上, 与目前最佳的多尺度拼图重构网络(MJR-Net)模型相比, MTFNet模型的Top-1准确率分别提高了0.44、1.01和0.66个百分点, 取得了更好的识别性能。

关键词: 食品图像分类, 局部二值模式, 挤压和激励网络, 细节注意力, 自注意力

Abstract:

With improvements in living standards, the demand for a healthy diet is increasing daily, and the problem of food image recognition has become an important research topic. Owing to the different processing and cooking methods of food, the shape and color of similar food vary, and different types of food may present similar visual characteristics. Hence, the recognition of food images is more challenging than general image recognition. To solve these problems, a multi-feature fusion food image classification network, MTFNet, is proposed. First, the R, G, and B color channel data of the image are fused with the texture features corresponding to the local binary mode as the input of the backbone Squeeze and Excite Network (SENet). A detail attention module is then proposed to mine the weights of each channel at different positions, which can enhance the local information of the feature map of each layer and improve its local representation ability. Subsequently, the self-attention mechanism is applied to calculate the self-attention weights between each channel of the feature map, which can mine the correlation between the feature maps and extract the global features of the image. Finally, the locally enhanced and global features are concatenated and fused to classify the images. The experimental results indicate that the Top-1 accuracy of the MTFNet model is improved by 0.44, 1.01, and 0.66 percentage points on the ETH Food101, ChineseFoodNet, and ISIA Food-500 food image datasets, respectively, as compared with Multi-scale Jigsaw Reconstruction Network (MJR-Net), achieving the best recognition performance.

Key words: food image classification, Local Binary Pattern (LBP), Squeeze and Excite Network (SENet), detail attention, self-attention

叶志鹏, 姜枫. 基于多特征融合的食品图像分类[J]. 计算机工程, 2024, 50(12): 254-264.

YE Zhipeng, JIANG Feng. Food Image Classification Based on Multi-Feature Fusion[J]. Computer Engineering, 2024, 50(12): 254-264.

https://www.ecice06.com/CN/Y2024/V50/I12/254

图/表 11

图1 MTFNet模型框架

Fig.1 Framework of the MTFNet model

图2 LBP算法原理图

Fig.2 Schematic diagram of the LBP algorithm

图3 SENet-154模型结构

Fig.3 SENet-154 model structure

图4 细节注意力机制原理图

Fig.4 Schematic diagram of the detail attention mechanism

图5 4组类别的食品图像样例图

Fig.5 Sample images of the food of four category groups

图6 细节注意力模块消融可视化实验结果

Fig.6 Visualization experiment results of the detail attention module ablation

图7 特征图可视化实验结果

Fig.7 Visualization experiment results of the feature maps

参考文献 40

1	闵巍庆, 刘林虎, 刘宇昕, 等. 食品图像识别方法综述. 计算机学报, 2022, 45 (3): 542- 566. doi: 10.11897/SP.J.1016.2022.00542
	MIN W Q , LIU L H , LIU Y X , et al. Review of food image recognition methods. Chinese Journal of Computers, 2022, 45 (3): 542- 566. doi: 10.11897/SP.J.1016.2022.00542
2	霍光, 林大为, 刘元宁, 等. 基于轻量级卷积神经网络的小样本虹膜图像分割. 吉林大学学报(理学版), 2023, 61 (3): 583- 591. doi: 10.13413/j.cnki.jdxblxb.2022078
	HUO G , LIN D W , LIU Y N , et al. Small-sample iris image segmentation based on lightweight convolutional neural networks. Journal of Jilin University (Science Edition), 2023, 61 (3): 583- 591. doi: 10.13413/j.cnki.jdxblxb.2022078
3	MING Z Y, CHEN J J, CAO Y, et al. Food photo recognition for dietary tracking: system and experiment[C]//Proceedings of International Conference on Multimedia Modeling. Berlin, Germany: Springer, 2018: 129-141.
4	ZHU L , LI Z B , LI C , et al. High performance vegetable classification from images based on AlexNet deep learning model. International Journal of Agricultural and Biological Engineering, 2018, 11 (4): 190- 196. doi: 10.25165/j.ijabe.20181104.2690
5	YANG H , KANG S , PARK C , et al. A hierarchical deep model for food classification from photographs. KSII Transactions on Internet and Information Systems, 2020, 14 (4): 1704- 1720. doi: 10.3837/tiis.2020.04.016
6	申志军, 穆丽娜, 高静, 等. 细粒度图像分类综述. 计算机应用, 2023, 43 (1): 51- 60. doi: 10.11772/j.issn.1001-9081.2021122090
	SHEN Z J , MU L N , GAO J , et al. Review of fine-grained image categorization. Journal of Computer Applications, 2023, 43 (1): 51- 60. doi: 10.11772/j.issn.1001-9081.2021122090
7	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2023-09-12]. http://arxiv.org/abs/1409.1556.
8	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA. IEEE Press, 2016: 770-778.
9	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA. IEEE Press, 2018: 7132-7141.
10	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[EB/OL]. [2023-09-12]. http://arxiv.org/abs/2010.11929.
11	MIN W Q, LIU L H, WANG Z L, et al. ISIA food-500: a dataset for large-scale food recognition via stacked global-local attention network[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York, USA: ACM Press, 2020: 393-401.
12	刘宇昕, 闵巍庆, 蒋树强, 等. 多尺度拼图重构网络的食品图像识别. 软件学报, 2022, 33 (11): 4379- 4395. doi: 10.13328/j.cnki.jos.006325
	LIU Y X , MIN W Q , JIANG S Q , et al. Food image recognition based on multi-scale jigsaw reconstruction network. Journal of Software, 2022, 33 (11): 4379- 4395. doi: 10.13328/j.cnki.jos.006325
13	LECUN Y, BENGIO Y. Convolutional networks for images, speech, and time series[EB/OL]. [2023-09-12]. http://pdfs.semanticscholar.org/e26c/c4a1c717653f323715d751c8dea7461aa105.pdf
14	HOCHREITER S , SCHMIDHUBER J . Long short-term memory. Neural Computation, 1997, 9 (8): 1735- 1780. doi: 10.1162/neco.1997.9.8.1735
15	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL]. [2023-09-12]. http://arxiv.org/abs/1706.03762.
16	GEIRHOS R, RUBISCH P, MICHAELIS C, et al. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness[EB/OL]. [2023-09-12]. http://arxiv.org/abs/1811.12231.
17	NASEER M, RANASINGHE K, KHAN S, et al. Intriguing properties of vision Transformers[EB/OL]. [2023-09-12]. http://arxiv.org/abs/2105.10497.
18	LIU Z, LIN Y T, CAO Y, et al. Swin Transformer: hierarchical vision Transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2021: 9992-10002.
19	YUAN L , HOU Q B , JIANG Z H , et al. VOLO: vision outlooker for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45 (5): 6575- 6586.
20	CHU X X, TIAN Z, WANG Y Q, et al. Twins: revisiting the design of spatial attention in vision Transformers[EB/OL]. [2023-09-12]. http://arxiv.org/abs/2104.13840.
21	DONAHUE J, JIA Y Q, VINYALS O, et al. DeCAF: a deep convolutional activation feature for generic visual recognition[EB/OL]. [2023-09-12]. http://arxiv.org/abs/1310.1531.
22	LIN T Y, ROYCHOWDHURY A, MAJI S. Bilinear CNNs for fine-grained visual recognition[EB/OL]. [2023-09-12]. http://arxiv.org/abs/1504.07889.
23	KONG S, FOWLKES C. Low-rank bilinear pooling for fine-grained classification[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2017: 7025-7034.
24	LI Y H, WANG N Y, LIU J Y, et al. Factorized bilinear models for image recognition[C]//Proceedings of the IEEE International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2017: 2098-2106.
25	GAO Y, BEIJBOM O, ZHANG N, et al. Compact bilinear pooling[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2016: 317-326.
26	JADERBERG M, SIMONYAN K, ZISSERMAN A, et al. Spatial Transformer networks[EB/OL]. [2023-09-12]. http://arxiv.org/abs/1506.02025.
27	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 3-19.
28	BOSSARD L, GUILLAUMIN M, VAN GOOL L. Food-101-mining discriminative components with random forests[C]// Proceedings of the European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 446-461.
29	HE H S , KONG F Y , TAN J D . DietCam: multiview food recognition using a multikernel SVM. IEEE Journal of Biomedical and Health Informatics, 2016, 20 (3): 848- 855. doi: 10.1109/JBHI.2015.2419251
30	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2015: 1-9.
31	HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2017: 2261-2269.
32	KAGAYA H, AIZAWA K, OGAWA M. Food detection and recognition using convolutional neural network[C]//Proceedings of the 22nd ACM international conference on Multimedia. New York, USA: ACM Press, 2014: 1085-1088.
33	YANAI K, KAWANO Y. Food image recognition using deep convolutional network with pre-training and fine-tuning[C]//Proceedings of the IEEE International Conference on Multimedia [WT《Times New Roman》]& Expo Workshops (ICMEW). Washington D.C., USA: IEEE Press, 2015: 1-6.
34	METWALLI A S, SHEN W, WU C Q. Food image recognition based on densely connected convolutional neural networks[C]//Proceedings of the International Conference on Artificial Intelligence in Information and Communication (ICAIIC). Washington D.C., USA: IEEE Press, 2020: 27-32.
35	FENG S , WANG Y G , GONG J H , et al. A fine-grained recognition technique for identifying Chinese food images. Heliyon, 2023, 9 (11): e21565. doi: 10.1016/j.heliyon.2023.e21565
36	LIU X D, ZHU Y H, LIU L H, et al. Feature-Suppressed Contrast for self-supervised food pre-training[C]//Proceedings of the Proceedings of the 31st ACM International Conference on Multimedia. New York, USA: ACM Press, 2023: 4359-4367.
37	BETTADAPURA V, THOMAZ E, PARNAMI A, et al. Leveraging context to support automated food recognition in restaurants[C]//Proceedings of the IEEE Winter Conference on Applications of Computer Vision. Washington D.C., USA: IEEE Press, 2015: 580-587.
38	WANG Z L , MIN W Q , LI Z , et al. Ingredient-guided region discovery and relationship modeling for food category-ingredient prediction. IEEE Transactions on Image Processing, 2022, 31, 5214- 5226. doi: 10.1109/TIP.2022.3193763
39	CHEN X, ZHU Y, ZHOU H, et al. ChineseFoodNet: a large-scale image dataset for Chinese food recognition[EB/OL]. [2023-09-12]. http://arxiv.org/abs/1705.02743.
40	SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization[C]//Proceedings of the IEEE International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2017: 618-626.

[1]	杨明强, 卢健. 基于跨模态注意力的目标语音提取[J]. 计算机工程, 2024, 50(9): 121-129.
[2]	蔡俊民, 梁正友, 孙宇, 陈子奥. 基于可变形三维图卷积的轻量级点云分类研究[J]. 计算机工程, 2024, 50(9): 255-265.
[3]	陈瀚, 赵春蕾, 蒋昊达, 王春东. 基于融合模型与语义网络的App用户意图识别研究[J]. 计算机工程, 2024, 50(8): 50-63.
[4]	陈明, 牛燕菲, 段莉, 高铁梁, 楚杨阳, 曹洁. 基于残差自注意力和分离集合匹配的高效端到端航天器组件检测[J]. 计算机工程, 2024, 50(8): 301-309.
[5]	王夙喆, 张雪英, 陈晓玉, 李凤莲, 吴泽林. 基于有效注意力和GAN结合的脑卒中EEG增强算法[J]. 计算机工程, 2024, 50(8): 336-344.
[6]	贺姗, 蔺素珍, 王彦博, 李大威. 基于特征融合的多波段图像描述生成方法[J]. 计算机工程, 2024, 50(6): 236-244.
[7]	谢帅康, 熊风光, 朱新杰, 宋宁栋, 李文清, 王廷凤. 基于空间可变形Transformer的三维点云配准方法[J]. 计算机工程, 2024, 50(3): 224-232.
[8]	徐浩宸, 刘满华. 基于多层次自注意力网络的人脸特征点检测[J]. 计算机工程, 2024, 50(2): 239-246.
[9]	王正家, 胡飞飞, 张成娟, 雷卓, 何涛. 引入轻量级Transformer的自适应窗口立体匹配算法[J]. 计算机工程, 2024, 50(2): 256-265.
[10]	郭祥振, 李思潼, 卢锐, 郭森, 崔学荣, 杨钢. 基于多任务联合注意力的结肠息肉分割网络[J]. 计算机工程, 2024, 50(2): 327-336.
[11]	吕文锐, 普园媛, 赵征鹏, 张衡, 阳秋霞. 基于字形约束和注意力的艺术字体风格迁移[J]. 计算机工程, 2024, 50(12): 306-317.
[12]	杨兴耀, 李志林, 张祖莲, 于炯, 陈嘉颖, 王东晓. 基于层间融合滤波器与社交神经引文网络的推荐算法[J]. 计算机工程, 2024, 50(11): 98-106.
[13]	王文静, 范涛, 王国中, 赵海武. 基于SA-BPNN多模态融合的教学质量评价方法[J]. 计算机工程, 2024, 50(11): 390-398.
[14]	杨瑞君, 秦晋京, 程燕. 基于生成对抗网络的自然场景低照度增强模型[J]. 计算机工程, 2024, 50(1): 279-288.
[15]	曹广硕, 黄瑞章, 陈艳平, 秦永彬. 基于多模态学习的乳腺癌生存预测研究[J]. 计算机工程, 2024, 50(1): 296-305.

选择文件类型/文献管理软件名称

选择包含的内容