Research On Food Nutrition Evaluation Methods Based On Multimodal Learning

With the development of the economy and the continuous improvement of people’s living standards,food safety and health have become issues of sustained public concern.The nutritional content of food is an important indicator of its healthiness.The "Chinese Resident Nutrition and Chronic Disease Report" released by the State Council in 2022 pointed out that unhealthy lifestyles still prevail among Chinese residents,with the overweight and obesity rate of adult residents exceeding 50%,and the incidence of chronic diseases such as hypertension and diabetes is on the rise.Understanding and evaluating the nutritional content of food is of great significance for public self-health management and disease prevention.The evaluation of food nutritional content has become a hot topic in various fields such as food science,computer vision,and nutrition and health.The number of professional food nutrition assessors cannot meet the demand for daily food nutrition assessment,and the accuracy of intelligent dietary assessment applications and food nutrition estimation systems is still insufficient.To alleviate the shortage of nutritional assessment professionals and improve the accuracy and efficiency of nutritional assessment,this thesis takes food images in the catering environment as the research object and proposes an automated food nutrient assessment method based on computer vision,combining multimodal fusion and deep learning methods.The specific work is as follows:(1)This thesis proposes an automated food nutrition evaluation method,Swin-Nutrition,which aims to predict the nutrient content of food from its RGB image,with the goal of improving the accuracy of nutrient estimation by capturing the feature information in the image.SwinNutrition utilizes Swin Transformer as the backbone network for feature extraction and features fusion module and nutrient prediction module for nutrient content evaluation.To capture more effective feature representations,a pyramid structure-based feature fusion module is designed to fuse multi-scale feature maps,enhancing feature representation and improving the accuracy of nutrient content prediction.(2)To overcome the limitations of single RGB image-based nutrient prediction,this thesis proposes a multi-modal fusion-based food nutrition evaluation method,RGB-D Net.The current food nutrition evaluation methods either rely on a single RGB image or simple fusion of RGB and depth images,ignoring the differences in the features of different modal images and the commonalities and characteristics between them.The proposed method uses a multi-modal feature fusion module(MMFF)to fully integrate the rich feature information from RGB and depth images,and a multi-scale feature fusion method to fuse features at different resolutions,improving the accuracy of nutrition evaluation.(3)Extensive experiments on nutrition evaluation datasets demonstrate the feasibility and effectiveness of the proposed food nutrient estimation methods.Visual analysis of the prediction results for four nutrients intuitively demonstrate the reliability of the model predictions.In summary,this thesis proposes two automatic and efficient nutrition evaluation models to address the existing issues in food nutrient assessment.Extensive experimental evaluations have demonstrated that the proposed research methods improve the accuracy of nutrient prediction and provide a novel perspective for the further development of food nutrient evaluation.This research also meets the needs of people for dietary monitoring and ensures a balanced diet for the public.Moreover,this study promotes the deployment of automated food nutrient evaluation methods in daily life.