Adaptive Multi-scale Region-wise Method Based On Multi-label Classification For Mixed-dish Recognition

Mixed dish image is a special type of food image,which belongs to multi-label data,while is different from general multi-label data as all of its dishes are placed in the same container.The irregular shape and overlap of dishes,as well as confounding between dish-boundaries pose great challenges for recognition tasks.Nowadays,with the increasing attention and pursuit of healthy lifestyles,various related web applications and APPs are emerging endlessly.Therefore,as the underlying technology of branch applications such as dietary records,nutrition analysis,and recipe recommendations,food image recognition is increasingly more and more attention.Food recognition tasks have evolved from early manual feature extraction and classification to later end-to-end deep models,and the data processed also ranges from single-label images at the beginning to multi-label images where each of the dishes are in different containers.Currently,there is no targeted study on mixed dish where all dishes are in the same container.In addition,existing deep models usually treat food recognition issue as object detection tasks,and require strong supervision information to obtain prediction results.This article has conducted in-depth research on the problem of mixed-dish recognition,and proposed an adaptive region-wise multi-scale solution which processes the challenging mixed-dish data from the perspective of multi-label classification instead of object detection,based on such intuitive assumption that "each region should contain a part of a certain type of dish",features are divided region by region,and the position information reflected in the feature map is fully used in the process.Also,based on objective facts such as "size and degree of mixing of ingredients distinct in each dish",and "shooting distances during data collection are different",a multi-scale fusion of features with different granularities is conducted.Compared with object detection schemes,the proposed method does not require any bounding-box annotation or pixel-level annotation,but only image-level annotation,which greatly saves manpower and avoids the introduction of subjective factors in the labeling process.Its network structure is obviously simpler,and can achieve good recognition results under weak supervision.Compared with the naive multi-label classification scheme,performance of our method has been significantly improved,excellent results of over 100% and 40% year-on-year growth rate of F1 score on two real mixed-dish data sets have been achieved,respectively.In this paper,vertical comparison experiments are performed on two mixed-dish data sets with a series of improved methods,and horizontal comparison experiments are performed on five data sets including three other common multi-label data sets.The rich and informative experimental results strongly verified the effectiveness,pertinence and certain universality of the proposed method.