Research On The Effectiveness Of Performance Measures For Cross-Modal Hashing

In recent years,the unprecedented amount of multimedia data has been generated on the Internet due to the prevalence of mobile devices and the development of social media sites.The core manifestation of multimedia data is that one instance can be presented through different media types,and there exists a certain degree of semantic correlation between data of different media types.Therefore,the demand for large-scale cross-media data retrieval is increasing day by day.At the same time,cross-media retrieval has become an important class of applications that can often significantly improve the user experience because it is more in line with realistic application scenarios.In such applications,people use a query instance of one modality（e.g.,image）to retrieval samples of another modality（e.g.,text）that are semantically related to it.To efficiently deal with the problem of large-scale multimedia data retrieval,many cross-media retrieval schemes have been proposed.Among them,cross-modal hashing methods have received attention from many researchers because of their low storage cost and fast retrieval speed.Performance measure is an important part of cross-modal hashing,and an effective evaluation strategy is very important in performance evaluation.Mean Average Precision（MAP）is the most widely used performance measure for cross-modal hashing methods.However,after research we found that the MAP measure ignores the multi-label information and the semantic hierarchy of labels in the evaluation process,which makes the MAP score does not fully reflect the true quality of the top-K retrieval results of cross-modal hashing,and thus the MAP performance measure cannot truly and effectively reflect the performance of cross-modal hashing methods.In view of this,this paper revisits the existing performance measures of cross-modal hashing methods and proposes a new performance measure.Specifically,this paper conducts research in the following aspects.（1）An in-depth analysis of the existing performance measures of cross-modal hashing methods is conducted,and the shortcomings of MAP measure and Normalized Discounted Cumulative Gains（NDCG）measure are summarized respectively by taking real retrieval results as examples.（2）Based on the label co-occurrence probability matrix,the category label relationships are classified into three categories: inclusion,similarity and uncorrelation,and a label weight formulation algorithm is proposed based on the relationships between labels and the cooccurrence probability matrix,and different weights are assigned to each different label.（3）With the re-adjusted label weights as the core,a new performance measure called Normalized Weighted Discounted Cumulative Gains（NWDCG）is proposed by extending NDCG.This performance measure considers the mult-label information and the semantic hierarchy between labels,and solves the limitation of existing performance measures.（4）To verify the effectiveness of the NWDCG performance measure,extensive experiments were conducted on three publicly available datasets using three popular cross-modal hashing methods.The superiority of the NWDCG performance measure is demonstrated by visualizing the comparison between the NWDCG and the two existing performance measures.In addition,this paper summarizes the experimental results of the three evaluation methods and reports the overall performance.