A Study On Offline Evaluation Methods Of Information Retrieval Based On User Behavior Model

Information retrieval has become an indispensable technology in Internet services to alleviate the contradiction between the limited cognitive ability of users and the vast amount of information available on the Internet.In the research field related to information retrieval,the study of its offline evaluation methods has received much attention from researchers.To make the evaluation results of offline evaluation methods match the experience of real users as closely as possible,the introduction of user behavior models in the design of offline evaluation metrics has become a long-standing research hotspot in information retrieval evaluation.However,the existing offline evaluation methods still have some limitations.On the one hand,the existing offline evaluation metrics do not fully consider the influence of cognitive factors on user behavior when constructing user behavior models.On the other hand,the previous work lacks complete validation of the consistency of offline evaluation metrics in describing user behavior and measuring user satisfaction.This thesis carries out systematic research on the offline evaluation methods of information retrieval based on user behavior models.Through in-depth mining of users’ search cognitive process,this thesis focuses on solving a series of challenging issues in the design of offline evaluation metrics.(1)To address the shortcomings of existing search interaction models that cannot accurately explain users’ stopping behavior,we propose a browsing interaction model that dynamically models users’ expected gains and costs and design a unified framework for offline evaluation metrics based on the model.The framework not only integrates a variety of existing evaluation metrics but also derives new metrics that are more closely aligned with user satisfaction.(2)To address the problem that traditional relevance assessment methods are difficult to apply directly to image search scenarios,we construct a search utility assessment model that combines topic relevance and image quality and design image search evaluation metrics that are more consistent with user experience.This work reveals the impact of different dimensions on utility assessment of image results and provides more user-friendly utility assessment solutions for image search performance evaluation under different intents.(3)To address the problem that existing session search behavior models do not consider the exploratory evolution of users’ information needs,we introduce the recency effect from cognitive psychology into the utility accumulation model and design the corresponding session-level evaluation metrics.The work provides an explanation for the accumulation of users’ utility in search sessions and enables better measurement of users’ satisfaction with the entire search session.(4)To address the problem that user satisfaction is difficult to collect in real-world scenarios,we systematically verify the consistency of offline evaluation metrics in describing user behavior and measuring user satisfaction.This work not only identifies the inherent connection between the optimization of offline evaluation metrics and the construction of user behavior models but also provides empirical evidence for the consistency between user behavior and user satisfaction.To conduct the above research,we construct a series of datasets on search user behavior and satisfaction feedback based on user study methods such as laboratory studies and field studies.These datasets not only validate the effectiveness of the proposed related offline evaluation methods for guiding the performance improvement of search systems but also help researchers to advance their research in the field of information retrieval evaluation.