PROJECT TITLE :

Computing Maximized Effectiveness Distance for Recall-Based Metrics - 2018

ABSTRACT:

Given an effectiveness metric M(·), 2 ordered document rankings X one and X two generated by a score-based information retrieval activity, and relevance labels in regard to some subset (possibly empty) of the documents appearing within the 2 rankings, Tan and Clarke's Maximized Effectiveness Distance (MED) computes the best difference in metric score that may be achieved that's according to all provided data, crystallized via a collection of relevance assignments to the unlabeled documents such that |M(X 1 ) - M(X 2 )| is maximized. The closer the maximized effectiveness distance is to zero, the a lot of similar X one and X2 will be thought of to be from the point of read of the metric M(·). Here, we tend to think about issues that arise when Tan and Clarke's definitions are applied to recall-based metrics, notably normalized discounted cumulative gain (NDCG), and average precision (AP). In specific, we have a tendency to show that MED can be applied to NDCG without requiring an a priori assumption in regard to the full variety of relevant documents; we additionally show that creating such an assumption leads to different outcomes for each NDCG and average precision (AP) compared to when no such assumption is created.