Comparing University Rankings Based On Rank Aggregation

A university ranking is a quantitative assessment of the performance of a program,activity,institution or system of higher education,based on an agreed-upon methodology,also known as the university league table,academic ranking,institution ranking,etc.With the globalization of higher education,the success of university ranking has been witnessed in the past decade by facts such as the emergence of the ever-increasingly new university ranking systems,the increasing number of workshops and conferences,and the huge clicks on the university rankers'websites.Therefore,various kinds and levels of university ranking systems in the range of nation,region,and the world appeared in the public.Despite there are certain limitations,university ranking systems are widely used around the world to help various stakeholders to solve their problem and achieve their goals.The discrepancies among various university ranking systems drive us to compare and analysis them.Qualitative and quantitative comparative analysis have been conducted by some research works,such as comparing qualitatively the methodologies used by different university ranking systems,investigating the similarities or correlations both between university ranking systems and between annual results from the same university ranking system using various measures,and exploring the degree of consistency of universities using their positional range.All the works have limitations:(1)the measures are not very scientific and reasonable in the investigation of similarities or correlations,and they lack the corresponding pre-processing of data;(2)they don't consider the statistical information in the exploration of university consistency by which their results couldn't reflect the real pictures objectively and correctly,and highly impacted by particular ranking systems.In order to solve the above problems,especially for the consistency analysis of universities,this paper compares and analyzes university ranking systems based on rank aggregation,and aims to provide a more comprehensive framework of university ranking system comparison under which we can study the similarities or correlations both between university ranking systems and between annual results from the same university ranking system by using more reasonable measures and appropriate data pre-processing,and we can also investigate the variation among all ranking systems and the degree of consistency among domestic universities by rank aggregation methods and traditional statistical methods.Besides,in order to find the best proper rank aggregation algorithm,this paper presents a data generation mechanism to provide artificial and controllable rank data,use more reasonable measures to evaluate the performance of 9 different rank aggregation algorithms.The data used is the 2017 results of 14 domestic and global university ranking systems.We find that:(1)Based on the overlapping universities between two university ranking systems,the similarity results will generally be much higher than that when we also consider those universities only assigned a rank in one of the above two systems,whose degree of declines depend on the pre-processing progress and used similarity measures.Since the RBO measure satisfies all the characteristics of an indefinite rank similarity measure,the similarity results by?_gis generally higher than that by RBO,because?_ggives a higher penalty to disjoint partial lists than reverse partial lists,uses simple counts to penalize disagreements between lists,but not considers the relative position of such disagreements.Besides,the RBO value among 4 domestic university ranking systems is higher than those among them and other global university ranking systems,which indicates that national university ranking systems will be more similar to each other.Even though the methodologies used by different university ranking systems vary from each other,there are still median to high similarity among them.(2)In the length-fixed pre-processing process group,the size of the overlapping and similarity by?_gand the M measure both increase,which are consistent with the previous research.However,the similarity by RBO measure decreases with increasing length.The comparison means that the chosen similarity measure has a remarkable impact on the final similarity results.(3)In the length-fixed pre-processing group,the university consistency pattern by traditional method is like a non-monotonic distribution(subjected to the corresponding UAR),but its falling part does not fall into the area of "highly consistent"or "reasonably consistent".For that by rank aggregation,the consistency difference of universities generally increases linearly even though a slight decline can be found in some datasets.In the university-fixed pre-processing group,the university consistency pattern by traditional method is also like a scattered non-monotonic distribution.Because of the rerank process,the university consistency pattern based on ranking aggregation is also non-monotonic,but be more concentrated.Comparing and analyzing university ranking systems is very important.On one hand,it is conducive to improving the university ranking systems,making them more scientific and reasonable,and promoting the healthy development of the university ranking.In the long term,it is conducive to guaranteeing and improving the quality of higher education,because the university ranking is one of the effective tools for the quality assurance and promotion of higher education.On the other hand,it is beneficial for the stakeholders to use the university ranking systems more dialectically and rationally,so that the university ranking can better serve the public.