A Study On Resource Indexing For Distributed Storage

The expansion of the Internet leads to the rapid growth of information, which brings about urgent needs for rapid, efficient and reliable mass storage systems. Traditional distributed storage systems are usually deployed on stable and reliable nodes; therefore they cannot make reasonable use of the storage space and computing resources of nodes in the P2P network. How to make full use of these idle storage space and computing resources to construct a distributed storage system based on the P2P technologies has become a hot topic in the distributed storage research area.In this paper, the characteristics of existing distributed storage systems and resource indexing technologies are studied; and two resource indexing frameworks, namely ROIN, an efficient reputation-oriented inverted index framework for the P2P network, and LBGR, a load balancing resource indexing framework based on grouping and reputation, are presented.In ROIN, resources are indexed by a two-level DHT network, which supports fuzzy search for resources. With the help of the reputation system, nodes with high reputation will be selected as index nodes. In this way, metadata will be stored on reliable nodes, improving the reliability of the system. A group of experiments concerning the search coverage rate, the search latency, the load of index nodes and the network disturbance of the ROIN framework are carried out. A series of experimental evaluations and comparisons are made between ROIN, PCIR, Pastry, and GlusterFS. Upon the experimental results and evaluations, we conclude that applying inverted indexing and selecting a suitable number of nodes with high reputation as index nodes will bring about a robust and efficient resource indexing framework.In order to deal with problems such as data inconsistency in the distributed resources indexing system, we present LBGR, a novel resource indexing framework. In LBGR, the grouping and reputation strategies are adopted, and resources can be indexed by the DHT structure. A node will be evaluated by its reputation, and a number of nodes with high reputation will be selected as super nodes. By applying a mechanism based on grouping and virtual nodes, LBGR solves the load balance problem which exists in DTH-based systems. In LBGR, groups can balance their loads according to their capacities by migrating virtual nodes. The experimental results show that the presented load balance algorithm works well, and is comparable with the centralized algorithm, while it requires less load information of the system. By analyzing the results of experiments between LBGR and GlusterFS, LBGR shows its advantage on dealing with the dynamic nature of the P2P network.The above two resource indexing frameworks can run in P2P networks. The experiments prove that they can provide reliable and efficient services and have practical values.