Research On Adaptive Index Method For Real-time Search On Microblogs

With the rapid development of Micro-blogging, the growing scale of microblogs and query requests puts forward higher requirements on the performance of real-time search on microblogs, seeking for an efficient and stable real-time index method to improve the performance of real-time search on microblogs becomes crucial. The traditional index methods cannot adapt to the occasion of index data updated frequently, and the existing index methods on microblogs are frustrated by the index fragmentation and exaggerate merging overhead, resulting in the real-time search on microblogs taking a lot of time in querying and updating microblogs.An adaptive index method for real-time search on microblogs is proposed to remedy this defect. The method adopts a dynamically adjustable Log-Structured index structure with Adaptive Merging Strategy(LS-AMS) for real-time search on microblogs. LS-AMS structure consists of an inverted index buffer and a sequence of dynamically adjustable index packages with exponentially increasing sizes. These index packages manage their invented indices using adaptive merging strategy, which can reduce the merging overhead to improve query efficiency and can adjust the index structure adaptively to reduce the effects of changeful environmental factors on the query performance based on the variation of environment factors, such as the arrival rate of query requests, arrival rate of new microblogs etc..According to the results of the performance test comparing with the index structures Multi-Segment and Log-Structured adopted in the popular real-time index methods currently, LS-AMS decreases at least eight percent on the query time without increasing the update time of the index methods. What`s more, LS-AMS is less affected by the variation of the arrive rate of the query requests. Experiments show that the adaptive index method is of great help to the promotion of query performance and stability of the real-time index on microblogs.