Mass Distributed Log Retrieval Technology

With the rapid development in network technology information, many departments out of security reasons save the user's information in the form of text preserved for the future of the security audit. However, this quantity is often very huge. In order to be able to find the tremendous amount of text data they need, people need a highly effective search tool. Therefore, how to efficiently store and query the text of this unstructured data is a problem worthy of study, but also of great practical significance.In this paper, we analyze and compare two kinds of today's popular search engines. And select the appropriate and efficient search engine for the particularity of the log file search system. Then proposed an effective timestamp indexing strategy for log file system.The last, based on previous two steps, we design a complete log file retrieval system, and analyze and compare two kinds of serial and parallel search techniques.Finally, build a 10 million log-scale data platform, and measure respectively serial and parallel search's efficiency.And then the experimental data obtained for the parallel retrieval efficiency for providing reliable evidence.