Research On The Legal Responsibility Of Data Mining By Search Engine Under The Robots.Txt

By setting the robots.txt, websites limit the search engine crawler to protect their sensitive business data and user privacy. Meanwhile, robots.txt possess the instrumental value of benefit balancing:it draws the boundary of rights to balance the interests of websites, users and the public. Accordingly, it has been generally complied with in the world as a soft constraint standardizing the data mining behaviors of search engines. With the rapid development of the technology, business model of search engines has become increasingly complicated and diversified, and thus makes the growing lag of legal system prominent. In the era of big data, data’s multiple value has been founded, which brings the search industry huge market space, but also intensifying competition. Chasing interests, search engines always break or ignore the practice of robots.txt for grabbing more data, while the misuse of robots.txt by websites emerging in endlessly, which finally causes chain reactions of infringement and vicious competition.Therefore, identifying the legal nature and value of robots.txt, determining the boundary of fair use and abuse of it, clarifying the distribution of rights and obligations between the relevant subjects, having those search engines take the legal responsibilities of robots.txt violation are of great significance. Based on the relative cyber law cases, the article analyzes the legal nature and efficacy of robots.txt from the perspective of copyright law and competition law, etc. and discuss the search engines’legal responsibilities of violating robots.txt with the fair use standard. Finally, both from the legislative and practical considerations, some relevant legal advice have been provided.The article argues that setting robots.txt is industry practice. In judicial practice, robots.txt should be given the legal efficacy as customary law of business, and so the meaning of copyright law. After the signing of the Internet search engine service self-discipline convention, robots.txt rises to self-discipline specification that it shall be generally complied with in China. From the perspective of legal behavior, robots.txt is a consensual contract. The setting of it by websites is issuing an offer, while data grabbing of search engine crawlers equals to acceptance, by which the contract is established and has legally binding upon the both parties. Hence, violation or ignoring of robots.txt during data mining results to concurrence of default liability, tortious liability and competition liability. On the other hand, it will be regarded as implied license to the search engine if no robots.txt exists. Due to the legal status combination of the search engine under new business modes, the division of ISP, ICP and corresponding imputation principles should be weaken. It is better to have the data mining behaviors classified by correlation with robots.txt. Under the fair use of robots.txt, we should have the legal responsibility of search engines identified in accordance with this "behavior standard" respectively in imputation principles application and both fact and value judgment, with business model combined.