Erasure Code Overwrite Design And Implementation In Stor Age System Ceph

In the era of cloud computing and big data, huge amounts of data has become the company’s core assets, as the basis for storing data, distributed storage system usually built on common servers. In tradition, distributed storage system through a multi-copies storage tolerate hardware and system errors, multi-copies can get a good performance in massively parallel computing scenarios, but it will bring many times the cost of storage. Erasure codes can achieve higher fault tolerance with smaller cost, effectively reduce storage costs.Erasure codes in existing distributed storage only support additional write, can not write any cover, which usually used to store data with the cold scene. To study more wider application of erasure codes in distributed systems, in this paper we the design and implementation of erasure codes overwrite system based on open-source distributed system Ceph. It supports anywhere offset write, update process remains strong data consistency and proposed some points of optimization after performance test which can enhance the system’s read and write throughput. The main work and research can be summarized as follows:1. Based on distributed storage CEPH, designs and implements erasure code over-write system, the update process to ensure transactional, all nodes either all up-dates, or else all is not updated, automatically roll back the update when fails and automatically recovery from node failure without human intervention;2. Using adaptive overwrite programs to support mixed writing scene. Combined reconstruction write-update and delta write update features, system adaptive over-write for selecting the optimal size of the update program to reduce disk write overhead and network overhead;3. Use write caching mechanism to reduce the coverage process, greatly improving the efficiency of overwrite. Erasure codes overwrite process requires pre-read then write, in the same file frequently overwrite, read and write locking severely reduced overall efficiency. By caching the data block, the pre-reading process can be skipped when the cache hit. At the same time, the design of the client’s cache isolation, isolation of different size of the update of the client, help to improve the cache hit rate;4. Use Key Value storage to management temporary objects of overwrite, can im-prove space utilization, accelerate allocation and space reclaim. Effectively im-prove the efficiency of reading temporary state data, improve the overall perfor-mance of read and write.