Design And Implemention Of Deduplication On Flash-based SSD

With the time of big data coming,the data explosion makes great challenges to storage system.Meanwhile,the rapid development of new storage medium,especially Flash memory,makes storage system more active.Although Flash-based solid state drives(SSDs)have unique advantages on performance,they are limited by their lifespan limitation in many applications.According to properties of Flash-based SSDs,I design a deduplication system in order to improve performance and lifespan of SSDs by employing deduplication in SSDs.Firstly,since related works about deduplication in SSDs are based on simulation,I implement deduplication in a SSDs board by a FPGA chip in order to truly evaluate advantages and weaknesses of deduplication in SSDs.Based on properties of hardware,I design group hash technique and indirect address mapping mechanism.After that,I evaluate the effectiveness of deduplication,finding that the high overhead of fingerprint computing of traditional deduplication will introduce a significant impact on performance of SSDs.In order to improve performance bottleneck of SSDs introduced by traditional deduplication system,I propose NF-Dedupe,a new deduplication scheme that improves performance and lifespan of SSDs.It utilizes CRC32 hash function to computer light fingerprints for pages,which is a light-weight function.And find potentially deduplicated pages by comparing their lignt fingerprints.NF-Dedupe determines whether a write page is duplicated or not by comparing the write page with its potential duplicated page read from underlying flash chips byte by byte,rather than relying on the comparison of fingerprints.We evaluate the NF-Dedupe via trace-driven simulations.Experimental results have shown that NF-Dedupe outperforms the other approaches and can achieve the deduplication rate ranging from 5.3% to 29.9% and the write latency is improved by a factor of up to 21% with an average of 12%.In conclusion,we find a reasonable way to implement deduplication in SSDs and combine the advantages of them,which wil be beneficial for development of SSDs.