Design And Implementation Of Distributed Video Deduplication Storage System

With the increasing application of cloud services,it is inevitable that data redundancy appears in cloud storage.Currently,less than 1/5 of cloud content is unique,which results in the waste of cloud storage space.In order to save storage space in the cloud,the technology of data deduplication began to emerge.Traditional deduplication technology mainly judges whether two video data are duplicated by comparing whether they are completely matched.However,in the elimination of video duplicate data,because the video storage content with similar perceived content is different with the difference of video resolution and video quality,this kind of duplicate video is also the object that needs to be de-duplicated.Traditional similarity detection and deduplication techniques are mainly aimed at files,and do not consider the particularity of video.Therefore,the studies on deduplication based on video perceptual content similarity detection has gradually become a hot spot.In order to achieve more efficient video deduplication,this paper proposes a video-aware hash fingerprint similarity detection method to improve the granularity of deduplication and reduce the consumption of system memory and CPU resources caused by video deduplication.The input video is preprocessed in a perceptual sequence generator to generate a perceptual hash fingerprint of the video;the eigenvalue generator extracts eigenvalues based on N-transform SFs.On the one hand,it considers the locality of fine-grained features to reduce the number of hash fingerprints that need to be traversed for extracting eigenvalues,on the other hand,it considers the similarity of frame storage data in small video intervals to reduce the length of video fingerprints that need to be traversed;finally,a specific data structure is used to judge whether the eigenvalues of the video fingerprint are the same as those of the existing video fingerprint in the system,so as to achieve the purpose of judging duplication.Based on the deduplication model,a distributed video deduplication storage system with complete functions is built to facilitate the actual operation and use of users.The video deduplication algorithm is applied to the video uploading function of users,so as to achieve deduplication in the inline stage,and reduce the memory occupation of video data to the system.Through the front-end interface,it is convenient for users to operate the system directly,the bottom compressed data reduces storage,the cached data is convenient for users to obtain,the data encryption module prevents information leakage,and the data migration strategy is formulated to improve scalability.Finally,the function and performance of the system are tested to verify that the distributed video deduplication storage system has perfect functions and strong robustness.