Research On Deep Learning Based Inverse Halftoning Method For Multi-type Halftone Images

The halftoning technology refers to the technique of converting a continuous tone image into a binary image which visually approximate a continuous tone image and widely used in the following fields such as printing industry,publishing,textile,electronic display,etc.Generally,these widely existing halftone images have to be first converted into continuous tone images in order to be used again.Inverse halftoning technology is a special digital image restoration technology that converts halftone images into continuous tone images.Therefore,researches on inverse halftoning method have important values both for academic research and practical application in image processing,image printing reproduction and machine vision.Due to the inevitable loss of image information in the process of halftoning,inverse halftoning is an ill-posed problem.At the same time,halftone dot noise removal and detail restoration are contradictory to each other.Different types of halftone images have inconsistency in dot shape and distribution,and scanned halftone images can not provide image supervision information.These challenging problems become the technical difficulties to be overcome in inverse halftoning method.Aiming at these problems,this dissertation studies deep learning based inverse halftoning methods.The main work and contributions are summarized as follows:1)Generates an halftone image dataset for studying inverse halftoningTo promote the study of inverse halftoning,we generate a publicly halftone image dataset including a multi-type digital halftone image dataset(MDHD)and a scanned halftone image dataset(SHD).The MDHD contains continuous tone images and their corresponding 4 types of halftone images with a total of 23445 images,which are divided into training dataset(2375),validation dataset(500)and test dataset(20570).The SHD contains 8,000 scanned halftone images divided into training dataset(6400 images),validation dataset(800 images)and test dataset(800 images).Images contained in both MDHD and SHD are common in life,including a variety of objects,such as human images,plant images,animal images,building images,etc.The creation of SHD and MDHD provides a guarantee for the restoration of multi-type halftone images in this dissertation,and also lays a solid foundation for other researchers to carry out related research.2)Proposes a multistage and multi-resolution network for single-type digital halftone image restorationThe existing methods have the problems that the image details are not clear enough when removing halftone noise,and the restored image color is different.To solve these problems,a multistage multi-resolution network(MM-Net)is proposed based on such a prior that different subnetworks focus on restoring different image information.Firstly,the multi-resolution convolutional neural network is used to achieve dot removal,and then both the dense residual block and detail loss function are combined to achieve detail enhancement.Finally,the image high-level semantic information is used to achieve global adjustment including color restoration.Three indexes of the Peak Signal-to-Noise Ratio,the Structural Similarity and the color difference are used to evaluate the method on three public datasets.Our experimental results show that compared with the existing inverse halftoning methods,the proposed MM-Net achieves the best results in both objective indexes and subjective visualization performance:the structural similarity is improved by 0.02-0.11,the peak signal-to-noise ratio is improved by 1.61-6.8dB,and the color difference is reduced by 0.48-2.73.3)Proposes a high quality restoration model for multi-type digital halftone imagesMost of the current inverse halftoning methods were aimed at the restoration of the singletype of the digital halftone image,which could not take into account the halftone noise removal and detail restoration for different types of digital halftone images.To solve this problem,an inverse halftoning method based on multi-scale generative adversarial network(MS-GAN)is proposed.Firstly,a multi-scale feature extraction module is employed to enrich the relationship between the current pixel and its surrounding neighborhood.The channel attention mechanism adaptively redistributes the weight of the fused features,so as to selectively pay attention to more important information.Then,the detail enhancement network is applied to enhance the image details.Finally,a multi-scale discriminator is used to further optimize the inverse halftone image to enhance the visual performance of the inverse halftone image.Experiments show that for the halftone images generated by BDD,the peak signal-to-noise ratio of the restored inverse halftone images is improved by 0.25-0.61dB while the structural similarity is improved by 0-0.03.For the halftone image generated by KDD,the peak signal-to-noise ratio of the restored inverse halftone image is improved by 1.14-1.28dB and the structural similarity is improved by 0.03-0.04.For the halftone image generated by DBS,the peak signal-to-noise ratio of the restored inverse halftone image is improved by 0.45-0.72dB while the structural similarity is improved by 0-0.01.Only when processing halftone images generated by FSDD,the performance of MS-GAN is slightly inferior to MM-Net.4)Proposes a Staged Transformer-fused inverse halftoning method for Scanned halftone image restorationIn view of the complex degradation of scanned halftone images affected by paper and ink,and the absence of corresponding label images,in order to obtain high-quality scanned inverse halftone images,a Staged Transformer-fused inverse halftoning method(ST-Net)is proposed based on the idea of problem simplification.Firstly,an unsupervised degradation network is trained to transform a continuous tone image into a scanned-like halftone image.Then,the Transformer-fused inverse halftoning model is trained by applying pairwise datasets to generate scanned inverse halftone images.Experimental results show that compared with the existing scanned inverse halftoning methods,the scanned inverse halftone images obtained by the Staged Transformer-fused inverse halftoning method achieve better results in both subjective effect and objective indexes:the Mean Opinion Rank is reduced by 0.67-2.28,the Clarity is improved by 1.97-11.74,and the Frechet Inception Distance is reduced by 1.80-10.96.