Adaptive Generation Of Fundus Images Reading Report Based On Semantic And Visual Features

Fundus diseases refer to the lesions in the posterior part of the eyeball.Fundus examination is very important,many diseases such as diabetes,hypertension,etc.can be reflected from the fundus image,which is generally a necessary examination item in ophthalmology department of each hospital.It usually takes about 30 minutes from taking fundus images to the doctor giving diagnosis report and explaining the patient’s condition.In the case of a large number of patients in the hospital,the whole process may be delayed to 1-2 days.The use of artificial intelligence method to make the machine automatically generate fundus image reading report will shorten the doctor’s diagnosis time,and give the doctor a hint,at the same time,it will be convenient for patients to retain the text basis of the current condition.It is an important and valuable research direction to apply artificial intelligence technology to automatic generation of medical image reading report.At present,there are many researches on the automatic generation of medical image diagnosis reports,most of which are based on image description methods.Traditional image description techniques are mainly based on retrieval and template generation,which cannot effectively extract abstract features from images and have no generalization.With the continuous development of computer vision and natural language processing technology,the coder decoder model with attention mechanism has achieved good results in the automatic generation of image reports.For the problem of automatic generation of image reading report in the medical field,only the description of specific disease needs to refer to the visual features in the image,and the non-visual statements such as doctor’s advice in the report do not need to refer to the visual information in the image to generate.In order to solve the above problems,this paper proposes an adaptive generation model of eye bottom image film reading report based on semantic and visual features.Specifically:（1）The fundus image feature extraction model based on deep convolution neural network is constructed.Because the quality of fundus image itself is not uniform and the characteristics of fundus image are specific,we design a preprocessing method for fundus image data,and processed the text data with Chinese word segmentation and word frequency statistics,and then set up comparative experiments to verify the influence of various factors on the extraction of fundus image features by DCNN network.Finally,the experimental results show that the fundus features extracted by DCNN are practical and effective,which can bring accurate feature expression for the subsequent generation of fundus image reading report task.（2）A decision adaptive image report generation model based on Adaptive Attention is proposed.In order to solve the problem that it is not necessary to refer to visual features when generating non visual sentences,this paper proposes an improved method based on adaptive attention image description model.We propose a new decision-making mechanism to adaptively decide whether to use visual information in the process of generating description.The model uses the residual neural network which is proved to be effective in extracting the features of fundus image as the encoder to extract the image features,uses the LSTM network which is pre trained by the Chinese description data set as the decoder to generate the image report,and triggers the index through the decision module to control whether to use the method with attention mechanism when generating each word.The experimental results show that the performance of the model is better than that of the adaptive attention model on the fundus image data set provided by a hospital,and it can generate the fundus image reading report more accurately.（3）A decision self-updating image report generation model based on reinforcement learning is proposed.Traditional image description methods use cross entropy loss to generate sentences that are as similar as manual annotation,which can easily cause exposure bias and loss evaluation mismatch Based on the decision-making adaptive fundus image reading report generation model proposed in this study,we introduce reinforcement learning to solve these two problems.The method directly optimizes the CIDEr evaluation criteria to further train the model and enables the decision-making method to carry out reinforcement self-renewal.The experimental results show that the performance of the decision adaptive model is further improved by introducing reinforcement learning into the decision self-updating method.