Research On Automatic Generation Algorithm Of Medical Image Report Based On Deep Learning

Medical imaging(such as X-rays)is an important basis for doctors to screen and diagnose patients.Doctors read images and write medical reports based on their own experience.The medical report reflects the doctor’s process of analyzing and interpreting images.Besides,it comprehensively and meticulously describes the morphological features(especially abnormal points)of important organs and tissues in the area seen,and makes a judgment of the type and severity of the disease,providing a critical reference for future treatments.The writing of medical reports is a highly specialized task that must be completed by experienced doctors.Due to the lack of such doctors(especially in local hospitals)and the increase in the number of patients,a doctor will face a huge amount of workload which can lead to inefficiency and bad quality of work.To solve these problems,researchers have proposed the automatic generation of medical imaging reports(AGMIR),which uses computers to automatically analyze images and generate diagnostic reports.In recent years,with the substantial increase in computer computing power and the development of deep learning,AGMIR research has made great progress,and many advanced models have emerged.However,the existing models generally have the following problems:(1)Lack of innovations in the model structure.Existing models mostly imitate mature models in the field of computer vision,lacking targeted improvements on medical report generation.(2)The diagnosis accuracy of generated reports by the model is low,and the model’s ability to perceive abnormal points is insufficient.(3)Most models only focus on the "Findings",neglecting the "Impression",and their results are not complete.To solve the deficiencies of the existing models,this paper draws on the latest research results of deep learning and proposes a novel and systematic solution for AGMIR.The algorithm framework is divided into three components:disease diagnosis model(DDM),diagnosis description generation model(FGN),and diagnosis conclusion generation model(IGM).DDM is constructed based on a graph convolutional network(GCN)and combines visual and semantic features to implement multi-label disease diagnosis.FGM is used to generate Findings.It uses a two-level attention mechanism and decoder architecture based on the upstream DDM.Detailly,it incorporates DDM map features and disease prediction information,and more importantly,it is optimized with reinforcement learning strategies,taking into account diagnosis accuracy and text fluency.IGM is used to generate a diagnosis conclusion(Impression)from the summary of the diagnosis description.The prediction by DDM,the generated findings by FGM,and the generated impression by IGM constitute the final diagnosis result.To comprehensively test the performance of the proposed algorithm model,we conduct experiments on two commonly used medical image report data sets(IU X-Ray and MIMIC-CXR)with a variety of evaluation metrics and make detailed comparisons between our model and other advanced models.The experimental results show that:(1)the DDM model can effectively model the internal correlation between disease labels,and the accuracy of multi-label disease classification is greatly improved;(2)The feature fusion method of FGM has greatly improved the fluency scores and diagnosis accuracy.After combining with the reinforcement learning strategy,the performance improvement is more significant;(3)IGM can effectively summarize the diagnosis description generated by the upstream model FGM to ensure the completeness of the report.