Study On The Short Form Of Evaluation Index System For Readability Of Online Health Information

Objectives This project developed a “Short Form of Evaluation Index System for Readability of Online Health Information” with good reliability and validity,clear weight and suitable for the application of health education practitioners.It provides a measurement tool for health education practitioners to accurately and quickly evaluate and quantify the readability level of network health information.Methods The research was divided into three phases.Stage #1: Development of the “Short Form of Evaluation Index System for Readability of Online Health Information” test version.The process is based on the “Online Health Information Readability Evaluation Index System”(hereinafter referred to as the “original index system”)obtained previously by our research group,and in consideration of the feedback from clinical nurses on the applicability evaluation of the original index system and the weight coefficient of each index.We adopt weight judgment and inductive analysis methods to analyze the original index system,propose the indexes to be simplified,and compose the expert consultation questionnaire.Subsequently,the improved Delphi method is performed.We combine the feedback of senior experts and the discussion of our research group,and determine the simplified indexes and the “Short Form of Evaluation Index System for Readability of Online Health Information”(hereinafter referred to as the “Short Form of Evaluation Index”).Stage #2: Weight determination of the “Short Form of Evaluation Index System for Readability of Online Health Information”.We construct a hierarchical structure model of the “Short Form of Evaluation Index” according to the analytic hierarchy process.Referring to the experts’ feedback in the second-round correspondence,we calculate the mean score differences of the importance of each first-and second-level index,obtain the Saaty’s scale,and establish a first-level-index judgment matrix and seven second-level-index judgment matrices.Then,we employ the software yaahp 10.3 to determine the combined weight coefficient of each index in the “Short Form of Evaluation Index” and perform the consistency test,to complete the weight calculation of the “Short Form of Evaluation Index”.Stage #3: Reliability and validity evaluations of “The Short Form of Evaluation Index System for Readability of Online Health Information”.Three clinical nurses are invited to re-evaluate the readability of the 30 diabetes health education webpages included in the application stage of the original index system with the “Short Form of Evaluation Index” test version,10 webpages for each nurse respectively.The internal consistency of the “Short Form of Evaluation Index” is verified by Cronbach’s α coefficient.The inter-rater reliability of the “Short Form of Evaluation Index” is verified by Cohen’s Kappa coefficient derived from evaluations of all webpages above by two researchers.In this study,we investigate the health literacy level of community residents,and then pick out the respondents with health literacy scores in high-and low-score groups.They are invited to complete a web knowledge test questionnaire after reading the web content.We collect the accuracy of the answers of both high-and low-score groups,and then analyze the correlation between correct rates and readability measurements,as well as the correlation between the original index system and the “Short Form of Evaluation Index” on the readability evaluation results of the above webpages,resulting in the completion of readability test.ResultsStage #1: The research group select 8 indexes to be simplified from the 32 second-level indexes of the original index system,and accordingly compose an improved Delphi consultation questionnaire.Of the first-level and second-level indexes in the “Short Form of Evaluation Index” test version,the importance scores are 4.41 – 5.00 and 4.00 – 5.00,the coefficients of variation are 0 – 0.14 and 0 – 0.283,the expert authority coefficients are 0.892 and 0.888,and the expert coordination coefficients of both index levels are 0.314 and 0.257(P<0.001).The improved Delphi method is carried out in two rounds,where 17 experts complete both rounds of consultation with a high degree of expert enthusiasm.In the feedback of the experts,there are 20 indexes that meet the retention criteria(approval rate of experts ≥ 70%).Based on the experts’ opinions and the professional meaning of the indexes,two additional indicators are retained.As a result,the “Short Form of Evaluation Index” test version lists 7 first-level indexes which include 22 second-level indexes.Stage #2: In the “Short Form of Evaluation Index”,the weights of the seven first-level indexes are: “literal expression”(0.2647)= “content design”(0.2647)> “operability”(0.1619)> “layout”(0.1073)> “visual assistance”(0.0876)> “typography”(0.0627)> “numerical application”(0.0512).The three highest combined weights among the 22 second-level indexes are: “using easy-to-understand words”(0.1351),“clear and specific behavior guidance”(0.1214),and “scientific and accurate information content”(0.1026).Both index levels have passed the consistency test(CR<0.1),indicating that the weight determination are reliable.Stage #3: The Cronbach’s α coefficient is 0.790 of the result that three clinical nurses evaluate the webpage with the “Short Form of Evaluation Index”.The overall Cohen’s Kappa coefficient is 0.857,and the Cohen’s Kappa coefficient of each index is 0.636 – 1.The correlation coefficients between the readability of webpages and the correct rates are 0.507 and 0.497(P<0.05),for high-score and low-score respondents respectively.The correlation coefficient of the readability score of the identical materials between the “Short Form of Evaluation Index” and the original index system is 0.770(P<0.001).The reliability and validity test results of the “Short Form of Evaluation Index” are close to the original index system.Conclusions In this project,we adopt weight judgment,inductive analysis,and improved Delphi methods to develop the “Short Form of Evaluation Index System for Readability of Online Health Information”.The process is scientific and reliable with high reliability and validity,and the weights of the indexes are reasonable.The “Short Form of Evaluation Index” can reduce the workload effectively.It serves as a powerful tool for health education practitioners represented by clinical nurses when they are screening and compiling high-readability health education materials,and lay a foundation for the improvement of the public’s health education effect and health literacy level.