Research On Computer-aided Interpretation Of Oracle Bone Inscriptions Based On Multi-modal Deep Learning For Image And Text

Oracle Bone Inscriptions(OBIs),one of the four earliest writing systems,is a treasure of our country’s history and culture,as well as the world’s intangible cultural heritage.The study of OBIs is of great significance to the inheritance of Chinese history and culture.It has been more than 100 years since the discovery of OBIs,and great achievements have been made in the interpretation of OBIs.However,the existing manual interpretation methods have encountered bottlenecks.It is a breakthrough to combine artificial intelligence technology with OBIs.However,most of the existing researches focus on Oracle bone character recognition,conjugation,semantic translation,etc.The research on the interpretation of OBIs is not comprehensive and in-depth,and there is no complete computer system to assist experts in the interpretation work.In this paper,we use deep learning techniques to carry on research on computer-aided interpretation of OBIs from multi-modal for image and text.The main innovative work is as follows:(1)Deep learning requires big data,but the existing Oracle data is scarce,so this paper carries out data augmentation in both image and text modalities.On the one hand,we use other Chinese glyph data to enhance the image modal of OBIs.On the other hand,both OBIs and Modern Chinese characters are divided into Ideographic Description Sequence(IDS).In order to alleviate the scarcity of Oracle text data,model can learn the semantics of radicals and configuration symbols from large-scale ancient Chinese data.The above data augmentation methods solve the usability of deep learning models.(2)Turning the interpretation into an abstract image retrieval problem in terms of image modal.Aiming at the structural characteristics of Chinese characters,a global and local feature fusion network based on Triplet architecture and transfer learning is proposed.Bronze are similar in glyph to OBIs.Based on this,inscriptions similarity in glyph to OBIs can be found as candidate sets to assist interpretation.The experimental result shows that our model has a good effect.(3)The interpretation of text modal is regarded as a classification problem.Based on the semantic similarity between Modern IDS and Oracle IDS,we propose a semantic model based on the components of Chinese characters(partial components and configuration symbols),and design an end-to-end candidate model for interpretation based on the probability transformation of partial components and multiple Oracle context information.The model predicts Modern Chinese characters and their probabilities by inputting IDS information of OBIs.The experimental result shows its effectiveness.Finally,a comprehensive interpretation model is designed based on the above research,and we build a computer-aided interpretation system of OBIs based on a browser-server architecture.Experts can obtain Bronze images with similar glyphs and Modern Chinese character sets from the two modalities,or obtain candidates through a comprehensive model,which can be used to assist experts in decision-making.