Research On Image Recommendation For Lyrics Based On Deep Learning

Songs are a very significant method that people use to express emotions and attitudes.Traditionally,music is an auditory art.People can feel the change of mood of the song by the change of melody and the ups and downs of tones.With the development of computer technology and the improvement of living standards,people are not only satisfied with ‘listening' to music,but also eager to ‘see' music.Therefore,the music visualization work gradually enters people's vision.Most of the previous music visualization work is based on the audio feature of the song,and visualizes the music by showing changes in frequency and pitch with the help of visual spectrum.However,it is too tedious to pass the visual spectrum.It is much more interesting and intuitive to visualize the process of music transformation through image transformation,and it is beneficial for people to understand music more deeply.Specifically,we found that there are elements in the lyrics,an important form of the song,that can be characterized by images.And the emotions expressed by the lyrics also correspond to the emotions in the images.Therefore,we use the lyrics as the medium of the song to establish the relationship between the song and the image to complete the task of music visualization.It is necessary to build a high-quality dataset for the deep learning,a data-driven technology.In view of the fact that there is no publicly data set related to lyrics matching,we have produced a lyrics-image dataset with emotional tags,and conducted a lot of experiments on it.We prove that the method of visualizing music through lyrics matching is effective.Therefore,the contribution of our work mainly includes the following aspects: firstly,we use the lyrics as a medium to establish the relationship between the song and the image to achieve music visualization;secondly,we establish a lyrics-image data set with emotional tags,and propose a feasible method of using multi-modal fusion method to illustrating;considering the emotional correspondence between images and lyrics,we propose to use multi-task learning method to help improve the accuracy of lyrics mapping and improve the quality of the image;on this basis,we find that there is no direct descriptiveness between the lyrics text and the image.in order to improve the matching accuracy,we propose to use the attention mechanism to increase the weight of related content between lyrics and images.Finally,according to our lyrics-image retrieval model,we have completed the development of lyrics mapping application.The music video is formed by the following elements : lyrics information with time stamp,the best candidate image for each lyrics segment,and the audio file synchronized with the lyric file in time.By this way,we can achieve the purpose of music visualization.