Audiovisual Multi-dimensional Integration Research Based On The Bouba-kiki Effect

The Bouba-kiki effect(BK effect)refers to people's tendency to associate round shapes with the sound "bouba",and sharp shapes with the sound "kiki".This tendency exists in different languages and cultures.Previous studies only paid attention to the sharpness characteristics of the shape,but never paid attention to the blockiness and compactness characteristics of the shape.Therefore,this research explores the integration relationship between the three dimensions of shape features and speech and the automaticity of them through three experiments.Experiment 1 firstly explores whether the shape's segmentability feature and compactness feature are matched with Bouba-type speech and Kiki-type speech in the BK effect.Two small experiments are included.In the experiment,a 5-point score task was used,and the subjects were required to complete the task of scoring voice and shape features.Experiment 1a found that people tend to match Bouba-type voices with shapes with low-segmentability features,and kiki-type voices with shapes with highsegmentability features.The results of experiment 1b show that people tend to match Bouba-type speech with shapes with high compactness features,and Kiki-type voices with shapes with low compactness features.Experiment 2 further explored whether there are voices that match the three shape feature dimensions: spikiness,segmentability,and compactness,including three small experiments.The paradigm of Experiment 2 is the same as Experiment 1,except that the phonetic material of Experiment 2 is a CV non-word composed of vowels and consonants.The results show that in the dimension of the spikiness of shapes feature,people tend to associate front vowels or unvoiced consonants with shapes with high spikiness,and associate back vowels or voiced consonants with shapes with low spikiness.Consistent with previous researches.In terms of the feature dimension of shape segmentability,people tend to associate non-words that combine voiced consonants and back vowels with shapes that have a high degree of segmentability,while those that combine unvoiced consonants and front vowels are associated with segmentation.In the dimension of compactness features,people tend to associate nonwords composed of voiced consonants and post-vowels with high compactness.Nonwords that combine unvoiced consonants and pre-vowels are associated with shapes with low compactness.Experiment 3 uses event-related potentials(ERPs)with high time resolution,adopts the Oddball paradigm,and uses mismatch negativity(MMN)to explore the brain Whether the sound and shape are immediately integrated and the processing automation.In the experiment,the amplitude of MMN induced under audiovisual conditions was significantly higher than that induced under auditory conditions.Further analysis found that under audiovisual conditions,the amplitude of MMN induced by incongruent audiovisual conditions is significantly more negative than the amplitude of MMN induced under audiovisual congruent conditions.This result shows that the BK effect is automatically processed,and the human brain can immediately integrate the information of the voice features and shape features in the early stage.The above results indicate that different phonetic features and shape features are related to the spikiness,segmentability and compactness.This shows that the BK effect is not an example of the integration between shape and sound,and that there is a general correlation between speech and shape.In addition,the BK effect is processed automatically,which occurs in the early stages of processing.These research results are of great significance for revealing the mapping rules between sound and shape and the brain's cognitive process of phonetic symbolism.