Bi-Directional Attention For Joint Instance And Semantic Segmentation In Point Clouds

In recent years,the rapid development of VR/AR,autonomous driving and intelligent robotics cannot be separated from the task of 3D scene recognition and understanding.For the above fields,the task of recognizing and understanding threedimensional scenes is just like the visual system,which guides the machine to interact with the three-dimensional real world.If the visual system can recognize and understand the scene as much as possible,it will be of great help to the subsequent interactive algorithms.Classical scene recognition and understanding tasks include semantic segmentation,object detection,instance segmentation,etc.These tasks are different levels of understanding of the scene.Semantic segmentation is to identify the semantic category of each point in the scene.The core task of object detection is to identify the exact position of the object in the scene.In this paper,the main task of instance segmentation of point cloud is to identify the semantic category and instance attribution of each point in the scene,that is,to identify and understand the scene from the finest granularity and get the semantic category and location information of each object in the scene,the machine's ability to perceive the real world in the visual system will be closer to that of humans.Although instance segmentation and semantic segmentation of point cloud are two levels of understanding of the scene,these two tasks are closely related.As is the simplest observation,the point that belongs to the same instance must belong to the same semantics,the point that belongs to different semantics must belong to different instances.If the direct correlation between the two tasks can be reasonably used,they will promote each other.Based on this kind of observation,the segmentation method proposed by many papers is to simultaneously process the two tasks and take advantage of multi-tasking learning to achieve a better segmentation effect.However,most of them only consider simple strategies,such as concatenating semantic feature matrix and instance feature matrix element by element,or directly adding the semantic feature matrix and the instance feature matrix,which may not lead to mutual promotion of the two tasks,but introduce some new risk of feature conflict;or introduce some traditional clustering method which is s non-differentiable and will break the integrity of the entire back-propagation chain.To solve the above problems,this paper proposes a bi-directional attention module to realize the mutual promotion of the two tasks.This module uses a feature similarity matrix to extract non-local information from the feature matrix of a task(that is,the similarity between the features of a point and those of the rest of the scene),and will pass this information to another task,to guide the segmentation of another task,to do so to avoid the two tasks of feature fusion directly,to avoid the potential function of rejection and task conflict,at the same time not only won't destroy the whole process of back propagation network will continue to implement in the process of back propagation of two tasks of information transmission,and achieve better segmentation effect.In order to prove the segmentation effect,we conducted segmentation experiments on three mainstream point cloud data sets,all of which achieved the stateof-the-art results.In addition,efficiency experiments are carried out to prove the superiority of the proposed method in terms of time and space complexity compared with the existing information transfer and aggregation module ASIS.Moreover,ablation experiments and mechanism analysis were conducted to further understand and analyze the mechanism and inherent advantages of the bi-directional attention module.