Semantic Analysis Of Object Interaction For Robot Autonomous Task Execution

As an intelligent machine capable of working semi-autonomously or autonomously,interacting with objects in the environment is one of the key crucial functions necessary for robots to replace humans in executing tasks.The degree of dependence of robots on humans in the process of interacting with objects reflects the degree of autonomy of the robot.To further improve the automation of robot interaction with objects in the working environment,this dissertation focuses on the key technologies and algorithms required for robots to interact with objects in the environment in order to complete user assigned tasks.When executing specific task,the robot goes through multiple steps: 1)Select tools that are suitable for completing the assigned task based on their functionality;2)Determine whether the selected tool is currently available for interaction in the environment;3)Predict the interaction regions of the tool when interacting with objects and the motion trajectories of the mechanical claw for completing tasks;4)Predict the grasping posture of the tool;5)Use the tool to perform the assigned task.To accomplish these steps,the robot needs to understand the functionality and usability of objects in the scene,the objects’ manipulability and applicability,know the interaction regions on the objects,plan the movement trajectory while interacting with the objects,and determine the position and rotational angle required for grasping the tool or object.Research efforts in object function understanding,affordance reasoning,interaction region prediction,motion trajectory prediction,and grasp detection have provided phased solutions for enabling robots to interact with objects.However,existing research works are often independent,lacking a holistic consideration for the complete process of robot task execution.In this dissertation,from the perspective of robots effectively grasping tools and executing tasks with high quality,semantic of task-oriented interaction with objects is studied.The research focuses on the semantic analysis of object interactions for autonomous task execution by robots,primarily targeting everyday tasks in daily life(such as “slicing bread” and “hammering nails”).In this context,the semantic analysis of object interactions primarily investigates how robots autonomously select interactive objects,the interaction areas during actual interaction with objects,the grasping poses of mechanical claws,and the motion trajectories of mechanical claws during task completion,including the following research content:Firstly,to address the issue of insufficient consideration of fine-grained attributes of tasks and tool functionalities in existing tool recommendation methods,this dissertation constructed a Fine-grained Tool-Task dataset and proposed a Fine-grained Tool Recommendation Network based on commonsense knowledge.The network classifies tool and manipulated object images with multiple labels based on multiple granularity semantics,constraining the feature distances between tools and manipulated objects that can complete the same fine-grained task to be smaller than those that cannot complete fine-grained tasks.Experiments demonstrate that the Finegrained Tool Recommendation Network can recommend the most suitable fine-grained tool for a specified task and can suggest alternative tools when the fine-grained tool is not available.Secondly,to address the limitations of existing graph-based object affordance reasoning methods in fully integrating contextual information,a graph attention-based Visual Affordance Reasoning Network was proposed in this dissertation.The significance of graph attention mechanisms in object affordance reasoning was analyzed.The network constructs affordance context based on graphs and learns edge weights using graph attention mechanism.Experimental results show that the graph attention-based affordance reasoning network effectively enhances the accuracy of object affordance reasoning.Thirdly,to address the issue that independently predicting object interaction regions and motion trajectories are not conducive to robots automatically executing tasks,this dissertation expanded the first-person interaction video dataset “epic-kitchens” and introduced an Interaction Region and Motion Trajectory prediction Network.The network simultaneously predicts interaction regions and motion trajectories,incorporating object categories as constraints into interaction region prediction to enhance accuracy.Experiments demonstrate that the network achieves the best performance on the interaction region prediction dataset “epic-kitchens”.Fourthly,to address the problem that existing grasp detection methods neglect the impact of grasp position on tool use or only predicting fixed-category tool-object grasp pose,this dissertation constructed Grasp Detection dataset for Handheld Tools.Also,this dissertation proposed a Tool Usage Friendly Lightweight grasp detection network.Experiments show that the proposed network predicts reasonable grasp detection rectangle for handheld tools of any category in real-world scenarios,and achieves competitive results compared to existing methods on the publicly available grasp detection dataset “Cornell Grasping”,with a small number of parameters.Fifthly,a prototype system for analyzing the semantic of object interaction was developed by integrating the above algorithms.Based on robotic hardware such as Kinova robotic arm,the effectiveness of the proposed algorithms in supporting for enabling robots to automatically perform specific tasks in real-world scenarios was verified.