FOT-simulation Testing Method For Automated Driving Systems Based On Naturalistic Driving Data

Testing and evaluation is a supporting technique for the development of automated driving systems,and a scenario-based development,testing,and evaluation process for automated systems has been developed,in which the testing and evaluation process is based on the "three-pillar approach" that includes simulation testing,field test and field operational test(FOT).Simulation test can realize fast tests and has the advantages of low cost and high robustness.The field test method improves the reliability of the test results by offering a real road environment.However,both the simulation test method and the field test method use fixed trajectories of background vehicles in test scenarios,which differ greatly from the driving behavior of human drivers and lead to the lack of behavioral correlation among vehicles.Interaction and game.As the third pillar,the FOT compensates for the lack of reality of test scenarios in the simulation test and field test.However,the FOT test cycle is long and costly and the FOT can only be carried with the prototype at the end of the development process of automated driving systems.Therefore,it cannot meet the fast development requirements of the agile development of autonomous driving systems.In order to solve the above-mentioned problems faced by the ‘three-pillar method’,this paper proposes the FOT simulation test method,which refers to the method of testing the automated driving system under FOT-like realistic test scenarios generated in this paper.As a scenario-based simulation test method,the FOT simulation test improves the reality of test scenarios by introducing human-like and stochastic driving behavior models and build a behavioral-correlation-friendly test platform.Therefore,the FOT simulation test has both the advantage of FOT,i.e.,the high reality of testing scenarios,and the advantage of simulation test,i.e.,high test effiency and strong robustness.The FOT simulation test process consists of four steps: test requirement analysis,test scenarios generation,test execution,and performance evaluation.This paper studies the methods of each step,especially the test scenario generation method,and tests a highway pilot system.Test requirements analysis limits the scope of test scenarios by analyzing the operational design domain and dynamic driving tasks of the test object as well as the test purpose.Test scenario generation is the main feature that distinguishes the FOT simulation test method from other test methods.First,the scenario hierarchical structure including functional scenario,logical scenario,and specific scenario and the generation method of each scenario layer are analyzed according to ISO/NP 34502 and other relevant international standards.Then,a test scenario model based on the high-order Markov decision process was built to analysis how the behavioral correlation of the test scenario and the human-like and stochastic background vehicle behavior influence the reality of the test scenario and the reliability of the test results.Finally,based on the analysis of the test requirements of the highway pilot system,the requirements for building specific test scenarios for the following and cut-in scenarios are proposed respectively.To generate car-following specific test scenarios,the probability distribution of characteristic parameters and the correlation between the parameters are analyzed with naturalistic driving data.The initial states of test scenarios are generated accordingly.Then,a car-following behavior model was built with the long-short-term-memory model,quantile regression,and kernel density estimation method.Simulation verified the stochasticity of the proposed model.And the estimated behavior of the proposed model is more similar to human drivers than IDM,in terms of acceleration and speed,in discrete and continuous scenarios.Compared with the benchmark car-following model,the proposed model can generate a driving environment more similar to the real traffic environment,indicating that the proposed model qualifies the background vehicle car following behavior model in the FOT simulation test method.In the meantime,in order to solve the problem that the number of dangerous scenarios data in the natural driving data is too limited to train a driving behavior model,a transfer-learning-based modeling method is proposed for modeling car-following behavior in dangerous scenarios.A car-following behavior model for dangerous scenarios is built with several hidden layers transferred from the model trained with normal driving data and then trained with dangerous driving data.In this way,the proposed model got the common knowledge of driving from the normal driving model.Experiments show that the model can generate more human-like behaviors than the benchmark model in dangerous scenarios.Therefore,the model can improve the reality of dangerous scenarios in the test environment.To generate cut-in specific test scenarios,the probability distribution of characteristic parameters and the correlation between parameters were analyzed with naturalistic driving data.The initial states of test scenarios are generated accordingly.A Bernoulli sampling based lane-change decision model and a sine function based lanechange execution model were designed.Experiment shows that the proposed model can generate stochastic lane change behavior and the characteristics of the generated scenarios,such as lane-change frequency,lane-change direction,and lateral acceleration,are similar to those of human drivers.Therefore,the proposed model qualifies the background vehicle lane-change behavior model in the FOT simulation test method.The test execution is modeled and analyzed as the Markov Chain and the test platform is designed with a traffic state update section to ensure the behavioral correlation.To evaluate the safety and comfort of automated vehicles,an evaluation method is proposed to quantitatively score the performance with the driving behavior of human drivers as the benchmark.Finally,a highway pilot system was FOTsimulation-tested on the proposed platform and evaluated with the proposed method.