Automated Driving Systems (ADS) have made great achievements in recent years thanks to the efforts from both academia and industry. A typical ADS is composed of multiple modules, including sensing, perception, planning, and control, which brings together the latest advances in different domains. Despite these achievements, safety assurance of ADS is of great significance, since unsafe behavior of ADS can bring catastrophic consequences. Testing has been recognized as an important system validation approach that aims to expose unsafe system behavior; however, in the context of ADS, it is extremely challenging to devise effective testing techniques, due to the high complexity and multidisciplinarity of the systems. There has been great much literature that focuses on the testing of ADS, and a number of surveys have also emerged to summarize the technical advances. Most of the surveys focus on the system-level testing performed within software simulators, and they thereby ignore the distinct features of different modules. In this paper, we provide a comprehensive survey on the existing ADS testing literature, which takes into account both module-level and system-level testing. Specifically, we make the following contributions: (1) we survey the module-level testing techniques for ADS and highlight the technical differences affected by the features of different modules; (2) we also survey the system-level testing techniques, with focuses on the empirical studies that summarize the issues occurring in system development or deployment, the problems due to the collaborations between different modules, and the gap between ADS testing in simulators and the real world; (3) we identify the challenges and opportunities in ADS testing, which pave the path to the future research in this field.

https://dl.acm.org/doi/pdf/10.1145/3579642

A Survey on Automated Driving System Testing: Landscapes and Trends

Autonomous vehicle (AV) systems must be comprehensively tested and evaluated before they can be deployed. High-fidelity simulators such as CARLA or LGSVL allow this to be done safely in very realistic and highly customizable environments. Existing testing approaches, however, fail to test simulated AVs systematically, as they focus on specific scenarios and oracles (e.g., lane following scenario with the “no collision” requirement) and lack any coverage criteria measures. In this paper, we propose <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math></inline-formula> , a framework for systematically testing AV systems against customizable correctness specifications. Designed modularly to support different simulators, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math></inline-formula> consists of two new languages for specifying dynamic properties of scenes (e.g. changing pedestrian behaviour after waypoints) and fine-grained assertions about the AV's journey. <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math></inline-formula> further supports multiple fuzzing algorithms that automatically search for test cases that violate these assertions, using robustness and coverage measures as fitness metrics. We evaluated the implementation of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math></inline-formula> for the LGSVL+Apollo simulation environment, finding 19 kinds of issues in Apollo, which indicate that the open-source Apollo does not perform well in complex intersections and lane-changing related scenarios. 

Specification-based Autonomous Driving System Testing

Automated driving has the potential to revolutionize personal, public, and freight mobility. Besides the enormous challenge of perception, i.e. accurately perceiving the environment using available sensor data, automated driving comprises planning a safe, comfortable, and efficient motion trajectory. To promote safety and progress, many works rely on modules that predict the future motion of surrounding traffic. Modular automated driving systems commonly handle prediction and planning as sequential separate tasks. While this accounts for the influence of surrounding traffic on the ego-vehicle, it fails to anticipate the reactions of traffic participants to the ego-vehicle's behavior. Recent works suggest that integrating prediction and planning in an interdependent joint step is necessary to achieve safe, efficient, and comfortable driving. While various models implement such integrated systems, a comprehensive overview and theoretical understanding of different principles are lacking. We systematically review state-of-the-art deep learning-based prediction, planning, and integrated prediction and planning models. Different facets of the integration ranging from model architecture and model design to behavioral aspects are considered and related to each other. Moreover, we discuss the implications, strengths, and limitations of different integration methods. By pointing out research gaps, describing relevant future challenges, and highlighting trends in the research field, we identify promising directions for future research. 

pdf/rethinking-integration-of-prediction-and-planning-in-deep-3drwrjaaug.pdf

Rethinking Integration of Prediction and Planning in Deep Learning-Based Automated Driving Systems: A Review

Ensuring safety and performance requirements of automated driving systems is a major challenge for the automotive industry. One way to tackle this challenge is a simulationbased approach. However, to rely on results generated by using this approach, the simulation needs to fulfill certain quality criteria depending on the intended usage. Hence, quality must be measured and determined at many different levels and areas of the testing and developing landscape, providing information with varying abstraction degrees. Additionally, quality not only has to be assessed for the automated driving system but also for the simulation models that simulate the vehicles components. Only if the simulation models have a certain quality level can they be used for the simulation. The taxonomy presented in this paper provides a better understanding of the concept of quality during the development and test process. It introduces the possibility to systematically evaluate whether development steps in this process need to be repeated or further assessed.

A taxonomy for quality in simulation-based development and testing of automated driving systems.

Context: Autonomous Vehicles (AVs) will transform the way we live and work. Several benefits are envisaged, including: reduction in traffic deaths, drop in harmful emissions, improvement in fuel economy, reduction in travel time, and consumer savings. Indeed, the trend in the automobile industry is the development of AVs in preparation for the introduction and mass implementation of driverless vehicles. However, given the complexity and increasing connectivity of the AV, the challenges for effective and efficient development are immense. Many problems are related to misconceptions in the requirements engineering phase for AVs. Hence, a Requirements Engineering (RE) process is crucial in the development of AVs. Objective: The purpose of this work is to identify and analyse the current RE approaches used for AVs development. The analysis is based on answers related to the type of RE problems addressed by the study, the RE phases covered by the approach, requirements modelling styles used, the type of requirements described in the study, the specific AVs considered in the study, and the open problems reported. Method: We conducted a Systematic Literature Review (SLR) as the basis for our work. Results: Our SLR draws on 31 studies in which we identified the RE problems addressed by the studies and the phases of the RE processes considered. We also uncovered the languages and description styles used to describe the requirementes of the AVs. Special attention was paid to the non-functional requirements of interest. Different types of AVs were identified. Last but not least, several challenges were revealed. Conclusions: This paper reports the current state of the art of RE for AVs and identifies some open issues that deserve further investigation.

Requirements engineering for autonomous vehicles: a systematic literature review

Ensuring the functional correctness and safety of autonomous vehicles is a major challenge for the automotive industry. However, exhaustive physical test drives are not feasible, as billions of driven kilometers would be required to obtain reliable results. Scenario-based testing is an approach to tackle this problem and reduce necessary test drives by replacing driven kilometers with simulations of relevant or interesting scenarios. These scenarios can be generated or extracted from recorded data with machine learning algorithms or created by experts. In this paper, we propose a novel graphical scenario modeling language. The graphical framework allows experts to create new scenarios or review ones designed by other experts or generated by machine learning algorithms. The scenario description is modeled as a graph and based on behavior trees. It supports different abstraction levels of scenario description during software and test development. Additionally, the graph-based structure provides modularity and reusable sub-scenarios, an important use case in scenario modeling. A graphical visualization of the scenario enhances comprehensibility for different users. The presented approach eases the scenario creation process and increases the usage of scenarios within development and testing processes.

SceML: a graphical modeling framework for scenario-based testing of autonomous vehicles

Capturing the Variety of Urban Logical Scenarios from Bird-view Trajectories.

Mastering a specified operational design domain (ODD) and the continuous increase in automation both require comprehensive perception, prediction, and planning algorithms, thus sophisticated automated vehicle (AV) system architectures. Scenario-based testing is emerging as a promising technique attempting to efficiently run tests on all critical situations for AVs’ verification and validation. ODD, AV architectures, and scenario-based testing represent three evolving development worlds with their specific characteristics, peculiarities, and point of view, which must be consolidated and harmonized to enable a seamless design, verification, and validation process. For that reason, this paper proposes an ontology aiming at consistently reconciling the worlds of ODD, AV architectures, and scenario-based testing utilizing cross-relationships. Consequently, the ontology-based approach offers systematic guidance for an ODD definition, the design of a vehicle system architecture for automated driving, and the derivation of corresponding scenario-based test cases during the development process according to the V-model. A case study including the use case “follow lane on a motorway” for the lateral control of an active lane keeping system is exhibited, applying the introduced ontology. Therein it is also shown how the ontology-based approach enhances test efficiency and incremental development. The paper concludes by highlighting the corresponding advantages and limitations of the presented work.

Towards an Ontology That Reconciles the Operational Design Domain, Scenario-based Testing, and Automated Vehicle Architectures

The validation of Advanced Driver Assistance Systems (ADAS) and Automated Driving Systems (ADS) is a major challenge for the automotive industry. Scenario based testing is a promising validation approach, but relies on a scenario database with a high coverage of possible scenarios. Especially in urban areas, the presence of different traffic objects and vulnerable road users (VRU) in combination with semi-structured environments leads to a huge number of possible scenarios. In this paper, a clustering method for grouping real-world driving sequences into semantically similar sequences is introduced. The method allows to deal with the complexity due to the varying number of traffic objects and different sequence lengths and can provide insights over realworld probabilities of driving scenarios. For comparison of different driving sequences, dynamic time warping (DTW) is leveraged. The method is applied to the vehicle trajectories of four urban intersections from the inD dataset and tested with different evaluations. Results show, that an analysis of raw trajectories provides useful insights and is an important component on the path towards a sufficient scenario database.

Trajectory-Based Clustering of Real-World Urban Driving Sequences with Multiple Traffic Objects

Ensuring the functional correctness and safety of autonomous vehicles is a major challenge for the automotive industry. However, exhaustive physical test drives are not feasible, as billions of driven kilometers would be required to obtain reliable results. Scenariobased testing is an approach to tackle this problem and reduce necessary test drives by replacing driven kilometers with simulations of relevant or interesting scenarios. These scenarios can be generated or extracted from recorded data with machine learning algorithms or created by experts. In this paper, we propose a novel graphical scenario modeling language. The graphical framework allows experts to create new scenarios or review ones designed by other experts or generated by machine learning algorithms. The scenario description is modeled as a graph and based on behavior trees. It supports different abstraction levels of scenario description during software and test development. Additionally, the graphbased structure provides modularity and reusable sub-scenarios, an important use case in scenario modeling. A graphical visualization of the scenario enhances comprehensibility for different users. The presented approach eases the scenario creation process and increases the usage of scenarios within development and testing processes.

SceML - A Graphical Modeling Framework for Scenario-based Testing of Autonomous Vehicles

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Thilo Braun

Author Tools

Chat about Author