Human-Like Decision-Making of Autonomous Vehicles in Dynamic Traffic Scenarios

I. Introduction

WITH the development of artificial intelligence, autonomous driving has a significant influence on human lifestyle and travel modes. It brings about a revolution in intelligent transportation [1]. Researchers are no longer limited to low-level autonomous driving assistance systems (ADAS) and have begun to explore the realization of Level 4 and Level 5 autonomous driving as defined by the society of automotive engineers (SAE) [2]. This means that the running process of autonomous vehicles will be completely independent of human drivers.

While there are already mature solutions available for simple driving scenarios, autonomous driving applications must inevitably face the challenge of dynamic and complex scenarios. Unexpected events such as sudden acceleration or deceleration of other vehicles, pedestrians crossing the street, sudden weather changes, and vehicle malfunctions leading to road congestion, can occur. Given the unpredictability of these situations, the autonomous driving system must be able to make appropriate judgments and decisions. It is necessary for autonomous vehicles to approach or even exceed the performance of human drivers.

Human-like decision-making is a key concept in autonomous driving technology. It enables autonomous driving systems to make correct judgments and decisions in complex traffic environments. Achieving human-like decision-making requires dealing with various uncertain factors in dynamic traffic scenes, meeting the needs of passengers and other road users, and ensuring efficiency and safety.

Autonomous vehicles with human-like characteristics have the potential to outperform human drivers due to their faster reaction times, highly accurate decision-making abilities, and greater adaptability. The high-performance computing unit can process vast amounts of data and make real-time decisions based on it, reducing the impact of human error. What is more, autonomous vehicles have the ability to continuously learn and enhance their performance. As time progresses, they will surpass human drivers in terms of performance, leading to improved efficiency and safety on the road.

Although there are differences in implementation, researchers have reached a consensus on the advantages of human-like driving. The specific advantages of human-like driving strategies can be summarized as follows: 1) to help drivers of surrounding vehicles predict the behavior of autonomous vehicles based on existing experience, thus improving overall driving safety; 2) to improve the performance of the human-machine cooperative driving system, because if autonomous vehicles make decisions in a predictable way, other human drivers will be easier to manipulate; and 3) to personalize the driving style of autonomous vehicles to provide services for different types of passengers and vehicles.

Implementing human-like decision-making requires significant algorithmic and technical support. For example, imitation learning can extract driving experience, reinforcement learning algorithms can formulate optimal driving strategies, and Bayesian networks can handle uncertain factors. It is also necessary to consider multiple factors, such as road safety, passenger needs, and road user expectations, to ensure that the decisions made by the autonomous driving system are safe, efficient, and reasonable.

In order to explore how to develop decision-making algorithms to meet public needs, researchers from different academic backgrounds study human-like driving from different perspectives [3]–[5]. However, these efforts have not yet formed a unified theoretical framework. The decision-making ability of some early autonomous driving systems can only support the completion of simple driving tasks [6], [7]. It is easy to make mistakes in complex and dynamic traffic scenarios, which is far from human capabilities. With the development of artificial neural network technology and the improvement of computing power, the study of learning-based decision-making has gradually increased. Learning-based approaches are outcome-oriented by using large amounts of instructional data or environmental interactions to train models, and they can reach the level of human drivers in terms of results and guide the vehicle to complete driving tasks [8]. However, current learning-based methods have some common problems, such as poor interpretation. When faced with complex scenarios, even the best current system will make mistakes, such as being unable to avoid obstacles or even cause serious traffic accidents.

Therefore, it is necessary to build a system with both performance and mechanism in one framework that combines reasoning and learning. First of all, the concept of human drivers’ driving behavior should be established, and the driving intentions of oneself and surrounding vehicles should be understood. Then autonomous vehicles should have driving styles and characteristics that can be adjusted as needed.

To promote research of human-like driving, many other factors should be considered in addition to the method itself, such as the design of driving simulators, driving datasets, and evaluation metrics. Due to the danger of unmanned testing in real traffic scenarios, driving simulation and the collection of driving datasets are essential for high-level decision-making methods. High-quality driving datasets and driving simulations are required to ensure the usability of algorithms in real scenarios. Hence it is necessary to summarize key elements and provide some help for the research in related fields.

This paper focuses on the above aspects of human-like decision-making. Some key questions are raised to help readers fully understand the current research. In particular, the following issues are discussed in detail: 1) the intelligence level of decision-making algorithms for autonomous driving and their imitation of human drivers; 2) the elements required for driving datasets, simulation platforms, and testing scenarios to verify human-like decision-making; 3) the evaluation metrics of human likeness, personalized driving, and the application of decision-making algorithms in real traffic scenarios; and 4) the potential research topics of human-like driving systems. In fact, each type of existing decision-making algorithm has a certain degree of human-likeness in mechanism or performance, but it cannot completely reproduce the driving style of human drivers. Although the designs of existing driving simulators, driving datasets, and evaluation indicators are relatively complete, a unified specification has not yet been formed. Driving simulations should collect driving data of autonomous vehicles in a virtual traffic environment to train decision-making algorithms and compare autonomous vehicles with humans in a risk-free environment. In the future, the application of artificial intelligence will result in breakthroughs in the decision-making of autonomous driving. It is expected that the hierarchical structure combined with intuitive-logical reasoning can meet the needs of human-like driving.

This paper is organized as follows. The background and the basic knowledge of human-like decision-making are introduced in Section II, while existing work on the decisionmaking of autonomous vehicles is reviewed and the challenges faced by current decision-making algorithms are discussed in Section III. The development of driving datasets and simulators is presented in Section IV, and related topics on humanlike decision-making are discussed in Section V. Finally, some potential future research directions are considered in Section VI.

II. Background

A Driving Behavior and Human-Like Driving

Human driving decisions are affected by many factors including knowledge, experience, personality, cultural background, and environmental factors. Some of these factors contribute to the safety of autonomous driving systems, while other factors such as emotional disorders may lead to negative outcomes. It was pointed out that professional knowledge and driving experience can effectively reduce driving violations [9], [10], which is an important guide. The impact of driving risk on driving behavior should also be considered. If algorithms can simulate the human perception of driving risk in a real environment, it will be of great significance for building an intelligent human-like driving system.

The similarity between artificial intelligence systems and humans is a research hotspot. As artificial intelligence becomes increasingly important in areas focused on humans, there is a growing understanding that both reasoning and emotions must be considered. Even social responsibility is something that AI systems need to possess [11]. Although there are some debates about whether autonomous driving strategies need to be consistent with that of human beings, the potential advantages of human-like strategies cannot be ignored, such as comfort and driving safety. Therefore, researchers study human-like intelligence, such as hybrid augmented intelligence [12], which introduces human roles or cognitive models into intelligent systems. This is a feasible and important growth model of artificial intelligence, which provides further inspiration for the design of human-like decision-making systems [13].

B Cognitive Architecture Models

Cognitive science is the theoretical basis for the development of human-like driving decision-making systems, and it has important guiding significance. Since most readers of this paper are developers of autonomous driving systems and may not be familiar with cognitive science, some basic concepts and existing research progress are provided herein.

The cognitive architecture model demonstrates and simulates the functions of human brains, rather than their physics and structure. For current research on automatic driving, understanding the cognitive architecture model is more important than understanding the physiological model of human brains. The related research can be divided into symbolist, connectionism architecture, and hybrid architecture, among which the representative examples are Soar [15], adaptive resonance theory (ART) [16], and the adaptive characteristics of thought-rational (ACT-R) [4]. In particular, ACT-R specifically points out that the human cognitive process requires the participation of four different modules: goal, vision, action, and commonly used descriptive knowledge modules. Many models describing driving behavior are based on ACT-R [17], [18].

Due to the complexity of cognitive processes, there is no single cognitive framework that can explain them all. However, some researchers have suggested that a general cognitive architecture model should include perception, movement, working memory, descriptive memory, and long-term memory [19]. This is an important reference for establishing a general cognitive architecture model. Furthermore, cognitive architecture models can be combined with other physiological models to expand their applicability and coverage [20]. In addition to the basic elements of cognitive architecture such as memory, perception, and execution modules, a complete cognitive framework should include long-term memory modules for storing prior knowledge and a target module for processing goal-oriented driving tasks.

In order to simulate parallel activities more effectively, the queuing network-model human processor (QN-MHP) was adopted [21], [22]. In fact, the QN-MHP integrates two complementary methods: the queuing network method and the symbolic method (taking ACT-R as an example). Therefore, the QN-MHP provides a framework for modeling and real-time generation of concurrent activities. Such a model can intuitively reflect the ability of the human cognitive system, but each subunit of the system needs to be carefully designed.

C Dynamic Traffic Scenarios

Dynamic traffic environments contain various factors of uncertainty that can impact the operation and efficiency of the entire transportation system. Here are some examples:

1) Vehicle speed and direction are unpredictable. Some vehicles may be speeding, while others may be traveling slowly. This uncertainty can lead to traffic congestion and accidents.

2) The position and movement direction of pedestrians and other obstacles are also uncertain. Pedestrians may suddenly run onto the road or cross the road. Other obstacles, such as bicycles or motorcycles, may also appear on the road. This uncertainty can lead to traffic accidents.

3) Weather and road conditions also influence traffic. For instance, on rainy days, the road surface becomes slippery, making it harder for vehicles to travel. This uncertainty can cause traffic congestion and accidents.

4) Vehicle breakdowns or stops are also uncertainties. Vehicles may suddenly malfunction or stop on the road, waiting for repairs. This uncertainty can lead to traffic congestion and jams.

Therefore, it is crucial to understand and consider these uncertainties to ensure efficient operation and safety of the transportation system. The layered models are widely used to classify different information into different layers to describe dynamic traffic scenarios in autonomous driving applications. A typical layered model includes the following five layers [23]:

Layer 1: Road-Level (road structure).

Layer 2: Traffic Infrastructure (traffic lights, etc.).

Layer 3: Temporary manipulation of Layers 1 and 2 (road maintenance, temporary closure).

Layer 4: Objects (dynamic vehicles and pedestrians).

Layer 5: Environment (weather, etc.).

The first two layers (i.e., Layers 1 and 2) describe static information, while the last three layers (i.e., Layers 3–5) describe dynamic information. As shown in Fig. 1, the latest traffic scenario model of the PEGASUS project [14] includes a digital communication layer in addition to the traditional model. This layer describes the information that relies on communication technologies such as vehicle to everything (V2X) and high definition (HD) maps. There are a large number of traffic participants whose behavior will affect the safety of vehicles in the dynamic traffic scenario. In addition, temporary operations and complex environmental conditions should be considered. Therefore, it is necessary to thoroughly test representative scenario types or instances and pay special attention to the performance of algorithms in boundary scenarios and hazards [24].

Figure  1.  Multi-layered model of a dynamic traffic scenario proposed by the PEGASUS project [14].

DownLoad: Full-Size Img PowerPoint

III. Human-Likeness of Existing Decision-Making Methods

Traditional decision-making methods for autonomous vehicles can be classified as either logic-based or statistics-based. Decisions defined by logical and statistical models are considered “rational”. In recent years, the application of artificial intelligence technology has allowed autonomous vehicles to break through these categories. Algorithms such as imitation learning and reinforcement learning can be used to directly learn driving strategies from driving data or environmental interactions. This section provides a comprehensive analysis of traditional and artificial intelligence-based decision-making methods, including their design and performance levels, which are compared to humans.

A Logic Based Methods

1) Finite State Machine: The finite state machine (FSM) comprises a set of states and transition relationships between each state, where the internal rules between each state are executed through conditional judgments. The structure of the FSM is divided into three types: series, parallel, and hybrid. Among them, the hybrid structure is the most widely used in the decision-making of autonomous driving. As shown in Fig. 2, the final decision result is calculated through state estimation and target selection in the hybrid structure of the FSM. The results of different submodules can be arbitrated according to priority, which combines all aspects of risk theory to model human-like driving. In most cases, the hybrid structure of the FSM has satisfactory performance, but it is still not a perfect solution. For the rule base, it is a major challenge to fully cover all potential situations that may occur in complex traffic scenarios. If the status enters an area not covered by the rule base, the vehicle can no longer be guaranteed to operate safely.

Figure  2.  Example of decision-making based on a finite state machine.

DownLoad: Full-Size Img PowerPoint

2) Fuzzy Reasoning: Fuzzy reasoning refers to the imitation of human thinking. It can effectively express qualitative and empirical information that has unclear boundaries. The decision-making system based on fuzzy reasoning has been widely used in the fields of medicine [25], agriculture [26], and social services [27]. Fuzzy logic provides a higher level of explanation for decision-making [28]. It realizes flexible rule design and can better adapt to a fuzzy traffic environment [29]. The main challenge of fuzzy reasoning is the lack of logical traceability and the inability to accurately determine the state of vehicles based on their behavior. Combining fuzzy logic with other complementary algorithms can improve the performance of the decision-making system.

B Statistical-Based Methods

1) Bayesian Network: As for data processing, the Bayesian network is good at classifying the probability of events and analyzing the reliability of events. The Bayesian network not only uses the probability description of the time-space relationship between vehicles but also incorporates the uncertainty of input data into the threat assessment of vehicles. It can efficiently represent uncertain events, such as estimating the probability of a vehicle collision [30]–[32].

Bayesian network-based methods offer advantages in dealing with uncertainty, which can improve the performance and reliability of autonomous driving systems. However, in practical applications, the computational cost of this algorithm is typically high, leading to long computation times. Additionally, the complexity of the algorithm also increases due to the need to balance multiple uncertain factors.

2) Markov Decision Process: Markov decision process (MDP) provides a framework for modeling decisions with partially random and partially controlled outcomes. In particular, partially observable MDP (POMDP) introduces perception uncertainty because driver intentions cannot be directly measured [33]. It has been found that POMDP is suitable for autonomous driving to make real-time decisions [34], [35]. However, obtaining a general solution to a specific problem is difficult with the original POMDP. Therefore, the multipolicy decision-making (MPDM) method has received extensive attention [36], [37].

C Artificial Intelligence Methods

1) Game Theory Based Methods: Game theory describes the process of interaction in information structures. Compared to other methods, game theory emphasizes the interaction between drivers and more accurately reflects their driving behavior [38], [39]. Some researchers believe that the potential of game theory is underestimated [40]. With the development of sensor perception and dataset simulation, more accurate traffic participant intentions can be gradually obtained to achieve a complete game, and it is easy to combine with other lane-changing models. In fact, the core of game theory is utility. The application of game theory in lane changing was reviewed in [40], based on the difference between the utility revenue function and the convergence strategy.

Compared to traditional decision-making methods, game theory makes more reasonable choices by considering the interaction between autonomous vehicles and the surrounding environment. Therefore, decision-making based on game theory has research potential.

2) Reinforcement Learning: Reinforcement learning optimizes sequential driving strategies through feedback evaluation while interacting with the environment. The basic reinforcement learning is modeled as MDP [41], where the goal is to find the optimal strategy $\pi^{*}$, which can achieve the highest discount reward expectation after the implementation of the strategy as follows:

Model-free methods, such as deep Q-networks (DQN) [42] and depth deterministic strategy gradient (DDPG) [43], are applied to generate advanced driving strategies without learning the dynamic model of the environment [44], [45]. Meanwhile, model-based methods make decisions by learning the dynamic model of the environment, thus reducing the time and cost required for actual interaction in the learning process [46]. Additionally, reinforcement learning can integrate recurrent neural networks to consider the POMDP for decision-making, making it more suitable for uncertain driving environments [47]. All of these works have effectively completed decision-making tasks under specific traffic scenarios.

Indeed, reinforcement learning is a powerful technology of artificial intelligence, but some problems must be discussed and resolved. First, the maximization of the reward function is usually different in different environmental definitions. Only carefully constructed rewards can be used for training human-like intelligence [48]. In addition, it is quite difficult to improve the learning efficiency of reinforcement learning. Although the learning efficiency can be improved to some extent by adopting the exploration strategy, it is still necessary to develop more efficient reinforcement learning algorithms.

In real-world applications, there is no doubt that auto- nomous vehicles must be absolutely accurate, which conflicts with the “trial and error” approach used in the training process. To deal with extreme scenarios in the real world and ensure driving safety, a complex architecture with safety mechanisms should be adopted.

3) Imitation Learning: The purpose of imitation learning is to learn expected behavior and perform tasks through expert data. By assuming that the best action is given by the expert dataset D consisting of the state-action pair $(s, a)$, the strategy $\pi_{\theta}(s)$ is trained to be as close as possible to the expert strategy $\pi^{*}(s)$ by

where $P(s|\theta)$ is the state distribution of the trained strategy $\pi_{\theta}$, and ${\cal{L}}$ is the loss function.

Behavior cloning treats the task of imitation learning as supervised learning. Fig. 3 depicts a convolutional neural network (CNN)-based “end-to-end” decision-making model designed by NVIDIA [49]. Research shows that the deep neural network can learn and obtain intermediate features that are meaningful for vehicle decision-making while ignoring a large amount of redundant information. Based on NVIDIA and other works [50], a full convolutional network with long and short-term memory (FCN-LSTM) was used to create a general model. The FCN-LSTM network extracts the temporal and spatial features of images and considers the semantics of visual features and their related time series to obtain decisions similar to human driver decisions. Then, based on the traditional imitation learning framework, Intel proposed conditional imitation learning to complete the internal “expert intention” modeling (including planning objectives, intentions, and prior knowledge) [51]. In Intel’s model, environment images and state observations during training are used, and the “expert intention” is introduced as a control input. Therefore, driving behavior can be separated from specific tasks, and the response to driving commands can handle the uncertainty of observed behavior.

Figure  3.  “End-to-end” decision-making framework based on convolutional neural networks designed by NVIDIA [49].

DownLoad: Full-Size Img PowerPoint

Inverse reinforcement learning is an “indirect” method that first learns the “intention” of human drivers through demonstration, and then generates a strategy based on the learned hidden goals. Compared with behavior cloning, the middle layer of inverse reinforcement learning can be designed to significantly improve the interpretability of the entire system. Several studies have been conducted to extract driving habits from human driving datasets. When designing the reward function, ride comfort [52], safety [53], and traffic efficiency [54] are emphasized. Furthermore, various attempts have been made to build decision-making systems, such as generating cost maps and combining model predictive controllers for decision-making [55].

In particular, generative adversarial imitation learning (GAIL) has achieved remarkable results in imitating complex behaviors in high-dimensional environments [56]. It has verified its capabilities on the real vehicle dataset [57]. Compared with the behavior cloning method, GAIL can effectively reduce the cumulative error and make the behavior more robust to fluctuations in the trajectory, bringing it closer to real behavior. On this basis, the algorithm combines parameter sharing trust region policy optimization and GAIL (PS-GAIL), which is more stable and has fewer collisions in long-term interaction [58]. Considering the insufficient response of multi-agent imitation learning to sudden emergency driving, the customized reward further improves the local interaction performance of agents and teaches them emergency avoidance abilities in complex scenarios [59].

In the above, different decision-making methods are reviewed. Now, their human-likeness can be summarized from the following aspects, as shown in Table I:

Table  I.  Comparison of Adaptability of Decision-Making Methods in Practical Application

Method	Reference	Combination of deterministic and fuzzy logic	Adaptability to unknown environments	Consideration of random factors	Learning ability
Finite state machines	[60]–[62]	+	+	+	+
Fuzzy reasoning	[63]–[65]	+++	++	+	+
Bayesian network	[30]–[32]	+	++	+++	+
Markov decision process	[34]–[37]	+	+++	++	+
Game theory approach	[38], [39]	++	++	++	++
Reinforcement learning	[44]–[47]	+	++	++	++
Imitation learning	[49]–[51]	+	+	++	+++
Note: +++: Good performance; ++: Average performance; +: Poor performance.

 | Show Table

DownLoad: CSV

a) Combination of deterministic and fuzzy logic: This is related to the human being’s ability to handle complex driving decisions with limited computing power.

b) Adaptability to unknown environments: In order to accurately and comprehensively understand its surrounding environment, autonomous vehicles should have effective decision-making algorithms to model and process some of the observed states.

c) Consideration of random factors: The adaptability of random factors is of great significance to the safety, intelligence, and reliability of autonomous driving.

d) Learning ability: In order to improve decision-making capabilities, autonomous vehicles should learn from expert data or self-learning in environmental interactions.

As shown in Table I, different methods focus on these four aspects respectively. Various decision-making algorithms have unique advantages and are applicable to different traffic situations. For instance, FSMs are suitable for discrete state and action spaces, such as lane changing, emergency braking, and steering. However, they cannot calculate continuous control outputs directly. Statistical-based methods are more suitable for scenarios with a large number of traffic participants in the environment, where predicting motion patterns is difficult, such as mixed traffic of motorized and non-motorized vehicles. Learning-based methods are applicable to almost all decision scenarios, depending on the specific algorithm selection and the overall design of the decision-making system. In conclusion, the autonomous driving decision-making system comprises multiple algorithms that work in combination, rather than a single type of algorithm. Therefore, the combination of different types of models is crucial for developing advanced human-like decision-making methods.

IV. Datasets and Simulation Platforms

It is of great significance to quantify and evaluate the decision-making ability of autonomous vehicles. The current evaluation of autonomous driving can be divided into three types: 1) dataset-based evaluation, 2) simulation-based evaluation, and 3) real-world-based evaluation [1]. Here, we review the relevant research on human driving datasets and autonomous driving simulation platforms.

A Human Driving Datasets

As more and more decision-making algorithms learn human driving habits from vehicle driving data, it is necessary to collect interactive vehicle data from the real world. Evaluating the performance of decision-making algorithms by comparing them with human driving data is also important. Table II lists the commonly used datasets for autonomous driving, and the main characteristics of these databases are as follows.

Table  II.  Comparison of Some Representative Human Driving Datasets and Their Characteristics

Dataset	Reference	Description
NGSIM	[66]	One of the most popular driving datasets; raw video information; high precision and wide coverage.
HighD	[67]	Accurate vehicle motions; dynamic urban scenarios are not included.
Lyft	[68]	Detailed movements for traffic participants.
INTERACTION	[69]	High-precision semantic maps, data on crisis situations.
HDD	[70]	Record driving scenes with a long time span and a large spatial range.
Argoverse	[71]	Provides relatively rich semantic map information.
Commonroad	[72]	The first driving dataset for planning.
nuPlan	[73]	The latest release dataset dedicated to ML-mased planning; including benchmarks and simulated traffic scenarios.

 | Show Table

DownLoad: CSV

1) NGSIM [66]: The “next generation simulation” project (NGSIM) includes a dataset of vehicle-road collaborations. The NGSIM dataset was collected from various regions, including structured road intersections, high-speed entrances and exits, and other hotspots of vehicle-road cooperation research. The original video information is used to generate track data for each vehicle in traffic flow. Many human-like driving algorithms have been tested and validated on the NGSIM dataset [59], [82].

2) HighD [67]: Compared with the dataset of NGSIM, the large-scale naturalistic vehicle trajectory dataset from German highways (HighD) attempts to the accuracy of vehicle motion information is improved. However, urban scenes containing intensive and highly interactive behaviors, such as roundabouts and unsignalized intersections, are not included in NGSIM and HighD.

3) Lyft [68]: Lyft includes sensing data and prediction data. It covers the movement logs of cars, cyclists, pedestrians, and other traffic agents encountered by their autonomous fleets.

4) INTERACTION [69]: The INTERnational, Adversarial and Cooperative moTION (INTERACTION) dataset was proposed for semantic mapping of interactive driving scenarios with different driving cultures. It not only focuses on the integrity of high-precision semantic map information and interactive entities, but also contains crisis data, which can effectively promote research on extreme scenarios.

5) HDD [70]: The Honda Research Institute Driving Dataset (HDD) is a public dataset based on onboard sensors, recording driving scenes with a longer time span and larger space range.

6) Argoverse [71]: Argoverse is also a public dataset based on onboard sensors. It provides relatively rich semantic map information. However, due to the limitations of onboard sensors, the motion track of surrounding objects is incomplete. The inaccuracy of other vehicles’ positions around the environment makes it more difficult to confirm the aerial view.

7) Commonroad [72]: Composable benchmarks for motion planning on roads (CommonRoad) are the first planning benchmarks with driving dataset for autonomous driving. In CommonRoad, real-world scenarios in the dataset can be easily converted into simulation scenarios to test the performance of decision-making algorithms offline.

8) nuPlan [73]: nuPlan is a closed-loop ML-based planning benchmark. It includes the latest released dataset dedicated to machine learning-based planning methods. The dataset of nuPlan can effectively evaluate the performance of long-term planning and provides a closed-loop simulation framework and metrics for specific scenarios.

As described above, the latest driving datasets, such as Commonroad [72] and nuPlan [73], already possess the basic elements for training and evaluating human-like driving decision-making methods. However, the data in complex scenarios should be further supplemented and improved, and data on extreme cases is still insufficient.

B Simulation Platforms

1) Classic Driving Simulators: Simulation plays an important role in data collection and experimental verification [83]. Considering the risks involved in recording driving data in certain scenarios, driving simulators should be used to supplement extreme (even crash) data to reduce the security risks of testing in the real world. Therefore, researchers have higher requirements for driving simulation, which promotes the development of high-fidelity simulators. Table III lists the driving simulations commonly used in autonomous driving.

Table  III.  Comparison of Some Representative Autonomous Driving Simulators and Their Characteristics

Simulator	Reference	Description
TORCS	[74]	The open racing car simulator (TORCS) is an open-source racing simulator which is also widely used in verification for autonomous driving algorithms. However, the functions of TORCS are limited, and the motion characteristics of the simulated vehicle in TORCS are significantly different from those of a real vehicle.
CARLA	[75]	Car learning to act (CARLA) is an open-source simulator that provides virtual scenes required to develop autonomous driving algorithms, including urban layouts, buildings, and vehicles. CARLA supports the free configuration of simulation scenarios and onboard sensors and controls both static and dynamic participants.
CarSim	[76]	The CAR-following SIMulation model (CarSim) is specifically designed for vehicle dynamics that simulate the response of a vehicle to its driver, road surface, and aerodynamic input. Due to CarSim’s high-fidelity vehicle dynamics, it is usually combined with other simulation software to build autonomous driving simulation systems [77].
Gazebo	[78]	Gazebo is a built-in simulation software of the robot operating system (ROS) [79], which offers the ability to accurately and efficiently simulate robots. Since ROS has been widely applied in the development of autonomous driving [80], Gazebo-based simulators have become increasingly popular.
ADAPS	[81]	Autonomous driving via principled simulations (ADAPS) learns autonomous driving strategies from vehicle accidents, which includes a high-fidelity simulation platform for testing and a simulation platform for accident analysis. This framework provides a more effective online framework for learning that significantly reduces the number of iterations required to learn autonomous driving strategies.

 | Show Table

DownLoad: CSV

The complete system for evaluating and testing decision-making capabilities based on driving simulators includes modules for collecting driving data from human drivers, identifying driving behaviors, and evaluating these behaviors. In recent times, researchers have started testing self-driving algorithms in simulations before putting them into practice in the real world. Different solutions were compared to obtain more convincing results [84], [85]. Moreover, simulation testing has been combined with real-world testing to develop and validate decision algorithms.

In practice, several necessary factors should be considered when building a simulation platform to evaluate and test decision-making capabilities. These factors include: 1) a dynamic vehicle model that accurately reflects the kinematics and dynamics characteristics of the actual vehicle; 2) various typical traffic scenarios; and 3) tests conducted with and without drivers under the same test conditions, as well as driving behavior evaluation based on vehicle movement. As shown in Fig. 4, the autonomous driving test simulation system should be a unified framework that includes offline simulation, real-time hardware-in-loop simulation, and driving simulators. The system not only needs to analyze driving behavior through collected driving data, but also needs to analyze the driving behavior of autonomous vehicles controlled by algorithms, evaluating their decision-making abilities under the same criteria.

Figure  4.  An example of a simulation platform for autonomous driving.

DownLoad: Full-Size Img PowerPoint

2) Mixed Reality Based Driving Simulators: Despite the interrelationship of virtual simulation, it is necessary to find a method to provide a more comprehensive and realistic representation of traffic scenarios, in order to convincingly verify the decision-making algorithms related to autonomous driving. Mixed reality (MR) is a combination of real and virtual elements, in which physical and digital objects coexist and interact in real-time. It retains the real-world scene restoration with security and element diversity. In recent years, many MR-based methods have been proposed for autonomous driving [86], and some platforms have been developed to verify the decision-making algorithm. Compared to traditional simulators, the results are more convincing.

In addition, virtual reality (VR) technology is used to simulate the virtual scene, where pedestrians interact with autonomous vehicles. By wearing VR devices, pedestrians can explore their responses to autonomous vehicles in different scenarios [87]. Ride comfort can also be investigated by collecting information about users’ immersion in VR-based driving simulations [88].

The main problems of the current MR-based driving simulator are as follows:

a) Consistent with reality: When drivers or pedestrians use mixed reality equipment, the simulation system needs to provide sufficient support to display and interpret the scene. This ensures that the response of the human tester is consistent with the real-world situation.

b) Usability: Compared to the combination of a traditional driving simulator and a large screen display, MR equipment is more difficult to use. Many users are not familiar with MR equipment and may deviate from the real world due to unskilled operation.

c) Standardization: Most MR-based simulators lack a common framework and unified metrics, which is not conducive to comparing different works.

V. Discussions

The latest research shows that current technology effectively solves decision-making tasks for different levels of autonomous vehicles. However, many challenges remain unresolved. In this section, we will discuss some important issues.

A Personalized Driving

As one of the important purposes of human-like driving, personalized driving deserves further discussion. Driving style is defined as the driver’s relatively stable, long-term, and inherent behavioral tendencies. It integrates the driver’s psychological thinking and behavioral mode. The future research trend is to consider personal characteristics to improve vehicle safety and passenger acceptance [89].

Driving styles can generally be divided into three categories:

1) Radical driving style: This style takes some risks to achieve the driving goal, accelerates and decelerates violently and frequently, and shows more aggressive driving behavior.

2) Careful driving style: This style places safety above achieving driving goals, showing more conservative acceleration and deceleration, or preferring to stay in the current lane or give way to other vehicles.

3) Moderate driving style: This style is between radical and cautious.

Obviously, such classification is rough and lacks unified standards. The driving styles of different individuals are significantly different. Thus, designing a reasonable scale to distinguish different driving styles is an important prerequisite for personalized driving. Recently, some works on driving styles have been conducted to define and simulate the driving behavior of humans or autonomous vehicles in specific environments or traffic scenarios, such as lane changing, car following, and intersections [90]–[92]. The features and scales selected in these works are also related to specific driving scenarios. In addition, the designed passenger preference measurement adopts parameters directly related to vehicle movement, including the preferred horizontal and vertical growth areas and the maximum allowable acceleration [93].

In fact, the public’s preference for driving style tends to be personalized, but there are also some rules. For example, it was found that passengers tend to be cautious when seated in autonomous vehicles [94]. When people can not control their driving, they prefer a cautious style. In addition, autonomous vehicles can use collaboration to perceive and share information that human drivers cannot directly obtain. Therefore, autonomous vehicles may be more confident than humans. These behaviors may be seen as causing public discomfort, even if they do not pose a danger.

B Human-Likeness

The similarity between algorithms and human decisions has been studied from different perspectives. Therefore, a unified indicator is needed to evaluate the human similarity of the algorithm, and the principle and driving performance should be considered to accurately evaluate the human similarity of the algorithm.

As described in Section III, the existing decision-making methods are evaluated for their human likeness through the following four aspects: 1) combination of deterministic and fuzzy logic, 2) adaptability to unknown environments, 3) consideration of random factors, and 4) learning ability. In addition, interpretability is also an important basis to evaluate the human similarity of the algorithm. It is the degree to which humans understand how to make decisions [95]. The higher the interpretability of the decision-making model, the easier it is for humans to understand why they make certain decisions. Only when the algorithm can be understood and interpreted by humans, can trust be established between humans and the model.

Furthermore, the driving performance of human drivers is an important reference for evaluating human likeness from the perspective of driving performance. In order to demonstrate the potential of their models to be similar to human driving behavior, some researchers have explained the “human-like” nature of their methods by directly comparing them with human driving data [96]. The similarity between them shows the similarity between intelligent decision-making algorithms and human decision-making processes. In general, the evaluation of human-like driving performance includes the following indicators: 1) driving comfort, 2) driving safety, 3) similarity with human demonstration trajectories, and 4) characteristics relative to other traffic participants.

C Real World Applications

There is no doubt that safety is the most important challenge and critical factor in autonomous driving. In the real world, traffic participants usually exhibit dynamic and uncertain behavior, so the autonomous vehicle must be able to handle all possible situations. Otherwise, decision-making will no longer be reliable and may even lead to serious traffic accidents. The data-driven method can improve the traversal depth of traffic scenarios, but it requires plenty of data to ensure safe decision-making. Therefore, it is necessary to design road tests to collect data covering environmental uncertainties. In addition, researchers should understand the interaction between autonomous vehicles and other elements in the traffic environment through road testing. Potential collision risks should be discovered in advance by designing some typical extreme scenarios.

Due to the danger of real-world testing, it is important to develop and utilize high-fidelity driving simulators. Unfortunately, current simulated traffic scenarios are still unrealistic and cannot fully cover all possible situations in the real world. Although it cannot be guaranteed that the algorithms that meet safety requirements in the simulation environment will still be safe when transplanted to the real world, it can be predicted that driving simulators will play an increasingly important role in the development of decision-making algorithms.

D Can Human-Like Driving Strategies Overcome the Limitations of Human Drivers?

Human drivers face limitations that can impact their ability to navigate roads safely. These limitations include distractions, fatigue, emotions, and subjective behavior. These factors impair their judgment, decision-making abilities, and reaction times, leading to slower responses, delayed decisions, and an increased likelihood of accidents.

The purpose of researchers developing human-like driving algorithms is to learn the driving style and good driving habits of human drivers, rather than learning the problems caused by the shortcomings of human drivers. In contrast, autonomous vehicles process vast amounts of data and make real-time decisions. They are not subject to the same limitations as human drivers, which means they can make more accurate and efficient decisions in dynamic traffic environments, taking into account various factors of uncertainty and ensuring the safety of all road users.

It should be noted that there is currently no single type of humanoid driving decision algorithm that can completely overcome the limitations of human drivers. To achieve a more accurate, efficient, and reliable autonomous driving system that surpasses the performance of human drivers in all aspects, it is necessary to analyze the psychology, decision-making models, and decision-making processes of human driving and use a combination of various advanced algorithms.

VI. Future Research

A Combination of Intuition and Logical Reasoning

The decision-making based solely on logical reasoning is insufficient to deal with complex, dynamic traffic scenarios [97]. Intuitive reasoning enables human beings to effectively avoid risks. Therefore, it is important to combine intuition and logical reasoning in order to adapt to dynamic traffic scenarios and make them comparable to human capabilities. One of the representative methods of this combination is fuzzy reinforcement learning, which is crucial to the realization of general decision-making for autonomous vehicles.

Fig. 5 shows the process of an adaptive cruise controller implemented through fuzzy reinforcement learning. Fuzzy reinforcement learning uses a membership function to transform continuous states into fuzzy semantic states; it then can generate complete rules after training. According to the rule base, the continuous output is obtained by weighted summation. In traditional fuzzy systems, the rule base is usually determined by a large amount of expert knowledge, and reinforcement learning can independently learn expert knowledge through the interaction between the agent and the environment, which plays a key role in the practical application of the system.

Figure  5.  Decision-making framework based on fuzzy reinforcement learning for adapt cruise control.

DownLoad: Full-Size Img PowerPoint

In order to verify the effectiveness of the proposed framework shown in Fig. 5, several test scenarios are designed on the driving simulator [98]. When the front vehicle accelerates violently, the human-like decision-making system will give appropriate acceleration to smoothly control the autonomous vehicle to catch up with the vehicle ahead. Although the speed will have a certain overshoot in this process, the acceleration is relatively stable, which ensures the safety of vehicle following tasks. On the contrary, when the vehicle ahead decelerates, the proposed method controls the autonomous vehicle so that it decelerates quickly without collision risk, so that the distance between the vehicles will never be less than the safe distance to ensure the safety of the autonomous vehicle. The driving behaviors learned from the proposed framework are not predefined but are fully learned through the process of interaction with the environment. It shows the good potential of combining logic and intuitive reasoning to learn human driving behavior.

On the basis of traditional reinforcement learning, the rapid development of deep reinforcement learning in recent years is crucial to developing a decision-making method that combines intuition and logical reasoning. For example, the success of deep reinforcement learning in AlphaGo [99]–[101] and electronic games [102]–[104] has proven its potential to build intelligent human-like decision-making systems. It has been pointed out that rewards are enough to drive intelligent behaviors studied in nature and artificial intelligence, including knowledge, learning, perception, social intelligence, language, generalization, and imitation [48]. Therefore, the combination of deep reinforcement learning and traditional model-based methods is an important direction of current research on the human-like decision-making technology of autonomous driving.

B Hierarchical Decision-Making Framework

In the “end-to-end” model, only low-dimensional control commands or sparse rewards are used for learning. Due to this limitation, behavioral intelligence may not be well displayed. One possible way to improve this is through hierarchical decision-making, which can combine planning, decision-making, and path-tracking control into a complete module. The decision-making framework includes several levels, such as task, maneuver, and motions, where different levels refer to the decision-making process for human beings to achieve different goals. Furthermore, the hierarchical decision-making framework combines semantic and numerical methods to ensure safety and compliance with traffic rules, while providing common functions to solve new complex scenarios. In addition, the effective combination of rules and learning allows the network to focus on problems that do not exist in the rule base. For example, a hierarchical decision-making system was proposed to combine reinforcement learning with a sampling-based motion planner, which is a reasonable method [105].

VII. Conclusions

With the development of artificial intelligence, people have higher expectations for the possibility of autonomous driving. Exploring human-like decision-making technology is an important research direction for the development of autonomous driving. Thus, it is very important to improve the ability of the decision-making system to build a reasonable driving model so that the autonomous vehicle can learn human expert knowledge and driving habits.

In this paper, the development of human-like driving technology was reviewed, where the advantages and disadvantages of various decision-making methods were analyzed, and related problems were discussed. In addition, the relevant research on driving datasets and simulation platforms was also reviewed for the development of advanced decision-making algorithms. Finally, several issues related to human-like driving were discussed, and potential research directions were presented as well.

It is an arduous challenge to develop human-like driving so that autonomous vehicles have decision-making ability similar to human beings, but it has great potential value in academic and industrial applications. We firmly believe that the joint efforts of academia and industry will help promote the rapid development of automatic driving technology.