Learning-based cooperative decision-making and control for multiple autonomous vehicles in unsignalized intersections

2.1. MAVs Markov decision process modeling

The MAVs' cooperative decision-making at intersections is modeled as a decentralized partially observable Markov decision process (Dec-POMDP), which is given by the following tuple:

As shown in Figure 1, a T-junction without traffic signals involves six AVs requiring cooperative decision-making, including left-turning vehicles, right-going straight vehicles, and left-going straight vehicles. The state space of the $$ N $$ AVs at the intersection is denoted by $$ \mathcal{S} $$. Taking the left-turning AV as an example to illustrate the sample collection process, the AVs in the straight-going lane are controlled using the car-following model^[35], providing state information for the left-turning AV. When the left-turning AV enters the decision-making zone, its state at discrete time $$ k $$ is defined as $$ s_i(k) = [v_i, v_{ri}, d_{ri}, v_{fi}, d_{fi}] \in \mathcal{S} $$, including its own velocity $$ v_i $$, velocities of the closest front vehicle $$ v_{fi} $$ and rear vehicle $$ v_{ri} $$, and distances to those vehicles $$ d_{fi} $$ (front) and $$ d_{ri} $$ (rear). The action space for vehicle $$ i $$ is denoted as $$ \mathcal{A}_i $$. At time step $$ k $$, vehicle $$ i $$ selects an action $$ a_i(k) \in \mathcal{A}_i $$ corresponding to one of the three behaviors: decelerating, maintaining current speed, or accelerating. The state transition probability $$ P $$ is given by $$ s_i(k+1) \sim P(\cdot \mid s_i(k), {a}_i(k)) $$, indicating the probability-based transition from state $$ s_i(k) $$ and action $$ {a}_i(k) $$ at time step $$ k $$ to the next state $$ s_i(k+1) $$. The reward function $$ R_i $$ reflects the traffic conditions of MAVs at the intersection.

Figure 1. Illustration of MAVs decision-making modeling. MAVs: Multiple autonomous vehicles.

2.2. Coordination graphs

Probabilistic graphical model (PGM) integrates probability theory with graph theory and constitutes a pivotal research direction in machine learning^[36]. PGM primarily focuses on the representation, inference, and learning of directed graphical models and undirected graphical models. A core objective in artificial intelligence is to extract implicit, fundamental knowledge from observed data, and PGM serves as a powerful methodology to achieve this goal. In MAVs systems, nodes of the probabilistic graph represent vehicles, while edges with connecting relationships describe the local performance of joint actions taken by adjacent vehicles, inferring the posterior distribution of joint actions based on current vehicle information. Probabilistic graph-based approaches can effectively model coupling relationships among MAVs, which facilitate the construction of large-scale artificial intelligence systems to address complex real-world problems.

The coordinated RL method introduces a coordination graph to establish connecting edges among agents with cooperative relationships^[37]. The messages $$ m_{ij}(a_j) $$ transmitted along adjacent edges $$ (i, j)\in E $$ can fully describe the coordination among multiple agents, as illustrated in Figure 2. By incorporating coordination graphs into multi-agent systems, the representational capacity of the value function is enhanced, which effectively addresses decision-making problems. In this case, the global state-action value function is expressed as

Figure 2. Message passing graph.

where $$ p_i(s_i, a_i) $$ represents the local utility function of agent $$ i $$ after taking action $$ a_i $$, and $$ p_{ij} $$ represents the collaborative utility function of neighboring agents $$ i $$ and $$ j $$ after they take actions $$ a_i $$ and $$ a_j $$, respectively.

The Ref^[36] demonstrates a direct duality between computing the maximum posteriori probability in PGM and solving the optimal joint action in coordination graphs. Consequently, the belief propagation algorithm, used for inference in PGM, is also applicable to multi-agent collaborative decision-making using coordination graphs. In Figure 2, agent $$ i $$ sends a message $$ m_{ij}(a_j) $$ to neighbor $$ j $$, which represents the maximum reward that agent $$ i $$ can obtain after agent $$ j $$ takes action $$ a_j $$^[38]. The message value is expressed as

where $$ \Gamma(i) $$ denotes the set of neighbors for the $$ i $$-th agent, $$ \bar{k} \in \Gamma(i) \backslash j $$ indicates that $$ \bar{k} $$ belongs to the set of neighbors, excluding $$ j $$. When there are cycles in the graph, $$ c_{ij} = \frac{1}{|A_k|}\sum_k{m_{ik}}(a_k) $$, otherwise, $$ c_{ij}=0 $$. Before convergence, agent $$ i $$ aggregates the messages received from its neighbors, but it does not require enumerating the joint action spaces of neighboring agents. After several iterations of message passing and updates among neighboring agents, the messages $$ m_{ij}(a_j) $$ converge to a fixed point. In each iteration, the state-action value function of agent $$ i $$ is represented as

which indicates that the state-action value function of agent $$ i $$ consists of the local utility function $$ p_i(a_i) $$ and the message values from different subtrees with neighbor agent $$ j $$ as the root. The optimal decision action for agent $$ i $$ can be derived as

Building upon the aforementioned mechanism, this paper proposes a LCDMC method for MAVs. The local utility function $$ p_i(a_i) $$ of vehicle $$ i $$ is constructed via a KLSPI RL algorithm, while the local collaborative utility function $$ p_{ij} $$ characterizes the performance of joint actions. A coordination graph-based message passing mechanism is employed to resolve the cooperative decision-making control problem for MAVs at unsignalized intersections.

3.1. The overall framework of the proposed method

The proposed algorithm primarily consists of two stages: offline policy learning and online deployment implementation. The algorithmic framework is depicted in Figure 3. In the offline training phase, samples are collected using a random policy. An approximate linear dependency (ALD) analysis is employed to learn model features and construct the kernel dictionary from the high-dimensional state space, thereby reducing the dimensionality of the samples. Then, a sparse kernel-based least-squares RL method is adopted to learn the policy. During the online deployment phase, when AV $$ i $$ enters the decision-making zone, it forms interactions with neighboring AVs $$ j $$, $$ j \in \Gamma(i) $$. A collaborative utility function $$ p_{ij}(a_i, a_j) $$ is defined to evaluate the joint actions between AVs $$ i $$ and $$ j $$. Based on its current state $$ s_i $$, the AV applies the offline-learned policy to compute the local utility function $$ p_i( a_i) $$ for each candidate action. Following the message-passing Equation (3), the action of AV $$ i $$ is derived by the local utility $$ p_i( a_i) $$ and collaborative utility function $$ p_{ij}(a_i, a_j) $$.

Figure 3. The block diagram of the proposed LCDMC algorithm. LCDMC: Learning-based cooperative decision-making and control.

Based on coordination graph theory, the global state-action value function is given by Equation (2). This formulation demonstrates that optimal decisions for MAVs can be computed in a distributed manner by iteratively exchanging messages along the coordination graph edges, thus avoiding exhaustive searches over the joint-action space. The proposed method contrasts with AV $$ i $$ directly computing actions via KLSPI. The decision module's output speed serves as the reference input, and an online RHRL method computes an optimal control law to ensure tracking of a reference trajectory. Gradient-based iterative updates of the critic and actor networks improve the real-time performance of the system.

3.2. Kernel-based efficient learning for local utility function

This section elaborates on the proposed LCDMC method for MAVs, which learns the local utility function via KLSPI. During the offline phase, extensive samples are pre-collected at unsignalized intersections. An optimized local decision-making policy is derived in a data-driven manner and subsequently deployed online for real-time decision-making.

In the coordinated traversal of multiple vehicles at the unsignalized intersection shown in Figure 1, the tuple for multi-vehicle cooperative decision-making modeling is defined in Equation (1). Under the current state $$ s_i(k) $$, the AV updates its state upon executing action $$ a_i(k) $$, while the reward function is designed to reflect the traffic dynamics of MAVs at the intersection. If a left-turning AV safely traverses the straight lane within 5 \text{s}, sampling terminates, and the reward is assigned as $$ R_i = -\varepsilon t $$, where $$ \varepsilon $$ is an adjustable positive constant. If the AV collides with neighboring vehicles on the straight lane, the reward function incurs a significant penalty $$ R_i = -1, 000 $$. If the vehicle remains stationary at the initial starting point without collision, the reward is set to $$ R_i = -400 $$. The reward function is given in Equation (6), and the sample $$ [s_i(k), a_i(k), R_i(k), s_i(k+1)] $$ is collected. AV samples in adjacent lanes are collected similarly. Table 1 lists the sampling parameters, where sd, kv, and acc represent deceleration, constant-speed maintenance, and acceleration, respectively.

Upon acquiring a substantial set of samples, the KLSPI algorithm is employed for policy learning. This section extends sparse kernel-based feature representation methods^[39] to cooperative decision-making in MAVs. In the proposed LCDMC framework, the local utility function of the AV is represented via basis functions constructed from feature vectors. A kernel sparsification technique is applied to extract features from the high-dimensional samples, yielding representative subsamples through ALD analysis. Assuming the kernel function $$ \kappa(\cdot, \cdot) $$ satisfies Mercer's condition, there exists a mapping $$ \phi(\cdot) $$ from the sample space to the Hilbert space, that is,

where $$ \langle\cdot, \cdot \rangle $$ denotes the inner product in the Hilbert space. This implies that all inner products can be computed via the kernel function, without explicit knowledge of the mapping $$ \phi(\cdot) $$.

Then, the representative subsample is obtained using the ALD analysis technique. Sample sparsification is implemented via ALD in two steps. Firstly, the distance between sample $$ s_i(t) $$ and feature point $$ s_i(q) $$ is calculated using

where $$ c=[c_1, c_2, \cdots , c_{t -1}]^\top\in \mathbb{R} ^{t -1} $$, $$ D_i(t)\in \mathbb{R} ^{t-1} $$ denotes the current kernel dictionary. Secondly, the dictionary is updated. $$ \mu _i $$ is set to quantify the degree of sample sparsification. If the distance between $$ s_i(t) $$ and $$ s_i(q) $$ is not less than $$ \mu _i $$, it implies that insufficient representational capacity of the current kernel dictionary for the given sample. Consequently, $$ s_i(t) $$ is incorporated into the dictionary, $$ D_i(t)=D_i(t-1)\cup {s}_i(t) $$. Otherwise, the dictionary $$ D_i(t) $$ remains unchanged.

The feature construction module utilizes the subsampled points generated previously to construct the basis function for each original sample as $$ K(s_i(k)) = [\kappa (s_i(k), s_i(t_1)), \kappa (s_i(k), s_i(t_2)), \cdots, \kappa (s_i(k), s_i(t_n))]^\top \in \mathbb{R}^n $$, where $$ n $$ denotes the dimensionality of the feature space. $$ \kappa(x_1, x_2)=\exp(-\frac{\|x-y\|^2}{2 \iota^2}) $$, $$ \iota $$ is the kernel width. The local utility function is approximated using

where $$ W_{ic} \in \mathbb{R}^n $$ is the weight vector of the network. The temporal difference (TD) error of RL is given as $$ \eta_i(k) = \hat{p}_i(s_i(k)) - \bar{p}_i( s_i(k) ) $$, where $$ \bar{p}_i(s_i(k)) =s_i^\top(k)\mathcal{Q} s_i(k)+u_i^\top(k)\mathcal{R} u_i(k)+\gamma_i \hat{p}_i(k+1) $$, $$ \mathcal{Q} $$ and $$ \mathcal{R} $$ are positive definite weight matrices, with $$ \gamma_i $$ denoting the discount factor.

By combining Equation (9) and rewriting TD error, we have:

By multiplying both sides of Equation (10) by $$ K(s_i(k)) $$ and defining

where $$ \bar{A} $$ is a full-rank matrix. By minimizing the TD error, the optimal network weights can be obtained through the least-squares method, which is:

Based on the above analysis, the KLSPI method is employed to learn local policies for AVs from collected high-dimensional samples. This method enables real-time derivation of the local utility function $$ p_i $$ during the online deployment phase, which constitutes a component of the $$ i $$-th vehicle's local state-action value function. Then, the optimized decision action can be computed online based on the derived function.

3.3. Online decision-making using coordination graphs

In the proposed LCDMC method for MAVs, a cooperative relationship is established among vehicles across different lanes. The Max-plus message passing mechanism is utilized to iteratively propagate local joint-action rewards through cooperative communication edges, enabling optimized decision-making for MAVs.

The cooperative interactions among AVs are inherently dynamic due to the multi-vehicle traffic flow dynamics. At unsignalized intersections, an undirected edge between AVs $$ i $$ and $$ j $$ is established if their paths may intersect. The potential path intersection is determined based on the distance and speed of the preceding and following vehicles once an AV enters the decision zone^[40]. As shown in Figure 4, when a left-turning AV enters the decision zone, it observes the speed and distance of the nearest leading and following AVs in the straight lane. If no AV is detectable within the perception range, the distance and speed of neighboring vehicles are set to a predefined large constant. The position coordinate of AV $$ i $$ is represented as $$ z_i = [x_i, y_i]^\top $$. Based on potential path intersections, the neighbors of vehicle $$ i $$ are defined as follows

Figure 4. Cooperative relationships in MAV systems. MAV: Multiple autonomous vehicle.

where $$ A_i $$ and $$ A_j $$ denote the decision zones of vehicles $$ i $$ and $$ j $$ in their respective lanes, as shown by the shaded regions in Figure 4.

After AV $$ i $$ acquires coordination edges and neighboring vehicles, the local collaborative state is represented as $$ \varPsi_i(t)=[ z_i^\top, z_j^\top]^\top $$, where $$ j\in\mathcal{N}_i $$. Upon executing the joint action, the collaborative state transitions to $$ \varPsi_i(k+1) $$. The collaborative utility function between adjacent AVs in the current state is defined as

where $$ \bar{z}_h(k+1), ( h=i, j) $$ represents the predicted distance of vehicle $$ h $$ to the target position $$ z_m=[x_m, y_m]^{\top} $$, and

where $$ dx_h(k+1) = x_h(k+1) - x_m $$, and $$ dy_h(k+1) = y_h(k+1) - y_m $$, with $$ h= i, j $$.

The term $$ p_{ij}(a_i, a_j) $$ characterizes the influence of joint actions between AVs in cooperative settings. Utilizing dynamic coordination graphs, the Max-plus message propagation mechanism enables AVs to determine optimal joint actions that maximize the predicted distance. Specifically, the mechanism prioritizes vehicles closer to their target positions for intersection crossing, while guaranteeing collision avoidance. This approach facilitates safe and coordinated multi-vehicle navigation at unsignalized intersections.

The computational complexity of the coordination graph primarily stems from two components: coordination edge updates and message propagation. Coordination edge updates refer to the distance computation between vehicles during each path intersection detection, while message propagation denotes the iterative process of the Max-plus algorithm along each edge until convergence. The complexity is $$ O( km(n+m) |\epsilon |) $$, where $$ k $$ denotes the number of message-passing iterations, $$ m $$ is the action space size of a single agent, $$ N $$ and $$ |\epsilon | $$ represent the number of agents and coordination edges^[36], respectively.

The forming and dissolving of edges in the coordination graph are primarily determined by the vehicles' traversal tasks and spatial correlations. The specific rules are as follows. Edge forming triggered by paths intersect: each lane at an intersection has a designated decision zone. When a vehicle enters this zone, its onboard sensors detect the speed and position of leading and following vehicles in adjacent lanes. If the detected vehicles' trajectories spatially intersect, a coordination edge is established, indicating the need for cooperative decision-making. Edge dissolving triggered by complete traversal or distance threshold: the system removes the corresponding coordination edge when an AV either completes cooperative traversal or exits the decision zone, thereby reducing computational complexity.

3.4. RHRL controller for real-time trajectory tracking

A RHRL approach is designed to achieve trajectory tracking control of the AVs in this section, with the reference velocity profile determined through multi-vehicle cooperative decision-making. The kinematic model of an Ackermann-steering AV is given as

where $$ x_i, y_i, \varphi_i $$ represent the vehicle's x-coordinate, y-coordinate, and yaw angle in the global coordinate system, respectively; and $$ \delta _{i} $$, $$ v_i $$, $$ L $$ denote the front wheel steering angle, longitudinal velocity, and wheelbase, respectively. The tracking error is defined as $$ \bar{e}(k ) =z_i(k )-r_i(k ) $$, where $$ z_i=[ x_i, y_i, \varphi_i] ^\top $$, $$ r_i=[ x_{ir}, y_{ir}, \varphi _{ir}] ^\top $$ represents the actual and reference trajectory. Furthermore, transforming the tracking error into the local coordinate system yields

By combining Equations (16) and (17), the tracking error dynamics can be discretized as follows:

where $$ v_{ir} $$ denotes the reference velocity corresponding to the decision action, $$ \Delta t $$ represents the sampling interval, and $$ \delta_{ir} $$ indicates the front wheel steering angle of the reference trajectory. For the tracking control problem of the nonlinear discrete-time system (16), a finite horizon performance index is defined as

where $$ U_i(e_i(\tau), u_i(\tau)) $$ represents the reward function for vehicle $$ i $$, $$ U_i(e_i(\tau) , u_i(\tau) )=e_i^\top(\tau) \hat{\mathcal{Q}}_ie_i(\tau) +u_i^\top(\tau)\hat{\mathcal{R}}_iu_i(\tau) $$, where $$ \hat{\mathcal{Q}} $$ and $$ \hat{\mathcal{R}} $$ are predefined positive definite weighting matrices. The prediction horizon length $$ H $$ is a positive integer. The terminal penalty function is given as $$ e_i^\top(k+H)P_ie_i(k+H) $$, and the terminal penalty matrix $$ P_i $$ can be obtained by solving the Riccati equation for the linearized discrete-time system^[41].

At the sampling instant $$ k $$, the learning strategy is implemented over the prediction horizon $$ [k, k+H-1] $$, and the corresponding value function is expressed as

The costate of the value function is defined as $$ \lambda_i(\tau)=\partial V_i(\tau)/\partial e_i(\tau) $$, and the optimal costate is given as

Then, the optimal control output is defined as

Due to the difficulty in obtaining analytical expressions of the control input for nonlinear systems, an actor-critic RL framework is adopted to learn the policy within the prediction horizon. This framework utilizes a value iteration approach, in which neural networks are employed to approximate the costate function and the optimal control input. The structure of the critic network is expressed as:

where $$ W _{ci} $$ and $$ \sigma_c ( \cdot ) $$ represent the weight matrices from the hidden layer to the output layer, and the activation function of the critic network, respectively. The TD error of the critic network is defined as $$ \eta _{ci}(\tau)={\hat{\lambda}_i}(e_i(\tau))-{\bar{\lambda}_i}(e_i(\tau)) $$, where

The critic network approximates the optimal costate by minimizing the error function $$ {E_{ci}}(\tau) = \frac{1}{2}{( {{\eta _{ci}(\tau)}} )^2} $$. Using the gradient descent method, the update rule for the network weights is formulated as

where $$ 0 < \alpha_c < 1 $$ is the learning rate of the critic network.

The actor network of the AV is a three-layer neural network that takes the tracking error $$ e_i $$ as input and approximates the control input $$ u_i $$. Its structure is as follows

where $$ W _{ai} $$ and $$ \sigma_a ( \cdot ) $$ represent the weight matrices from the hidden layer to the output layer, and the activation function of the actor network, respectively. The approximation error of the actor network is defined as $$ \eta _{ai}(\tau) =\hat{u}_i(\tau) -\bar{u}_i(\tau) $$, with $$ \bar{u}_i(\tau)=-\frac{1}{2}\hat{\mathcal{R}}_i^{-1}\left[\frac{\partial ( e_i (\tau+1) )}{\partial u_i(\tau)}\right] ^\top\hat{\lambda}_i(\tau+1) $$. The actor network approximates the optimal control input by minimizing the error function $$ {E_{ai}(\tau)} = \frac{1}{2}{( {{\eta _{ai}(\tau)}} )^2} $$. Based on the gradient descent method, the update rule for the network weights is given as

where $$ 0 < \alpha_a < 1 $$ is the learning rate of the actor network.

Employing the RHRL approach, an optimized control sequence is learned within the prediction horizon at time step $$ k $$, and the first element is applied to the system. The actor-critic networks maintain continuity between adjacent prediction horizons by leveraging prior experience during learning, thereby accelerating policy convergence and improving real-time computational efficiency.

The online learning computational complexity of the RHRL approach stems from Equations (25) and (27), which is approximately $$ O(H(n_{ci}+n_{ui}+n_i)n_i) $$, where $$ H $$, $$ n_i $$, $$ n_{ci} $$, and $$ n_{ui} $$ denote the prediction horizon length, the dimension of input layer, the number of hidden layer neurons in the critic and the actor network, respectively^[41]. Additionally, the proposed approach obtains optimized control policies with an explicit structure, enabling direct deployment of the learned policy with reduced complexity $$ O(n_{ui}n_i) $$, thus meeting the real-time control requirements of AVs.

Based on the above analysis, the offline learning phase of the proposed method employs KLSPI to learn local policies from high-dimensional collected samples, which improve the computational efficiency and policy convergence. During the online deployment phase, when AV $$ i $$ enters the decision zone, communication edges with neighboring vehicles $$ j $$ on adjacent lanes are established. A collaborative utility function $$ p_{ij}(a_i, a_j) $$ is introduced, which characterizes the performance of joint actions between vehicles $$ i $$ and $$ j $$. Simultaneously, vehicle $$ i $$ deploys the learned policy and obtains the local utility function $$ p_i(a_i) $$ for different decisions. The final decisions are determined by combining the local utility function $$ p_i(a_i) $$ with the collaborative utility function $$ p_{ij}(a_i, a_j) $$ through the iterative message passing Equation (3). The optimized decision output serves as the reference signal for RHRL-based trajectory tracking control, ensuring safe and efficient intersection traversal. The proposed algorithm is summarized in Algorithm 1.

4.1. Scenario 1: MAVs coordination at an unsignalized T-shaped intersection

In scenario 1, AVs in Lane 1 perform left turns, while AVs in Lanes 2 and 3 continue straight, as depicted in Figure 5. During navigation, vehicles estimate the motion states of neighboring AVs. By establishing coordination edges with neighbors, they compute optimal actions for safe intersection traversal. The state definitions for AVs in scenario 1 are specified as follows: When an AV in Lane 1 enters decision zone $$ A1 $$, it establishes coordination edges with AVs in Lane 2. By observing the velocity and relative distances of vehicles in Lane 2, the state vector $$ s_1(k) = [v_1, v_{f2}, d_{f2}, v_{r2}, d_{r2}] $$ is recorded. Upon entering decision zone $$ A2 $$, the AV in Lane 2 establishes coordination edges with AVs in Lane 1, acquiring its state vector $$ s_2(k) = [v_2, v_{f1}, d_{f1}, v_{r1}, d_{r1}] $$. When an AV in Lane 1 enters decision zone $$ A3 $$, it establishes coordination edges with AVs in Lane 3 and acquires its state vector $$ s_1(k) = [v_1, v_{f3}, d_{f3}, v_{r3}, d_{r3}] $$. Similarly, when an AV in Lane 3 enters decision zone $$ A4 $$, it observes the states of vehicles in the left-turn lane and establishes coordination. The dashed bounding boxes mark AVs that have completed their traversal tasks.

Figure 5. Illustration of MAVs intersection traversal scenario 1. MAVs: Multiple autonomous vehicles.

During the sampling phase for scenario 1, vehicle states are sampled according to the parameters in Table 1. Each vehicle randomly selects an action (acceleration, deceleration, or maintaining speed) and executes it for 5 s. The updated state vector $$ s_i(k+1) $$ is then recorded, including the ego vehicle's velocity, neighboring vehicles' velocities, and their relative distances. The reward $$ R_i(k) $$ is computed based on the state transition via Equation (6), yielding the experience tuple $$ [s_i(k), a_i(k), s_i(k+1), R_i(k)] $$ for policy learning. The minimum safe distance is set to 3 $$ \text{m} $$, with collisions detected when longitudinal inter-vehicle distances in adjacent lanes fall below this threshold. Upon collecting $$ N_{\text{sam}} $$ samples per vehicle, the ALD method performs kernel sparsification to extract model features, with training parameters listed in Table 2. The kernel dictionary dimensions obtained via ALD are 193, 240, and 323 for the three lanes, respectively. The policy converges after 7 offline training iterations, with the weight convergence shown in Figure 6. The learned policy is subsequently deployed at unsignalized intersections to evaluate the performance of the proposed LCDMC algorithm. Figure 7 shows the steering angle responses for vehicles across different lanes, where red and green curves represent leading and following AVs, respectively.

Figure 6. Convergence process of local policy learning via KLSPI. KLSPI: Kernel-based least-squares policy iteration.

Figure 7. The steering angles of the leading and following AVs on different lanes in scenario 1. AVs: Autonomous vehicles.

Figure 8 demonstrates the cooperative decision-making and motion trajectories at typical time instants in scenario 1, where sd and acc represent deceleration and acceleration, respectively. Initially, vehicles in each lane are randomly positioned within $$ [0, 50] \text{m} $$ of the intersection, with randomly generated velocities in $$ [0, 10] \text{m/s} $$. Before entering the decision zones, all vehicles maintain tracking control with a constant speed of $$ 8 $$ m/s. At $$ t $$ = 1.3 s, the red-marked vehicle c0 on Lane 1 enters the decision zone $$ A1 $$ and detects the blue-marked vehicle c2 on Lane 2 near the convergence point. By integrating $$ p_i $$ with the collaborative utility function $$ p_{ij} $$, the Max-plus algorithm derives optimal actions: c0 decelerates while c2 accelerates through the zone. At $$ t $$ = 3.3 s, vehicle c0 enters decision zone $$ A_3 $$ and observes a preceding vehicle on Lane 3. Given that c0 is closer to the convergence point than the trailing vehicle c5 on Lane 3, the control strategy determines that c0 should accelerate while c5 should decelerate. At $$ t $$ = 4.4 s, black-marked vehicle c3 on Lane 1 enters the decision zone. Detecting that its preceding vehicle is still traversing the zone and no trailing vehicles are within sight, the LCDMC assigns deceleration to c3 and acceleration to green-marked vehicle c1. At $$ t $$ = 4.9 s, c3 observes that the preceding vehicle c1 has cleared the intersection, while c1 detects vehicle c5 ahead with no trailing vehicles. Both c3 and c1 execute acceleration. By $$ t $$ = 7 s, all AVs have successfully negotiated the intersection.

Figure 8. The cooperative decision-making and motion trajectories of MAVs at typical time instants in scenario 1. MAVs: Multiple autonomous vehicles.

From Figures 7 and 8, it can be observed that the following vehicles dynamically adjust their acceleration based on the velocity and distance of the leading vehicles, while considering their own speed. This behavior is consistent with the car-following model^[35] and effectively prevents collisions.

The performance metrics of the algorithms are statistically analyzed to demonstrate the superiority of the proposed approach. Each algorithm is evaluated over 1, 000 test trials. The performance metrics, including average execution time and its standard deviation, collision rate, failure rate, and comfort level, are presented in Table 3. Traversal time denotes the duration for all vehicles to completely pass the intersection and reach their target lanes. Collision rate counts the percentage of collision occurrences across 1, 000 trials. Failure rate represents the percentage of trials in which vehicles remain within the intersection after 10 s, failing to complete traversal. Comfort is quantified as the average root mean square (RMS) of vehicle accelerations, where lower values indicate higher comfort. Independent learning is a decentralized RL approach in which multiple agents independently learn their state-action value functions without coordination or communication^[38], treating others as part of the environment and updating their functions based on individual observations and rewards. The independent learning method^[38] deploys identical policies to those in the proposed LCDMC approach, it does not consider the performance of joint actions.

As demonstrated in Table 3, the proposed LCDMC method achieves higher traffic throughput efficiency compared to independent learning approaches at unsignalized intersections. This improvement is attributed to LCDMC's cooperative decision-making mechanism, which evaluates joint-action performance through the Max-plus message propagation mechanism, facilitating timely and rational decisions that reduce traversal time. In contrast, in independent learning approaches, vehicles prioritize self-interest by maximizing individual $$ p_i $$ values, leading to aggressive traversal attempts and frequent collisions. Conversely, the collaborative utility function in LCDMC is designed to prevent collisions while improving overall efficiency. Across 1, 000 trials, independent learning exhibited larger standard deviations in intersection traversal time and more frequent exceedances of the 10 s timeout threshold. The LCDMC algorithm achieved smoother velocity profiles and superior comfort metrics. These results demonstrate that LCDMC outperforms independent learning in multi-vehicle intersection scenarios in terms of traffic throughput efficiency, collision avoidance and motion stability.

4.2. Scenario 2: MAVs coordination at a complex unsignalized T-shaped intersection

Scenario 2 extends scenario 1 by introducing more complexity in the intersection traffic conditions. In this scenario, AVs in Lane 2 proceed straight, while AVs in Lane 1 and Lane 3 perform left turns, as illustrated in Figure 9. During traversal, AVs in different lanes must consider the motion states of neighboring vehicles to establish cooperative relationships and make rational decisions for safe intersection crossing. The learned policy is deployed in the unsignalized T-junction scenario to evaluate the performance of the LCDMC algorithm. Figure 10 shows the coordinated decision-making and motion trajectories at typical timestamps in scenario 2.

Figure 9. Illustration of MAVs intersection traversal scenario 2. MAVs: Multiple autonomous vehicles.

Figure 10. The cooperative decision-making and motion trajectories of MAVs at typical time instants in scenario 2. MAVs: Multiple autonomous vehicles.

Similar to scenario 1, we conducted 1, 000 test trials for cooperative decision-making for MAVs in scenario 2 to validate the algorithm's superiority. Statistical results of performance metrics are listed in Table 4. The results demonstrate that LCDMC achieves higher traffic throughput efficiency than independent learning at unsignalized intersections. This improvement stems from LCDMC's coordinated decision-making mechanism, which evaluates joint-action performance during policy deployment and leverages message propagation to achieve time-optimal decisions, which reduce traversal time. In contrast, the vehicle action policy in independent learning prioritizes individual rewards, leading to aggressive maneuvers and frequent collisions. LCDMC mitigates this issue through a joint-action utility function that reflects the joint-action performance. Across 1, 000 trials, LCDMC achieved a lower collision rate compared to independent learning. It should be noted that LCDMC's collision avoidance strategy occasionally requires deceleration commands, resulting in slightly worse comfort metrics than independent learning. Overall, the results demonstrate LCDMC's superiority in multi-vehicle coordination efficiency and safety for complex intersection navigation.

4.3. Scenario 3: MAVs coordination at an unsignalized cross intersection

This subsection evaluates the proposed LCDMC algorithm for unsignalized cross intersections. The scenario is illustrated in Figure 11, where AVs perform straight-through, right-turn, and left-turn maneuvers in mixed traffic. Effective navigation requires cross-lane coordination mechanisms to enable optimized decision-making, ensuring safe and efficient crossing. Figure 12 depicts the coordinated decision-making process and resulting motion trajectories at typical time instants in scenario 3.

Figure 11. Illustration of MAVs intersection traversal scenario 3. MAVs: Multiple autonomous vehicles.

Figure 12. The cooperative decision-making and motion trajectories of MAVs at typical time instants in scenario 3. MAVs: Multiple autonomous vehicles.

Subsequently, the performance metrics of different algorithms are calculated to validate the superiority of the proposed approach. Each method undergoes 100 trials, and the performance metrics are recorded in Table 5. The results show that independent learning exhibits a higher standard deviation in traversal time compared to the proposed LCDMC algorithm. Independent learning yields a collision rate of $$ 26\% $$ and a failure rate of $$ 14.5\% $$, where failure is defined as exceeding the 13 s timeout threshold. LCDMC achieves better comfort performance compared to independent learning. Overall, LCDMC outperforms independent learning for MAVs in intersection scenarios in terms of traffic throughput efficiency, collision avoidance and driving smoothness. These results align with scenarios 1 and 2.

The LCDMC algorithm integrates KLSPI to compute the local utility function $$ p_i $$. Meanwhile, the collaborative utility function $$ p_{ij}(a_i, a_j) $$ predicts the joint-action performance of AVs in adjacent lanes, enabling coordinated decision-making. Message propagation across coordination graphs drives an optimized decision policy. The resulting decision is converted into a reference velocity for AV trajectory tracking control via a RHRL approach. Multiple simulation scenarios demonstrate that the proposed LCDMC method improves policy learning efficiency and ensures safe and efficient cooperative navigation of MAVs at unsignalized intersections.

Authors' contributions

Made equal contributions to conception and design of the study and simulation and interpretation: Zhang, R.; Xu, X.

Technical and material support, draft improvement: Lu, Y.

Conceptualization, supervision: Xu, X.; Zhang, X.

Supervision, draft improvement: Ma, Q.

Availability of data and materials

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Financial support and sponsorship

None.

Conflicts of interest

Xu, X. (Xin Xu) is Associate Editor of the journal Intelligence & Robotics. Xu, X. (Xin Xu) was not involved in any steps of editorial processing, notably including reviewers' selection, manuscript handling and decision making. The other authors declare that there are no conflicts of interest.

Ethical approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Copyright

© The Author(s) 2025.