UAV maneuver decision-making via deep reinforcement learning for short-range air combat

2.1. UAV motion model

The three-degree-of-freedom UAV model is considered because the main consideration in short-range air combat problem is the position and velocity relationship between the two sides^[3]. The motion model is established in the ground coordinate system as East-North-Up (ENU) coordinate system, which is shown in Figure 1.

Figure 1. The UAV's motion model in the ground coordinate system. UAV: unmanned aerial vehicle

To simplify the problem, assume that the velocity direction is fixed with the -axis of the body coordinate system, and the UAV's motion model is shown as^{[14, 16]}

where $$ x $$ , $$ y $$ and $$ z $$ are the UAV's position coordinate values and $$ {\textbf{p}} = {\left[ {x, y, z } \right]^T} $$ , $$ v = \left\| {\textbf{v}} \right\| $$ is the velocity, $$ \dot x $$ , $$ \dot y $$ and $$ \dot z $$ are the values of $$ {\textbf{v}} $$ on the ground coordinate axes, $$ \gamma $$ , $$ \theta $$ and $$ \psi $$ represent the flight-path bank angle, flight-path angle and heading angle respectively, $$ g $$ is the acceleration of gravity, $$ {n_x} $$ is the overload in velocity direction, and $$ {n_z} $$ is the normal overload. Noting that the heading angle $$ \psi $$ is the angle between $$ {\textbf{v'}} $$ , the projection of $$ {\textbf{v}} $$ on the $$ xoy $$ plane, and $$ ox $$ axis. The basic control parameters $$ {n_x} $$ , $$ {n_z} $$ and $$ \gamma $$ in the motion model can be expressed as a control input vector $$ {\textbf{u}} = {\left[ {{n_x}, {n_z}, \gamma } \right]^T} \in {\mathbb{R}^3} $$ , which can be used to control the UAV's position and velocity in the air combat maneuver decision making^[16]. At the same time, the UAV's motion must satisfy the constraints which are expressed as^[3]

where subscript $$ min $$ and $$ max $$ mean the minimum and maximum values, and the parameters are set as $$ {v_{\min }} = 30{m \mathord{\left/ {\vphantom {m s}} \right. } s} $$ , $$ {v_{\max }} = 150{m \mathord{\left/ {\vphantom {m s}} \right. } s} $$ , $$ {\theta _{\min }} = - {\pi \mathord{\left/ {\vphantom {\pi 4}} \right. } 4} $$ , $$ {\theta _{\max }} = {\pi \mathord{\left/ {\vphantom {\pi 4}} \right. } 4} $$ , $$ {n_{x\min }} = - 1 $$ , $$ {n_{x\max }} = 2.5 $$ , $$ {n_{z\min }} = - 4 $$ and $$ {n_{z\max }} = 4 $$ . These constraints are judged and processed when the control input vector $$ {\textbf{u}} $$ is input to the UAV motion model and the UAV state, including $$ v $$ , $$ \theta $$ and $$ \psi $$ , is updated.

By giving the control parameters' values and UAV's state at the time step $$ t $$ , the UAV's state at the time step $$ t+1 $$ can be easily obtained by the Runge-Kutta method^[14].

2.2. Air combat environment

In the one-to-one short-range air combat environment, there are two UAVs, divided into red and blue. The red aims to gain the advantage situation over the blue until the blue side is destroyed by its weapon, and the blue aims to do the opposite^[4]. In this paper, the red UAV is controlled by the proposed decision method based on DRL algorithm. The relationship between the red and blue sides during the battle, which is shown in Figure 2, is mainly described by both sides' velocity vectors, the red's velocity $$ {{\textbf{v}}_r} $$ and the blue's velocity $$ {{\textbf{v}}_b} $$ , and the relative position vector D, which can be expressed as

Figure 2. The relationship between the red and blue sides during the battle.

where $$ {{\textbf{p}}_r} $$ and $$ {{\textbf{p}}_b} $$ are the red and blue's position vectors respectively.

The angle $$ {\varphi _r} $$ , named the attacking angle, is between $$ {{\textbf{v}}_r} $$ and $$ {\textbf{D}} $$ , and the formula can be expressed as^[16]

During the confrontation, the red has a chance to attack and deal damage to the blue only if the blue is in its attacking area. $$ {\varphi _r} $$ can be used to describe whether the blue in the red's attacking area and the conditions can be described as

where $$ {D_{att, \min }} $$ and $$ {D_{att, \max }} $$ are the minimum and maximum attacking distances, and $$ {\varphi _{att, \max }} $$ is the maximum attacking angle that has a chance to deal damage.

The angle $$ {\varphi _b} $$ , named escaping angle, is between $$ {{\textbf{v}}_b} $$ and $$ {\textbf{D}} $$ , and the formula is expressed as^[16]

During the confrontation, the red not only needs to keep the blue in its attacking area but also tries to avoid being attacked by the blue. Thus, the advantage area for the red is defined behind the blue and can be described as

where $$ {D_{adv, \min }} $$ and $$ {D_{adv, \max }} $$ are the minimum and maximum advantage distances, and $$ {\varphi _{esp, \max }} $$ is the maximum escaping angle that the red is in its advantage area.

During air combat, the UAV has limited attacking resource and needs attack only under certain conditions. The enemy should be both within the red's attacking area and hard to escape. These conditions can be described as^[15]

where $$ {\varphi _{esp, att}} $$ is the maximum escape angle for the enemy if the red wants to make a successful attack. On the other hand, not every attack could cause damage to the other side, and it will probably take more than an attack to destroy the other side. Thus, it is assumed that the UAVs have blood value $$ {B_0} $$ when initialized, and every attack has the probability $$ {p_{att}} $$ to cause some damage $$ \Delta B $$ , which is shown as

where $$ p \in \left[ {0, 1} \right] $$ is a random number, $$ {B_1} $$ , $$ {B_2} $$ , $$ {B_3} $$ are the reduced blood values after an attack, and $$ {p_{att1}} $$ and $$ {p_{att2}} $$ are the threshold of probability.

Note that Figure 1 is shown from the red perspective, and the relative relationship between the two sides can also be defined from the blue perspective in the same way. In the view of the blue, distance vector $$ {\textbf{D '}} $$ , attacking angle $$ {\varphi '_b} $$ , and escaping angle $$ {\varphi '_r} $$ are defined like Equations (4-8), and the attacking conditions are defined as

2.3. Air combat process

For an episode of one-to-one short-range air combat, the red and the blue are initiated, then confront in the combat environment until the conditions for the end of the episode are satisfied^[17]. An episode ends when the red or the blue is damaged or the maximum decision step $$ {t_{\max }} $$ is reached. If the UAV is out of the range of height or its blood value $$ B $$ is reduced to zero, this UAV will be judged as damaged. There are three types of results for the red, win, loss and draw, at the end of an episode. The red wins over the blue only when the blue is damaged before reaching the $$ {t_{\max }} $$ . Similarly, the red is lost when it is damaged before the blue within $$ {t_{\max }} $$ time steps. The draw result means that both sides are still undamaged when the maximum decision step $$ {t_{\max }} $$ is reached. The duration for each step is $$ {t_{step}} $$ . At every step in an episode, the red and the blue will make a maneuver decision by its action policy respectively to get the $$ {{\textbf{u}}_r} $$ and $$ {{\textbf{u}}_b} $$ . During a step, the UAVs will continually execute $$ {{\textbf{u}}_r} $$ and $$ {{\textbf{u}}_b} $$ until another maneuver decision-making starts. The flag $$ done $$ shows when the episode ends. In this paper, the action policy of the red is actor network which is updated by the PPO algorithm, and the blue's action policy is described in section 3.6. The air combat framework is shown in Figure 3 and the process in an episode is shown in Algorithm 1.

Figure 3. Short-range air combat framework.

3.1. PPO algorithm and improvement

The PPO algorithm is a type of DRL algorithm that has been used in many types of problems. In this part, the basic PPO algorithm and its usage in short-range air combat are introduced.

3.1.2. Improved points

To improve the training effect, some improvement points are adopted in this paper, and the framework of PPO algorithm for short-range air combat is shown in Figure 4.

Figure 4. The PPO algorithm framework for short-range air combat.

The first improvement point is considering the historical combat data when making decision. During the confrontation, the red must gradually accumulate the situational advantages over the blue and finally beat it. Therefore, decision on current action should take into account the previous air combat situations. The link between the action decision and historical air combat experience is established by adding the gate recurrent unit (GRU)^[22] to the neural network. The GRU is a type of recurrent neural network (RNN), which can adaptively capture the time's dependencies in different scales^[22], similar to long short-term memory (LSTM). However, the GRU is easier to train and more efficient than LSTM. The GRU layer, the hidden layer using GRU, is used in both actor network and critic network. The networks' inputs, observation and state, are firstly processed by the fully connected network to extract the inputs' features. Then the features are fused with historical features by the GRU layer to obtain the integrated features considering the historical situational features. The specific process is shown in Figure 4.

The second improvement point is to differentiate the neural network inputs. For the basic PPO algorithm, the critic network uses the same input as the actor network, which is named state space^[20]. However, the actor and critic networks play different roles in the algorithm. The actor network's state space is in the view of the red because the actor network's input is the red's observation of the air combat environment. On the other hand, the critic network is used to evaluate the output of the actor network by its output, the critic value, based on the current air combat situation^[23]. Thus, the input of critic network can be more objective and include some information that cannot be observed by the red. In this vein, the observation $$ o $$ from the observation generator and the state $$ s $$ from the state generator are designed for actor network and critic network respectively, as shown in Figure 4.

The third improvement point is a variable-sized experience buffer. For each time step, the experiences are stored in the experience buffer in order. During the training, it is divided into thousands of episodes, and each episode is over if it satisfies the ending condition. Thus, the length of each episode may be different. To make the training more general, the network parameters are updated until enough experiences are stored. However, the return calculation should be calculated on the basis of the complete experiences. Therefore, when the networks are updated, the numbers of experiences, which are larger than the number of the minimum experience for training $$ N $$ , are different to make sure that the same episode's experiences are stored in the buffer.

The fourth improvement point is the phased training process, which is discussed in section 3.6.

3.2. State space design

The state space of air combat should contain the information of the red and the blue UAVs, and a suitable designed state space can speed up the convergence of training. In this part, two state spaces designed for actor network and critic network are introduced.

3.3. Action space design

In the air combat problem, the UAV's actions are always summed up as a maneuver library, which consists of a series of tactical actions, such as high yo-yo, cobra maneuvering and so on^[3]. Pilots can choose from the library according to the combat situation. However, the establishment of the library is difficult and complex, and these tactical actions can be disassembled into basic actions. Thus, fifteen basic actions $$ {a_1}, {a_2}, \cdots , {a_{15}} $$ are adopted to form the action space $$ A $$ , which includes five types of directions, forward, upward, downward, left turn and right turn, and three types of speed control, maintenance, acceleration and deceleration^[25]. Every basic action in the designed library is a set of values of control parameters, $$ {\left[ {{n_x}, {n_z}, \gamma } \right]^T} $$ , for the UAV motion model. The designed maneuver library is shown in Table 1. Therefore, the action space is discrete and its dimension is 15.

3.4. Reward function design

The aim of RL is to maximize the cumulative reward obtained from the environment. Therefore, the reward function is the bridge to communicate the training result requirements to the DRL algorithm and its design is extremely important^[25]. In this paper, the reward function is well-designed and divided into three parts: dense reward, event reward and end-game reward. Different types of rewards are triggered under different conditions to transmit different expectations. In this part, the reward function designed for short-range air combat is introduced.

3.5. Action policy for the Blue

During the training, a policy is adopted as the blue's action policy, which consists of prediction and decision. In the prediction step, the blue will predict which action the red will do at the next time step and then estimate the red's position and velocity at the next time step based on the predicted action. At the decision step, the blue will find which action it should take to confront the red. To find which action is better, the thread function $$ T $$ is defined^[27], which consists of angle thread $$ {T_\varphi } $$ , velocity thread $$ {T_v} $$ , distance thread $$ {T_d} $$ and height thread $$ {T_h} $$ . Hence, $$ T $$ is calculated by

Noting that the definitions of $$ {T_\varphi } $$ , $$ {T_v} $$ , $$ {T_d} $$ and $$ {T_h} $$ are the same as in ref.^[27]. The blue's action policy is described as Algorithm 3.

3.6. Training process

In the confrontation training, the red and the blue confront each other for thousands of episodes in the air combat environment. Every episode works as Algorithm 1 and for every step the blue makes the maneuver decision as Algorithm 3. To train the red's action policy with the DRL algorithm, the experience for every step is stored. When satisfying the training conditions, the experiences will be used to update the red's action policy. To make sure the training is successful and to obtain satisfactory results, the training is divided into three phases, basic, dominant and balanced^[3]. The initial states for these phases are shown in Table 2.

Table 2

The initial states of the UAVs in three training phases

	Camp	$$ \textbf{x (m}) $$	$$ \textbf{y (m)} $$	$$ \textbf{z (m)} $$	$$ \textbf{v (m/s)} $$	$$ \theta ( \deg ) $$	$$ \psi \left( {\deg } \right) $$
Basic	Red	$$ \left[ {200, 1800} \right] $$	$$ \left[ {1500, 3500} \right] $$	$$ \left[ {700, 2500} \right] $$	60	0	$$ \left[ { - 0.5\pi , 0.5\pi } \right] $$
	Blue	$$ \left[ {1500, 2200} \right] $$	$$ \left[ {1500, 3500} \right] $$	$$ \left[ {700, 2500} \right] $$	60	0	$$ \left[ { - 0.3\pi , 0.3\pi } \right] $$
Dominant	Red	$$ \left[ {200, 1800} \right] $$	$$ \left[ {1500, 3500} \right] $$	$$ \left[ {700, 2500} \right] $$	60	0	$$ \left[ { - 0.5\pi , 0.5\pi } \right] $$
	Blue	$$ \left[ {1500, 2200} \right] $$	$$ \left[ {1500, 3500} \right] $$	$$ \left[ {700, 2500} \right] $$	60	0	$$ \left[ { - 0.3\pi , 0.3\pi } \right] $$
Balanced	Red	$$ \left[ {800, 4800} \right] $$	$$ \left[ {0, 4500} \right] $$	$$ \left[ {800, 3500} \right] $$	60	0	$$ \left[ { 0 , 2\pi } \right] $$
	Blue	$$ \left[ {800, 4800} \right] $$	$$ \left[ {0, 4500} \right] $$	$$ \left[ {800, 3500} \right] $$	60	0	$$ \left[ { 0, 2\pi } \right] $$

The three phases constitute a progressive relationship, which means the later training is based on the training results of the former training. The red's actor network and critic network are loaded with the former trained networks' parameters before starting the training.

4.1. Parameters setting

The hyperparameters setting for the DRL algorithm is shown in Table 3^[18].

The parameters of the designed air combat environment are set as follows^[15]. For the attacking distance, it is set as $$ {D_{att, \min }} = 40m $$ and $$ {D_{att, \max }} = 900m $$ . The maximum attacking angle $$ {\varphi _{att, \max }} = {\pi \mathord{\left/ {\vphantom {\pi 6}} \right. } 6} $$ and the maximum escape angle when attacking $$ {\varphi _{esp, att}} = {\pi \mathord{\left/ {\vphantom {\pi 3}} \right. } 3} $$ . For the advantage area, it is set as $$ {D_{adv, \min }} = 40m $$ , $$ {D_{adv, \max }} = 1300m $$ and $$ {\varphi _{esp, \max }} = {\pi \mathord{\left/ {\vphantom {\pi 3}} \right. } 3} $$ . For the blood, the probabilities are set as $$ {p_{att1}} = 0.1 $$ , $$ {p_{att2}} = 0.4 $$ and $$ {p_{att}} = 0.8 $$ , and the damage values are set as $$ {B_0} = 300 $$ , $$ {B_1} = 51 $$ , $$ {B_2} = 21 $$ and $$ {B_3} = 11 $$ . For an episode, the maximum decision step $$ {t_{\max }} = 400s $$ and the time step $$ {t_{step}} = 0.5s $$ . The threshold values are set as $$ {D_{th}} = {H_{th}} = 5000m $$ . For the reward function, the weights are set as $$ {r_a} = 0.15 $$ , $$ {r_d} = 0.6 $$ , $$ {r_v} = 0.1 $$ , $$ {r_h} = 0.15 $$ , $$ {w_{dense}} = 0.05 $$ and $$ {w_{adv}} = 0.05 $$ , the parameters about height are set as $$ {H_{\max }} = 500m $$ , $$ {H_{adv}} = 300m $$ , $$ {H_{att}} = 100m $$ and $$ {H_{\min }} = - 300m $$ , and the parameters in end-game reward are set as $$ {r_0} = 5 $$ , $$ {r_1} = 3 $$ , $$ {r_2} = 6 $$ and $$ {r_{loss}} = 15 $$ .

4.2. Training results in the phases

The four cases are trained with the hyperparameters in Table 3 and initial states in Table 2. Four cases are compared in this paper, which are case I PPO, case II PPO with GRU, case III PPO with GRU and state input and case IV PPO with state input. The state input means the inputs of actor and critic networks are different, and if there is no GRU, the GRU layer in Figure 4 is replaced with linear layer with 128 units and ReLU.

In the basic phase, the UAVs' states are shown in the first line of Table 2. The blue always performs the forward and maintain action $$ {a_0} $$ and the red is initialized behind the blue. Four cases are trained with 30000 episodes respectively. In the dominant phase, the UAVs' states are shown in the second line of Table 2. The blue uses the policy as Algorithm 2 and red is also initialized behind the blue. Four cases are trained with 40000 episodes respectively, and every case trains based on their own training result in the basic phase. In the balanced phase, the UAVs' states are shown in the third line of Table 2, and the initial states of the red and the blue are the same. Four cases are trained with 60000 episodes respectively, and every case trains based on their own training result in the dominant phase. The returns of every episode in the three phases are shown in Figure 5.

Figure 5. The episode returns of every episode in the three phases while training. A: the episode returns in the basic phase; B: the episode returns in the dominant phase; C: the episode returns in the balanced phase.

It can be seen that the PPO algorithm can go to converge faster when it is combined with GRU. In the basic phase, a relatively simple scenario, all four cases can easily find a better action policy to enclose the blue and beat it. And in the more complex scenario, it can find a better policy faster with GRU. But there are also larger episode reward variations in its infancy. Therefore, the state input is introduced to reduce the episode reward variations, which will synchronize to slightly reduce the final episode reward.

In addition, to test the final training result for the four cases, the training results in the balanced phase are reloaded and the initial states of the UAVs are set as

And the maneuvering trajectories are shown in Figure 6. The steps for the four cases are 146, 90, 123 and 193. For Figure 6B, it is shown that the blue and the red collide. It is obvious that the red can beat the blue in a smaller space range when trained with PPO with GRU, and by adding state input, the red can be more flexible to avoid collision. But the cost is the increase in time steps, which explains the decrease of final episode reward. In case III, the red uses hover and altitude variations to lure the blue closer and gain an advantage situation, instead of pursuing the blue.

Figure 6. The maneuvering trajectories in the training test for four cases. A: the maneuvering trajectories for the case I PPO; B: maneuvering trajectories for case II PPO with GRU; C: maneuvering trajectories for case III PPO with GRU and state input; D: maneuvering trajectories for case IV PPO with state input.

4.3. Confrontation tests

To compare the final training result of four cases, the confrontation tests for the four cases are conducted in this part when all of the training is finish. To accelerate the confrontation tests, the $$ {B_0} $$ is set as 100 during the test, and the UAVs' initial states are the same as they are in the balanced phases. The tie air combat result is divided into two types, tie win if $$ {B_r} > {B_b} $$ and tie loss if $$ {B_r} \leqslant {B_b} $$ . All of the tests are conducted in the air combat environment for 100 episodes, and the results are shown in Table 4.

It can be seen that by combining PPO with GRU and state input, the UAV can get a more flexible and intelligent action policy even though the training process is the same. It is proved that training the action policy by the PPO with proposed improve points can help the UAV gain an advantage situation more quickly and greater operational capability in short-range air combat confrontation, and the action policy can be more intelligent to adapt to the blue's uncertain policy.

Authors' contributions

Made substantial contributions to the research, idea generation, software development, and conduct the DRL experiments. Wrote and edited the original draft: Zheng Z

Performed process guidance responsibility for the planning and execution of the study, as well as the evolution of overarching research aims, critical review, and material support. Review and revise the original draft: Duan H

Availability of data and materials

Not applicable.

Financial support and sponsorship

This work was partially supported by National Natural Science Foundation of China under grant #U20B2071, #91948204, #T2121003 and #U1913602.

Conflicts of interest

All authors declared that there are no conflicts of interest.

Ethical approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Copyright

© The Author(s) 2023.