Reinforcement learning-based attitude control for a quadrotor UAV system with performance constraints

2.1. Kinematics and dynamics of the UAV attitude system with actuator faults

The structural diagram of a quadrotor UAV is shown in Figure 1, which includes two coordinate frames: the inertial frame OXYZ and the body-fixed frame $$ O_BX_BY_BZ_B $$. The orientation of the body-fixed frame relative to the inertial frame is represented by modified Rodrigues parameters (MRPs)^[32]. Accordingly, the attitude kinematics and dynamics of the quadrotor UAV are described as

Figure 1. Structural diagram of a quadrotor UAV. UAV: Unmanned aerial vehicle.

where $$ \sigma=e $$tan$$ (\Phi/4) $$, Φ is the Euler angle, e is the Euler axis, and $$ \omega=[\omega_x, \omega_y, \omega_z]^T $$ denotes the angular velocity of the quadrotor UAV. $$ J\in\mathbb{R}^{3\times3} $$ is the inertia matrix, $$ \tau_a\in\mathbb{R}^3 $$ denotes the control torque, and the kinematic matrix $$ G(\sigma) $$ is given by

It should be noted that the MRP representation possesses a coordinate singularity at $$ \Phi = \pm360^\circ $$, where the term $$ \text{tan}(\Phi/4) $$ diverges. However, for the quadrotor UAV attitude control task considered in this study, the rotation angle Φ is assumed to remain within a restricted operating region $$ (|\Phi|<360^\circ) $$, which is well-suited for most flight maneuvers. Under this condition, the MRPs provide a more compact and efficient parameterization compared to quaternions or Euler angles, and the singularity issue can be effectively avoided through proper initial condition settings and performance constraints. Therefore, Equations (1) and (2) can be reformulated as

where

which satisfy the following properties.

Property 1^[32]. $$ M(\sigma) $$ is bounded as follows:

where $$ k_m $$ and $$ k_n $$ are positive constants.

Property 2^[33]. $$ \dot{M}(\sigma)-2C(\sigma, \dot{\sigma}) $$ is skew-symmetric. For the skew-symmetric matrix $$ \dot{M}(\sigma)-2C(\sigma, \dot{\sigma}) $$, it holds that $$ \zeta^T(\dot{M}(\sigma)-2C(\sigma, \dot{\sigma}))\zeta=0 $$, with $$ \zeta\in\mathbb{R}^{3} $$.

In this paper, the actuator fault model, which encompasses efficiency loss and bias-type faults, is expressed as^[34]

where $$ \tau_i(t) $$ is the designed control input, $$ \kappa_i $$ is an unknown variable in the range [0, 1) representing loss of control effectiveness. The positive loss matrix is defined as $$ \kappa_f=\text{diag}(1-\kappa_1, ..., 1-\kappa_n)\in\mathbb{R}^{n\times n} $$. $$ \tau_o(t)=[\tau_{o, 1}(t), \tau_{o, 2}(t), \tau_{o, 3}(t)]^T $$ is the unknown bias fault function. Here, $$ t_i $$ represents the failure time of each actuator.

By incorporating Equation (9) into Equation (4), the system dynamics under actuator faults can be rewritten as

where $$ \tau=[\tau_1, \tau_2, \tau_3]^T $$ denotes the control torque vector, with $$ \tau_1 $$, $$ \tau_2 $$, and $$ \tau_3 $$ representing the torques along the roll, pitch, and yaw axes, respectively.

In summary, Equation (9) indicates that the i-th actuator fails from time $$ t_i $$. Before the occurrence of actuator failure, i.e., for $$ t<t_{f, i} $$, if $$ \kappa_i=0 $$ and $$ \tau_{o, i}=0 $$, the actuator is considered to operate under fault-free conditions, and the control torque $$ \tau $$ can be fully applied. When $$ 0<\kappa_i<1 $$, the actuator is considered to experience partial loss of effectiveness (PLOE). If $$ \kappa_i=1 $$, the actuator is considered to experience total loss of effectiveness; however, this extreme case is not considered in the current study, and $$ \kappa_i $$ is assumed to be strictly less than 1.

Moreover, it should be noted that faults in practical systems are difficult to repair or eliminate. Therefore, the transition from a fault-free state to a faulty state is considered unidirectional. In addition, each actuator is assumed to experience at most one fault.

2.2. Prescribed performance

To enhance attitude tracking performance, a prescribed performance constraint is applied to the UAV system to confine the tracking error within a predefined envelope, thereby ensuring that both transient and steady-state performance requirements are satisfied.

Defintion 1^[29]. A continuous function $$ \eta_i(t):\mathbb{R}_+ \rightarrow \mathbb{R}_+ $$ is defined as the performance function, which satisfies the following conditions:

(1) $$ \eta_i(t) $$ is positive and monotonically decreasing;

(2) $$ \lim\limits_{t \to \infty}\eta_i(t)=\eta_{i\infty}, and \lim\limits_{t \to 0}\eta_i(t)=\eta_{i0} $$. The performance function is specified as

where $$ \eta_{i0} $$ is the initial error bound, $$ \eta_{i\infty} $$ is the steady-state error bound, and $$ l>0 $$ determines the convergence rate. Accordingly, the prescribed tracking performance is achieved provided that the tracking error $$ e_i $$ satisfies

where $$ \underline{\omega}_i $$ and $$ \overline{\omega}_i $$ are positive constants.

2.3. FLS

An FLS consists of four components: the fuzzifier, the fuzzy inference engine, the defuzzifier, and the knowledge base. The knowledge base comprises a set of IF-THEN fuzzy rules. The i-th rule is described as follows^[12]: $$ R_i: $$ If $$ s_1 $$ is $$ P_1^i $$ and $$ s_2 $$ is $$ P_2^i $$ and $$ \cdot\cdot\cdot $$ and $$ s_n $$ is $$ P_n^i $$, then y is $$ G_i, i=1, 2, ..., N_r $$. The FLS can be expressed as

where $$ s=[s_1, ..., s_n]^T\in\mathbb{R}^n $$ represents the input vector and $$ y\in \mathbb{R} $$ represents the system output. $$ N_r>1 $$ denotes the number of fuzzy rules. $$ P_j^i $$ and $$ G_i $$ represent the fuzzy sets. $$ \mu_{P_j^{i}}(s_j) $$ denotes the membership functions related to $$ s_j $$, defined as $$ \mu_{P_j^{i}}(s_j) = \exp[-(s_j-c_{ji})^2/(2a^2) $$].

The fuzzy basis function is defined as

Define $$ \Theta(s)=[\Theta_1(s), \Theta_2(s), ..., \Theta_{N_r}(s)]^T $$ and $$ W=[\overline{y}_1, \overline{y}_2, ..., \overline{y}_{N_r}]^T $$. Then, the following expression is obtained

Lemma 1^[12]. For any continuous function $$ h(s) $$ defined over a compact set $$ \Omega_{s^*} $$, there exists an FLS such that the following inequality holds

where $$ \epsilon $$ denotes the function reconstruction error, bounded by $$ |\epsilon| \leq \bar{\epsilon} $$, with $$ \bar{\epsilon} $$ being an unknown positive bound.

3.1. Critic design

For the quadrotor UAV system, let $$ \sigma_r $$ denote the desired attitude parameters derived from the reference Euler angles. The tracking error is defined as

The long-term tracking cost function is defined as

where T denotes the time constant, and $$ f(e_p(t)) $$ is an appropriately chosen function to satisfy the control requirements. Specifically, it is defined as $$ f(e_p)= $$ tanh$$ (e_p^TQe_p) $$, with $$ Q\in\mathbb{R}^{N\times N} $$ being a positive matrix. Therefore, $$ f(t)<b_r=1 $$.

Due to dependence on future system information, directly solving $$ F(t) $$ becomes intractable for complex UAV systems subject to prescribed performance constraints and actuator faults. To overcome this difficulty, a critic FLS is designed to approximate $$ F(t) $$, expressed as

where $$ {W}_c^{*}\in\mathbb{R}^{h_c} $$ denotes the ideal weight vector of the critic FLS. Here, $$ h_c $$ is the number of fuzzy basis functions, and $$ \epsilon_c $$ represents the reconstruction error. The input $$ s_c $$ is defined as $$ s_c= $$ tanh $$ (e_p) $$.

According to Equations (18) and (19), the continuous-time temporal difference error is defined as

where $$ \hat{F}(t)=\hat{W}_c^T\Theta_c(s_c) $$ with weight estimation error $$ \tilde{W}_c=\hat{W}_c-W_c^* $$, and $$ \hat{W}_c $$ denotes the actual weight of the critic FLS. The error objective function is defined as $$ E_c=\frac{1}{2}e_c^2 $$. Using the gradient descent method, we obtain

where $$ \rho_c $$ denotes the learning rate, and $$ \nabla\Theta $$ represents the gradient of $$ \Theta_c(s_c) $$ relative to $$ s_c $$.

Considering the symmetry of the updating law Equation (21) and that the boundary condition for $$ J(t) $$ is defined at $$ T\to\infty $$, it is preferable to update past approximations to ensure that future estimations remain unaffected. Furthermore, as the gradient information $$ \nabla\Theta_c $$ is difficult to obtain, a store of past temporal difference errors is utilized to perform a backward Euler approximation of $$ F(t) $$. Accordingly, we have

where $$ \Delta{t} $$ is a time interval constant. Substituting Equation (22) into Equation (20) yields

Then, we obtain

According to the gradient method, the critic updating law can be rewritten as

In addition, to enhance the robustness of the algorithm against system uncertainties and external disturbances, a 𝜉-correction term is incorporated. This term ensures the boundedness of the estimated parameters even in the absence of the strict persistent excitation (PE) condition^[35]. Therefore, the final updating law for the critic FLS is given by

3.2. Actor design

To address the prescribed performance constraint, an error transformation function is introduced to reformulate it into an unconstrained equivalent form. To this end, a smooth and strictly increasing function $$ S(Z) $$ is defined as

where Z is the transformed error. The error transformation function is defined as

Because $$ S(Z) $$ is smooth and strictly increasing, its inverse function is

where $$ H=\frac{e_p}{\eta} $$. The time derivative of Z is given by:

where $$ r=\frac{\partial S^{-1} }{\partial H} \frac{1}{\eta } $$, $$ v=Ae_p $$, and $$ A=\text{diag}(\frac{\dot{\eta}_1}{\eta_1}, \frac{\dot{\eta}_2}{\eta_2}, ..., \frac{\dot{\eta}_N}{\eta_N}) $$.

Accordingly, the Lyapunov function candidate $$ L_1 $$ is constructed as

The time derivative of $$ L_1 $$ is given by

According to Equation (17), we have $$ \dot{e}_p=\dot{\sigma}-\dot{\sigma}_r $$. Define a new error variable $$ e_q=\dot{\sigma}-\alpha $$, where $$ \alpha(t) $$ denotes a virtual control variable. Consequently, the time derivative of $$ \dot{L}_1 $$ can be expressed as

Thus, the virtual control variable $$ \alpha $$ is formulated as

where $$ K_1 = \text{diag}(k_1, \dots, k_N) > 0 $$, and $$ k_i $$ is selected such that $$ k_i > \frac{\dot{\eta}_i}{\eta_i} $$. According to Equation (30), we have

where $$ K_1-A>0 $$. Subsequently, the Lyapunov function candidate $$ L_2 $$ is given by

Taking the time derivative of $$ L_2 $$, we obtain

To ensure the negative definiteness of $$ \dot{L}_2 $$, the control input is designed as

Due to model uncertainties, the exact values of $$ M(\sigma) $$ and $$ C(\sigma, \dot{\sigma}) $$ in Equation (4) are generally unavailable. Moreover, the bias fault $$ \tau_o $$ and the actuator efficiency degradation factor $$ \kappa $$ in Equation (9) are unknown in practice. Consequently, the term $$ {\kappa_f}^{-1} $$ in Equation (38) cannot be implemented directly. To address this problem, an adaptive parameter $$ \hat{\varrho} $$ is introduced to estimate the unknown inverse efficiency gain $$ \varrho^*={\kappa_f}^{-1}\in \mathbb{R}^{3\times3} $$. Define the approximation error as $$ \tilde{\varrho}=\hat{\varrho}-\varrho^* $$ and let $$ \tau_c = -Z-K_2e_q+C\alpha+M\dot{\alpha}-\tau_{o} $$ denote the nominal control signal. Based on the above settings, the control input can be redesigned as

To analyze the closed-loop stability and derive the adaptive mechanisms, the following Lyapunov function candidate $$ L_3 $$ is designed as

where $$ \Gamma = \text{diag}(\gamma_1, \gamma_2, \gamma_3) \in \mathbb{R}^{3 \times 3} $$ is a positive definite learning rate matrix. Since the fault parameter matrix captures the dynamics of three independent attitude channels, the trace form is employed to aggregate the estimation errors of all dimensions into a scalar energy function.

In order to handle the relationship between the vector inner product and the matrix trace during the derivation, the following lemma is introduced:

Lemma 2. For any two vectors $$ x, y \in \mathbb{R}^n $$ and a diagonal matrix $$ D \in \mathbb{R}^{n \times n} $$, the following relationship holds

Taking the time derivative of $$ L_3 $$, we obtain

Applying Lemma 2 with $$ x=e_q $$, $$ D=\kappa_f\tilde{\varrho} $$, and $$ y=\tau_c $$, the scalar term can be rewritten as

Thus, $$ \dot{L}_3 $$ can be expressed as

To eliminate the indefinite terms associated with $$ \tilde{\varrho} $$ and ensure $$ \dot{L}_3 \leq 0 $$, the bracketed term inside the trace is required to be zero. Accordingly, the adaptive law $$ \hat{\varrho} \in \mathbb{R}^{3 \times 3} $$ is designed as

Meanwhile, the actor component is augmented with an FLS, which not only approximates the uncertain nonlinear terms in the system dynamics but also accounts for bias faults. This design enables implicit and adaptive fault compensation. The ideal approximation performance of the actor FLS is expressed as

where $$ W_a^{*T}\in \mathbb{R}^{h_a\times N} $$ denotes the ideal weight matrix. Define the approximation error as $$ \tilde{W}_a=\hat{W}_a-W_a^* $$. Here, $$ h_a $$ denotes the number of fuzzy basis functions in the actor component. $$ \epsilon_a\in\mathbb{R}^N $$ is the function reconstruction error, and $$ s_a=[\omega, \sigma, \dot{\sigma}, \sigma_r, \dot{\sigma}_r]^T $$. Therefore, the nominal control signal is expressed as

where $$ \hat{W}_a $$ denotes the actual weight of the actor FLS. The adaptive law for the efficiency estimation matrix $$ \hat{\varrho} $$ is rewritten as

where $$ \varepsilon $$ is a positive design parameter introduced to ensure the boundedness of the adaptive law. According to Equations (39) and (47), the control input can be redesigned as

With the adaptive mechanism Equation (48) compensating for the actuator efficiency degradation, the remaining fault tolerance capability relies on the actor FLS. By treating the unknown bias fault $$ \tau_{o} $$ and system uncertainties as a unified approximation target, the actor FLS implicitly compensates for these effects without requiring explicit fault diagnosis.

To realize this implicit compensation while simultaneously optimizing system performance, the objective of the actor FLS is to drive the estimated cost function $$ \hat{F}(t) $$ toward the desired value $$ F_d=0 $$. Accordingly, the actor error is defined as

where $$ K_V=[K_{V1}, ..., K_{VN}]^T\in \mathbb{R}^N $$ with $$ K_{Vi}>0 $$. Define the error function $$ E_a=\frac{1}{2}e_a^Te_a $$. By using the gradient descent method, the updating law of $$ \dot{\hat{W}}_a $$ is formulated as

where $$ \rho_a $$ represents a positive learning rate.

Since $$ \tilde{W}_a $$ is unknown, the updating law in Equation (51) cannot be directly implemented. To this end, a 𝜉-modification term is employed to reformulate the updating law for $$ \hat{W}_a $$ as

where $$ \xi_a $$ is a positive constant. The 𝜉-modification term enhances the robustness of the UAV system and ensures the boundedness of $$ \hat{W}_a $$ without requiring the PE condition. Therefore, the proposed RL-based fuzzy adaptive fault-tolerant attitude control algorithm is detailed in Algorithm 1.

Then, a theorem is obtained as follows.

Theorem 1 For a quadrotor UAV system subject to performance constraints and actuator faults, the proposed RL-based fuzzy adaptive fault-tolerant attitude control strategy ensures stability and tracking performance. The error signals $$ e_p $$, $$ e_q $$, $$ \tilde{\varrho} $$, $$ \tilde{W}_c $$, and $$ \tilde{W}_a $$ are semiglobally uniformly ultimately bounded, and all error signals remain within their respective compact sets. The compact sets $$ \Psi_1 $$, $$ \Psi_2 $$, $$ \Psi_3 $$, $$ \Psi_4 $$, and $$ \Psi_5 $$ are defined as follows:

Remark 1 It is important to emphasize the forward invariance of the prescribed performance corridor. According to the Lyapunov analysis in Theorem 1, the candidate function $$ L(t) $$ is bounded for all $$ t \ge 0 $$, satisfying $$ L(t) \le L(0) + \mathcal{G}/\mathcal{P} $$, as derived in Equation (69). Since $$ L_1 = \frac{1}{2}Z^T r^{-1} Z $$ is a constitutive part of $$ L(t) $$, the transformed error vector $$ Z(t) $$ remains semiglobally uniformly ultimately bounded (i.e., $$ Z \in \mathcal{L}_\infty $$). Recalling the error transformation $$ Z = S^{-1}(H) $$ defined in Equation (29), it is clear that $$ |Z| \to \infty $$ only if the normalized error $$ H(t) $$ approaches the boundaries $$ \pm 1 $$. Consequently, the guaranteed boundedness of $$ Z(t) $$ strictly implies that $$ -1 < H(t) < 1 $$ for all $$ t > 0 $$, or equivalently, $$ -\underline{\omega}_i \eta_i(t) < e_{pi}(t) < \bar{\omega}_i \eta_i(t) $$. This theoretical result ensures that the tracking error $$ e_p(t) $$ is strictly confined within the predefined performance boundaries and cannot reach the singularity points, provided the initial condition $$ e_p(0) $$ lies within the prescribed corridor.

Proof: For a UAV system subject to performance constraints, the following Lyapunov function candidate is constructed

Based on Equations (37), (48) and (49), we obtain

To facilitate subsequent analysis, we introduce

Based on the updating law Equation (52), we have

where $$ b_a, b_c $$ and $$ b_K $$ are constants that meet the following conditions: $$ \|\Theta_a\|\leq b_a, \|\Theta_c\|\leq b_c, \|K_V\|=b_K $$. Furthermore, $$ \mathcal{G}_a $$ is given by

The parameter $$ \xi_a $$ should satisfy the condition $$ \xi_a-3b_a^2 > 0 $$.

Define $$ \Delta \Theta_c(t) = \left( 1 - \frac{\Delta t}{T} \right) \Theta_c(Z_c(t)) - \Theta_c(Z_c(t - \Delta t)) $$ and $$ \mu = 1/\Delta t. $$ Furthermore, $$ \Delta \Theta_c(t) $$ satisfies $$ \|\Delta \Theta_c(t)\| \leq ( 2 - \frac{\Delta t}{T} ) b_c = b_\Delta $$.

According to Equation (26), the time derivative of $$ L_c $$ is given by

According to Equations (62) and (64), we can conclude that

where

with $$ r_a=\xi_a-3b_a^2 $$ and $$ r_c=\xi_c+\mu b_\Delta^2-b_K^2 b_c^2-1 $$. Meanwhile, the corresponding parameters $$ K_1, K_2, \xi_a $$, and $$ \xi_c $$ should be chosen appropriately to satisfy $$ \lambda_{\min}(K_1-A)>0, \lambda_{\min}(K_2-I)>0, r_a>0, r_c>0 $$.

Multiplying both sides of Equation (65) by $$ e^{\mathcal{P} t} $$ yields

By integrating both sides of Equation (68), we obtain

According to Equations (58) and (69), there is

Therefore, the tracking error is ultimately bounded as

where $$ \nu=L(0) + \mathcal{G}/\mathcal{P} $$. In addition, we obtain

Authors' contributions

Conceived the research concept, designed the methodology, established the simulation platform, drafted the original manuscript, prepared the visualizations, and obtained funding support: Ouyang, Y.

Participated in data analysis, performed the main simulations, conducted simulation validation, contributed to drafting the original manuscript, and revised the manuscript: Ma, C.

Conducted data curation, assisted in result analysis, performed data visualization, and polished the manuscript: Wang, Y.

Provided technical support, assisted in establishing the simulation platform, and revised the manuscript: Su, Y.

Supervised the entire project, managed project administration, thoroughly reviewed and edited the manuscript, and validated all results: He, X.

Availability of data and materials

The data supporting the findings of this study are available from the corresponding author upon reasonable request.

AI and AI-assisted tools statement

Not applicable.

Financial support and sponsorship

This work was supported by the National Natural Science Foundation of China (Grant Nos. 62303008, 62303012, 62495083, and 62236002).

Conflicts of interest

All authors declared that there are no conflicts of interest.

Ethical approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Copyright

© The Author(s) 2026.