Reinforcement learning-based optimal adaptive fuzzy control for nonlinear multi-agent systems with prescribed performance

1. INTRODUCTION

Optimal control is achieved by designing a control protocol that not only achieves the system control objectives but also minimizes the system cost. The field of optimal control has garnered significant scholarly interest in recent years. The optimal controller can be deduced from the solution of the Hamilton-Jacobi-Bellman (HJB) equation ^[1]. For linear systems, this is actually solving the Riccati equation. However, for nonlinear systems, the HJB equation is a partial differential equation containing multiple nonlinear terms, and it is conceivably challenging to solve the equation directly. One approach that can be implemented is dynamic programming (DP) ^[2–5]. However, this approach becomes less feasible for high-dimensional systems since it is a backward, offline computational process, which significantly increases the computational complexity in high-dimensional scenarios. As a form of machine learning, reinforcement learning (RL) arguably opens up another avenue to solve the problem ^[6–9]. The most commonly used RL algorithms make use of the actor-critic structure, in which the actor interacts with the environment, and the critic evaluates the actions of the actor and provides feedback; in this way, the actor performs the next task again. Subsequently, RL has been employed in various nonlinear systems for adaptive control, leading to remarkable outcomes^[10–14]. For example, an optimal adaptive controller for nonlinear systems with control gain functions was proposed to achieve not only tracking control but also the optimal performance of systems^[10]. In^[12], the problem of tracking control of nonlinear systems with input constraints was investigated. In^[14], an optimal observer-based adaptive control scheme for nonlinear stochastic systems with input and state constraints was proposed.

In the meanwhile, since most physical models in practical applications can be represented by nonlinear systems, the study of nonlinear systems is very important and has yielded rich results^[15–18]. In recent years, multi-agent systems (MASs) have garnered significant attention from scholars due to their capability to perform tasks that surpass the capabilities of a single agent. The consensus problem in MASs refers to achieving a state of agreement or convergence among multiple agents through the design of control protocols, which is a fundamental problem in the design and control of MASs. Over the past decades, the problem of consensus control for MASs has been extensively studied, leading to significant advancements^[19–30]. In^[23], a consensus control scheme incorporating a modified disturbance observer was designed to achieve fixed-time tracking control of nonlinear MASs with unknown disturbances. In^[27], an event-triggered control distributed scheme was proposed to address the problem of asymptotic tracking for nonlinear MASs with uncertain leaders. In recent times, there has been a surge of interest in incorporating RL into MASs. It is an interesting and challenging problem and has produced some excellent results^[31–38]. For instance, in^[33], an optimal backstepping consensus control protocol based on RL was introduced for nonlinear strict-feedback MASs, which not only exhibits algorithmic simplicity but also relaxes the need for two general conditions: known dynamics and persistence excitation. In^[34], an optimal RL-based event-triggered controller was proposed for nonlinear stochastic systems.

On the other hand, the concept of prescribed performance control (PPC) has emerged as a prominent research topic in the control community, initially proposed by Bechlioulis and Rovithakis^[39]. Transient and steady-state performance, which is often neglected by conventional control schemes that solely ensure closed-loop stability, is the primary concern of prescribed performance. The PPC strategy aims to align the actual system performance achieved after execution with the desired or prescribed performance criteria and has yielded remarkable outcomes^[40–43]. By utilizing exponential performance functions, both the nonlinear switched systems in^[40] and the non-triangularly structured systems in ^[42] were able to achieve the desired rate of convergence. To overcome the issue of "differential explosion" caused by repeated derivation, dynamic surface control schemes were proposed to implement the tracking control of systems in ^[41] and ^[43], respectively. Recently, there has been significant research focused on integrating RL into PPC^{[44, 45]}. However, it is noticed that all of the above results depend on the initial conditions; i.e., at the initial moment, the initial error needs to be made within a prescribed boundary by properly setting the initial values.

Motivated by the discussions above, this article focuses on the optimal adaptive consensus tracking control problem for leader-follower nonlinear MASs subject to prescribed performance constraints. The main contributions of the article are as follows.

(1) Based on the actor-critic structure of RL, the proposed consensus tracking control scheme can achieve optimal control of MASs while an excellent tracking effect is guaranteed. Compared with ^[10–12], the proposed algorithm is simpler to implement since it does not require system dynamic and persistent excitation conditions.

(2) In contrast to existing performance functions ^[40–43], most of which rely on initial value conditions, an improved performance function is introduced such that the proposed consensus tracking control scheme is able to force the convergence of the consensus tracking error to a prescribed region without the requirement of initial value conditions.

(3) Compared with the traditional backstepping control scheme ^{[24, 25]}, the dynamic surface technique is adopted, which effectively avoids the problem of "differential explosion" caused by multiple derivations of the virtual controller and makes the control structure simpler.

2. PRELIMINARIES AND PROBLEM FORMULATION

3. DESIGN PROCEDURE AND MAIN RESULTS

The controller design is conducted based on the following coordinate transformations

where $$ \hat{\alpha}_{i, j-1} $$ denotes the approximate optimal virtual controller, $$ {\bar\alpha}_{i, j} $$ denotes the output of the first-order filter, and $$ \rho_{i, j} $$ is defined as the filtering error. The filtering dynamic is introduced as

where $$ \kappa_{i, j-1} $$ is a positive constant.

To facilitate brevity, the following definitions are provided before the commencement of the design steps. For $$ i=1, \cdots, N, j=1, \cdots, n $$, $$ \tilde\theta_{a, ij}=\theta_{a, ij}-\hat \theta_{a, ij}$$, $$\tilde\theta_{c, ij}=\theta_{c, ij}-\hat \theta_{c, ij}$$, $$\tilde\vartheta_{i, j}=\vartheta_{i, j}-\hat \vartheta_{i, j}$$ where $$ \hat \theta_{a, ij} $$, $$ \hat \theta_{c, ij} $$, and $$ \hat \vartheta_{i, j} $$ are the estimations of $$ \theta_{a, ij}, \theta_{c, ij} $$, and $$ \vartheta_{i, j} $$, respectively. $$ \lambda^{\min}_{i, j} $$ denotes the minimum eigenvalue of $$ S_{i, j}S_{i, j}^T $$. The set $$ \varOmega $$ is the tight set that contains zero, and $$ \psi(\varOmega) $$ represents the admissible control. The adaptive laws of actor FLSs and critic FLSs, adaptive law $$ {\hat\vartheta}_{i, j} $$ are designed to be

where $$ \sigma_{a, ij} $$, $$ \sigma_{c, ij} $$ and $$ \sigma_{i, j} $$ are design parameters.

Step 1: Derivation with respect to $$ e_i $$ gives

Define the performance index function for the first subsystem of the agent $$ i $$ as

where $$ c_{i, 1}(\zeta_i, \alpha_{i, 1})=\zeta_i^2+\alpha_{i, 1}^2 $$ represents the value function.

The optimal performance index function is expressed as

Considering $$ x_{i, 2} $$ as the optimal virtual controller $$ \alpha^*_{i, 1} $$, then the HJB equation is given as

where $$ \eta_i=\dfrac{\chi}{\sqrt{(e_i+\chi)}(e_i+\chi)} $$, $$ v_i=\dfrac{1+\varpi_i^2}{(1-\varpi_i^2)^2} $$.

Define the Bellman residual as

The approximate optimal virtual controller is expected to guarantee that $$ \Theta $$ tends to zero. The following positive function is given by

Since equation $$ H_{i, 1}(\zeta_i, \alpha^*_{i, 1}, V_{i, 1}^*)=0 $$ has a unique solution, it is easy to deduce that $$ I_{i, 1}=0 $$ is equivalent to $$ \frac{\partial H_{i, 1}(\zeta_i, \alpha^*_{i, 1}, V_{i, 1}^*)}{\partial \hat\theta_{a, i1}}=0 $$. Then, from equations (12) and (13), we get

Thus, the designed adaptive laws $$ \dot {\hat\theta}_{a, i1} $$ and $$ \dot {\hat\theta}_{c, i1} $$ enable $$ H_{i, 1}(\zeta_i, \alpha^*_{i, 1}, V_{i, 1}^*)=0 $$ to be satisfied.

By calculating $$ \frac{\partial H_{i, 1}}{\partial\alpha^*_{i, 1}}=0 $$, it yields

To obtain the optimal virtual controller, we decompose $$ \frac{\partial V_{i, 1}^*}{\partial \zeta_i} $$ to derive the following equation

where $$ c_{i, 1} $$ is a design parameter.

Substituting equation (24) into equations (23), it follows that

Since $$ V_{i, 1}^o $$ is an unknown term, by applying Lemma 2, there exists an FLS such that

where $$ \theta^*_{i, 1} $$ refers to the optimal weights, $$ S_{i, 1} $$ refers to the basis function, and $$ \varepsilon_{i, 1} $$ is the approximation error.

Substituting equation (27) into equations (24) and (26) results in

The $$ \hat \theta_{a, i1} $$ and $$ \hat \theta_{c, i1} $$ of the actor FLS and the critic FLS are used to estimate the unknown weights $$ \theta^*_{i, 1} $$, respectively, to obtain

Construct the following Lyapunov candidate function

The derivation of $$ V_{i, 1} $$ yields

Define $$ F_{i, 1} $$ to be the

By Lemma 2, there exists an FLS approximation to $$ F_{i, 1} $$ that results in

where $$ \vartheta_{i, 1}^* $$ is the ideal weight, $$ \phi_{i, 1} $$ is the basis function, and $$ \epsilon_{i, 1} $$ is the approximation error satisfying $$ \epsilon_{i, 1}\leq \bar\epsilon_{i, 1} $$ with $$ \bar\epsilon_{i, 1}>0 $$.

With the help of Young's inequality, one has

Substituting equations (30), (36), (37), and (38) into equation (33) yields

According to Young's inequality, it can be derived that

By means of equations (40)-(44), $$ \dot V_{i, 1} $$ becomes

Then, it can be further bounded as

where $$ \Upsilon_{i, 1}=\frac{1}{2}\bar\epsilon_{i, 1}^2+\frac{\sigma_{i, 1}}{2}\vartheta_{i, 1}^2 +\frac{\sigma_{a, i1}}{2}\theta_{a, i1}^TS_{i, 1}S_{i, 1}^T \theta_{a, i1}+\frac{\sigma_{c, i1}}{2}\theta_{c, i1}^TS_{i, 1}S_{i, 1}^T \theta_{c, i1}-$$$$(\frac{\sigma_{a, i1}}{2}-\frac{1}{4})\hat\theta_{a, i1}^TS_{i, 1}S_{i, 1}^T\hat \theta_{a, i1} -\frac{\sigma_{c, i1}}{2}\hat\theta_{c, i1}^TS_{i, 1}S_{i, 1}^T\hat \theta_{c, i1} $$.

Step j$$ (2\leq j \leq n-1) $$: From equations (9) and (10), it holds that

The performance index function is chosen to be

where $$ c_{i, j}(z_i, \alpha_{i, j})=z_{i, j}^2+\alpha_{i, j}^2 $$ represents the value function.

Considering $$ x_{i, j+1} $$ as the optimal virtual controller $$ \alpha^*_{i, j} $$, then the optimal performance index function has the form

The HJB equation can be written as

By calculating $$ \frac{\partial H_{i, j}}{\partial\alpha^*_{i, j}}=0 $$, it yields

Then, decompose $$ \frac{\partial V_{i, j}^*}{\partial z_{i, j}} $$ into the following two parts

where $$ c_{i, j} $$ is a design parameter.

From equation (52), it is easy to introduce

Since $$ V_{i, j}^o $$ is an unknown term, by applying Lemma 2, there exists an FLS such that

where $$ \theta^*_{i, j} $$ refers to the optimal weights, $$ S_{i, j} $$ refers to the basis function, and $$ \varepsilon_{i, j} $$ is the approximation error.

According to the above equation, we can get

The $$ \hat \theta_{a, ij} $$ and $$ \hat \theta_{c, ij} $$ of the actor FLS and the critic FLS are used to estimate the unknown weights $$ \theta^*_{i, j} $$, respectively, to obtain

The candidate Lyapunov function function is chosen as

Derivation of $$ V_{i, j} $$ yields

From equations (10) and (11), it holds that

Define $$ M_i=-\dot{\hat\alpha}_{i, j-1} $$, which is a bounded continuous function.

By Lemma 2, there exists an FLS satisfying

where $$ F_{i, j}=f_{i, j}-\dot {\bar \alpha}_{i, j} $$, $$ \vartheta_{i, j}^* $$ is the ideal weight, $$ \phi_{i, j} $$ is the basis function, and $$ \epsilon_{i, j} $$ is the approximation error satisfying $$ \epsilon_{i, j}\leq \bar\epsilon_{i, j} $$ with $$ \bar\epsilon_{i, j}>0 $$.

With the help of Young's inequality, the following inequality holds

Similar to the first step, it has

From equations (58), (62), and (64)-(72), $$ \dot V_{i, j} $$ can be directly bounded as

where $$ \Upsilon_{i, j}=\frac{1}{2}\bar\epsilon_{i, j}^2+\frac{\sigma_{i, j}}{2}\vartheta_{i, j}^2 +\frac{\sigma_{a, ij}}{2}\theta_{a, ij}^TS_{i, j}S_{i, j}^T \theta_{a, ij}+\frac{\sigma_{c, ij}}{2}\theta_{c, ij}^TS_{i, j}S_{i, j}^T \theta_{c, ij}-(\frac{\sigma_{a, ij}}{2}-\frac{1}{4})\hat\theta_{a, ij}^TS_{i, j}S_{i, j}^T\hat \theta_{a, ij} $$$$-\frac{\sigma_{c, ij}}{2}\hat\theta_{c, ij}^TS_{i, j}S_{i, j}^T\hat \theta_{c, ij}+\frac{1}{2}M_{i}^2 $$ and $$ \varkappa_i= \begin{cases} \frac{(d_i+b_i)^2\eta_i^2}{2}, & j=1, \\ 1, & j=2, \cdots, n-1. \end{cases} $$Step n: The performance index function for the last subsystem is defined as

where $$ c_{i, n}(z_n, \alpha_{i, n})=z_{i, n}^2+u_{i}^{*2} $$ represents the value function.

The optimal performance index function is

The HJB equation is introduced as

By calculating $$ \frac{\partial H_{i, n}}{\partial u^*_{i}}=0 $$, we get

$$ \frac{\partial V_{i, n}^*}{\partial z_{i, n}} $$ can be decomposed into

where $$ c_{i, n} $$ is a design parameter. Since $$ V_{i, n}^o $$ is an unknown term, by applying Lemma 2, one gets

Then, the facts below are easily available

The $$ \hat \theta_{a, in} $$ and $$ \hat \theta_{c, in} $$ of the actor FLS and the critic FLS are used to estimate the unknown weights $$ \theta^*_{i, n} $$, respectively, to obtain

Candidate Lyapunov function is chosen to be the

The time derivative of $$ V_{i, n} $$ is given by

By Lemma 2, there exists an FLS satisfying

where $$ F_{i, n}=f_{i, n}-\dot {\bar \alpha}_{i, n} $$, and $$ \epsilon_{i, n}\leq \bar\epsilon_{i, n} $$ with $$ \bar\epsilon_{i, n}>0 $$.

Similar to the previous steps, $$ \dot V_{i, n} $$ can be bounded as

where $$ \Upsilon_{i, n}=\frac{1}{2}\bar\epsilon_{i, n}^2+\frac{\sigma_{i, n}}{2}$$$$\vartheta_{i, n}^2 +\frac{\sigma_{a, in}}{2}\theta_{a, in}^TS_{i, n}S_{i, n}^T $$$$\theta_{a, in}+\frac{\sigma_{c, in}}{2}\theta_{c, in}^TS_{i, n}S_{i, n}^T \theta_{c, in}-(\frac{\sigma_{a, in}}{2}-$$$$\frac{1}{4})\hat\theta_{a, in}^TS_{i, n}S_{i, n}^T\hat \theta_{a, in} -$$$$\frac{\sigma_{c, in}}{2}\hat\theta_{c, in}^TS_{i, n}S_{i, n}^T\hat \theta_{c, in}+\frac{1}{2}M_{i}^2 $$.

Define $$ E_i=\min\{c_{i, 1}-\frac{7}{4}, c_{i, 2}-\frac{7}{4}-\frac{(d_i+b_i)^2\eta_i^2}{2}, c_{i, m}-\frac{9}{4}, \frac{\sigma_{i, m}}{2}, (\sigma_{a, im}-$$$$\frac{\sigma_{c, im}}{2})\lambda^{\min}_{i, m}, \lambda^{\min}_{i, m}\frac{\sigma_{c, im}}{2}\tilde\theta_{c, im}^T\} $$, $$ \Upsilon_i=\sum_{m=1}^{n}\Upsilon_{i, m} $$, $$ \dot V_{i, n} $$ can then be written as

Theorem 1: Consider a nonlinear MAS under Assumptions 1-2, with the optimal virtual controller choice of equations (30), (58), and (83), the adaptive laws of equations (14) and (15), the actor FLS choice of equation (12), and the critic FLS choice of (13). Then, we select the parameter $$ \sigma_{a, im}>\frac{\sigma_{c, im}}{2}>\frac{\sigma_{a, im}}{2}>0$$, $$c_{i, 1}>\frac{7}{4}, c_{i, 2}>\frac{7}{4}+\frac{(d_i+b_i)^2\eta_i^2}{2}$$, $$c_{i, m}>\frac{9}{4}, \frac{1}{\kappa_i}>\frac{1}{2}+\frac{(d_i+b_i)^2\eta_i^2}{2}$$, $$\frac{1}{\kappa_i}>1$$. Thus, we can conclude the following results:

(1) All signals in the closed-loop system are bounded.

(2) Consensus tracking error is within predefined bounds.

Proof: The total Liapunov function for all agents is selected to be

The derivative of V with respect to time is

where $$ E=\min{E_i, i=1, 2, \cdots, N}, \Upsilon=\varSigma_{i=1}^N{\Upsilon_i} $$.

Obviously, it can be inferred that

From (91), we know that $$ \zeta_i $$ is bounded by

This means that the consensus tracking error can be bounded within prescribed bounds, i.e., $$ \Gamma(-\varphi(t))\leq e_i\leq \Gamma(\varphi(t)) $$. Then, we can easily confirm $$ ||y-\bar 1y_r||\leq\frac{||\Gamma(\varphi(t))||}{{\underline\sigma}(\mathscr L+\mathscr B)} $$, where $$ {\underline\sigma}(\mathscr L+\mathscr B) $$ represents the smallest singular value of the matrix $$ \mathscr L+\mathscr B $$.

On the other hand, it is known from equation (92) that $$ \tilde\theta_{a, ij}, \tilde\theta_{c, ij}^T, \tilde\vartheta_{i, j}, \rho_{i, j} $$, and $$ z_{i, j} $$ are bounded. Since $$ \hat \alpha_{i, j} $$ is a function consisting of bounded signals, $$ \hat \alpha_{i, j} $$ is bounded. Based on the definition of $$ \tilde\theta_{a, ij}=\theta_{a, ij}-\hat \theta_{a, ij}, \tilde\theta_{c, ij}=\theta_{c, ij}-\hat \theta_{c, ij}, \tilde\vartheta_{i, j}=\vartheta_{i, j}-\hat \vartheta_{i, j} $$, it is evident that $$ \hat \theta_{a, ij}, \hat \theta_{c, ij} $$, and $$ \hat \vartheta_{i, j} $$ are bounded. Thus, it can be concluded that all signals are bounded.

4. SIMULATION EXAMPLES

Consider the nonlinear MASs with four following agents and a leader, whose dynamics model is represented as

where $$ i=1, 2, 3, 4, f_{i, 2}=0.01\sin(0.5(x_{i, 1}-x_{i, 2})) $$. The reference output trajectory is set to $$ y_r=\sin(0.5t) $$. The communication topology is displayed in Figure 1, through which the Laplace matrix $$ \mathscr L $$ is easily obtained as

Figure 1. The communication topology.

Select the fuzzy membership function as

The time-varying function $$ \omega_i(t) $$ is chosen as $$ \omega_i(t)=\dfrac{1}{(1-l_i)\exp(-1.5t)+l_i} $$. The initial state values are selected as $$ x_{i, 1}(0)=[0.11, 0.09, 0.13, 0.1] $$, $$ x_{i, 2}(0)=[0.11, 0.2, 0.18, 0.11] $$, $$ \hat \theta_{c, i1}(0)=[0.2, 0.2, 0.22, 0.12] $$, $$ \hat \theta_{c, i2}(0)=[0.2, 0.2, 0.12, 0.12] $$, $$ \hat \theta_{a, i1}(0)=[1.4, 1, 1, 1] $$, $$ \hat \theta_{a, i2}(0)=[1.2, 1.2, 1, 1] $$, $$ \hat \vartheta_{i, j}(0)=1 $$, $$ \bar \alpha_{i, j}(0)=0 $$. The design parameters are selected as $$ l_i=0.2 $$, $$ \chi=5 $$, $$ c_{i, j}=9 $$, $$ \kappa_{i, 2}=0.02 $$, $$ \sigma_{c, ij}=7 $$, $$ \sigma_{a, ij}=5 $$, $$ \sigma_{11}=\sigma_{12}=\sigma_{21}=\sigma_{22}=\sigma_{31}=21 $$, and $$ \sigma_{32}=\sigma_{41}=\sigma_{42}=11 $$. The resulting simulations are presented in Figures 2-6. Figure 2 depicts the output trajectories of follower agents and the reference trajectory, demonstrating the guaranteed well-tracking performance under the designed control protocol. Figure 3 shows the consensus tracking errors of agents and performance constraint bounds, from which it can be seen that the constraint has never been violated. Figure 4 provides the designed control protocol. Figure 5 depicts the trajectories of adaptive parameter $$ \theta_{c, ij} $$. Figure 6 portrays the trajectories of adaptive parameters $$ \theta_{a, ij} $$. The trajectories of adaptive parameter $$ \vartheta_{i, j} $$ is plotted in Figure 7. The above leads show that adaptive parameters $$ \theta_{c, ij} $$, $$ \theta_{a, ij} $$, and $$ \vartheta_{i, j} $$ are bounded. Based on the aforementioned results, it is evident that our control objectives have been successfully achieved.

Figure 2. The trajectories of the system output state $$ y_i(i=1, 2, 3, 4) $$ and the reference signal $$ y_r $$.

Figure 3. System consensus tracking error $$e_i(i=1, 2, 3, 4)$$ and constraint functions $$\Gamma(\varphi(t)), \Gamma(-\varphi(t))$$.

Figure 4. The trajectories of the control protocols $$u_i(i=1, 2, 3, 4)$$.

Figure 5. Responses of $$\theta_{c, ij}(i=1, 2, 3, 4;k=1, 2)$$.

Figure 6. Responses of $$\theta_{a, ij}(i=1, 2, 3, 4;k=1, 2)$$.

Figure 7. Responses of $$\vartheta_{i, j}(i=1, 2, 3, 4;k=1, 2)$$.

5. CONCLUSIONS

In this paper, the problem of optimal adaptive consensus tracking control for nonlinear MASs with prescribed performance has been addressed. Firstly, a time-varying scalar function is introduced such that the designed performance function bypasses the initial value conditions. Based on the error transformation function, an unconstrained system is obtained. Subsequently, a RL-based consensus control scheme based on optimal control theory and dynamic surface technique has been proposed. Finally, it is shown that the stability of the closed-loop system and the error constraints are not violated. In practice, the systems are always subject to various uncertain constraints, such as actuator faults and input dead zones, which will have a large impact on the performance of systems. Therefore, designing a properly performance-constrained optimal control scheme considering the above situations is a topic for further research in the future.

DECLARATIONS