Optimal consensus control of Markov jump multi-agent systems under switching topology

2.1. Notation

The symbol $$ \otimes $$ represents the Kronecker product. ℝⁿ denotes the set of n-dimensional real vectors. R^n×n is the set of n × n real matrices. I_N is the N-dimensional identity matrix. λ_min{·} and λ_max{·} represent the smallest and largest eigenvalues of ·, respectively.

2.2. Graph theory

Define a directed graph $$ \vartheta=(\mathcal{V},\mathcal{E},\mathcal{A}) $$ to represent the interactive relationships between a leader and N followers. $$ \mathcal{V} $$ = {v_j : j = 1, …, N} and $$ \mathcal{E} $$ = {(v_i, v_j)|i, j ∈ $$ \mathcal{V} $$} indicate the set of nodes and the set of edges, respectively. Within this graph, the associated adjacency matrix is $$ \mathcal{A} $$ = [a_ij] ∈ ℝ^N×N, where the element a_ij > 0 if the edge (v_i, v_j) ∈ $$ \mathcal{E} $$ exists, with agent j acts as the neighbor of agent i under this condition, whereas a_ij = 0. The neighborhood set of v_i is defined as N_i ={j|(i, j) ∈ $$ \mathcal{E} $$}. The degree matrix is denoted as $$ \mathcal{D} $$ = diag{d₁, d₂, …, d_N}, where d_i = $$ \sum_{j \in N_i} a_{ij} $$. Then, the Laplacian matrix is expressed as $$ \mathcal{L}=\mathcal{D}-\mathcal{A} $$. The pinning matrix $$ \mathcal{B} $$ = diag {b₁, b₂, …, b_N} characterizes the connection relationship between followers and the leader, where b_i = 1 if agent i receives information from the leader directly; otherwise b_i = 0. The graph contains a directed spanning tree if a path exists from the leader to all followers.

For a time-variant topology, the dynamic communication structure at time step k is $$ \vartheta $$(k) = ($$ \mathcal{V} $$, $$ \mathcal{E} $$(k), $$ \mathcal{A} $$(k)), and all possible topological forms in the system constitute a finite set {$$ \vartheta $$₁, $$ \vartheta $$₂, …, $$ \vartheta $$_f}. A Markov switching signal $$ \varsigma $$(k) : [0, ∞] → ℓ ∈ F $$ \triangleq $$ {1, 2, …, f} is utilized to describe the switching of the topologies. Accordingly, a_ij^ℓ, b_i^ℓ, and l_ii^ℓ = Σ_j=1^Na_ij are elements of the adjacency matrix $$ \mathcal{A} $$, the pinning matrix $$ \mathcal{B} $$, and the Laplacian matrix $$ \mathcal{L} $$ in the topology ℓ, respectively. It is said that $$ \vartheta $$(k) = $$ \vartheta $$_ℓ if and only if $$ \varsigma $$(k) = ℓ. $$ \vartheta $$_un is indicated as $$ \vartheta_{un} = \bigcup_{\ell \in F} \vartheta_\ell $$, which is employed to represent the union of all possible topologies. Ω = [Ω_ij]_f×f indicates the transition probability matrix of a Markov process, where Ω_ij =Pr{$$ \varsigma $$(k+1) = j|$$ \varsigma $$(k) = i}, $$ \forall $$i, j ∈ F. χ is the stationary distribution for {$$ \varsigma $$(k)}, then Ω^Tχ = χ.

Assumption 1 Under the framework of Markov switching topology, the associated union graph is assumed to contain a directed spanning tree, with the switching signal $$ \varsigma $$(k) governed by an ergodic Markov chain.

2.3. Problem formulation

Consider MASs consisting of one leader and N followers. The dynamic evolution equations for each follower and the leader can be expressed as follows:

where x_i(k) ∈ ℝⁿ is the state of the i-th agent, x₀(k) ∈ ℝⁿ is the state of the leader and u_i(k) ∈ ℝ^m denotes the system input. g represents a simplified symbol for a Markov chain {g(k)}, which takes values in a finite set $$ \mathcal{S} $$ = {1, 2, 3, …, m}.

The transition probability of the Markov chain{g(k)} is defined as

where π_gp is the transition probability from mode g to mode p at the next time step with π_gp ≥ 0 for all g, p ∈ S and Σ_p=1^mπ_gp = 1 for each mode g.

Denote the tracking error of the local neighborhood for the i-th agent by

Then, the dynamic evolution equation for the error system of agent i is expressed as follows:

Remark 1 It is worth noting that both A_g and B_g depend on the Markov jump process g(k), while a_ij^ℓ, b_i^ℓ, and l_ii^ℓ rely on the switching signal $$ \varsigma $$(k) of the topology. Since the probability distribution of the signal $$ \varsigma $$(k) for topology switching follows a stationary distribution, the topology graph among agents is guaranteed not to remain in a fixed mode indefinitely. This property further ensures the joint connectivity of the Markov switching topology graph considered in this paper. In addition, the Markov jumping signal g(k) and the Markov switching signal $$ \varsigma $$(k) are independent. Note that if they are not independent, the calculation of transition probabilities during modal transitions may be affected, which could in turn influence the reconstruction of the cost function and the analysis of system stability. This issue is left for future research.

To characterize the cost of the agent, the cost function of the i-th agent in mode g is defined as follows:

where Q_ig is the semidefinite matrix and R_ig is the positive definite matrix.

Remark 2 In the cost function (5), the utility function C_ig of each agent consists of two components: the tracking error (3) and the energy consumed by the control input u_i(k). This paper aims to make all followers reach consensus with the leader and minimize the cost function (5) of agent i at the same time. To achieve the optimization goal of the system, it is crucial to select appropriate weight parameters in the utility function C_ig. According to the classical linear quadratic regulator theory^[33], the weighting matrix Q_ig directly affects the convergence rate of the system states, while R_ig is closely related to the control input energy. Therefore, practical implementation requires a reasonable balance between system energy consumption and convergence performance.

Based on the above analysis, the main objective of this paper is to design an optimal controller u_i(k) that enables the MASs to achieve optimal consensus.

2.4. Problem transformation

In order to solve the optimal control problem in Markov jump MASs, a cost function is defined for each agent under different modes in Equation (5). Since mode jumps can affect system performance and complicate the stability analysis, the cost function is reformulated. Meanwhile, the optimal control policy is difficult to express in a linear state-feedback form due to the NCTs. For this reason, the method of restructuring the cost function is adopted to settle the problem caused by the jumping parameter. Subsequently, a global approach is proposed to address the interference caused by NCTs.

Inspired by^[16], the transition probability (2) is utilized to reconstruct the cost function for each agent, as shown below:

Hence, the control problem for MASs is formulated as a multiobjective optimization problem. The above multi-objective problem is transformed into a single-objective problem by employing the weighted sum method. The reconstructed cost function is written as

where λ_g > 0 is a weight coefficient and Σ_g=1^mλ_g = 1. λ_g represents the weights associated with the system parameter modes. Its values can be generated by using the transition probabilities π_gp between system modes. Moreover, in practice, users can influence system performance by adjusting λ_g.

With the Equations (6) and (7), the cost function of each agent is derived as follows:

where D_g = Σ_p=1^mλ_pπ_pg > 0. c_ig(e_i(k), u_i(k)) = e_i(k)^T$$ \mathcal{Q} $$_ie_i(k) + u_i(k)^T$$ \mathcal{R} $$_iu_i(k) is the utility function, where $$ \mathcal{Q} $$_i = Σ_g=1^mD_gQ_ig and $$ \mathcal{R} $$_i = Σ_g=1^mD_gR_ig.

Due to the interference from the NCTs, it is difficult to directly establish a linear relationship between the controller u_i(k) and the tracking error e_i(k). To overcome this limitation, the global forms of the controller and error are proposed to establish a linear relationship between the controller and the error system.

The global error vector E(k) and the global control vector U(k) are defined as

From the Equation (4), the dynamic evolution equation of the global error system is established as follows:

where $$ \bar{A} $$_g = I_N $$ \otimes $$ A_g and $$ \bar{B} $$_ℓg = ($$ \mathcal{L} $$^ℓ + $$ \mathcal{B} $$^ℓ) × B_g.

3.1. Optimal consensus control

From the Equations (8) and (9), the global cost function is derived as below:

where the matrix Q, the matrix R, and the global kernel matrix P are expressed as follows:

where the matrix P_ii is the kernel matrix of the cost function V_i(e_i(k)) and is the positive definite matrix. Since the cost function can be expressed in a quadratic form V(E(k)) = E^T(k)PE(k), Equation (11) is rewritten as

By Bellman’s optimality principle, the optimal cost function V^*(E(k)) is defined as

Then, the Hamilton function is constructed as follows:

Moreover, the first-order necessary condition requires that $$ \frac{\partial H[E(k),U(k)]}{\partial U(k)} $$ = 0. Furthermore, the equation $$ (R+{\bar B}_{\ell g}^TP{\bar B}_{\ell g})U(k)+{\bar B}^T_{\ell g}P{\bar A}_gE(k) $$ = 0 holds and the global optimal controller is described as

where $$ {\bar K} $$^*_ℓg is the global optimal controller gain and P^* is the unique solution to the ARE is expressed as

Remark 3 The researchers in^[23,28,34] derived the corresponding Hamilton function by constructing a cost function for each agent and determined the optimal control policy. A local neighborhood tracking error system is established. It is worth noting that, due to the influence of the NCTs, the methods of^[23,28,34] cannot derive the ARE based on the local cost function, which makes it difficult to derive the optimal control policy. To this end, a global cost function is developed to derive the global ARE in this paper. Subsequently, GPI and GVI algorithms are proposed to obtain approximate solutions for the global ARE and thereby determine the optimal controller. Compared with the aforementioned literature, the GPI and GVI algorithms developed in this paper are intended to address the problem that the ARE cannot be established due to NCTs. In addition, since matrix P in Equation (17) exhibits nonlinear characteristics, its exact solution is difficult to obtain.

Remark 4 The problem of optimal consensus control of MASs is considered in^[28], where the optimal control policy is derived as u_i^*(k) = -$$ \frac{\alpha}{2} $$(d_i + b_i)R_ii^-1B_i^T$$ \frac{\partial V_i^*(e_i(k+1))}{\partial e_i(k+1)} $$. By expanding the gradient section, one can obtain

Notably, the above equation contains the term $$ \sum\limits_{j\in\mathcal{N}_i} $$a_ijB_ju_j(k). This causes e_j(k) (assume u_j(k) = K_je_j(k)) to appear when converting the Bellman equation through a quadratic value function, thereby preventing the successful derivation of the ARE. To tackle this issue, this paper defines the global cost function and successfully derives the global ARE suitable for the system.

3.2. GPI algorithm and GVI algorithm

To calculate the solution of the global ARE, GPI and GVI algorithms are proposed below. Proofs of the convergence of the two algorithms will be provided in Section 3.3.

Combining the Equations (13) and (16), the Lyapunov equation is given by

Definition 1 (Admissible Control) If there exists a control policy U(k) that ensures the cost function (11) is bounded and makes the system (10) stable, then this policy is an admissible control.

The specific implementation details of the proposed GPI and GVI algorithms are detailed in Algorithm 1 and Algorithm 2, respectively.

Algorithm 1.

Algorithm 2.

Since the GPI algorithm requires the prior determination of an admissible policy, it exhibits a certain degree of conservatism. In comparison, the VI algorithm allows the utilization of any initial policy. Hence, the following algorithm is designed to solve for an approximate solution of ARE.

Remark 5 In contrast to traditional local controller ui^*(k) = -(R_i + B_i^TP_iB_i)^-1B_i^TP_i^*A_ie_i(k), the global optimal controller is provided in the Equation (16). The reason is that it is difficult for traditional local controllers to utilize the ARE approach for solving the optimal controller, whereas the global controller in (16) mitigates the influence of NCTs. In fact, the controller u_i(k) of each agent can also be derived in this paper: u_i(k) = -K_iie_i(k) - Σ_j=1^NK_ije_j(k), where K_ii = (l_ii^ℓ + b_i^ℓ)[$$ \mathcal{R} $$_i + (l_ii^ℓ + b_i^ℓ)²B_g^TP_iiB_g]^-1B_g^TP_iiA_g and K_ij = (l_ii^ℓ + b_i^ℓ)a_ij^ℓ[$$ \mathcal{R} $$_i + (l_ii^ℓ + b_i^ℓ)²B_g^TP_iiB_g]^-1B_g^TP_iiB_gK_j; K_j is the optimal controller gain for the j-th agent and its specific form is u_j(k) = -K_je_j(k). Furthermore, the global controller gain $$ {\bar K} $$_ℓg in (16) is expressed in the form of K_ii and K_ij as follows:

3.3. Properties of iterative algorithms and stability of system

This subsection analyzes the algorithms’ monotonicity and determines their convergence properties. Then, the proof of the stability for the closed-loop system is given.

Lemma 1 The iterative cost function V^l(E(k)) of GPI is a monotonically nonincreasing sequence, that is, the following relationship holds: V^l+1(E(k)) ≤ V^l(E(k)).

Proof The matrix ψ^l is designed to be

Hence, the Equation (19) can be written as

A function of a matrix is defined as follows:

According to Equation (16), the following conclusion can be drawn: $$ (R+{\bar B}_{\ell g}^TP^l{\bar B}_{\ell g}){\bar K}^{l+1}_{\ell g}-{\bar B}_{\ell g}P^l{\bar A}_g $$ = 0.

A matrix $$ \Pi $$ is expressed as $$ \Pi $$ = R + $$ \bar {B} $$_ℓg^TP^l$$ \bar {B} $$_ℓg and the Equation (25) can be rewritten as follows:

• Assuming that K = $$ {\bar K}^{l+1}_{\ell g} $$, the Equation (25) can be expressed as Φ(P^l, $$ {\bar K} $$_ℓg^l+1) = Q + $$ {\bar{A}_{g}}^TP^l\bar{A}_{g}-({\bar K}^{l+1}_{\ell g})^T \Pi ({\bar K}^{l+1}_{\ell g}) $$;
• When K is the gain of any controller, the following condition is true:

Due to the relationship that $$ \Pi \succeq $$ 0, it concludes that Φ(P^l, $$ {\bar K} $$_ℓg^l) $$ \succeq $$ Φ(P^l, $$ {\bar K} $$_ℓg^l+1). The difference form of the kernel matrix for the cost function is expressed by

Therefore, it can be concluded that: P^l $$ \succeq $$ P^l+1. Using the relationship between the cost function and the kernel matrix in Equation (13), the monotonicity of the cost function V^l(E(k)) follows.

Lemma 2 Define the iterative cost function V^l of GVI as

If the kernel matrix P^l+1 and controller gain $$ {\bar K} $$_ℓg^l are updated based on (21) and (22), the following statements hold:

• If P¹ - P⁰ $$ \succ $$ 0, then V^l+1(E(k)) ≥ V^l(E(k)).
• If P¹ - P⁰ $$ \prec $$ 0, then V^l+1(E(k)) ≤ V^l(E(k)).

Proof It is first demonstrated that the cost function sequence is nondecreasing under the assumption P¹ - P⁰ $$ \succ $$ 0. Define a new sequence γ^l with γ⁰(E(k)) = V⁰(E(k)) = E^T(k)P⁰E(k) = 0 as follows:

Due to P¹ - P⁰ $$ \succ $$ 0, it can derive E^T(k)(P¹ - P⁰)E(k) ≥ 0. From γ⁰(E(k)) = V⁰(E(k)), we have

which means

Assuming that V^l(E(k)) ≥ γ^l-1(E(k)) is true, it follows that V^l(E(k+1)) ≥ γ^l-1(E(k+1)) holds. Considering

which indicates

Based on the Equation (21), a function is given by

Select the matrix M as the kernel matrix of the Equation (30), the following equation holds: γ^l(E(k)) = E^T(k)M^lE(k). By the conditions defined above, one can conclude that: M⁰ = P⁰. Subsequently, assume that the relationship between M^l and P^l satisfies M^l $$ \succeq $$ P^l when l ≥ 1. When condition K = $$ {\bar K} $$_ℓg^l is selected, the following conclusion can be obtained:

From the Equation (21) and the Equation (36), the following conclusion is obtained: M^l+1 $$ \succeq $$ P^l+1, which means γ^l+1(E(k)) ≥ V^l+1(E(k)). Therefore, the inequality γ^l(E(k)) ≥ V^l(E(k)) holds.

Then, the following conclusion can be derived:

which implies

Hence, the first part of the proof is completed by mathematical induction.

Next, prove that the sequence of the cost function is nonincreasing by assuming P¹ - P⁰ $$ \prec $$ 0. Define a new sequence Γ^l with Γ¹(E(k)) = V¹(E(k)) = E^T(k)P¹E(k) as follows:

Due to P¹ - P⁰ $$ \prec $$ 0, it can derive E^T(k)(P¹ - P⁰)E(k) ≤ 0. From Γ¹(E(k)) = V¹(E(k)), we have

which yields

Assuming that Γ^l(E(k)) ≤ V^l-1(E(k)) is true, it follows that Γ^l(E(k+1)) ≤ V^l-1(E(k+1)) holds.

Considering the following equation:

which means

The proof of Γ^l+1(E(k)) ≥ V^l+1(E(k)) is similar to the proof given above and will not be repeated here. Then, the inequality can be represented as

Therefore, the second part is proven to be complete using mathematical induction.

Remark 6 In^[30], the monotonicity of the VI algorithm is proved by constructing two specific functions and comparing them with the cost function, but a complete proof of γ^l(E(k)) ≥ V^l(E(k)) is not provided. A rigorous proof of this inequality is given in Lemma 2 of this paper. The proof shows that the monotone property of the cost function relies on the comparison between P¹ and P⁰. Once this relationship is established, the monotonic behavior is preserved throughout the iterations.

Theorem 1 Suppose the sequences $$ {\bar K} $$_ℓg^l and P^l are computed by Algorithm 1 and Algorithm 2. Then, the iterative controller gain $$ {\bar K} $$_ℓg^l and the iterative kernel matrix P^l converge to the optimal controller gain $$ {\bar K} $$_ℓg^* and the optimal kernel matrix P^*.

Proof Since the iterative cost function in Algorithm 1 and the iterative cost function in Algorithm 2 when P¹ - P⁰ $$ \prec $$ 0 are both nonincreasing, the convergence of these two cases is analyzed first. With Lemma 1, Lemma 2, and Equation (13), the nonincreasing property of the iterative kernel matrix P^l is obtained as P^l+1 $$ \preceq $$ P^l. Subsequently, it can be derived from Equation (18) that P^l $$ \succeq $$ 0.

As the matrix sequence {P^l} is monotonically bounded, there exists a limit lim_l→∞ P^l = $$ \underline{P} $$. According to Algorithms 1 and 2, $$ {\bar K} $$_ℓg^l = $$ \underline{K} $$ when l → ∞. Then, the Equations (17) and (19) are transformed as

As a result, the limit matrix $$ \underline{P} $$ satisfies the ARE. Since the proof of case P¹ - P⁰ $$ \succ $$ 0 in Lemma 2 is similar to the above process, it will not be repeated in the subsequent derivation. Under the above analysis, it is concluded that the iterative controller gain $$ {\bar K} $$_ℓg^l and iterative kernel matrix P^l converge to the optimal solutions $$ {\bar K} $$_ℓg^* and P^*, respectively.

Next, the convergence of the iterative kernel matrix P^l in Algorithm 2 when P¹ - P⁰ $$ \succ $$ 0 is analyzed. From Equation (30), the following equation can be derived as

where γ⁰(E(k + l + 1)) = 0. Thus, the inequality is shown from (45) as follows:

By the Equation (37), it follows that V^l+1(E(k)) ≤ γ^l+1(E(k)) ≤ $$ \bar {V} $$. Furthermore, the relationship is derived as P^l+1 $$ \preceq \bar {P} $$, where $$ \bar {P} $$ satisfies the equation $$ \bar {V} $$ = E^T(k)$$ \bar {P} $$E(k). Hence, it is obtained that $$ \bar {P} $$ is an upper bound of P^l. Then, the convergence of the iterative kernel matrix P^l is proven by Equation (13) and Lemma 2. Therefore, it is concluded that the iterative controller gain $$ \bar {K} $$_ℓg^l and iterative kernel matrix P^l converge to the optimal solutions $$ \bar {K} $$_ℓg^* and P^*, respectively.

Theorem 2 Assuming that the U^*(k) is provided by the Equation (16) and V^*(E(k)) complies with the Equation (14), then the system (10) converges to an asymptotically stable state, and the consensus of the agents is reached.

Proof Select optimal cost function V^*(E(k)) as a Lyapunov function, whose specific form as

where Q $$ \succeq $$ 0 and R $$ \succ $$ 0. Subsequently, by analyzing the Equation (48), the kernel matrix P^* of V^*(E(k)) satisfies P^* $$ \succ $$ 0. Define the minimum eigenvalue $$ \underline{p} $$ = λ_min{P^*} > 0 and the maximum eigenvalue $$ \overline {p} $$ = λ_max{P^*} > 0 of matrix P^*, respectively. Thus, the following equation is derived:

Furthermore, from the inequality $$ \underline {p} $$||E(k)||² ≤ V^*(E(k)), it follows that V^*(E(k)) → ∞ when ||E(k)|| → ∞. Then, V^*(E(k)) is radially unbounded.

From Equations (10), (16), and (23), the difference of the Lyapunov function is defined as

In particular, the above conditions imply that $$ {\bar {W}} $$_ℓg^* is a positive definite matrix and the minimum eigenvalue of $$ {\bar {W}} $$_ℓg^* is defined as $$ \underline{w} $$ =λ_min{$$ \bar {W} $$}_ℓg^* > 0. Then, the Equation (50) is expressed as the following form:

Combining Equation (51) and the Lyapunov stability theorem, the state of the error system (10) is asymptotically stable.