Abstract: Finding feasible, collision-free paths for multiagent systems can be challenging, particularly in non-communicating scenarios where each agent's intent (e.g. goal) is unobservable to the others. In particular, finding time efficient paths often requires anticipating interaction with neighboring agents, the process of which can be computationally prohibitive. This work presents a decentralized multiagent collision avoidance algorithm based on a novel application of deep reinforcement learning, which effectively offloads the online computation (for predicting interaction patterns) to an offline learning procedure. Specifically, the proposed approach develops a value network that encodes the estimated time to the goal given an agent's joint configuration (positions and velocities) with its neighbors. Use of the value network not only admits efficient (i.e., real-time implementable) queries for finding a collision-free velocity vector, but also considers the uncertainty in the other agents' motion. Simulation results show more than 26% improvement in paths quality (i.e., time to reach the goal) when compared with optimal reciprocal collision avoidance (ORCA), a state-of-the-art collision avoidance strategy.

Abstract: This paper addresses the distributed formation control problem of a networked multi-agent system (MAS) subject to limited communication resources. First, a dynamic event-triggered communication mechanism (DECM) is developed to schedule inter-agent communication such that some unnecessary data exchanges among agents can be reduced so as to achieve better resource efficiency. Different from most of the existing event-triggered communication mechanisms, wherein threshold parameters are fixed all the time, the threshold parameter in the developed event triggering condition is dynamically adjustable in accordance with a dynamic rule. It is numerically shown that the proposed DECM can achieve a better tradeoff between reducing inter-agent communication frequency and preserving an expected formation than some existing ones. Second, an event-triggered formation protocol is delicately proposed by using only locally triggered sampled data in a distributed manner. Based on the formation protocol, it is shown that the state formation control problem is cast into an asymptotic stability problem of a reduced-order closed-loop system. Then, criteria for designing desired formation protocol and communication mechanism are derived. Finally, the effectiveness and advantages of the proposed approach are demonstrated through a comparative study in multirobot formation control.

Abstract: We introduce the first goal-driven training for visual question answering and dialog agents. Specifically, we pose a cooperative ‘image guessing’ game between two agents – Q-BOT and A-BOT– who communicate in natural language dialog so that Q-BOT can select an unseen image from a lineup of images. We use deep reinforcement learning (RL) to learn the policies of these agents end-to-end – from pixels to multi-agent multi-round dialog to game reward.,,We demonstrate two experimental results.,,First, as a ‘sanity check’ demonstration of pure RL (from scratch), we show results on a synthetic world, where the agents communicate in ungrounded vocabularies, i.e., symbols with no pre-specified meanings (X, Y, Z). We find that two bots invent their own communication protocol and start using certain symbols to ask/answer about certain visual attributes (shape/color/style). Thus, we demonstrate the emergence of grounded language and communication among ‘visual’ dialog agents with no human supervision.,,Second, we conduct large-scale real-image experiments on the VisDial dataset [5], where we pretrain on dialog data with supervised learning (SL) and show that the RL finetuned agents significantly outperform supervised pretraining. Interestingly, the RL Q-BOT learns to ask questions that A-BOT is good at, ultimately resulting in more informative dialog and a better team.

Abstract: This paper investigates the optimal consensus control problem for discrete-time multi-agent systems with completely unknown dynamics by utilizing a data-driven reinforcement learning method. It is known that the optimal consensus control for multi-agent systems relies on the solution of the coupled Hamilton–Jacobi–Bellman equation, which is generally impossible to be solved analytically. Even worse, most real-world systems are too complicated to obtain accurate mathematical models. To overcome these deficiencies, a data-based adaptive dynamic programming method is presented using the current and past system data rather than the accurate system models also instead of the traditional identification scheme which would cause the approximation residual errors. First, we establish a discounted performance index and formulate the optimal consensus problem via Bellman optimality principle. Then, we introduce the policy iteration algorithm which motivates this paper. To implement the proposed online action-dependent heuristic dynamic programming method, two neural networks (NNs), 1) critic NN and 2) actor NN, are employed to approximate the iterative performance index functions and control policies, respectively, in real time. Finally, two simulation examples are provided to demonstrate the effectiveness of the proposed method.

Abstract: This paper addresses the output consensus problem of heterogeneous linear multi-agent systems. We first propose a novel distributed event-triggered control scheme. It is shown that, with the proposed control scheme, the output consensus problem can be solved if two matrix equations are satisfied. Then, we further propose a novel self-triggered control scheme, with which continuous monitoring is avoided. By introducing a fixed timer into both event- and self-triggered control schemes, Zeno behavior can be ruled out for each agent. The effectiveness of the event- and self-triggered control schemes is illustrated by an example.

Abstract: This technical note presents a continuous-time multi-agent system for distributed optimization with an additive objective function composed of individual objective functions subject to bound, equality, and inequality constraints. Each individual objective function is assumed to be convex in the region defined by its local bound constraints only without the need to be globally convex. All agents in the system communicate using a proportional-integral protocol with their output information instead of state information to reduce communication bandwidth. It is proved that all agents with any initial state can reach output consensus at an optimal solution to the given constrained optimization problem, provided that the graph describing the communication links among agents is undirected and connected. It is further proved that the system with only integral protocol is also convergent to the unique optimal solution if each individual objective function is strictly convex. Simulation results are presented to substantiate the theoretical results.

Abstract: This paper is concerned with sampled-data leader-following consensus of a group of agents with nonlinear characteristic. A distributed consensus protocol with probabilistic sampling in two sampling periods is proposed. First, a general consensus criterion is derived for multiagent systems under a directed graph. A number of results in several special cases without transmittal delays or with the deterministic sampling are obtained. Second, a dimension-reduced condition is obtained for multiagent systems under an undirected graph. It is shown that the leader-following consensus problem with stochastic sampling can be transferred into a master–slave synchronization problem with only one master system and two slave systems. The problem solving is independent of the number of agents, which greatly facilitates its application to large-scale networked agents. Third, the network design issue is further addressed, demonstrating the positive and active roles of the network structure in reaching consensus. Finally, two examples are given to verify the theoretical results.

Abstract: Recent reviews stated that the complex and context-dependent nature of human decision-making resulted in ad-hoc representations of human decision in agent-based land use change models (LUCC ABMs) and that these representations are often not explicitly grounded in theory. However, a systematic survey on the characteristics (e.g. uncertainty, adaptation, learning, interactions and heterogeneities of agents) of representing human decision-making in LUCC ABMs is missing. Therefore, the aim of this study is to inform this debate by reviewing 134 LUCC ABM papers. We show that most human decision sub-models are not explicitly based on a specific theory and if so they are mostly based on economic theories, such as the rational actor, and mainly ignoring other relevant disciplines. Consolidating and enlarging the theoretical basis for modelling human decision-making may be achieved by using a structural framework for modellers, re-using published decision models, learning from other disciplines and fostering collaboration with social scientists. Review on human decisions in agent-based models of land use change.Most human decision models are not based on theory.Used theories are mainly from economics and not from psychology.A framework to guide modellers how to implement human decision is needed.

Abstract: The energy landscape is experiencing accelerating change; centralized energy systems are being decarbonized, and transitioning towards distributed energy systems, facilitated by advances in power system management and information and communication technologies. This paper elaborates on these generations of energy systems by critically reviewing relevant authoritative literature. This includes a discussion of modern concepts such as ‘smart grid’, ‘microgrid’, ‘virtual power plant’ and ‘multi-energy system’, and the relationships between them, as well as the trends towards distributed intelligence and interoperability. Each of these emerging urban energy concepts holds merit when applied within a centralized grid paradigm, but very little research applies these approaches within the emerging energy landscape typified by a high penetration of distributed energy resources, prosumers (consumers and producers), interoperability, and big data. Given the ongoing boom in these fields, this will lead to new challenges and opportunities as the status-quo of energy systems changes dramatically. We argue that a new generation of holonic energy systems is required to orchestrate the interplay between these dense, diverse and distributed energy components. The paper therefore contributes a description of holonic energy systems and the implicit research required towards sustainability and resilience in the imminent energy landscape. This promotes the systemic features of autonomy, belonging, connectivity, diversity and emergence, and balances global and local system objectives, through adaptive control topologies and demand responsive energy management. Future research avenues are identified to support this transition regarding interoperability, secure distributed control and a system of systems approach.

Abstract: Agent-Based Computing is a diverse research domain concerned with the building of intelligent software based on the concept of "agents". In this paper, we use Scientometric analysis to analyze all sub-domains of agent-based computing. Our data consists of 1,064 journal articles indexed in the ISI web of knowledge published during a twenty year period: 1990-2010. These were retrieved using a topic search with various keywords commonly used in sub-domains of agent-based computing. In our proposed approach, we have employed a combination of two applications for analysis, namely Network Workbench and CiteSpace - wherein Network Workbench allowed for the analysis of complex network aspects of the domain, detailed visualization-based analysis of the bibliographic data was performed using CiteSpace. Our results include the identification of the largest cluster based on keywords, the timeline of publication of index terms, the core journals and key subject categories. We also identify the core authors, top countries of origin of the manuscripts along with core research institutes. Finally, our results have interestingly revealed the strong presence of agent-based computing in a number of non-computing related scientific domains including Life Sciences, Ecological Sciences and Social Sciences.

Abstract: In this paper, we consider the synchronization control problem for uncertain high-order nonlinear multi-agent systems in a leader-follower scheme, under a directed communication protocol. A robust decentralized control law of minimal complexity is proposed that achieves prescribed, arbitrarily fast and accurate synchronization of the following agents with the leader. The control protocol is decentralized in the sense that the control signal of each agent is calculated based solely on local relative state information from its neighborhood set. Additionally, no information regarding the agents' dynamic model is employed in the design procedure. Moreover, provided that the communication graph is connected and contrary to the related works on multi-agent systems, the controller-imposed transient and steady state performance bounds are fully decoupled from: 1) the underlying graph topology, 2) the control gains selection, and 3) the agents' model uncertainties, and are solely prescribed by certain designer-specified performance functions. Extensive simulation results clarify and verify the approach.

Abstract: This paper addresses the problem of leader-following consensus for networked multi-agent systems subject to limited communication resources and unknown-but-bounded process and measurement noise. First, a new distributed event-based communication mechanism on the basis of a time-varying threshold parameter is developed to schedule transmission of each sensor's measurement through a communication network so as to alleviate consecutive occupancy of communication resources. Second, a novel concept of set-membership leader-following consensus is put forward, through which the true states of all followers are guaranteed to always reside in a bounding ellipsoidal set of the leader's state. Third, in the case that full information of followers’ states are not measurable, a distributed observer-based consensus protocol is presented to provide a set-membership estimation of each follower's state. Then, based on a recursive computation of confidence state estimation ellipsoids and leader state ellipsoid, a delicate convex optimization algorithm in terms of recursive linear matrix inequalities is proposed to design desired consensus protocol and event-based mechanism. Finally, an illustrative example is given to show the effectiveness and advantage of the developed approach.

Abstract: This paper addresses the consensus problem for a continuous-time multiagent system (MAS) with Markovian network topologies and external disturbance. Different from some existing results, global jumping modes of the Markovian network topologies are not required to be completely available for consensus protocol design. A network topology mode regulator (NTMR) is first developed to decompose unavailable global modes into several overlapping groups, where overlapping groups refer to the scenario that there exist commonly shared local modes between any two distinct groups. The NTMR schedules which group modes each agent may access at every time step. Then a new group mode-dependent distributed consensus protocol on the basis of relative measurement outputs of neighboring agents is delicately constructed. In this sense, the proposed consensus protocol relies only on group and partial modes and eliminates the need for complete knowledge of global modes. Sufficient conditions on the existence of desired distributed consensus protocols are derived to ensure consensus of the MAS with a prescribed $H_{\infty }$ performance level. Two examples are provided to show the effectiveness of the proposed consensus protocol.

Abstract: Existing Nussbaum function based results on consensus of multi-agent systems require that the unknown control directions of all the agents should be the same. This note proposes an adaptive method to relax such a requirement to allow non-identical control directions, under the condition that some control directions are known. Technically, a novel idea is proposed to construct a new Nussbaum function, from which a conditional inequality is developed to handle time-varying input gains. Then, the inequality is integrated with adaptive control technique such that the proposed Nussbaum function for each agent is adaptively updated. Moreover, in addition to parametric uncertainties, each agent has non-parametric bounded modelling errors which may include external disturbances and approximation errors of static input nonlinearities. Even in the presence of such uncertainties, the proposed control scheme is still able to ensure the states of all the agents asymptotically reach perfect consensus. Finally, simulation study is performed to show the effectiveness of the proposed approach.

Abstract: In this technical note, the consensus control problem is investigated for a class of discrete time-varying stochastic multi-agent system subject to sensor saturations. An event-based mechanism is adopted where each agent updates the control input signal only when the pre-specified triggering condition is violated. To reflect the time-varying manner and characterize the transient consensus behavior, a new index for mean-square consensus is put forward to quantify the deviation level from individual agent to the average value of all agents’ states. For a fixed network topology, the aim of the proposed problem is to design time-varying output-feedback controllers such that, at each time step, the mean-square consensus index of the closed-loop multi-agent system satisfies the pre-specified upper bound constraints subject to certain triggering mechanism. Both the existence conditions and the explicit expression of the desired controllers are established by resorting to the solutions to a set of recursive matrix inequalities. An illustrative simulation example is utilized to demonstrate the usefulness of the proposed algorithms.

Abstract: The paper is concerned with the problem of distributed model predictive control (DMPC) for formation of multiple linear second-order agents with collision avoidance and obstacle avoidance. All the agents are permitted to implement optimization simultaneously at each time step. The assumed input trajectory and state trajectory are introduced to obtain a computationally tractable optimization problem in a distributed manner. As a result, a compatibility constraint is required to ensure the consistency between each agent׳s real operation and its plan and to establish the agreement among agents. The terminal ingredients are tailored by making use of the specific form of the system model and the control objective. The terminal set is ensured to be positively invariant with the designed terminal controller. The collision avoidance constraint and the obstacle avoidance constraint are satisfied for any state in the terminal set. The weighted matrix of the terminal cost is determined by solving a Lyapunov equation. Moreover, recursive feasibility of the resulting optimization problem is guaranteed and closed-loop stability of the whole system is ensured. Finally, a numerical example is given to illustrate the effectiveness of the proposed algorithm.

Abstract: In this paper, we consider the cooperative output regulation problem of heterogeneous linear multi-agent systems (MASs) by event-triggered control. We first develop an event-triggering mechanism for leader-following consensus of homogeneous MASs. Then by proposing an internal reference model for each agent, a novel distributed event-triggered control scheme is developed to solve the cooperative output regulation problem of heterogeneous MASs. Furthermore, a novel self-triggered control scheme is also proposed, such that continuous monitoring of measurement errors can be avoided. The feasibility of both proposed control schemes is studied by excluding Zeno behavior for each agent. An example is finally provided to demonstrate the effectiveness of the control schemes.

Abstract: A multi-agent system consisting of $N$ agents is considered. The problem of steering each agent from its initial position to a desired goal while avoiding collisions with obstacles and other agents is studied. This problem, referred to as the multi-agent collision avoidance problem , is formulated as a differential game. Dynamic feedback strategies that approximate the feedback Nash equilibrium solutions of the differential game are constructed and it is shown that, provided certain assumptions are satisfied, these guarantee that the agents reach their targets while avoiding collisions.

Abstract: This paper develops an off-policy reinforcement learning (RL) algorithm to solve optimal synchronization of multiagent systems. This is accomplished by using the framework of graphical games. In contrast to traditional control protocols, which require complete knowledge of agent dynamics, the proposed off-policy RL algorithm is a model-free approach, in that it solves the optimal synchronization problem without knowing any knowledge of the agent dynamics. A prescribed control policy, called behavior policy, is applied to each agent to generate and collect data for learning. An off-policy Bellman equation is derived for each agent to learn the value function for the policy under evaluation, called target policy, and find an improved policy, simultaneously. Actor and critic neural networks along with least-square approach are employed to approximate target control policies and value functions using the data generated by applying prescribed behavior policies. Finally, an off-policy RL algorithm is presented that is implemented in real time and gives the approximate optimal control policy for each agent using only measured data. It is shown that the optimal distributed policies found by the proposed algorithm satisfy the global Nash equilibrium and synchronize all agents to the leader. Simulation results illustrate the effectiveness of the proposed method.

Abstract: In this technical note, a distributed velocity-constrained consensus problem is studied for discrete-time multi-agent systems, where each agent's velocity is constrained to lie in a nonconvex set. A distributed constrained control algorithm is proposed to enable all agents to converge to a common point using only local information. The gains of the algorithm for all agents need not to be the same or predesigned and can be adjusted by each agent itself based on its own and neighbors' information. It is shown that the algorithm is robust to arbitrarily bounded communication delays and arbitrarily switching communication graphs provided that the union of the graphs has directed spanning trees among each certain time interval. The analysis approach is based on multiple novel model transformations, proper control parameter selections, boundedness analysis of state-dependent stochastic matrices1, exploitation of the convexity of stochastic matrices, and the joint connectivity of the communication graphs. Numerical examples are included to illustrate the theoretical results.

Abstract: This paper presents an intelligent controller for “load frequency control (LFC)” application in “smart grid (SG)”environment having changes in communication topology (CT) via a multi-agent system (MAS) technology. In this study, network-induced effects, time delay, and change in CT have been addressed to examine the system performance in a closed loop. An event-triggered control method is used to reduce the communication burden in a network. An intelligent controller based on reinforcement learning consists of two levels, estimator agent and controller agent, in each multi-area system. Particle swarm optimization is used to tune the controller parameters. Furthermore, the proposed control strategy and system architecture as MAS for LFC in SG are analyzed in detail, verified for various load conditions and different network configurations. In addition, mean-square error of the power system states with CT is also analyzed. The results of this study validate the feasibility of the proposed control, as well as the capability of the MAS for the operation of LFC in SG with changes in CT.

Abstract: This paper proposes a multiagent-based optimal microgrid control scheme using a fully distributed diffusion strategy. A two-level cooperative optimization multiagent system is adapted for distributed energy resources economic dispatch. The lower level implements an adaptive droop scheme based on online no-load frequency adjustments. The upper level implements distributed communication using diffusion between neighboring agents for optimal microgrid management. The proposed control scheme enables peer-to-peer communication among the agents without the necessity of a centralized controller, and simultaneously performs resource optimization while regulating the system frequency. The results are compared with centralized and consensus-based optimization algorithms. We have concluded that the proposed algorithm is superior over consensus algorithms in terms of convergence speed and utilizes reduced communication infrastructure compared to centralized controllers. Simulation demonstrations were conducted along with experimental results from a hardware-based microgrid using an industrial multiagent framework. The simulation and experimental results show that the proposed method and the agent framework can be deployed in real-world microgrids and offer superior decision making on optimal microgrid control.

Abstract: This paper presents a distributed multiagent scheme to detect and identify cyber threats on the protection systems of power grids. The integration of information and communication technologies into existing power grids builds critical cyber-physical energy systems, in which digital relays are networked cyber-physical components subject to various cyber threats. Cyber attacks on protection systems may mimic real faults, cause component failure, and disable the communication links. Agents utilize both cyber and physical properties to reinforce the detection technique and further distinguish cyber attacks from physical faults. This paper also introduces the problem of secure communication protocols and highlights the comparative studies for enhancing the security of the protection systems. The proposed scheme is validated using a benchmark power system under various fault and cyber attack scenarios.

Abstract: This paper considers two classes of large population stochastic differential games connected to optimal and robust decentralized control of large-scale multiagent systems. The first problem ( P1 ) is one where each agent minimizes an exponentiated cost function, capturing risk-sensitive behavior, whereas in the second problem ( P2 ) each agent minimizes a worst-case risk-neutral cost function, where the “worst case” stems from the presence of an adversary entering each agent’s dynamics characterized by a stochastic differential equation. In both problems, the individual agents are coupled through the mean field term included in each agent’s cost function, which captures the average or mass behavior of the agents. We solve both P1 and P2 via mean field game theory. Specifically, we first solve a generic risk-sensitive optimal control problem and a generic stochastic zero-sum differential game, where the corresponding optimal controllers are applied by each agent to construct the mean field systems of P1 and P2 . We then characterize an approximated mass behavior effect on an individual agent via a fixed-point analysis of the mean field system. For each problem, P1 and P2 , we show that the approximated mass behavior is in fact the best estimate of the actual mass behavior in various senses as the population size, $N$ , goes to infinity. Moreover, we show that for finite $N$ , there exist $\epsilon$ - Nash equilibria for both P1 and P2 , where the corresponding individual Nash strategies are decentralized in terms of local state information and the approximated mass behavior. We also show that $\epsilon$ can be taken to be arbitrarily small when $N$ is sufficiently large. We show that the $\epsilon$ - Nash equilibria of P1 and P2 are partially equivalent in the sense that the individual Nash strategies share identical control laws, but the approximated mass behaviors for P1 and P2 are different, since in P2 , the mass behavior is also affected by the associated worst-case disturbance. Finally, we prove that the Nash equilibria for P1 and P2 both feature robustness, and as the parameter characterizing this robustness becomes infinite, the two Nash equilibria become identical and equivalent to that of the risk-neutral case, as in the one-agent risk-sensitive and robust control theory.

Abstract: This article describes a method of organizing the communication protocol, which allows agents of the multiagent system (MAS) to make decisions about their actions. Plan activities and interact with each other to perform tasks of modern industrial and business processes based on cyber-physical systems. The main attention is paid to those multi-agent systems, where autonomous agents — robots or smart things — participate in business processes among people, and their activities are organized in an unreliable and unknown environment. The article shows, how to organize a communication system between agents in a peer-to-peer network using the decentralized Ethereum Blochchain technology and smart contracts. The architecture of protocol of autonomous business activity, based on this communication method is given. As a result, the experience of implementation an autonomous economic system with unmanned aerial vehicles (UAV) is described.

Abstract: This paper focuses on the leader-following consensus control problem of multiagent systems in random vibration environment. The Ito stochastic systems with heterogeneous unknown dynamics and external disturbances are established to describe the agents in random vibration environment. The fuzzy logic systems are applied to approximate the unknown nonlinear dynamics, and one adaptive parameter is designed to decay the effect of external disturbances. We present a new distributed consensus controller for each follower agent only based on local information that is measured or received from its neighbors and itself. Under the consensus controller, we prove that all the follower agents can keep consensus with the leader, even though only a very small part of follower agents can measure or receive the state information of the leader. Furthermore, the states of all the follower agents are bounded in probability. Finally, the simulation results are provided to illustrate the effectiveness of the designed algorithm.