Abstract: The real-time recurrent learning algorithm is a gradient-following learning algorithm for completely recurrent networks running in continually sampled time. Here we use a series of simulation experiments to investigate the power and properties of this algorithm. In the recurrent networks studied here, any unit can be connected to any other, and any unit can receive external input. These networks run continually in the sense that they sample their inputs on every update cycle, and any unit can have a training target on any cycle. The storage required and computation time on each step are independent of time and are completely determined by the size of the network, so no prior knowledge of the temporal structure of the task being learned is required. The algorithm is nonlocal in the sense that each unit must have knowledge of the complete recurrent weight matrix and error vector. The algorithm is computationally intensive in sequential computers, requiring a storage capacity of the order of the thi...

Abstract: An extremely efficient algorithm is presented for generating D-optimal designs. It is shown that the underlying set function of the sequential design algorithm is submodular and thus the so-called accelerated greedy algorithm may be applied to this problem. While the new algorithm's statistical basis is identical to that of the Wynn and Federov algorithm, it requires significantly fewer function evaluations. The new algorithm is particularly useful when no prior information concerning the structure of the optimal design is available.

Abstract: A fast new algorithm is presented for training multilayer perceptrons as an alternative to the back-propagation algorithm. This new algorithm reduces the required training time considerably and overcomes many of the shortcomings presented by the conventional back-propagation algorithm. The new algorithm shortens the training time by several orders of magnitude for the pattern recognition type considered. In some cases improvement ratios of the new algorithm over the back-propagation algorithm run higher than 10,000, and it is not unlikely that this number may be further increased by considering patterns with higher resolution (more pixels per pattern). The new algorithm can also be implemented in a parallel architecture. One processor handles the network processing and the calculation of the modified back-propagation error signals, while the other processor handles the Kalman filter calculations. Aside from the speed advantage, the new algorithm is also more predictable in its training. The algorithm makes steady progress toward improving the mean squared error. In contrast, the back-propagation algorithm tends to reach a certain mean squared error and remain there for many iterations making little or no progress. At some point, it either rapidly converges, or jumps to a new level where it would again make little or no progress for many iterations. The convergence of the back-propagation algorithm depends heavily on the magnitude of the initial weights. If chosen incorrectly, the algorithm takes a long time to converge. The new algorithm on the other hand is much less sensitive to the initial weight setting. Furthermore, the adaptive nature of the Kalman gain makes the new algorithm much less likely to get caught in a state other than the global minimum. The new algorithm is much faster and more reliable than the back-propagation algorithm. It is very consistent in its training and is much less sensitive to the initial weight settings than the back-propagation algorithm. In every aspect the new algorithm outperforms the back-propagation algorithm for training multilayer perceptrons. (Abstract shortened with permission of author.)

Abstract: This paper presents a learning algorithm which may be used to train networks whose neurons may have discontinuous or nondifferentiable activation functions. The algorithm has been demonstrated using several different neuron activation functions. Although it shares several features with the error back-propagation algorithm, the heuristic derivation presented does not appeal to the highly mathematical derivation of the error back-propagation algorithm. The error back-propagation learning algorithm is shown to be at least reasonable. The learning algorithm derived could be argued to be successful just because of its similarity with the error back-propagation algorithm. Alternatively, it may be that the success of the error back-propagation algorithm, in that it does not seem to suffer from the problems normally associated with gradient descent procedures, is due to its similarity with the algorithm presented. >

Abstract: Summary form only given, as follows. A new supervised learning algorithm which does not require any gradient computation is presented. In the new gradient-free (G-F) algorithm, the error between the actual output and the desired output is not measured by the least-squared norm as in the backpropagation algorithm, but by the up-norm. In the G-F algorithm, the weights are updated in each iteration only after incorporating all the input patterns. The authors use the example of the XOR problem to evaluate the performance of the algorithm. A Monte-Carlo simulation is performed and the results obtained are encouraging. >

Abstract: The construction of prediction algorithms in a situation in which a learner faces a sequence of trials, with a prediction to be made in each, and the goal of the learner is to make few mistakes is studied. It is assumed that the learner has reason to believe that one of some pool of known algorithms will perform well but does not know which one. A simple and effective method, based on weighted voting, is introduced for constructing a compound algorithm in such a circumstance. It is called the weighted majority algorithm and is shown to be robust with respect to errors in the data. Various versions of the weighted majority algorithm are discussed, and error bounds for them that are closely related to the error bounds of the best algorithms of the pool are proved. >