Factorial code

Abstract: We investigate the consequences of maximizing information transfer in a simple neural network (one input layer, one output layer), focusing on the case of nonlinear transfer functions. We assume that both receptive fields (synaptic efficacies) and transfer functions can be adapted to the environment. The main result is that, for bounded and invertible transfer functions, in the case of a vanishing additive output noise, and no input noise, maximization of information (Linsker's infomax principle) leads to a factorial code-hence to the same solution as required by the redundancy-reduction principle of Barlow. We also show that this result is valid for linear and, more generally, unbounded, transfer functions, provided optimization is performed under an additive constraint, i.e. which can be written as a sum of terms, each one being specific to one output neuron. Finally, we study the effect of a non-zero input noise. We find that, to first order in the input noise, assumed to be small in comparison with th...

Abstract: Factorial learning, finding a statistically independent representation of a sensory “image”---a factorial code---is applied here to solve multilayer supervised learning problems that have traditionally required backpropagation. This lends support to Barlow's argument for factorial sensory processing, by demonstrating how it can solve actual pattern recognition problems. Two techniques for supervised factorial learning are explored, one of which gives a novel distributed solution requiring only positive examples. Also, a new nonlinear technique for factorial learning is introduced that uses neural networks based on almost reversible cellular automata. Due to the special functional connectivity of these networks---which resemble some biological microcircuits---learning requires only simple local algorithms. Also, supervised factorial learning is shown to be a viable alternative to backpropagation. One significant advantage is the existence of a measure for the performance of intermediate learning stages.

Abstract: Independent component analysis (ICA) is a statistical method for transforming an observable multi-dimensional random vector into components that are as statistically independent as possible from each other. Usually, the ICA framework assumes a model according to which the observations are generated (such as a linear transformation with additive noise). ICA over finite fields is a special case of ICA in which both the observations and the independent components are over a finite alphabet. In this paper, we consider a generalization of this framework in which an observation vector is decomposed to its independent components (as much as possible) with no prior assumption on the way it was generated. This generalization is also known as Barlow’s minimal redundancy representation problem and is considered an open problem. We propose several theorems and show that this hard problem can be accurately solved with a branch and bound search tree algorithm, or tightly approximated with a series of linear problems. Our contribution provides the first efficient set of solutions to Barlow’s problem. The minimal redundancy representation (also known as factorial code) has many applications, mainly in the fields of neural networks and deep learning. The binary ICA is also shown to have applications in several domains, including medical diagnosis, multi-cluster assignment, network tomography, and internet resource management. In this paper, we show that this formulation further applies to multiple disciplines in source coding, such as predictive coding, distributed source coding, and coding of large alphabet sources.

Abstract: In a number of practical scenarios, such as video conferencing and visual human/computer interaction, objects that belong to a well defined class are segmented, normalized, and encoded, after which they are stored and/or transmitted, and subsequently reconstructed. The Karhunen-Loeve transform (KLT) optimally concentrates the signal power in a relatively small number of uncorrelated coefficients. Nevertheless, it implicitly assumes a multidimensional Gaussian probability model, which is typically not correct. Here we show that, in the context of video sequences of human heads, the segmentation and normalization steps result in partial symmetries which force the KLT coefficients to lie close to low-dimensional manifolds in suitably chosen high-dimensional KLT subspaces. We show how this fact can be used to track the faces robustly, and to estimate their pose. We use vector quantization to discover those manifolds, and to build a factorial code that has a substantially lower dimensionality than KLT.