Syntactic Transformations on Distributed Representations

TL;DR: This paper explores the possibility of moving beyond implementation by exploiting holistic structure-sensitive operations on distributed representations using Pollack’s Recursive Auto-Associative Memory, demonstrating that the implicit structure present in these representations can be used for a kind of structure- sensitive processing unique to the connectionist domain.

Abstract: There has been much interest in the possibility of connectionist models whose representations can be endowed with compositional structure, and a variety of such models have been proposed. These models typically use distributed representations that arise from the functional composition of constituent parts. Functional composition and decomposition alone, however, yield only an implementation of classical symbolic theories. This paper explores the possibility of moving beyond implementation by exploiting holistic structure-sensitive operations on distributed representations. An experiment is performed using Pollack’s Recursive Auto-Associative Memory (RAAM). RAAM is used to construct distributed representations of syntactically structured sentences. A feed-forward network is then trained to operate directly on these representations, modeling syntactic transformations of the represented sentences. Successful training and generalization is obtained, demonstrating that the implicit structure present in these representations can be used for a kind of structure-sensitive processing unique to the connectionist domain.