Smoothing Regularizers for Projective Basis Function Networks

TL;DR: The new regularizers are shown to yield better generalization errors than weight decay when the implicit assumptions in the latter are wrong and to enable the direct enforcement of smoothness without the need for costly Monte-Carlo integrations of S(W, m).

Abstract: Smoothing regularizers for radial basis functions have been studied extensively, but no general smoothing regularizers for projective basis junctions (PBFs), such as the widely-used sigmoidal PBFs, have heretofore been proposed. We derive new classes of algebraically-simple mth-order smoothing regularizers for networks of the form f(W, x) = Σi=1N Ujg [xT vj + Vjo] + uo, with general projective basis functions g[ċ]. These regularizers are: RG(W,m) = Σi=1N Uj2||vj||2m-1 Global Form RL(W,m) = Σi=1N uj2||vj||2m Local Form These regularizers bound the corresponding mth-order smoothing integral S(W,m) = ∫dD xω(x) ||∂m f(W,x)/∂xm||2, where W denotes all the network weights {uj, uo, vj, vo}, and Ω(x) is a weighting function on the D-dimensional input space. The global and local cases are distinguished by different choices of Ω(x). The simple algebraic forms R(W, m) enable the direct enforcement of smoothness without the need for costly Monte-Carlo integrations of S(W, m). The new regularizers are shown to yield better generalization errors than weight decay when the implicit assumptions in the latter are wrong. Unlike weight decay, the new regularizers distinguish between the roles of the input and output weights and capture the interactions between them.