Optimal estimation

Abstract: This paper deals with arbitrarily distributed finite-power input signals observed through an additive Gaussian noise channel. It shows a new formula that connects the input-output mutual information and the minimum mean-square error (MMSE) achievable by optimal estimation of the input given the output. That is, the derivative of the mutual information (nats) with respect to the signal-to-noise ratio (SNR) is equal to half the MMSE, regardless of the input statistics. This relationship holds for both scalar and vector signals, as well as for discrete-time and continuous-time noncausal MMSE estimation. This fundamental information-theoretic result has an unexpected consequence in continuous-time nonlinear estimation: For any input signal with finite power, the causal filtering MMSE achieved at SNR is equal to the average value of the noncausal smoothing MMSE achieved with a channel whose SNR is chosen uniformly distributed between 0 and SNR.

Abstract: Properties of maximum likelihood estimates of cointegrated systems are studied. Alternative formulations are considered, including a new triangular system error correction mechanism. We demonstrate that full system maximum likelihood brings the problem of inference within the family covered by the locally asymptotically mixed normal asymptotic theory, provided all unit roots have been eliminated by specification and data transformation. Methodological issues provide a major focus of the paper. Our results favor use of full system estimation in error correction mechanisms or subsystem methods that are asymptotically equivalent. They also point to disadvantages in the use of unrestricted VAR's formulated in levels and of certain single equation approaches to estimation of error correction mechanisms. Copyright 1991 by The Econometric Society.

Abstract: This paper deals with arbitrarily distributed finite-power input signals observed through an additive Gaussian noise channel. It shows a new formula that connects the input-output mutual information and the minimum mean-square error (MMSE) achievable by optimal estimation of the input given the output. That is, the derivative of the mutual information (nats) with respect to the signal-to-noise ratio (SNR) is equal to half the MMSE, regardless of the input statistics. This relationship holds for both scalar and vector signals, as well as for discrete-time and continuous-time noncausal MMSE estimation. This fundamental information-theoretic result has an unexpected consequence in continuous-time nonlinear estimation: For any input signal with finite power, the causal filtering MMSE achieved at SNR is equal to the average value of the noncausal smoothing MMSE achieved with a channel whose signal-to-noise ratio is chosen uniformly distributed between 0 and SNR.