Complete Characterization of Gorbunov and Pinsker Nonanticipatory Epsilon Entropy of Multivariate Gaussian Sources: Structural Properties

Charalambos D. Charalambous, Themistoklis Charalambous, Christos Kourtellaris, Jan H. van Schuppen

Research output: Contribution to journalArticleScientificpeer-review

Abstract

This paper derives the optimal test channel distribution and the complete characterization of the classical Gorbunov and Pinsker [2], [3] nonanticipatory epsilon entropy of multivariate Gaussian Markov sources with square-error fidelity, which remained an open problem since 1974. The paper also formulates a state dependent nonanticipatory epsilon entropy, in which past reproductions are available to the decoder and not to the encoder, the test channel is specified with respect to an auxiliary (state) random process, and the reproduction process is a causal function of past reproduction and the auxiliary random process. This variation is analogous to the Wyner-Ziv and Wyner [4], [5] rate distortion function (RDF), of memoryless sources. It is shown that the operational rate of zero-delay codes, with past reproductions available to the decoder but not to the encoder is bounded below by the state dependent nonanticipatory epsilon entropy rate. For the case of multivariate Gaussian Markov sources with square-error fidelity, the optimal test channel distribution and the complete characterization of the state dependent of nonanticipatory epsilon entropy are derived, and also shown that that the two nonanticipatory epsilon entropies coincide. The derivations are new; they are based on structural properties of the stochastic realizations of the reproduction process that induce the optimal test channel distributions. They are derived using, achievable lower bounds on information theoretic measures, properties of mean-square estimation theory, Hadamard’s inequality, and canonical correlation coefficients of a tuple of multivariate jointly Gaussian random processes. Applications of the nonanticipatory epsilon entropy and its state dependent variation are discussed to the areas of control of unstable Gaussian systems over limited memory channels, design of causal estimators for Gaussian Markov sources with a fidelity criterion, computation of the rate loss of causal and zero-delay codes of Gaussian Markov sources with respect to non-causal codes.

Original languageEnglish
Number of pages25
JournalIEEE Transactions on Information Theory
DOIs
Publication statusE-pub ahead of print - 13 Nov 2021
MoE publication typeA1 Journal article-refereed

Keywords

  • Channel estimation
  • Codes
  • Decoding
  • Distortion
  • Entropy
  • Markov processes
  • multivariate Gaussian processes
  • Nonanticipatory epsilon entropy
  • Resource description framework
  • square-error fidelity criterion

Fingerprint

Dive into the research topics of 'Complete Characterization of Gorbunov and Pinsker Nonanticipatory Epsilon Entropy of Multivariate Gaussian Sources: Structural Properties'. Together they form a unique fingerprint.

Cite this