NSVQ: Noise Substitution in Vector Quantization for Machine Learning

Research output: Contribution to journalArticleScientificpeer-review

7 Citations (Scopus)
452 Downloads (Pure)

Abstract

Machine learning algorithms have been shown to be highly effective in solving optimization problems in a wide range of applications. Such algorithms typically use gradient descent with backprop- agation and the chain rule. Hence, the backpropagation fails if intermediate gradients are zero for some functions in the computational graph, because it causes the gradients to collapse when multiplying with zero. Vector quantization is one of those challenging functions for machine learning algorithms, since it is a piece-wise constant function and its gradient is zero almost everywhere. A typical solution is to apply the straight through estimator which simply copies the gradients over the vector quantization function in the backpropagation. Other solutions are based on smooth or stochastic approximation. This study proposes a vector quantization technique called NSVQ, which approximates the vector quantization behavior by substituting a multiplicative noise so that it can be used for machine learning problems. Specifically, the vector quantization error is replaced by product of the original error and a normalized noise vector, the samples of which are drawn from a zero-mean, unit-variance normal distribution. We test our proposed NSVQ in three scenarios with various types of applications. Based on the experiments, the proposed NSVQ achieves more accuracy and faster convergence in comparison to the straight through estimator, exponential moving averages, and the MiniBatchKmeans approaches.
Original languageEnglish
Pages (from-to)13598 - 13610
Number of pages13
JournalIEEE Access
Volume10
Early online dateJan 2022
DOIs
Publication statusPublished - 2022
MoE publication typeA1 Journal article-refereed

Keywords

  • backpropagation
  • gradient collapse
  • gradient propagation
  • noise substitution
  • vector quantization

Fingerprint

Dive into the research topics of 'NSVQ: Noise Substitution in Vector Quantization for Machine Learning'. Together they form a unique fingerprint.

Cite this