NSVQ: Noise Substitution in Vector Quantization for Machine Learning

Tutkimustuotos: LehtiartikkeliArticleScientificvertaisarvioitu

312 Lataukset (Pure)


Machine learning algorithms have been shown to be highly effective in solving optimization problems in a wide range of applications. Such algorithms typically use gradient descent with backprop- agation and the chain rule. Hence, the backpropagation fails if intermediate gradients are zero for some functions in the computational graph, because it causes the gradients to collapse when multiplying with zero. Vector quantization is one of those challenging functions for machine learning algorithms, since it is a piece-wise constant function and its gradient is zero almost everywhere. A typical solution is to apply the straight through estimator which simply copies the gradients over the vector quantization function in the backpropagation. Other solutions are based on smooth or stochastic approximation. This study proposes a vector quantization technique called NSVQ, which approximates the vector quantization behavior by substituting a multiplicative noise so that it can be used for machine learning problems. Specifically, the vector quantization error is replaced by product of the original error and a normalized noise vector, the samples of which are drawn from a zero-mean, unit-variance normal distribution. We test our proposed NSVQ in three scenarios with various types of applications. Based on the experiments, the proposed NSVQ achieves more accuracy and faster convergence in comparison to the straight through estimator, exponential moving averages, and the MiniBatchKmeans approaches.
Sivut13598 - 13610
JulkaisuIEEE Access
Varhainen verkossa julkaisun päivämäärätammik. 2022
DOI - pysyväislinkit
TilaJulkaistu - 2022
OKM-julkaisutyyppiA1 Julkaistu artikkeli, soviteltu


Sukella tutkimusaiheisiin 'NSVQ: Noise Substitution in Vector Quantization for Machine Learning'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä