Optimal Quantiser Performance with a Small Dither

Abstract

A quantiser is a non-smooth function and no inverse function exist that can be applied to correct for the error it introduces. Applying a dither signal and averaging to a quantiser produces a smooth image for which an inverse function does exist. This article describes methods for minimising the error after inverse compensation. We show that there is an optimal dither variance that minimises the error after inversion. Simple rules for choosing the optimal dither variance are presented. The error after inversion can be made arbitrarily small by increasing the averaging length. This can be done by oversampling the signal by the same factor as the number of averages. Quantisation of a dither signal with a continuous probability distribution, produces a discrete probability mass function. We discuss a method for recovering an unknown continuous probability distribution from the empirical discrete probability mass function of the quantised dither signal. This enables inverse compensation in systems where exact control of the dither signal is not possible, as inverse compensation requires information about the continuous probability distribution of the dither signal and the step-size of the quantiser.