In digital signal processing, quantization is the process of approximating a continuous range of values (or a very large set of possible discrete values) by a relatively-small set of discrete symbols or integer values. More specifically, a signal can be multi-dimensional and quantization need not be applied to all dimensions. A discrete signal need not necessarily be quantized (a pedantic point, but true nonetheless and can be a point of confusion). See ideal sampler.
A common use of quantization is in the conversion of a continuous signal into a discrete signal by sampling and then quantizing. Both of these steps are performed in analog-to-digital converters with the quantization level specified by a number of bits. A specific example would be compact disc (CD) audio which is sampled at 44,100 Hz and quantized with 16 bits (2 bytes) which can be one of 65,536 () possible values per sample.
The simplest and best-known form of quantization is referred to as scalar quantization, since it operates on scalar (as opposed to multi-dimensional vector) input data. In general, a scalar quantization operator can be represented as
where is a real number, is an integer, and and are arbitrary real-valued functions. The integer value is the representation that is typically stored or transmitted, and then the final interpretation is constructed using when the data is later interpreted. The integer value is sometimes referred to as the quantization index.
In computer audio and most other applications, a method known as uniform quantization is the most common. If is a real valued number between -1 and 1, a uniform quantization operator that uses M bits of precision to represent each quantization index can be expressed as
- .
In this case the and operators are just multiplying scale factors (one multiplier being the inverse of the other). The value is often referred to as the quantization step size. Using this quantization law and assuming that quantization noise is approximately uniformly distributed over the quantization step size (an assumption typically accurate for rapidly varying or high ) and assuming that the input signal to be quantized is approximately uniformly distributed over the entire interval from -1 to 1, the signal to noise ratio (SNR) of the quantization can be computed as
- .
From this equation, it is often said that the SNR is approximately 6 dB per bit.
In digital telephony, two popular quantization schemes are the 'A-law' (dominant in Europe) and 'µ-law' (dominant in North America and Japan). These schemes map discrete analog values to an 8-bit scale that is nearly linear for small values and then increases logarithmically as amplitude grows. Because the human ear's perception of loudness is roughly logarithmic, this provides a higher signal to noise ratio over the range of audible sound intensities for a given number of bits.
Compression
Quantization also plays a part in lossy data compression. One such lossy compression scheme is JPEG. During compression, the coefficients of the discrete cosine transform are quantized to facilitate the entropy encoding step. So by reducing the set of values (the post-quantized step of JPEG typically yields many zero values which be exploited to reduce the number of bits needed) by quantization, higher compression ratios can be achieved.