In digital signal processing, quantization is the process of approximating a continuous signal (or a very large set of possible discrete values) by a relatively-small set of discrete symbols or integer values. A common application of quantization is the conversion of an analog signal to a digital one via analog-to-digital conversion. The simplest and best-known form of quantization is referred to as scalar quantization, since it operates on scalar (as opposed to multi-dimensional vector) input data. In general, a scalar quantization operator can be represented as
where x is a real number, i = round(f(x)) is an integer, and f(x) and g(i) are arbitrary real-valued functions. The integer value i = round(f(x)) is the representation that is typically stored or transmitted, and then the final interpretation is constructed using g(i) when the data is later interpreted. The integer value i is sometimes referred to as the quantization index.
In computer audio and most other applications, a method known as uniform quantization is the most common. If x is a real valued number between -1 and 1, a uniform quantization operator that uses M bits of two's complement precision to represent each quantization index can be expressed as
- .
In this case the f(x) and g(i) operators are just multiplying scale factors (one multiplier being the inverse of the other). The value is often referred to as the quantization step size. Using this quantization law and assuming that quantization noise is approximately uniformly distributed over the quantization step size (an assumption typically accurate for rapidly varying x or high M) and assuming that the input signal x to be quantized is approximately uniformly distributed over the entire interval from -1 to 1, the signal to noise ratio of the quantization can be computed as
- .
From this equation, it is often said that the SNR is approximately 6 dB per bit.
In digital telephony, two popular quantization schemes are the 'A-law' (dominant in Europe) and 'µ-law' (dominant in North America and Japan). These schemes map discrete analog values to an 8 bit scale that is nearly linear for small values and then increases logarithmically as amplitude grows. Because the human ear's perception of loudness is roughly logarithmic, this provides a higher signal to noise ratio over the range of audible sound intensities for a given number of bits.