Comparison of Unicode encodings

This is an old revision of this page, as edited by Plugwash (talk | contribs) at 23:55, 15 August 2005. The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

This page compares Unicode encodings. Two situations are considered: eight-bit-clean environments and environments like Simple Mail Transfer Protocol that forbid use of byte values that have the high bit set. Originally such prohibitions were to allow for links that only used 7 data bits but they remain in the standards and so software must generate messages that comply with the restrictions. Standard Compression Scheme for Unicode and Binary Ordered Compression for Unicode are excluded from the comparison tables because it is difficult to simply quantify their size.

In summary

If space were the only consideration, UTF-32 would lose in almost every case since characters outside the basic multilingual plane are very rare, and one of the bytes of BMP characters in UTF-32 is always 0. For seven-bit environments UTF-7 clearly wins in terms of size over the combination of other Unicode encodings with quoted printable or base64. For eight-bit-clean environments things vary considerably depending on what code points are in the text.

In detail

The tables below list the number of bytes per code point for different Unicode ranges. Any additional comments needed are included in the table. The figures assume that overheads at the start and end of the block of text are negligible.

Eight-bit environments

Code range (hexadecimal) UTF-8 UTF-16 UTF-32 GB18030
000000 - 00007F 1 2 4 1
000080 - 0007FF 2 2 4 2 for stuff inherited from GB2312/GBK (e.g.
most Chinese stuff) 4 for everything else.
000800 - 00FFFF 3 2 4
010000 - 10FFFF 4 4 4 4

Seven-bit environments

This table may not cover every special case and so should be used for estimation and comparison only. To accurately determine the size of text in an encoding, see the actual specifications.

code range (hexadecimal) UTF-7 UTF-8 quoted printable UTF-8 base64 UTF-16 quoted printable UTF-16 base64 UTF-32 quoted printable UTF-32 base64 GB18030 quoted printable GB18030 base64
000000 - 000032 same as 000080-00FFFFFF 3 1⅓ 6 2⅔ 12 5⅓ 3 1⅓
000033 - 00003C 1 for "direct characters" and possibly "optional direct characters" (depending on the encoder setting) 2 for +, otherwise same as 000080-00FFFFFF 1 1⅓ 4 2⅔ 10 5⅓ 1 1⅓
00003D (equals sign) 3 1⅓ 6 2⅔ 12 5⅓ 3 1⅓
00003E - 00007E 1 1⅓ 4 2⅔ 10 5⅓ 1 1⅓
00007F 5 for an isolated case inside a run of single byte characters. For runs 2⅔ per character plus padding to make it a whole number of bytes plus two to start and finish the run 3 1⅓ 6 2⅔ 12 5⅓ 3 1⅓
000080 - 0007FF 6 2⅔ 2-6 depending on if the byte values need to be escaped 2⅔ 8-12 depending on if the final two byte values need to be escaped 5⅓ 4-6 for stuff inherited from GB2312/GBK (e.g.
most Chinese stuff) 6-10 for everything else.
2⅔ for stuff inherited from GB2312/GBK (e.g.
most Chinese stuff) 5⅓ for everything else.
000800 - 00FFFF 9 4 2⅔ 5⅓
010000 - 10FFFF same as two characters from above 12 5⅓ 8-12 depending on if the low bytes of the surrogates need to be escaped. 5⅓ 5⅓ 6-10 5⅓