Standard Compression Scheme for Unicode: Difference between revisions

Revision as of 05:56, 24 March 2006

Latest revision as of 09:33, 7 May 2025

(81 intermediate revisions by 49 users not shown)

Line 1:

{{TableShort description|Unicode Technical Standard}}

{{expand German|date=August 2013}}

The '''Standard Compression Scheme for Unicode''' (SCSU)<ref>{{cite web |url=https://www.unicode.org/reports/tr6/ |title=UTS #6: Compression Scheme for Unicode |date=2005-05-06 |quote=SCSU defines a compact encoding, which is sometimes useful. However, Unicode text is much more commonly stored and transmitted in [[UTF-8]] which is less compact (except for [[ASCII]]), much simpler, and does not present any security issues. For longer texts, general-purpose compression is effective and common.|access-date=2008-06-13}}</ref> is a [[Unicode]] Technical Standard for reducing the number of [[byte]]s needed to represent Unicode text, especially if that text uses mostly characters from one or a small number of per-language character blocks. It does so by dynamically mapping values in the range 128&ndash;255 to offsets within particular blocks of 128 characters. The initial conditions of the encoder mean that existing strings in [[ASCII]] and [[ISO-8859-1]] that do not contain C0 control codes other than NULL TAB CR and LF can be treated as SCSU strings. Since most alphabets do reside in blocks of contiguous Unicode codepoints, texts that use small alphabets and either ASCII punctuation or punctuation that fits within the window for the main alphabet can be encoded at one byte per character (plus setup overhead, which for common languages is often only 1 byte), most other punctuation can be encoded at 2 bytes per symbol through non-locking shifts. SCSU can also switch to [[UTF-16]] internally to handle non-alphabetic languages.

==History & use ==

The '''Standard Compression Scheme for Unicode''' (SCSU) is a [[Unicode]] Technical Standard to reduce the number of [[byte]]s needed to represent text, especially if that text uses mostly characters from a small number of Unicode blocks. It does so by dynamically mapping the values in the range 128&ndash;255 to blocks of 128 characters. Since most alphabets are in 128 contiguous Unicode codepoints, this allows for 1 byte per character (plus overhead) encoding for many text files. SCSU will also switch to UTF-16 internally to handle non-alphabetic languages.

[[Reuters]] originally developed SCSU, then under the name RCSU for Reuters Compression Scheme for Unicode.<ref>{{Cite web|url=https://unicode.org/iuc/iuc9/Friday2.html#b3|title = Ninth International Unicode Conference - Friday - Track B}}</ref><ref>{{Cite web|url=https://unicode.org/iuc/iuc10/program.html|title=Tenth International Unicode Conference - Conference Program}}</ref><ref>{{Cite web|url=https://unicode.org/reports/tr6-10.html|title=Compression Scheme for Unicode}}</ref><ref name=Ewellic>{{Cite web|url=https://www.unicode.org/notes/tn14/UnicodeCompression.pdf|title = A survey of Unicode compression}}</ref>

At first the Unicode Consortium considered it to be a character encoding,<ref>{{Cite web|url=https://unicode.org/reports/tr17/tr17-2.html|title = UTR#17: Character Encoding Model}}</ref> but in 1999 changed its mind: although it was still considered a transfer encoding syntax, for a while it was no longer considered a character encoding because different compressors might yield different outputs for the same text.<ref>https://unicode.org/reports/tr17/tr17-3.html#Transfer Encoding Syntax</ref> However, in 2004 this decision was reverted and now SCSU is considered a ''compressing'' character encoding scheme, as opposed to a simple or compound character encoding scheme.<ref>{{Cite web|url=https://unicode.org/L2/L2004/04288-tr17-5d2.html#CharacterEncodingScheme|title=UTR#17: Character Encoding Model|date=2004-07-14}}</ref><ref>{{Cite web |title=UTR#17: Unicode Character Encoding Model |url=https://unicode.org/reports/tr17/ |access-date=2023-11-14 |website=unicode.org}}</ref>

SCSU is not a resounding success. Few places need to compress enough Unicode text to make it worth using a poorly supported compression scheme. Treated purely as a compression format, it's inferior to most commonly used compression programs for texts over a few kilobytes. It can be used as a text encoding, but it's very hard to handle internally, and the percentage savings between SCSU and [[UTF-16]] or [[UTF-8]] drops after external compression, dramatically in the case of [[bzip2]] and other modern compression schemes. It does have the advantage that SCSU can compress texts that are only a few characters long, whereas most full-scale compressors need a few kilobytes of data to break even against their own overhead.

Roman Czyborra (of [[GNU Unifont]]) wrote a decompressor.<ref>{{Cite web| title=This is a deflator to UTF-8 output for input compressed in SCSU | url=https://czyborra.com/scsu/scsu.c | archive-url=https://web.archive.org/web/19990908230458/http://czyborra.com:80/scsu/scsu.c | archive-date=1999-09-08}}</ref> The IBM-contributed decompressor is found in [[International Components for Unicode]], along with a compressor written in Java.<ref>{{Cite web|url=https://github.com/unicode-org/icu/blob/3f043c7693e20c8cded76035918dad104e7256e3/icu4j/main/classes/charset/src/com/ibm/icu/charset/CharsetSCSU.java|title = International Components for Unicode|website = [[GitHub]]|date = 22 October 2021}}</ref> Simpler reference codecs are available as attachments to TR6.

[[Reuters]], the organization that floated the first draft of SCSU, is believed to use SCSU internally.

[[Symbian OS]], an operating system for mobile phones and other mobile devices, uses SCSU to serialize strings.

==External links==

* [http://www.unicode.org/reports/tr6/ UTS #6: Compression Scheme for Unicode]

*[http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id= SIL's freeware fonts, editors and documentation]

[[Microsoft SQL Server|SQL Server 2008 R2]] uses SCSU to compress Unicode values (there meaning from strings in [[UCS-2]] encoding) stored in ''nchar(n)'' and ''nvarchar(n)'' columns, achieving space savings between 15% and 50% (while [[UTF-8]] only has this 50% reduction for [[ASCII]] subset of Unicode), depending on the language of the data.<ref>{{cite web|url=https://docs.microsoft.com/en-us/previous-versions/sql/sql-server-2008-r2/ee240835(v=sql.105)|title=Unicode Compression Implementation (SQL Server 2008 R2 Books Online)|access-date=2008-08-18}}</ref>

== The scheme ==

[[Category:Unicode]]

The following sections briefly describe the anatomy of a compressed SCSU stream. For a full description (matching that of a decompressor), see the UTS #6 document.

=== Encoding modes ===

SCSU starts in the single-byte mode, which uses the compressed Window encoding. There exist commands to switch to a UTF-16BE "Unicode" mode, and to switch to the single-byte mode from that mode.

=== Window encoding ===

The core of SCSU lies in the windows for which the meanings of bytes 0x80-0xff are defined. There are eight static windows for simpler scripts and punctuation, and 6 types of dynamic windows (plus "half Unicode block" windows and custom windows for the supplementary planes) for scripts making use of more characters.

Both simple and dynamic windows are selected by special command characters. For individual characters that do not fit into the current block, command characters for quoting are provided.

== Comparison with general-purpose plain text compression schemes ==

Because UTF-16 or UTF-8 text might occupy more space than its equivalent in pre-Unicode encodings did, one might want to use compression such as SCSU to mitigate this problem.<ref>{{Cite web| title=Implementation Guidelines | url=https://unicode.org/versions/Unicode3.0.0/ch05.pdf | archive-url=https://web.archive.org/web/20150730234318/http://www.unicode.org/versions/Unicode3.0.0/ch05.pdf | archive-date=2015-07-30}}</ref> In comparison with general-purpose compressors, it is not necessarily advantageous to use SCSU.<ref name=Ewellic/> Also, while it can be used as a text encoding, because of the stateful nature of the algorithm difficulties may arise when using it as an internal text representation since basic text operations become non-trivial.

Treated purely as a compression algorithm, SCSU is inferior to most commonly used general-purpose algorithms for texts of over a few kilobytes.

SCSU does have the advantage that it can usefully compress texts that are only a few characters long, whereas most full-scale compressors need hundreds of bytes of data to break even against their own overhead. In [[Symbian OS]], SCSU is used even for Clipboard operations, e.g. Cut, Copy & Paste of small strings of text.

== In HTML ==

Supporting SCSU in [[HTML]] documents is prohibited by the [[W3C]]<ref>{{Cite web |url=https://www.w3.org/TR/html51/syntax.html#character-encodings |title=8.2.2.3. Character encodings |website=HTML 5.1 Standard |publisher=W3C}}</ref><ref>{{Cite web |url=https://www.w3.org/TR/html5/syntax.html#character-encodings |title=8.2.2.3. Character encodings |website=HTML 5 Standard |publisher=W3C}}</ref> and [[WHATWG]]<ref>{{Cite web |url=https://html.spec.whatwg.org/multipage/parsing.html#character-encodings |title=12.2.3.3 Character encodings |website=HTML Living Standard |publisher=WHATWG}}</ref> HTML standards because HTML wasn't designed with non-ASCII-compatible encodings in mind. In the past, [[cross-site scripting]] vulnerabilities due to browsers' poor handling of such encodings have been demonstrated.<ref>{{Cite web |url=https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta |title=&lt;meta> - HTML |website=MDN Web Docs |publisher=Mozilla|archiveurl=https://web.archive.org/web/20181003090922/https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta|archivedate=3 October 2018}}</ref>

== See also ==

* [[Binary Ordered Compression for Unicode]] (BOCU-1)

* [[International Components for Unicode]] A library that can convert between SCSU and other Unicode encodings

== References ==

<references />

{{Unicode navigation}}

{{Character encoding}}

{{DEFAULTSORT:Standard Compression Scheme For Unicode}}

[[Category:Unicode Transformation Formats]]