Content deleted Content added
security issues aren't in the compression scheme as such and addressed elsewhere in the article + ‘better’ depends very much on what you measure and in what circumstances (e.g. if you store the file as is, rather than in e.g. a .7z archive, and your metric is file size, SCSU will almost always be ‘better’ than UTF-8) |
sources for Reuters + ‘Reuters is believed to use SCSU internally’ makes no sense as a thing to state: they'll use the same technology as everybody else; if an employee is consulting a web page most likely UTF-8, if sending mail maybe even ASCII and if they're running an SQL Server instance it'll use its mix of UTF16LE and SCSU + character encoding or not? the zig-zag + Czyborra's decompressor + MSDN page moved + why compress in the first place + source for general purpose compression |
||
Line 1:
{{expand German|date=August 2013}}
The '''Standard Compression Scheme for Unicode''' (SCSU)<ref>{{cite web |url=https://www.unicode.org/reports/tr6/ |title=UTS #6: Compression Scheme for Unicode |date=2005-05-06 |quote=SCSU defines a compact encoding, which is sometimes useful. However, Unicode text is much more commonly stored and transmitted in [[UTF-8]] which is less compact (except for [[ASCII]]), much simpler, and does not present any security issues. For longer texts, general-purpose compression is effective and common.|access-date=2008-06-13}}</ref> is a [[Unicode]] Technical Standard for reducing the number of [[byte]]s needed to represent Unicode text, especially if that text uses mostly characters from one or a small number of per-language character blocks. It does so by dynamically mapping values in the range 128–255 to offsets within particular blocks of 128 characters. The initial conditions of the encoder mean that existing strings in [[ASCII]] and [[ISO-8859-1]] that do not contain C0 control codes other than NULL TAB CR and LF can be treated as SCSU strings. Since most alphabets do reside in blocks of contiguous Unicode codepoints, texts that use small alphabets and either ASCII punctuation or punctuation that fits within the window for the main alphabet can be encoded at one byte per character (plus setup overhead, which for common languages is often only 1 byte), most other punctuation can be encoded at 2 bytes per symbol through non-locking shifts. SCSU can also switch to [[UTF-16]] internally to handle non-alphabetic languages.
==History
[[Reuters]] originally developed SCSU, then under the name RSCU for Reuters Compression Scheme for Unicode.<ref>https://unicode.org/iuc/iuc9/Friday2.html#b3</ref><ref>https://unicode.org/iuc/iuc10/program.html</ref><ref>https://unicode.org/reports/tr6-10.html</ref><ref name=Ewellic>https://web.archive.org/web/20190508064108/ewellic.org/compression.html#scsu</ref>
At first the Unicode Consortium considered it to be a character encoding,<ref>https://unicode.org/reports/tr17/tr17-2.html</ref> but in 1999 changed its mind: although it was still considered a transfer encoding syntax, for a while it was no longer considered a character encoding because different compressors might yield different outputs for the same text.<ref>https://unicode.org/reports/tr17/tr17-3.html#Transfer Encoding Syntax</ref> However, in 2004 this decision was reverted and now SCSU is considered a ''compressing'' character encoding scheme, as opposed to a simple or compound character encoding scheme.<ref>https://unicode.org/L2/L2004/04288-tr17-5d2.html#CharacterEncodingScheme</ref>
[[Symbian OS]], an operating system for mobile phones and other mobile devices, uses SCSU to serialize strings.▼
Roman Czyborra (of [[GNU Unifont]]) wrote a decompressor.<ref>https://czyborra.com/scsu/scsu.c</ref>
▲[[Symbian OS]], an operating system for mobile phones and other mobile devices, uses SCSU to serialize strings.
[[Microsoft SQL Server|SQL Server 2008 R2]] uses SCSU to compress Unicode values (there meaning from strings in [[UCS-2]] encoding) stored in ''nchar(n)'' and ''nvarchar(n)'' columns, achieving space savings between 15% and 50% (while [[UTF-8]] only has this 50% reduction for [[ASCII]] subset of Unicode), depending on the language of the data.<ref>{{cite web|url=http://msdn.microsoft.com/en-us/library/ee240835(SQL.105).aspx|title=Unicode Compression Implementation (SQL Server 2008 R2 Books Online)|access-date=2008-08-18}}</ref>▼
▲[[Microsoft SQL Server|SQL Server 2008 R2]] uses SCSU to compress Unicode values (there meaning from strings in [[UCS-2]] encoding) stored in ''nchar(n)'' and ''nvarchar(n)'' columns, achieving space savings between 15% and 50% (while [[UTF-8]] only has this 50% reduction for [[ASCII]] subset of Unicode), depending on the language of the data.<ref>{{cite web|url=
== Comparison with general-purpose plain text compression schemes ==▼
▲== Comparison with general-purpose plain text compression schemes ==
Because UTF-16 or UTF-8 text might occupy more space than its equivalent in pre-Unicode encodings did, one might want to use compression such as SCSU to mitigate this problem.<ref>https://unicode.org/versions/Unicode3.0.0/ch05.pdf</ref> In comparison with general-purpose compressors, it is not necessarily advantageous to use SCSU.<ref name=Ewellic/> Also, while it can be used as a text encoding, because of the stateful nature of the algorithm difficulties may arise when using it as an internal text representation since basic text operations become non-trivial.
Treated purely as a compression algorithm, SCSU is inferior to most commonly used general-purpose algorithms for texts of over a few kilobytes.
| |||