Content deleted Content added
→Character duplication: Duplicate characters in Unicode |
m →Errors due to normalization differences: HTTP to HTTPS for SourceForge |
||
| (12 intermediate revisions by 7 users not shown) | |||
Line 1:
{{Short description|Aspect of the Unicode
{{Refimprove|date=November 2014}}
'''Unicode equivalence''' is the specification by the [[Unicode]] [[character (computing)|character]] encoding standard that some sequences of [[code point]]s represent essentially the same character. This feature was introduced in the standard to allow compatibility with
[[Unicode]] provides two such notions, [[canonical form|canonical]] equivalence and compatibility. [[Code point]] sequences that are defined as '''canonically equivalent''' are assumed to have the same appearance and meaning when printed or displayed. For example, the code point
Sequences that are defined as '''compatible''' are assumed to have possibly distinct appearances, but the same meaning in some contexts. Thus, for example, the code point U+FB00 (the [[typographic ligature]] "ff") is defined to be compatible—but not canonically equivalent—to the sequence U+0066 U+0066 (two Latin "f" letters). Compatible sequences may be treated the same way in some applications (such as [[sorting]] and [[index (database)|index]]ing), but not in others; and may be substituted for each other in some situations, but not in others. Sequences that are canonically equivalent are also compatible, but the opposite is not necessarily true.
Line 13:
===Character duplication===
{{Main|Duplicate characters in Unicode}}
For compatibility or other reasons, Unicode sometimes assigns two different code points to entities that are essentially the same character. For example, the
===Combining and precomposed characters===
For consistency with some older standards, Unicode provides single code points for many characters that could be viewed as modified forms of other characters (such as U+00F1 for "ñ" or U+00C5 for "Å") or as combinations of two or more characters (such as U+FB00 for the ligature "ff" or U+0132 for the [[Dutch alphabet|Dutch letter]] "[[IJ (digraph)|
For consistency with other standards, and for greater flexibility, Unicode also provides codes for many elements that are not used on their own, but are meant instead to modify or combine with a preceding [[base character]]. Examples of these [[combining character]]s are
In the context of Unicode, '''character composition''' is the process of replacing the code points of a base letter followed by one or more combining characters into a single [[precomposed character]]; and '''character decomposition''' is the opposite process.
Line 26:
====Example====
{| class="wikitable" align="center" width="50%" style="text-align: center;"
|+ |''Amélie'' with its two canonically equivalent [[Unicode]] forms ([[#Normal_forms|NFC and NFD]])
|- style="background-color:#ffeaea"
! style="width: 10em;" | NFC character
Line 42:
===Typographical non-interaction===
Some scripts regularly use multiple combining marks that do not, in general, interact typographically, and do not have precomposed characters for the combinations. Pairs of such non-interacting marks can be stored in either order. These alternative sequences are, in general, canonically equivalent. The rules that define their sequencing in the canonical form also define whether they are considered to interact.
===Typographic conventions===
Unicode provides code points for some characters or groups of characters which are modified only for aesthetic reasons (such as [[Typographic ligature|ligatures]], the [[half-width katakana]] characters, or the [[full-width]] Latin letters for use in Japanese texts), or to add new semantics without losing the original one (such as digits in [[subscript]] or [[superscript]] positions, or the circled digits (such as "①") inherited from some Japanese fonts). Such a sequence is considered compatible with the sequence of original (individual and unmodified) characters, for the benefit of applications where the appearance and added semantics are not relevant. However, the two sequences are not declared canonically equivalent, since the distinction has some semantic value and affects the rendering of the text.
===Encoding errors===
Line 57:
Unicode provides standard normalization algorithms that produce a unique (normal) code point sequence for all sequences that are equivalent; the equivalence criteria can be either canonical (NF) or compatibility (NFK). Since one can arbitrarily choose the [[representative (mathematics)|representative]] element of an [[equivalence class]], multiple canonical forms are possible for each equivalence criterion. Unicode provides two normal forms that are semantically meaningful for each of the two compatibility criteria: the composed forms NFC and NFKC, and the decomposed forms NFD and NFKD. Both the composed and decomposed forms impose a '''canonical ordering''' on the code point sequence, which is necessary for the normal forms to be unique.
In order to compare or search Unicode strings, software can use either composed or decomposed forms; this choice does not matter as long as it is the same for all strings involved in a search, comparison, etc. On the other hand, the choice of equivalence criteria can affect search results. For instance, some [[typographic ligature]]s like U+FB03 ({{char|ffi}}), [[Roman numerals]] like U+2168 ({{char|Ⅸ}}) and even [[Unicode subscripts and superscripts|subscripts and superscripts]], e.g. U+2075 ({{char|⁵}}) have their own Unicode code points. Canonical normalization (NF) does not affect any of these, but compatibility normalization (NFK) will decompose the ffi ligature into the constituent letters, so a search for U+0066 ({{char|f}}) as substring would succeed in an NFKC normalization of U+FB03 but not in NFC normalization of U+FB03. Likewise when searching for the Latin letter {{char|I}} (U+0049) in the precomposed Roman numeral {{char|Ⅸ}} (U+2168). Similarly, the superscript
Transforming superscripts into baseline equivalents may not be appropriate, however, for [[rich text]] software, because the superscript information is lost in the process. To allow for this distinction, the Unicode character database contains '''compatibility formatting tags''' that provide additional details on the compatibility transformation.<ref>{{cite web|url=https://www.unicode.org/reports/tr44/#Character_Decomposition_Mappings|title=UAX #44: Unicode Character Database|publisher=Unicode.org|access-date=20 November 2014}}</ref> In the case of typographic ligatures, this tag is simply <code><compat></code>, while for the superscript it is <code><super></code>. Rich text standards like [[HTML]] take into account the compatibility tags. For instance, HTML uses its own markup to position a U+0035 in a superscript position.<ref>{{cite web|url=http://unicode.org/reports/tr20/tr20-2.html#Compatibility|title=Unicode in XML and other Markup Languages|publisher=Unicode.org|access-date=20 November 2014}}</ref>
===Normal forms===
Line 82:
The normal forms are not [[closure (mathematics)|closed]] under string [[concatenation]].<ref> Per [http://www.unicode.org/faq/normalization.html#5 What should be done about concatenation]</ref> For defective Unicode strings starting with a Hangul vowel or trailing [[Hangul Jamo (Unicode block)|conjoining jamo]], concatenation can break Composition.
However, they are not [[injective function|injective]] (they map different original glyphs and sequences to the same normalized sequence) and thus also not [[bijection|bijective]] (
A single character (other than a Hangul syllable block) that will get replaced by another under normalization can be identified in the Unicode tables for having a non-empty compatibility field but lacking a compatibility tag.
Line 96:
==Errors due to normalization differences==
When two applications share Unicode data, but normalize them differently, errors and data loss can result. In one specific instance, [[OS X]] normalized Unicode filenames sent from the [[Netatalk]] and [[Samba (software)|Samba]] file- and printer-sharing software.
==See also==
| |||