Content deleted Content added
editing a link |
|||
Line 7:
== Segmentation problems ==
=== Word segmentation ===
Word segmentation is the problem of dividing a string of written language into its component [[word]]s.
In [[English language|English]] and many other languages using some form of the [[Latin alphabet]], the [[Space (punctuation)|space]] is a good approximation of a [[word divider]] (word [[delimiter]]). (Some examples where the space character alone may not be sufficient include contractions like ''can't'' for ''can not''.)
However the equivalent to this character is not found in all written scripts, and without it word segmentation is a difficult problem. Languages which do not have a trivial word segmentation process include [[Chinese language|Chinese]], [[Japanese language|Japanese]], where [[sentences]] but not words are delimited, [[Thai language|Thai]] and [[Lao language|Lao]], where phrases and sentences but not words are delimited, and [[Vietnamese language|Vietnamese]], where syllables but not words are delimited.
Line 18:
The [[Unicode Consortium]] has published a [http://www.unicode.org/reports/tr29/ Standard Annex] on Text Segmentation, exploring the issues of segmentation in multiscript texts.
'''Word splitting''' is the process of [[parsing]] [[concatenated]] text (i.e. text that contains no spaces or other word separators) to infer where word breaks exist.
Word splitting may also refer to the process of [[Syllabification|hyphenation]].
| |||