Text segmentation: Difference between revisions

Revision as of 17:21, 4 May 2006

Revision as of 09:23, 28 May 2006

Line 1:

'''Written text segmentation''' is the process of divingdividing written text into [[word]]s or other similar meaningful units. The term applies to [[human mind|mental]] processes used by humans when reading text, and to artificial processes implemented in [[computers]], which are the subject [[natural language processing]].

The problem is relatively trivial for written languages that have explicit word boudary markers, such as the word spaces of written [[English language|English]] or the distinctive initial, medial and final letter shapes of [[Arabic language|Arabic]]. When such clues are not consistently available, the task often requires fairly non-trivial techniques, such as statistical decision-making, large dictionaries, as well as consideration of syntactic and semantic constraints.