Text normalization: Difference between revisions

Revision as of 20:08, 28 October 2007

Revision as of 00:21, 15 November 2007

Line 8:

* converting all letters to lower or upper case

* removing punctuation

* removing letters with accent marks and other diacritics from letters

* expanding abbreviations

* removing [[stopwords]] or "too common" words

While this may be done manually, and usually is in the case of ad hoc and personal documents, many [[programming language]]s support mechanisms which enable text normalization.