Text normalization: Difference between revisions

Revision as of 23:19, 7 November 2012

Latest revision as of 14:00, 14 November 2024

(27 intermediate revisions by 24 users not shown)

Line 1:

{{Short description|Process of transforming text into a single canonical form}}

{{Use American English|date=March 2021}}

{{Use mdy dates|date=March 2021}}

{{Distinguish|word normalization|Unicode normalization}}

'''Text normalization''' is the process of transforming [[writing|text]] into a single [[canonical form]] that it might not have had before. Normalizing text before storing or processing it allows for [[separation of concerns]], since input is guaranteed to be consistent before operations are performed on it. Text normalization requires being aware of what type of text is to be normalized and how it is to be processed afterwards; there is no all-purpose normalization procedure.<ref name="cs506">{{cite web

| title = CS506/606: Txt Nrmlztn

| author = [[Richard Sproat]] and Steven Bedrick

| date = September 2011

| accessdate = October 2, 2012

| url = http://www.cslucsee.ogi.edu/~sproatr/Courses/TextNorm/}}</ref>

== Applications==

Text normalization is frequently used when converting [[speech synthesis|text to speech]]. [[Number]]s, [[Calendar date|date]]s, [[acronym]]s, and [[abbreviation]]s are non-standard "words" that need to be pronounced differently depending on context.<ref name="sproate">Sproat, R.; Black, A.; Chen, S.; Kumar, S.; OstendorfkOstendorf, M.; Richards, C. (2001). "Normalization of non-standard words." ''Computer Speech and Language'' '''15'''; 287–333. [[Digital object identifier|doi]]:[httphttps://dx.doi.org/10.1006/csla.2001.0169 10.1006/csla.2001.0169].</ref> For example:

* "$200" would be pronounced as "two hundred dollars" in English, but as "lua selau tālā" in Samoan.<ref>{{cite web

Line 17 ⟶ 20:

| accessdate = October 2, 2012

| url = http://mylanguages.org/samoan_numbers.php}}</ref>

* "vi" could be pronounced as "[[viViolet (name)|vie]]e," "[[VioletVi (nametext editor)|vee]]," or "[[Roman numerals|the sixth]]" depending on the surrounding words.<ref name="msdn">{{cite web

| title = Text-to-Speech Engines Text Normalization

| work = MSDN

Line 27 ⟶ 30:

== Techniques ==

For simple, context-independent normalization, such as removing non-[[alphanumeric]] characters or [[diacritical marks]], [[regular expressions]] would suffice. For example, the [[sed]] script <ttcode>sed -&nbsp;&#8209;e &nbsp;"s/\s+/ &nbsp;/g" &nbsp;&nbsp;''inputfile''</ttcode> would normalize runs of [[whitespace character]]s into a single space. More complex normalization requires correspondingly complicated algorithms, including [[___domain knowledge]] of the language and vocabulary being normalized. Among other approaches, text normalization has been modeled as a problem of tokenizing and tagging streams of text<ref name="tagging">Zhu, C.; Tang, J.; Li, H.; Ng , H.; Zhao, T. (2007). "A Unified Tagging Approach to Text Normalization." ''Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics''; 688–695. [[Digital object identifier|doi]]:[http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.72.8138 10.1.1.72.8138].</ref> and as a special case of machine translation.<ref name="mt">Filip, G.; Krzysztof, J.; Agnieszka, W.; Mikołaj, W. (2006). [httphttps://www.proceedings2006.imcsitannals-csis.org/proceedings/2006/pliks/202.pdf "Text Normalization as a Special Case of Machine Translation."] ''Proceedings of the International Multiconference on Computer Science and Information Technology'' '''1'''; 51–56.</ref><ref name="sm">Mosquera, A.; Lloret, E.; Moreda, P. (2012). [http://lrec.elra.info/proceedings/lrec2012/workshops/25.NLP4ITA-Proceedings.pdf#page=14 "Towards Facilitating the Accessibility of Web 2.0 Texts through Text Normalisation"] ''Proceedings of the LREC workshop: Natural Language Processing for Improving Textual Accessibility (NLP4ITA)''; 9-14</ref>

==Textual References scholarship==

In the field of [[textual scholarship]] and the editing of historic texts, the term "normalization" implies a degree of modernization and standardization – for example in the extension of [[scribal abbreviation]]s and the transliteration of the archaic [[glyph]]s typically found in manuscript and early printed sources. A ''normalized edition'' is therefore distinguished from a ''[[Diplomatics#Diplomatic editions and transcription|diplomatic edition]]'' (or ''semi-diplomatic edition''), in which some attempt is made to preserve these features. The aim is to strike an appropriate balance between, on the one hand, rigorous fidelity to the source text (including, for example, the preservation of enigmatic and ambiguous elements); and, on the other, producing a new text that will be comprehensible and accessible to the modern reader. The extent of normalization is therefore at the discretion of the editor, and will vary. Some editors, for example, choose to modernize archaic spellings and punctuation, but others do not.<ref>{{cite book |first=P. D. A. |last=Harvey |title=Editing Historical Records |publisher=British Library |place=London |year=2001 |isbn=0-7123-4684-8 |pages=40–46 }}</ref>

{{Reflist}}

== See also ==

* {{annotated link|Automated paraphrasing}}

* [[{{annotated link|Canonicalization]]}}

* {{annotated link|Text simplification}}

* [[{{annotated link|Unicode equivalence]]}}

== References ==

* [[Canonicalization]]

{{Reflist}}

* [[Unicode equivalence]]

[[Category:Natural language processing]]

{{compu-sci-stub}}

[[de:Normalisierung (Text)]]

[[pl:Normalizacja tekstu]]