Text normalization: Difference between revisions

Revision as of 02:48, 5 June 2018

Revision as of 14:28, 20 January 2020

Line 27:

== Techniques ==

For simple, context-independent normalization, such as removing non-[[alphanumeric]] characters or [[diacritical marks]], [[regular expressions]] would suffice. For example, the [[sed]] script <code>sed&nbsp;&#8209;e&nbsp;"s/\s+/&nbsp;/g"&nbsp;&nbsp;''inputfile''</code> would normalize runs of [[whitespace character]]s into a single space. More complex normalization requires correspondingly complicated algorithms, including [[___domain knowledge]] of the language and vocabulary being normalized. Among other approaches, text normalization has been modeled as a problem of tokenizing and tagging streams of text<ref name="tagging">Zhu, C.; Tang, J.; Li, H.; Ng , H.; Zhao, T. (2007). "A Unified Tagging Approach to Text Normalization." ''Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics''; 688–695. [[Digital object identifier|doi]]:[http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.72.8138 10.1.1.72.8138].</ref> and as a special case of machine translation.<ref name="mt">Filip, G.; Krzysztof, J.; Agnieszka, W.; Mikołaj, W. (2006). [httphttps://www.proceedings2006.imcsitfedcsis.org/archive/pliks/202.pdf "Text Normalization as a Special Case of Machine Translation."] ''Proceedings of the International Multiconference on Computer Science and Information Technology'' '''1'''; 51–56.</ref><ref name="sm">Mosquera, A.; Lloret, E.; Moreda, P. (2012). [http://lrec.elra.info/proceedings/lrec2012/workshops/25.NLP4ITA-Proceedings.pdf#page=14 "Towards Facilitating the Accessibility of Web 2.0 Texts through Text Normalisation"] ''Proceedings of the LREC workshop: Natural Language Processing for Improving Textual Accessibility (NLP4ITA)''; 9-14</ref>

== See also ==