Content deleted Content added
KolbertBot (talk | contribs) m Bot: HTTP→HTTPS |
m layout |
||
Line 28:
For simple, context-independent normalization, such as removing non-[[alphanumeric]] characters or [[diacritical marks]], [[regular expressions]] would suffice. For example, the [[sed]] script <tt>sed -e "s/\s+/ /g" ''inputfile''</tt> would normalize runs of [[whitespace character]]s into a single space. More complex normalization requires correspondingly complicated algorithms, including [[___domain knowledge]] of the language and vocabulary being normalized. Among other approaches, text normalization has been modeled as a problem of tokenizing and tagging streams of text<ref name="tagging">Zhu, C.; Tang, J.; Li, H.; Ng , H.; Zhao, T. (2007). "A Unified Tagging Approach to Text Normalization." ''Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics''; 688–695. [[Digital object identifier|doi]]:[http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.72.8138 10.1.1.72.8138].</ref> and as a special case of machine translation.<ref name="mt">Filip, G.; Krzysztof, J.; Agnieszka, W.; Mikołaj, W. (2006). [http://www.proceedings2006.imcsit.org/pliks/202.pdf "Text Normalization as a Special Case of Machine Translation."] ''Proceedings of the International Multiconference on Computer Science and Information Technology'' '''1'''; 51–56.</ref><ref name="sm">Mosquera, A.; Lloret, E.; Moreda, P. (2012). [http://lrec.elra.info/proceedings/lrec2012/workshops/25.NLP4ITA-Proceedings.pdf#page=14 "Towards Facilitating the Accessibility of Web 2.0 Texts through Text Normalisation"] ''Proceedings of the LREC workshop: Natural Language Processing for Improving Textual Accessibility (NLP4ITA)''; 9-14</ref>
== References ==▼
{{Reflist}}▼
== See also ==
Line 37 ⟶ 33:
* [[Canonicalization]]
* [[Unicode equivalence]]
▲== References ==
▲{{Reflist}}
[[Category:Natural language processing]]
| |||