Natural language processing: Difference between revisions

Revision as of 20:46, 22 February 2025

Revision as of 07:23, 4 March 2025

Line 39:

* the larger such a (probabilistic) language model is, the more accurate it becomes, in contrast to rule-based systems that can gain accuracy only by increasing the amount and complexity of the rules leading to [[intractable problem|intractability]] problems.

Rule-based systems are commonly used:

Although rule-based systems for manipulating symbols were still in use in 2020, they have become mostly obsolete with the advance of [[Large language model|LLM]]s in 2023.

Before that they were commonly used:

* when the amount of training data is insufficient to successfully apply machine learning methods, e.g., for the machine translation of low-resource languages such as provided by the [[Apertium]] system,

* for preprocessing in NLP pipelines, e.g., [[Tokenization (lexical analysis)|tokenization]], or