Language model: Difference between revisions

Revision as of 18:03, 5 February 2025

Revision as of 19:11, 5 February 2025

Line 7:

== History ==

[[Noam Chomsky]] did earlypioneering work on language models in the 1950s by developing a theory of [[Formal grammar|formal grammars]], which are alsobecame fundamental to the field of [[Programming language|programming languages]].<ref>{{Cite journal |last=Chomsky |first=N. |date=1956-09 |title=Three models for the description of language |url=https://ieeexplore.ieee.org/document/1056813 |journal=IRE Transactions on Information Theory |volume=2 |issue=3 |pages=113–124 |doi=10.1109/TIT.1956.1056813 |issn=2168-2712}}</ref>

LaterIn 1980, statistical approaches basedwere onexplored discrete representations wereand found to be more useful for many purposes than rule-based formal grammars. Discrete representations like [[Word n-gram language model|word ''n''-gram language models]], with probabilities for discrete combinations of words, made significant advances.

In the 2000s, continuous representations for words, such as [[Word2vec|word embeddings]], began to replace discrete representations.<ref>{{Cite news |date=2022-02-22 |title=The Nature Of Life, The Nature Of Thinking: Looking Back On Eugene Charniak’s Work And Life |url=https://cs.brown.edu/news/2022/02/22/the-nature-of-life-the-nature-of-thinking-looking-back-on-eugene-charniaks-work-and-life/ |archive-url=http://web.archive.org/web/20241103134558/https://cs.brown.edu/news/2022/02/22/the-nature-of-life-the-nature-of-thinking-looking-back-on-eugene-charniaks-work-and-life/ |archive-date=2024-11-03 |access-date=2025-02-05 |language=en}}</ref> Typically, the representation is a [[Real number|real-valued]] vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning, and common relationships between pairs of words like plurality or gender .

== Pure statistical models ==