Word n-gram language model: Difference between revisions

Revision as of 13:23, 10 August 2023

Revision as of 13:28, 10 August 2023

Line 114:

=== Skip-gram language model ===

Skip-gram language model is an attempt at overcoming the data sparsity problem that preceding (i.e. word ''n''-gram language model) faced. Words represented in an embedding vector were not necessarily consecutive anymore, but could leave gaps that are ''skipped'' over.<ref>{{cite web|url=http://homepages.inf.ed.ac.uk/ballison/pdf/lrec_skipgrams.pdf|title=A Closer Look at Skip-gram Modelling|author=David Guthrie|date=2006|display-authors=etal|access-date=27 April 2014|archive-url=https://web.archive.org/web/20170517144625/http://homepages.inf.ed.ac.uk/ballison/pdf/lrec_skipgrams.pdf|archive-date=17 May 2017|url-status=dead}}</ref>

Formally, a {{mvar|k}}-skip-{{mvar|n}}-gram is a length-{{mvar|n}} subsequence where the components occur at distance at most {{mvar|k}} from each other.

Line 125:

:''the in'', ''rain Spain'', ''in falls'', ''Spain mainly'', ''falls on'', ''mainly the'', and ''on plain''.

<!--Merged from skip gram-->

In skip-gram model, semantic relations between words are represented by [[linear combination]]s, capturing a form of [[compositionality]]. For example, in some such models, if {{mvar|v}} is the function that maps a word {{mvar|w}} to its {{mvar|n}}-d vector representation, then

<math display="block">v(\mathrm{king}) - v(\mathrm{male}) + v(\mathrm{female}) \approx v(\mathrm{queen})</math>

where ≈ is made precise by stipulating that its right-hand side must be the [[Nearest neighbor search|nearest neighbor]] of the value of the left-hand side.{{r|mikolov}}{{r|compositionality}}

== Syntactic ''n''-grams ==