Meta-learning (computer science): Difference between revisions

Revision as of 12:56, 9 September 2023

Revision as of 22:22, 24 September 2023

Line 14:

| year = 1986

| language = en

| chapter-url = httphttps://books.google.com/books?hl=en&lr=&id=f9RylgKpHZsC&oiq=fndutgoff&pg=PA107&ots=VeZN4AF3GH&sig=xDRArOx5vCkRaLTXdFFaAs9Qs78#v=onepage&q=utgoff&f=false

}}</ref> This means that it will only learn well if the bias matches the learning problem. A learning algorithm may perform very well in one ___domain, but not on the next. This poses strong restrictions on the use of [[machine learning]] or [[data mining]] techniques, since the relationship between the learning problem (often some kind of [[database]]) and the effectiveness of different learning algorithms is not yet understood.

Line 31:

==Common approaches==

There are three common approaches:<ref name="paper1">{{cite blogweb|url=https://lilianweng.github.io/lil-log/2018/11/30/meta-learning.html|first=Lilian|last=Weng|year=2018|title=Meta-Learning: Learning to Learn Fast|website=OpenAI Blog|date=30 November 2018|access-date=27 October 2019|language=en}}</ref>

# using (cyclic) networks with external or internal memory (model-based)

# learning effective distance metrics (metrics-based)

Line 40:

====Memory-Augmented Neural Networks====

A Memory-Augmented [[Neural Network]], or MANN for short, is claimed to be able to encode new information quickly and thus to adapt to new tasks after only a few examples.<ref name="paper2">{{cite paperdocument|url=http://proceedings.mlr.press/v48/santoro16.pdf|first1=Adam|last1=Santoro|first2=Sergey|last2=Bartunov|first3=Daan|last3=Wierstra|first4=Timothy|last4=Lillicrap|title=Meta-Learning with Memory-Augmented Neural Networks|publisher=Google DeepMind|access-date=29 October 2019|language=en}}</ref>

====Meta Networks====

Meta Networks (MetaNet) learns a meta-level knowledge across tasks and shifts its inductive biases via fast parameterization for rapid generalization.<ref name="paper3">{{cite arxivarXiv|first1=Tsendsuren|last1=Munkhdalai|first2=Hong|last2=Yu|year=2017|title=Meta Networks|eprint=1703.00837|class=cs.LG|language=en}}</ref>

===Metric-Based===

Line 49:

====Convolutional Siamese Neural Network====

[[Siamese neural network]] is composed of two twin networks whose output is jointly trained. There is a function above to learn the relationship between input data sample pairs. The two networks are the same, sharing the same weight and network parameters.<ref name="paper4">{{cite paperdocument|url=http://www.cs.toronto.edu/~rsalakhu/papers/oneshot1.pdf|first1=Gregory|last1=Koch|first2=Richard|last2=Zemel|first3=Ruslan|last3=Salakhutdinov|year=2015|title=Siamese Neural Networks for One-shot Image Recognition|publisher=Department of Computer Science, University of Toronto|___location=Toronto, Ontario, Canada|language=en}}</ref>

====Matching Networks====

Matching Networks learn a network that maps a small labelled support set and an unlabelled example to its label, obviating the need for fine-tuning to adapt to new class types.<ref name="paper5">{{cite paperdocument|url=http://papers.nips.cc/paper/6385-matching-networks-for-one-shot-learning.pdf|last1=Vinyals|first1=O.|last2=Blundell|first2=C.|last3=Lillicrap|first3=T.|last4=Kavukcuoglu|first4=K.|last5=Wierstra|first5=D.|year=2016|title=Matching networks for one shot learning|publisher=Google DeepMind|access-date=3 November 2019|language=en}}</ref>

====Relation Network====

The Relation Network (RN), is trained end-to-end from scratch. During meta-learning, it learns to learn a deep distance metric to compare a small number of images within episodes, each of which is designed to simulate the few-shot setting.<ref name="paper6">{{cite paperdocument|url=http://openaccess.thecvf.com/content_cvpr_2018/papers_backup/Sung_Learning_to_Compare_CVPR_2018_paper.pdf|last1=Sung|first1=F.|last2=Yang|first2=Y.|last3=Zhang|first3=L.|last4=Xiang|first4=T.|last5=Torr|first5=P. H. S.|last6=Hospedales|first6=T. M.|year=2018|title=Learning to compare: relation network for few-shot learning|language=en}}</ref>

====Prototypical Networks====

Prototypical Networks learn a [[metric space]] in which classification can be performed by computing distances to prototype representations of each class. Compared to recent approaches for few-shot learning, they reflect a simpler inductive bias that is beneficial in this limited-data regime, and achieve satisfied results.<ref name="paper7">{{cite paperdocument|url=http://papers.nips.cc/paper/6996-prototypical-networks-for-few-shot-learning.pdf|last1=Snell|first1=J.|last2=Swersky|first2=K.|last3=Zemel|first3=R. S.|year=2017|title=Prototypical networks for few-shot learning|language=en}}</ref>

===Optimization-Based===

Line 70:

====Reptile====

Reptile is a remarkably simple meta-learning optimization algorithm, given that both of its components rely on [[meta-optimization]] through gradient descent and both are model-agnostic.<ref name="paper10">{{cite arxivarXiv|first1=Alex|last1=Nichol|first2=Joshua|last2=Achiam|first3=John|last3=Schulman|year=2018|title=On First-Order Meta-Learning Algorithms|eprint=1803.02999|class=cs.LG|language=en}}</ref>

==Examples==