Deep learning: Difference between revisions

Revision as of 14:17, 23 May 2017

Revision as of 18:08, 24 May 2017

Line 4:

{{machine learning bar}}

'''Deep learning''' (also known as '''deep structured learning''', '''hierarchical learning''' or '''deep machine learning''') is the study of [[artificial neural networks]] and related [[machine learning]] [[algorithm]]s that contain more than one [[Multilayer_perceptron#Layers|hidden layer]]. These deep nets:<ref name="BOOK2014.1">{{cite journal |last1=Deng |first1=L. |last2=Yu |first2=D. |year=2014 |title=Deep Learning: Methods and Applications |url=http://research.microsoft.com/pubs/209355/DeepLearning-NowPublishing-Vol7-SIG-039.pdf |journal=Foundations and Trends in Signal Processing |volume=7 |issue=3–4 |pages=199–200 |doi=10.1561/2000000039}}</ref>

'''Deep learning''' (also known as '''deep structured learning''', '''hierarchical learning''' or '''deep machine learning''') is a class

of [[machine learning]] [[algorithm]]s that:<ref name="BOOK2014.1">{{cite journal |last1=Deng |first1=L. |last2=Yu |first2=D. |year=2014 |title=Deep Learning: Methods and Applications |url=http://research.microsoft.com/pubs/209355/DeepLearning-NowPublishing-Vol7-SIG-039.pdf |journal=Foundations and Trends in Signal Processing |volume=7 |issue=3–4 |pages=199–200 |doi=10.1561/2000000039}}</ref>

* use a cascade of many layers of [[Nonlinear filter|nonlinear processing]] units for [[feature extraction]] and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be [[Supervised learning|supervised]] or [[Unsupervised learning|unsupervised]] and applications include pattern analysis (unsupervised) and classification (supervised).

* are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher level features are derived from lower level features to form a hierarchical representation.

Line 19 ⟶ 18:

Various deep learning architectures such as [[#Deep_neural_network_architectures|deep neural network]]s, [[convolutional neural network|convolutional deep neural networks]], [[deep belief network]]s and [[recurrent neural network]]s have been applied to fields like [[computer vision]], [[automatic speech recognition]], [[natural language processing]], audio recognition and [[bioinformatics]] where they have been shown to produce state-of-the-art results on various tasks.

Although ''Deep learning'' has been characterized as a [[buzzword]], or a rebranding of [[neural network]]s.<ref>{{cite video |last=Collobert |first=R. |date=April 2011 |title=Deep Learning for Efficient Discriminative Parsing |url=http://videolectures.net/aistats2011_collobert_deep/ |website=VideoLectures.net |time=7min 45s}}</ref><ref>{{cite web |last=Gomes |first=L. |date=20 October 2014 |title=Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Other Huge Engineering Efforts |url=http://spectrum.ieee.org/robotics/artificial-intelligence/machinelearning-maestro-michael-jordan-on-the-delusions-of-big-data-and-other-huge-engineering-efforts |work=[[IEEE Spectrum]]}}</ref>, deep neural nets have demonstrated an ability to out-perform other machine learning algorithms on tasks such as object recognition in the field of [[computer vision]].

<ref name="krizhevsky2012"></ref>

== Introduction ==

Line 810 ⟶ 809:

According to LeCun,<ref name="lecun2016slides"/> in the early 2000s, in an industrial application CNNs already processed an estimated 10% to 20% of all the checks written in the US in the early 2000s.

Significant additional impact of deep learning in image or object recognition was felt in the years 2011–2012. Although CNNs trained by backpropagation had been around for decades,<ref name="LECUN1989"/> and GPU implementations of NNs for years,<ref name="jung2004"/> including CNNs,<ref name="chellapilla2006"/> fast implementations of CNNs with max-pooling on GPUs in the style of Dan Ciresan and colleagues<ref name="ciresan2011"/> were needed to make a dent in computer vision.<ref name="SCHIDHUB"/> In 2011, this approach achieved for the first time superhuman performance in a visual pattern recognition contest.<ref name="ciresan2011NN"/> Also in 2011, it won the ICDAR Chinese handwriting contest, and in May 2012, it won the ISBI image segmentation contest.<ref name="ciresan2012NIPS"/>

Until 2011, CNNs did not play a major role at computer vision conferences, but in June 2012, a paper by Dan Ciresan et al. at the leading conference CVPR<ref name="ciresan2011CVPR"/> showed how max-pooling CNNs on GPU can dramatically improve many vision benchmark records, sometimes with human-competitive or even superhuman performance. In October 2012, a similar system by Alex Krizhevsky in the team of Geoff Hinton<ref name="krizhevsky2012">Krizhevsky, A., Sutskever, I. and Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks.

{{cite journal|last1=Krizhevsky|first1=Alex|last2=Sutskever|first2=Ilya|last3=Hinton|first3=Geoffry|title=ImageNet Classification with Deep Convolutional Neural Networks|journal=NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada|date=2012|url=https://www.cs.toronto.edu/~kriz/imagenet_classification_with_deep_convolutional.pdf}}

NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada</ref> won the large-scale [[ImageNet competition]] by a significant margin over shallow machine learning methods. In November 2012, Ciresan et al.'s system also won the ICPR contest on analysis of large medical images for cancer detection, and in the following year also the MICCAI Grand Challenge on the same topic.<ref name="ciresan2013miccai">D. Ciresan, A. Giusti, L.M. Gambardella, J. Schmidhuber (2013). Mitosis Detection in Breast Cancer Histology Images using Deep Neural Networks. Proceedings MICCAI, 2013.</ref>

In 2013 and 2014, the error rate on the ImageNet task using deep learning was further reduced quickly, following a similar trend in large-scale speech recognition. Releases like the Wolfram Image Identification project continue to bring improvements in the technology to the public eye.<ref>{{Cite web|url=https://www.imageidentify.com/|title=The Wolfram Language Image Identification Project|website=www.imageidentify.com|access-date=2017-03-22}}</ref>