Pruning (artificial neural network): Difference between revisions

Revision as of 06:44, 19 May 2021

Latest revision as of 18:18, 26 June 2025

(23 intermediate revisions by 17 users not shown)

Line 1:

{{Short description|Trimming artificial neural networks to reduce computational overhead}}

{{Multiple issues|

{{other uses|Pruning (disambiguation)}}

{{Underlinked|date=August 2020}}

In [[deep learning]], '''pruning''' is the practice of removing [[parameter]]s from an existing [[Neural network (machine learning)|artificial neural network]].<ref>{{cite arXiv|last1=Blalock|first1=Davis|last2=Ortiz|first2=Jose Javier Gonzalez|last3=Frankle|first3=Jonathan|last4=Guttag|first4=John|date=2020-03-06|title=What is the State of Neural Network Pruning?|class=cs.LG|eprint=2003.03033}}</ref> The goal of this process is to reduce the size (parameter count) of the neural network (and therefore the [[computational resource]]s required to run it) whilst maintaining accuracy. This can be compared to the biological process of [[synaptic pruning]] which takes place in [[Mammal|mammalian]] brains during development.<ref>{{Cite journal |last1=Chechik |first1=Gal |last2=Meilijson |first2=Isaac |last3=Ruppin |first3=Eytan |date=October 1998 |title=Synaptic Pruning in Development: A Computational Account |url=https://ieeexplore.ieee.org/document/6790725 |journal=Neural Computation |volume=10 |issue=7 |pages=1759–1777 |doi=10.1162/089976698300017124 |pmid=9744896 |s2cid=14629275 |issn=0899-7667|url-access=subscription }}</ref>

{{Orphan|date=June 2020}}

}}

== Node (neuron) pruning ==

In the context of [[artificial neural network]], '''pruning''' is the practice of removing parameters (which may entail removing individual parameters, or parameters in groups such as by [[artificial neurons|neurons]]) from an existing network.<ref>{{cite arxiv|last1=Blalock|first1=Davis|last2=Ortiz|first2=Jose Javier Gonzalez|last3=Frankle|first3=Jonathan|last4=Guttag|first4=John|date=2020-03-06|title=What is the State of Neural Network Pruning?|class=cs.LG|eprint=2003.03033}}</ref> The goal of this process is to maintain accuracy of the network while increasing its efficiency. This can be done to reduce the computational resources required to run the neural network.

A basic algorithm for pruning is as follows:<ref>Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). ''Pruning convolutional neural networks for resource efficient inference''. arXiv preprint arXiv:1611.06440.</ref><ref>[https://jacobgil.github.io/deeplearning/pruning{{Cite web |last=Gildenblat |first=Jacob |date=2017-deep06-learning23 |title=Pruning deep neural networks to make them fast and small] |url=http://jacobgil.github.io/deeplearning/pruning-deep-learning |access-date=2024-02-04 |website=Github |language=en}}</ref>

A basic algorithm for pruning is as follows:<ref>Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). ''Pruning convolutional neural networks for resource efficient inference''. arXiv preprint arXiv:1611.06440.</ref><ref>[https://jacobgil.github.io/deeplearning/pruning-deep-learning Pruning deep neural networks to make them fast and small].</ref>

#Evaluate the importance of each neuron.

#Rank the neurons according to their importance (assuming there is a clearly defined measure for "importance").

#Remove the least important neuron.

#Check a termination condition (to be determined by the user) to see whether to continue pruning.

== Edge (weight) pruning ==

Most work on neural network pruning focuses on removing weights, namely, setting their values to zero. Early work suggested to also change the values of non-pruned weights.<ref>{{Cite journal |last1=Chechik |first1=Gal |last2=Meilijson |first2=Isaac |last3=Ruppin |first3=Eytan |date=April 2001 |title=Effective Neuronal Learning with Ineffective Hebbian Learning Rules |url=https://ieeexplore.ieee.org/document/6789989 |journal=Neural Computation |volume=13 |issue=4 |pages=817–840 |doi=10.1162/089976601300014367 |pmid=11255571 |s2cid=133186 |issn=0899-7667|url-access=subscription }}</ref>

== See also ==

* [[Knowledge distillation]]

* [[Neural Darwinism]]

== References ==

Line 18 ⟶ 23:

{{Compudeep-ailearning-stub}}