Pruning (artificial neural network): Difference between revisions

Revision as of 13:42, 29 August 2023

Latest revision as of 18:18, 26 June 2025

(14 intermediate revisions by 12 users not shown)

Line 1:

{{Short description|Trimming artificial neural networks to reduce computational overhead}}

{{Orphan|date=June 2020}}

{{other uses|Pruning (disambiguation)}}

In the context of [[artificialdeep neural networklearning]], '''pruning''' is the practice of removing [[Parameter|parametersparameter]]s (whichfrom mayan entail removing individual parameters, or parameters in groups such as byexisting [[artificialNeural neurons|neurons]])network from(machine anlearning)|artificial existingneural network]].<ref>{{cite arXiv|last1=Blalock|first1=Davis|last2=Ortiz|first2=Jose Javier Gonzalez|last3=Frankle|first3=Jonathan|last4=Guttag|first4=John|date=2020-03-06|title=What is the State of Neural Network Pruning?|class=cs.LG|eprint=2003.03033}}</ref> The goal of this process is to maintainreduce accuracythe size (parameter count) of the neural network while(and increasingtherefore itsthe [[efficiencycomputational resource]].s Thisrequired canto berun doneit) towhilst reducemaintaining theaccuracy. [[ComputationalThis resource|computationalcan resources]]be requiredcompared to run the neural network. Abiological process of [[synaptic pruning]] which takes place in the[[Mammal|mammalian]] brain of mammalsbrains during development .<ref>{{Cite journal |lastlast1=Chechik |firstfirst1=Gal |last2=Meilijson |first2=Isaac |last3=Ruppin |first3=Eytan |date=October 1998-10 |title=Synaptic Pruning in Development: A Computational Account |url=https://ieeexplore.ieee.org/abstract/document/6790725 |journal=Neural Computation |volume=10 |issue=7 |pages=1759–1777 |doi=10.1162/089976698300017124 |pmid=9744896 |s2cid=14629275 |issn=0899-7667|url-access=subscription }}</ref>.

In the context of [[artificial neural network]], '''pruning''' is the practice of removing [[Parameter|parameters]] (which may entail removing individual parameters, or parameters in groups such as by [[artificial neurons|neurons]]) from an existing network.<ref>{{cite arXiv|last1=Blalock|first1=Davis|last2=Ortiz|first2=Jose Javier Gonzalez|last3=Frankle|first3=Jonathan|last4=Guttag|first4=John|date=2020-03-06|title=What is the State of Neural Network Pruning?|class=cs.LG|eprint=2003.03033}}</ref> The goal of this process is to maintain accuracy of the network while increasing its [[efficiency]]. This can be done to reduce the [[Computational resource|computational resources]] required to run the neural network. A process of pruning takes place in the brain of mammals during development <ref>{{Cite journal |last=Chechik |first=Gal |last2=Meilijson |first2=Isaac |last3=Ruppin |first3=Eytan |date=1998-10 |title=Synaptic Pruning in Development: A Computational Account |url=https://ieeexplore.ieee.org/abstract/document/6790725 |journal=Neural Computation |volume=10 |issue=7 |pages=1759–1777 |doi=10.1162/089976698300017124 |issn=0899-7667}}</ref>.

== Node (neuron) pruning ==

A basic algorithm for pruning is as follows:<ref>Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). ''Pruning convolutional neural networks for resource efficient inference''. arXiv preprint arXiv:1611.06440.</ref><ref>[https://jacobgil.github.io/deeplearning/pruning{{Cite web |last=Gildenblat |first=Jacob |date=2017-deep06-learning23 |title=Pruning deep neural networks to make them fast and small] |url=http://jacobgil.github.io/deeplearning/pruning-deep-learning |access-date=2024-02-04 |website=Github |language=en}}</ref>

#Evaluate the importance of each neuron.

#Rank the neurons according to their importance (assuming there is a clearly defined measure for "importance").

#Remove the least important neuron.

#Check a termination condition (to be determined by the user) to see whether to continue pruning.

Recently a highly pruned three layer tree architecture, has achieved a similar success rate to that of LeNet-5 on the CIFAR-10 dataset with a lesser computational complexity.<ref>{{Cite journal |last1=Meir |first1=Yuval |last2=Ben-Noam |first2=Itamar |last3=Tzach |first3=Yarden |last4=Hodassman |first4=Shiri |last5=Kanter |first5=Ido |date=2023-01-30 |title=Learning on tree architectures outperforms a convolutional feedforward network |journal=Scientific Reports |language=en |volume=13 |issue=1 |pages=962 |doi=10.1038/s41598-023-27986-6 |issn=2045-2322 |pmc=9886946 |pmid=36717568}}</ref>

== Edge (weight) pruning ==

Most work on neural network pruning focuses on removing weights, namely, setting their values to zero. Early work suggested to also change the values of non-pruned weights .<ref>{{Cite journal |lastlast1=Chechik |firstfirst1=Gal |last2=Meilijson |first2=Isaac |last3=Ruppin |first3=Eytan |date=April 2001-04 |title=Effective Neuronal Learning with Ineffective Hebbian Learning Rules |url=https://ieeexplore.ieee.org/abstract/document/6789989 |journal=Neural Computation |volume=13 |issue=4 |pages=817–840 |doi=10.1162/089976601300014367 |pmid=11255571 |s2cid=133186 |issn=0899-7667|url-access=subscription }}</ref>.

Most work on neural network pruning focuses on removing weights, namely, setting their values to zero.

Early work suggested to also change the values of non-pruned weights <ref>{{Cite journal |last=Chechik |first=Gal |last2=Meilijson |first2=Isaac |last3=Ruppin |first3=Eytan |date=2001-04 |title=Effective Neuronal Learning with Ineffective Hebbian Learning Rules |url=https://ieeexplore.ieee.org/abstract/document/6789989 |journal=Neural Computation |volume=13 |issue=4 |pages=817–840 |doi=10.1162/089976601300014367 |issn=0899-7667}}</ref>.

== See also ==

* [[Knowledge distillation]]

* [[Neural Darwinism]]

== References ==

{{reflist}}

Line 21 ⟶ 23:

{{Compudeep-ailearning-stub}}