Pruning (artificial neural network): Difference between revisions

Revision as of 06:32, 19 May 2021

Revision as of 06:36, 19 May 2021

Line 18:

=== Optimum Brain Damage ===

The [https://nyuscholars.nyu.edu/en/publications/optimal-brain-damage Optimum Brain Damage pruning algorithm] (OBD), introduced by [http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf Le Cun, Denker and Solla] in 1990, is based on second order derivatives of the error function. The aim is to iteratively delete the weights whose deletion will result in the least increase of the error in the network. There is an important problem in the estimation of formula, is the size of the Hessian matrix. The calculation of the Hessian matrix is time-consuming, hence Le Cun et al. (1990) assume that the [[Hessian]] is diagonal. On the other hand, Hassibi and Stork (1993) argue that Hessian for every problem they have considered are strongly non-diagonal, and this leads OBD to eliminate the wrong weights.

=== Optimum Brain Surgeon pruning algorithm ===

The [https://papers.nips.cc/paper/647-second-order-derivatives-for-network-pruning-optimal-brain-surgeon.pdf Optimum Brain Surgeon pruning algorithm (OBS)], introduced by [[Hassibi]] and Stork in 1993), is a more complex form of Optimum Brain Damage (OBD). Although OBS and OBD are basically based on the same theoretical approach, OBS does not make any assumption about the form ofHessian matrix. Therefore, OBS is more complex and robust than OBD. It is claimed that optimum brain surgeon (OBS) is significantly better than magnitude based (MB) and optimal brain damage(OBD) techniques, and OBS permits the pruning of more weights than other methods for the same error on the training set, and thus yields better generalization on test data. The most important problem with OBS is that the inverse of the Hessian matrix has to be computed to judge saliency and weight change for every link. Therefore, this method is quite slow and takes much memory compared to the other methods.

== References ==