Talk:Rectified linear unit: Difference between revisions

Revision as of 18:32, 8 April 2021

Revision as of 18:34, 8 April 2021

Line 64:

== Is ReLu ever more than an activation function? ==

* I removed the following text from the article:

** A unit employing the rectifier is also called a '''rectified linear unit''' ('''ReLU''').<ref name="nair2010">{{cite conference |authors=Vinod Nair and [[Geoffrey Hinton]] |year=2010 |title=Rectified Linear Units Improve Restricted Boltzmann Machines |url=https://www.cs.toronto.edu/~hinton/absps/reluICML.pdf |conference=[[International Conference on Machine Learning|ICML]] }}</ref>

* I originally had replaced it with this text, but then I chose to remove the paragraph entirely:

A unit employing the rectifier is also called a '''rectified linear unit''' ('''ReLU''').<ref name="nair2010">{{cite conference |authors=Vinod Nair and [[Geoffrey Hinton]] |year=2010 |title=Rectified Linear Units Improve Restricted Boltzmann Machines |url=https://www.cs.toronto.edu/~hinton/absps/reluICML.pdf |conference=[[International Conference on Machine Learning|ICML]] }}</ref>

** Originally, the term '''rectified linear unit''' ('''ReLU''') referred to both the linear (fully-connected) layer and the activation function together,<ref name="nair2010">{{cite conference |authors=Vinod Nair and [[Geoffrey Hinton]] |year=2010 |title=Rectified Linear Units Improve Restricted Boltzmann Machines |url=https://www.cs.toronto.edu/~hinton/absps/reluICML.pdf |conference=[[International Conference on Machine Learning|ICML]] }}</ref> but it has become common to refer to just the activation function as the ReLU.<ref name="brownlee" /><ref name="medium-relu" />

* I'm not sure what the author of the original text meant by a "unit." When I read their reference, I could not discern whether it meant an activation function or a connection function (such as a fully-connected or convolutional function) followed by an activation function.

I originally had replaced it with this text, but then I chose to remove the paragraph entirely:

* Does anyone have expertise on this? Thank you!

Originally, the term '''rectified linear unit''' ('''ReLU''') referred to both the linear (fully-connected) layer and the activation function together,<ref name="nair2010">{{cite conference |authors=Vinod Nair and [[Geoffrey Hinton]] |year=2010 |title=Rectified Linear Units Improve Restricted Boltzmann Machines |url=https://www.cs.toronto.edu/~hinton/absps/reluICML.pdf |conference=[[International Conference on Machine Learning|ICML]] }}</ref> but it has become common to refer to just the activation function as the ReLU.<ref name="brownlee" /><ref name="medium-relu" />

I'm not sure what the author of the original text meant by a "unit." When I read their reference, I could not discern whether it meant an activation function or a connection function (such as a fully-connected or convolutional function) followed by an activation function.

Does anyone have expertise on this? Thank you!

--[[User:Yoderj|Yoderj]] ([[User talk:Yoderj|talk]]) 18:32, 8 April 2021 (UTC)