Sample complexity: Difference between revisions

Revision as of 11:29, 23 June 2016

Revision as of 11:44, 23 June 2016

Line 1:

{{Machine learning bar}}

The '''sample complexity''' of a [[machine learning]] algorithm represents the number of training-samples that it needs in order to successfully learn a target function.

The '''sample complexity''' of a [[machine learning]] algorithm represents the number of training-samples that it needs to successfully learn a target function.

More precisely, the sample complexity is the number of training-samples that we need to supply to the algorithm, so that the function returned by the algorithm is within an arbitrarily small error of the best possible function, with probability arbitrarily close to 1.

Line 7 ⟶ 6:

There are two variants of sample complexity:

* The weak variant fixes a particular input-output distribution;

* The strong variant takes the worst-case sample complexity over all input-output distributions.

A natural question in statistical learning is, for a given hypothesis space, whether the sample complexity is finite in the strong sense, that is, there is a bound on the number of samples needed so that an algorithm can approximately solve all possible learning problems on a particular input-output space no matter the distribution of data over that space. The No Free Lunch Theorem, discussed below, says that this is always impossible if the hypothesis space is not constrained.

The No Free Lunch theorem, discussed below, proves that, in general, the strong sample complexity is infinite. I.e, there is no algorithm that can learn the globally-optimal target function using a finite number of training samples.

However, if we are only interested in a particular class of target functions (e.g, only linear functions) then the sample complexity is finite, and it depends linearly on the [[VC dimension]] on the class of target functions.

<ref name=":0" />