Numeric precision in Microsoft Excel: Difference between revisions

Revision as of 02:48, 8 April 2013

Revision as of 14:02, 23 August 2013

Line 80:

Both the "compatibility" function STDEVP and the "consistency" function STDEV.P in Excel 2010 return the 0.5 population standard deviation for the given set of values.

===Subtraction of Subtraction Results===

Doing simple subtractions may lead to errors.<br />

As example we build a difference of the the cells <br />

:<math>A1: 28.552</math>

:<math>A2: 27.399</math>

:<math>A3: 26.246</math>

The equations<br />

:<math>B2: = A1 - A2 --> 1.1530</math>

:<math>B3: = A2 - A3 --> 1.1530</math>

Both seems to be the same right ?<br />

But if we calculate<br />

:<math>C3: = B2 - B3 --> -3.55271E-15</math>

The result we expected was 0.<br />

Very simple but wrong ?<br />

:<math>1.1530 - 1.1530 = -3.55271E-15</math>

QUOD EXCEL ERAT DEMONSTANDUM

===Round-off error===