Numeric precision in Microsoft Excel: Difference between revisions

Revision as of 11:33, 3 July 2010

Revision as of 17:58, 3 July 2010

Line 8:

|author=Steve Dalton |chapter=Table 2.3: Worksheet data types and limits |pages=13–14 |isbn=0470027975 |edition=2nd |publisher=Wiley |year=2007 |url=http://books.google.com/books?id=ABUSU9PWUuIC&pg=PA13}}

</ref>) Although Excel can display 30 decimal points, its precision for a specified number is confined to 15 [[significant figures]], and calculations may have an accuracy that is even less due to twothree issues: [[round-off error|round off]],<ref name=roundoff>

Round-off is the loss of accuracy when numbers that differ by small amounts are subtracted. Because each number has only fifteen significant digits, their difference is inaccurate when there aren't enough significant digits to express the difference.

</ref> [[truncation]], and [[Binary numeral system|binary storage]].

==Accuracy and binary storage==