Content deleted Content added
→Accuracy and binary storage: cleaned up excessive apostrophes |
added accumulation and (catastrophic-) cancellation to the issues causing inaccuracies |
||
Line 8:
|author=Steve Dalton |chapter=Table 2.3: Worksheet data types and limits |pages=13–14 |isbn=0-470-02797-5 |edition=2nd |publisher=Wiley |year=2007 |url=https://books.google.com/books?id=ABUSU9PWUuIC&pg=PA13}}
</ref>). Although Excel can display 30 decimal places, its precision for a specified number is confined to 15 [[significant figures]], and calculations may have an accuracy that is even less due to
Round-off is the loss of accuracy when numbers that differ by small amounts are subtracted. Because each number has only fifteen significant digits, their difference is inaccurate when there aren't enough significant digits to express the difference.
</ref> [[truncation]], and [[Binary numeral system|binary storage]], accumulation of the deviations of the operands in calculations, and worst: cancellation at subtractions resp. 'catastrophic cancellation' at subtraction of values with similar magnitude.
==Accuracy and binary storage==
| |||