Content deleted Content added
Phil Boswell (talk | contribs) m →Accuracy and binary storage: Spelling/grammar/punctuation/typographical correction |
Adding short description: "Details of data storage in a spreadsheet application" |
||
| (5 intermediate revisions by 5 users not shown) | |||
Line 1:
{{Short description|Details of data storage in a spreadsheet application}}
As with other spreadsheets, [[Microsoft Excel]] works only to limited accuracy because it retains only a certain number of figures to describe numbers (it has limited [[Arithmetic precision|precision]]). With some exceptions regarding erroneous values, infinities, and denormalized numbers, Excel calculates in [[double-precision floating-point format]] from the [[IEEE 754-2008|IEEE 754 specification]]<ref name=microsoft_spec>
{{cite web |title=Floating-point arithmetic may give inaccurate results in Excel |id=Revision 8.2
</ref> (besides numbers, Excel uses a few other data types<ref name=Dalton>
Line 48 ⟶ 49:
|title=Excel addition strangeness
|website=office-watch.com
|url=
}}
</ref>
Line 70 ⟶ 71:
|id=KB 78113
|website=Microsoft support
|date=6 June 2024
|url=http://support.microsoft.com/kb/78113/en-us
}} — A detailed explanation with examples of the binary/15 sig fig storage consequences.
Line 87 ⟶ 89:
|title=What every computer scientist should know about floating point
|journal=Computing Surveys
|doi=10.1145/103162.103163
|type=edited reprint
|id=E19957-01 / 806-3568
|via=Sun Microsystems
|url=http://docs.sun.com/source/806-3568/ncg_goldberg.html
|url-access=subscription
}} — Focuses upon examples of floating point representations of numbers.
* {{cite web
|title=Visual Basic and arithmetic precision
Line 112 ⟶ 116:
}}
1. The shortcomings in the {{code|{{=}} 1 + x - 1}} tasks are a combination of 'fp-math weaknesses' and 'how Excel handles it', especially Excel's rounding. Excel does some rounding and / or 'snap to zero' for most of its results, in average chopping the last 3 bits of the IEEE double representation. This behavior can be switched
▲1. The shortcomings in the {{code|{{=}} 1 + x - 1}} tasks are a combination of 'fp-math weaknesses' and 'how Excel handles it', especially Excel's rounding. Excel does some rounding and / or 'snap to zero' for most of its results, in average chopping the last 3 bits of the IEEE double representation. This behavior can be switched of by setting the formula in parentheses: {{code|{{=}} ( 1 + 2^-52 - 1 )}}. You will see that even that small value survives. Smaller values will pass away as there are only 53 bits to represent the value, for this case 1.0000000000 0000000000 0000000000 0000000000 0000000000 01, the first representing the {{code|1}}, and the last the {{code|2^-52}}.
2. It is not only clean powers of two surviving, but any combination of values constructed of bits which will be within the 53 bits once the decimal 1 is added. As most decimal values do not have a clean finite representation in binary they will suffer from 'round off' and 'cancellation' in tasks like the above.
Line 174 ⟶ 177:
|title=What every computer scientist should know about floating point
|journal=Computing Surveys
|doi=10.1145/103162.103163
|type=edited reprint
|id=E19957-01 / 806-3568
|via=Sun Microsystems
|url=https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
|url-access=subscription
}} — more or less 'the holy book' of fp-math
</ref>
who states:
Line 211 ⟶ 216:
|publisher=[[Academic Press, Inc.]]
|edition=8
|
|isbn=978-0-12-384933-5
|lccn=2014010276 <!-- |url=https://books.google.com/books?id=NjnLAwAAQBAJ
|access-date=2016-02-21-->
|page=25
Line 262 ⟶ 266:
|series=Department of Physics
|publisher=[[Lund University]]
|url=https://www.researchgate.net/publication/242226580
}}
</ref>
| |||