Content deleted Content added
Citation bot (talk | contribs) Altered url. URLs might have been anonymized. Add: doi, date. | Use this bot. Report bugs. | Suggested by Dominic3203 | Category:Numerical analysis | #UCB_Category 70/235 |
Adding short description: "Details of data storage in a spreadsheet application" |
||
| (2 intermediate revisions by 2 users not shown) | |||
Line 1:
{{Short description|Details of data storage in a spreadsheet application}}
As with other spreadsheets, [[Microsoft Excel]] works only to limited accuracy because it retains only a certain number of figures to describe numbers (it has limited [[Arithmetic precision|precision]]). With some exceptions regarding erroneous values, infinities, and denormalized numbers, Excel calculates in [[double-precision floating-point format]] from the [[IEEE 754-2008|IEEE 754 specification]]<ref name=microsoft_spec>
Line 93 ⟶ 94:
|via=Sun Microsystems
|url=http://docs.sun.com/source/806-3568/ncg_goldberg.html
|url-access=subscription
}} — Focuses upon examples of floating point representations of numbers.
* {{cite web
|title=Visual Basic and arithmetic precision
Line 114 ⟶ 116:
}}
1. The shortcomings in the {{code|{{=}} 1 + x - 1}} tasks are a combination of 'fp-math weaknesses' and 'how Excel handles it', especially Excel's rounding. Excel does some rounding and / or 'snap to zero' for most of its results, in average chopping the last 3 bits of the IEEE double representation. This behavior can be switched
2. It is not only clean powers of two surviving, but any combination of values constructed of bits which will be within the 53 bits once the decimal 1 is added. As most decimal values do not have a clean finite representation in binary they will suffer from 'round off' and 'cancellation' in tasks like the above.
Line 180 ⟶ 182:
|via=Sun Microsystems
|url=https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
|url-access=subscription
}} — more or less 'the holy book' of fp-math
</ref>
who states:
| |||