Talk:Floating-point arithmetic/Archive 4: Difference between revisions

Revision as of 07:06, 7 May 2012

Revision as of 07:04, 23 May 2012

Line 476:

I just reverted a bit about the Pilot Ace in history because it used software to emulate floating point. However it occrs to me that there might be something worthwhile in the bit about J.H.Wilkinson, ''Rounding errors in algebraic processes''. Is there evidence about who wrote a book about floating point or that this was a particular turning point? [[User:Dmcq|Dmcq]] ([[User talk:Dmcq|talk]]) 18:46, 6 February 2012 (UTC)

== IEEE 754 ==

I have added a section discussing the "big picture" on the rationale and use for the IEEE 754 features which often gets lost when discussing the details.

I plan to add specific references for the points made there (from Kahn's web site). It would be good to expand the examples and add additional ones as well.

[[User:Brianbjparker|Brianbjparker]] ([[User talk:Brianbjparker|talk]]) <span style="font-size: smaller;" class="autosigned">—Preceding [[Wikipedia:Signatures|undated]] comment added 11:22, 19 February 2012 (UTC).</span><!--Template:Undated--> <!--Autosigned by SineBot-->

:You need to cite something saying these were accepted rationales for it. Citations point to specific books journals or newspapers and preferably page number ranges. [[User:Dmcq|Dmcq]] ([[User talk:Dmcq|talk]]) 13:51, 19 February 2012 (UTC)

Added direct citations as requested.

[[User:Brianbjparker|Brianbjparker]] ([[User talk:Brianbjparker|talk]]) <span style="font-size: smaller;" class="autosigned">—Preceding [[Wikipedia:Signatures|undated]] comment added 18:20, 19 February 2012 (UTC).</span><!--Template:Undated--> <!--Autosigned by SineBot-->

:Thanks. My feeling about Kahan and his diatribe against Java is that he just doesn't get what programmers have to do when testing a program. Having a switch to enable lax typing of intermediate results where you know it ill only be run in environments you've tested is a good idea but that wasn't what Java was originally designed for. The section about extended precision there seems undue in length as I'm pretty certain other considerations like signed zero and denormal handling were the main original considerations where it differed from previous implementations. [[User:Dmcq|Dmcq]] ([[User talk:Dmcq|talk]]) 20:37, 19 February 2012 (UTC)

:Although I referenced Kahan's Java paper several times, I certainly didn't want this section to appear as a slight against Java. Kahan has several other papers discussing the need for extended precision that do not mention Java-- I will replace the current references with those in the near future, and try to trim it down (although I don't think that that reference is a diatribe against Java, just against its numerics). I certainly didn't want to get into the tradeoffs between improved numerical precision of results versus exact reproducibility in Java in this section. I do however think that it is important to clarify the intended use of the IEEE754 features in an introductory article like this, which can get lost in detailed descriptions of the features. In particular, I find that there is *wide* misunderstanding of the intended use of, and need for, extended precision amongst the programming community, particularly as extended precision was historically not supported in several RISC processors, and thus it is underused by programmers, even when targeting the x86 platform for e.g. HPC (even when these same programmers would carry additional significant figures for intermediate calculations if doing the same computations by hand, as alluded to in this section). Also, Kahan's descriptions of work on the design of the x87 (based on his experience designing HP calculators which use extended precision internally) makes it clear that extended precision was intended as a key feature (indeed a recommended feature) of IEEE754, compared with previous implementations.

[[User:Brianbjparker|Brianbjparker]] ([[User talk:Brianbjparker|talk]]) 00:56, 20 February 2012 (UTC)

:As far as I'm aware the main other rationales were

::To have a sound mathematical basis in that results were correctly rounded versions of accurate results and also so reasoning about the calculations would be easier.

::Round to even was used to improve accuracy. In fact this is much more important than extended precision if the double storage mode is only used for intermediate calculations. Using extended precision only gives bout one extra bit overall at the end if values in arrays are in doubles. The main reason I believe they were put in was it made calculating mathematical functions much easier and more accurate, they can also be used in inner routines with benefit.

::Biased rounding was put in I believe to support interval arithmetic - another part of being able to guarantee the results of calculations. [[User:Dmcq|Dmcq]] ([[User talk:Dmcq|talk]]) 15:43, 20 February 2012 (UTC)

:::''Using extended precision only gives bout one extra bit overall at the end if values in arrays are in doubles''. This is false in general; you must be thinking of some special cases where not many intermediate calculations happen before rounding to double for storage. For a counterexample, e.g. consider a loop to take a dot product of two double-precision arrays (not using Kahan summation etc.) [[User:Stevenj|— Steven G. Johnson]] ([[User talk:Stevenj|talk]]) 21:16, 20 February 2012 (UTC)

::::You would normally get very little advantage in that case over round to even with so few intermediate calculations. And for longer calculations round to even wins over just using a longer mantissa and rounding down. You only get a worthwhile gain if the storage is in extended precision. [[User:Dmcq|Dmcq]] ([[User talk:Dmcq|talk]]) 21:53, 20 February 2012 (UTC)

:::::That is certainly not the case in general. The examples you are thinking of are using simple exactly rounded single arithmetic expresions-- the advantage of extended precision is avoiding loss of precision in more complicated numerically unstable formulae-- e.g. it is easy to construct examples were even computing a quadratic formula discriminant can cause massive loss of ULP when computed in double but not in double extended. Several examples are given in the Kahan references. This is in addition to the advantage of the extended exponent in avoiding overflow in e.g. dot products. [[User:Brianbjparker|Brianbjparker]] ([[User talk:Brianbjparker|talk]]) 00:16, 22 February 2012 (UTC)

:::When you say ''Round to even was used to improve accuracy.'', I take it you are mainly referring to the exact rounding: breaking ties by round to even does avoid some additional statistic biases but it is rather subtle (might be worth mentioning the main text though..). [[User:Brianbjparker|Brianbjparker]] ([[User talk:Brianbjparker|talk]]) 00:16, 22 February 2012 (UTC)

::: ''Biased rounding was put in I believe to support interval arithmetic''. Yes, I believe directed rounding was included to support interval arithmetic, but also for debugging numerical stability issues-- if an algorithm gives drastically different results under round to + and - infinity then it is likely unstable. [[User:Brianbjparker|Brianbjparker]] ([[User talk:Brianbjparker|talk]]) 00:16, 22 February 2012 (UTC)

::: ''As far as I'm aware the main other rationales were... to have a sound mathematical basis in that results were correctly rounded versions of accurate results and also so reasoning about the calculations would be easier.''. Yes, the exact rounding is an important point-- I have added some additional text earlier in the article to expand on this. It is true that, like previous arithmetics, having a precise specification to allow expert numerical analysts to write robust libraries was an important consideration, but the unique aspect of IEEE-754 is that it was also aimed at a broad market of non-expert users and so I focused in the section on the robustness features relevant to that (I will add some text highlighting that aspect as well though). [[User:Brianbjparker|Brianbjparker]] ([[User talk:Brianbjparker|talk]]) 00:16, 22 February 2012 (UTC)

::::Well exact rounding, but I thought it better to specify the precise format they have. The point is that rounding rather than truncating is what really matters. With rounding the error only tends to go up with the number of computations as the square root of the number of operations whereas with directed rounding it goes up linearly. Even the reduction of bias by round to even matter in this. You alwayts get something else putting in a little bias so it is not as good as this but directed rounding is really bad. You're better off just perturbing the original figures for stability checking.

::::The mathematical basis makes it much easier to do things like construct longer precision arithmetic packages easily, in fact the fused multiply is particularly useful for this. [[User:Dmcq|Dmcq]] ([[User talk:Dmcq|talk]]) 00:27, 22 February 2012 (UTC)

:::::The use of directed rounding for diagnosis of stability issues is discussed here http://www.cs.berkeley.edu/~wkahan/Stnfrd50.pdf and in other references at that web site. It also discusses why perturbation alone is not as useful. IEEE 754-2008 annex B states this explicitly-- "B.2 Numerical sensitivity: Debuggers should be able to alter the attributes governing handling of rounding or exceptions inside subprograms, even if the source code for those subprograms is not available; dynamic modes might be used for this purpose. For instance, changing the rounding direction or precision during execution might help identify subprograms that are unusually sensitive to rounding, whether due to ill-condition of the problem being solved, instability in the algorithm chosen, or an algorithm designed to work in only one rounding- direction attribute. The ultimate goal is to determine responsibility for numerical misbehavior, especially in separately-compiled subprograms. The chosen means to achieve this ultimate goal is to facilitate the production of small reproducible test cases that elicit unexpected behavior." [[User:Brianbjparker|Brianbjparker]] ([[User talk:Brianbjparker|talk]]) 01:04, 22 February 2012 (UTC)

::::::The uses that somebody makes of features is quite a different thing from the rationale for why somebody would pay to have them implemented. The introduction to the standard gives a succinct summary of the main reasons for the standard. I'll just copy the latest here so you can see

:a) Facilitate movement of existing programs from diverse computers to those that adhere to this standard as well as among those that adhere to this standard.

:b) Enhance the capabilities and safety available to users and programmers who, although not expert in numerical methods, might well be attempting to produce numerically sophisticated programs.

:c) Encourage experts to develop and distribute robust and efficient numerical programs that are portable, by way of minor editing and recompilation, onto any computer that conforms to this standard and possesses adequate capacity. Together with language controls it should be possible to write programs that produce identical results on all conforming systems.

:d) Provide direct support for

::― execution-time diagnosis of anomalies

::― smoother handling of exceptions

::― interval arithmetic at a reasonable cost.

:e) Provide for development of

::― standard elementary functions such as exp and cos

::― high precision (multiword) arithmetic

::― coupled numerical and symbolic algebraic computation.

:f) Enable rather than preclude further refinements and extensions.

::::::There are other things but this is what the basic rationale was and is. Directed rounding was for interval arithmetic. [[User:Dmcq|Dmcq]] ([[User talk:Dmcq|talk]]) 01:56, 22 February 2012 (UTC)

:::::::Thanks. Actually, I believe that "d) Provide direct support for― execution-time diagnosis of anomalies" is referring to this use of directed rounding to diagnose numerical instability. Certainly Kahan makes it clear that he considered it a key usage from the early design of the x87. I agree that its use for interval arithmetic was also considered from the beginning. [[User:Brianbjparker|Brianbjparker]] ([[User talk:Brianbjparker|talk]]) 02:11, 22 February 2012 (UTC)

::::::::No that refers to identification and methods of notifying the various exceptions and the handling of the signalling and quiet NaNs. Your reference from 2007 does not support in any way that arbitrarily jiggling the calculations using directed rounding was considered as a reason to include directed rounding in the specification. He'd have been just laughed at if he had justified spending money on the 8087 for such a purpose when there are easy ways of doing something like that without any hardware assistance. [[User:Dmcq|Dmcq]] ([[User talk:Dmcq|talk]]) 08:23, 22 February 2012 (UTC)