[mdlug] More OOCalc math wonkiness

[Drew]

     Using OO Calc on another project, I ended up getting a zero 
result rather abruptly
when scaling down a Slip value. Increasing the precision of this 
parameter, so that the
display read .001 instead of 0, changed the results. So I did a 
little test in a fresh
spreadsheet:

Start at any cell. Enter a column of numbers as follows:

0.001
0.002
0.003
0.004

Do not change the precision of the display. The numbers just entered 
read all zeros, as the default is
two decimal places. (not significant figures!)

Now in another cell, add up all the numbers (=SUM(A1:A4)). They 
should add up to 0.01, which is within
the (default) precision of the cell format. However, the result displayed is 0.

Now change the precision of the cells with the list of numbers to at 
least three decimal places. The cell
with the addition formula then shows the correct value of 0.01.

Evidently, Calc is referring to cells to be used in formulas, not by 
any actual number stored separately,
as is expected from calculators, but rather by *whatever is 
displayed* by virtue of the Font format
setting. At least now I have a better handle on how to write the bug 
report. However, I'm not sure that this
isn't supposed to be a "feature"; and even so, fixing it is likely 
going to require completely rewriting
large sections of code, if the developers did what I think they did. 
Still, it's almost certainly responsible
for the off-the-wall results that I've been getting with the least 
square analysis.

BTW: I also tried this in Microsoft Excel. Excel gave the correct 
answer to within the precision setting
*of the formula cell* regardless of the precision settings of the 
cells being added. So Excel apparently
is at least getting this right.

----

- Drew.