[mdlug] More OOCalc math wonkiness

[Aaron Kulkis]

Drew wrote:
>      Using OO Calc on another project, I ended up getting a zero 
> result rather abruptly
> when scaling down a Slip value. Increasing the precision of this 
> parameter, so that the
> display read .001 instead of 0, changed the results. So I did a 
> little test in a fresh
> spreadsheet:
> 
> Start at any cell. Enter a column of numbers as follows:
> 
> 0.001
> 0.002
> 0.003
> 0.004
> 
> Do not change the precision of the display. The numbers just entered 
> read all zeros, as the default is
> two decimal places. (not significant figures!)
> 
> Now in another cell, add up all the numbers (=SUM(A1:A4)). They 
> should add up to 0.01, which is within
> the (default) precision of the cell format. However, the result displayed is 0.
> 
> Now change the precision of the cells with the list of numbers to at 
> least three decimal places. The cell
> with the addition formula then shows the correct value of 0.01.
> 
> Evidently, Calc is referring to cells to be used in formulas, not by 
> any actual number stored separately,
> as is expected from calculators, but rather by *whatever is 
> displayed* by virtue of the Font format
> setting. At least now I have a better handle on how to write the bug 
> report. However, I'm not sure that this
> isn't supposed to be a "feature"; and even so, fixing it is likely 
> going to require completely rewriting
> large sections of code, if the developers did what I think they did. 
> Still, it's almost certainly responsible
> for the off-the-wall results that I've been getting with the least 
> square analysis.
> 
> BTW: I also tried this in Microsoft Excel. Excel gave the correct 
> answer to within the precision setting
> *of the formula cell* regardless of the precision settings of the 
> cells being added. So Excel apparently
> is at least getting this right.

Have you sent a bug report to the OpenOffice.org people?