[mdlug] More OOCalc math wonkiness

[R. Kannan]

Does not happen in OpenOffice 2.0.2 (Novell Edition) on SuSE10.0

-----Original Message-----
>From: Ken Siersma <siersmak at gmail.com>
>Sent: Nov 27, 2007 2:09 PM
>To: MDLUG's Main discussion list <mdlug at mdlug.org>
>Subject: Re: [mdlug] More OOCalc math wonkiness
>
>Doesn't happen in OpenOffice 2.2.0, Ubuntu 7.04.
>
>On Nov 27, 2007 3:44 PM, Drew <drew4096 at gmail.com> wrote:
>
>>     Using OO Calc on another project, I ended up getting a zero
>> result rather abruptly
>> when scaling down a Slip value. Increasing the precision of this
>> parameter, so that the
>> display read .001 instead of 0, changed the results. So I did a
>> little test in a fresh
>> spreadsheet:
>>
>> Start at any cell. Enter a column of numbers as follows:
>>
>> 0.001
>> 0.002
>> 0.003
>> 0.004
>>
>> Do not change the precision of the display. The numbers just entered
>> read all zeros, as the default is
>> two decimal places. (not significant figures!)
>>
>> Now in another cell, add up all the numbers (=SUM(A1:A4)). They
>> should add up to 0.01, which is within
>> the (default) precision of the cell format. However, the result displayed
>> is 0.
>>
>> Now change the precision of the cells with the list of numbers to at
>> least three decimal places. The cell
>> with the addition formula then shows the correct value of 0.01.
>>
>> Evidently, Calc is referring to cells to be used in formulas, not by
>> any actual number stored separately,
>> as is expected from calculators, but rather by *whatever is
>> displayed* by virtue of the Font format
>> setting. At least now I have a better handle on how to write the bug
>> report. However, I'm not sure that this
>> isn't supposed to be a "feature"; and even so, fixing it is likely
>> going to require completely rewriting
>> large sections of code, if the developers did what I think they did.
>> Still, it's almost certainly responsible
>> for the off-the-wall results that I've been getting with the least
>> square analysis.
>>
>> BTW: I also tried this in Microsoft Excel. Excel gave the correct
>> answer to within the precision setting
>> *of the formula cell* regardless of the precision settings of the
>> cells being added. So Excel apparently
>> is at least getting this right.
>>
>> ----
>>
>> - Drew.
>>
>> _______________________________________________
>> mdlug mailing list
>> mdlug at mdlug.org
>> http://mdlug.org/mailman/listinfo/mdlug
>>
>_______________________________________________
>mdlug mailing list
>mdlug at mdlug.org
>http://mdlug.org/mailman/listinfo/mdlug


________________________________________
PeoplePC Online
A better way to Internet
http://www.peoplepc.com