THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

Alexander Kuznetsov

For better precision cast decimals before calculations

I frequently do calculations against decimal values. In some cases casting decimal values to float ASAP, prior to any calculations, yields better accuracy.  For example, in the following script both expressions should return 12100.0:

 

SELECT POWER(POWER(12100.0, 0.01),100), POWER(POWER(CAST(12100.0 AS FLOAT), 0.01),100)

--------------------------------------- ----------------------

13780.6                                 12100

 

 

Another example demonstrates poor accuracy of averaging decimals as compared to averaging floats:

 

CREATE TABLE #t(d DECIMAL(6,1));

GO

INSERT INTO #t

SELECT 1 UNION ALL SELECT 0 UNION ALL SELECT 0;

DECLARE @avg FLOAT;

SELECT @avg = AVG(d) FROM #t;

SELECT @avg AS [avg], @avg*3 AS imprecize_avg_by_3;

SELECT @avg = AVG(CAST(d AS FLOAT)) FROM #t;

SELECT @avg AS [avg], @avg*3 AS precize_avg_by_3;

 

avg                    imprecize_avg_by_3

---------------------- ----------------------

0.333333               0.999999

 

(1 row(s) affected)

 

avg                    precize_avg_by_3

---------------------- ----------------------

0.333333333333333      1

 

 

Of course, explicit casting to a decimal with higher precision is also an option:

 

DECLARE @avg FLOAT;

SELECT @avg = AVG(CAST(d AS DECIMAL(38,18))) FROM #t;

SELECT @avg AS [avg], @avg*3 AS precize_avg_by_3;

 

avg                    precize_avg_by_3

---------------------- ----------------------

0.333333333333333      1

 

Published Saturday, December 20, 2008 10:38 PM by Alexander Kuznetsov

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

David L. Penton said:

Calculations are all about significant figures.  Prior to your final calculation, as long as you properly carry your significant figures (http://en.wikipedia.org/wiki/Significant_figures) you are fine.  Then you can round to what is appropriate for your final answer.

December 20, 2008 11:48 PM
 

AK said:

Well, prior to my final calculation:

POWER(POWER(12100.0, 0.01),100)

I carry my significant figures all right, but the result is 13.88% wrong - 13780.6 instead of 12100 ;)

December 21, 2008 10:13 AM

Leave a Comment

(required) 
(required) 
Submit

About Alexander Kuznetsov

Alex Kuznetsov has been working with object oriented languages, mostly C# and C++, as well as with databases for more than a decade. He has worked with Sybase, SQL Server, Oracle and DB2. He regularly blogs on sqlblog.com, mostly about database unit testing, defensive programming, and query optimization. Alex has written a book entitled "Defensive Database Programming with Transact-SQL" and several articles on simple-talk.com and devx.com. Currently he works at DRW Trading Group in Chicago, where he leads a team of developers, practicing agile development, defensive programming, TDD, and database unit testing.

This Blog

Syndication

Powered by Community Server (Commercial Edition), by Telligent Systems
  Privacy Statement