THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

Linchi Shea

Checking out SQL Server via empirical data points

Elephant in the room: TPC-E

Over the past few months in several semi-formal occasions, I ran into folks from your well-known vendors (minus Microsoft). Some of the folks were from the vendors’ performance labs and were involved in conducting benchmark tests and publishing benchmark results. So naturally they were very happy and eager to talk about performance, I mean any database performance, until the topic of TPC-E came up. They might not be squirming. But it’s hard to be mistaken that they really didn’t want to talk about it or go anywhere near it, especially not on any sort of record.

That just felt weird!

On one hand, you have this industry standard database OLTP benchmark that all these firms were on board the Transaction Processing Performance Council in its official approval and release in February 2007. And it was supposed to eventually replace TPC-C. On the other hand, for all these years Microsoft has been the only database platform vendor in releasing any TPC-E benchmark results.

It has been four years since the benchmark’s release, and I don’t think everyone can still claim they are evaluating it and trying to work out the wrinkles. So there must be very solid reasons for the other DBMS vendors to not touch it. And yet no one wanted to discuss those reasons publicly. Sure, you hear things through the grapevine. But the silence from the official channels is just deafening!

Or have I missed something really obvious?

Published Saturday, January 07, 2012 1:21 AM by Linchi Shea
Filed under: ,

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

Scott Newman said:

Here's a blog post that may explain why you're not seeing a lot of other db vendors use TPC-E:

http://iablog.sybase.com/paulley/2008/10/the-state-of-tpc-e/

Not saying I buy most of it, but there are a couple good points.

January 7, 2012 9:44 AM
 

Stephen Colbert said:

You sir, get a "nod of the head".

January 7, 2012 11:34 AM
 

Peter Nordal said:

It is possible that C & E are sufficiently different, that optimizing for one requires sacrificing enough performance in the other to marketing-mandate an either-or choice.

MS was the new kid on the block when E was released, had little in the way of an installed base, hence little to lose, and chose the E path.

The others were already so far down the road to C optimization that changing focus to E would mean knowingly give up some C performance. Which is hard to do, when most revenue comes from current customers upgrading, hence needing to see a clear reason (improved benchmarks) to fork over for the new version.

January 7, 2012 4:14 PM
 

Linchi Shea said:

Peter;

It's pretty easy to guess what might be the reasons for not using TPC-E and why they didn't want to talk about it. And we do hear a lot of things informally. I have no problem that people wanted to protect what was in their best interest.

Still, I think this is done wrong. If they don't want to use it, why not fight that battle out at the benchmark design and evaluation phase? It's better to refuse to approve it or openly/officially challenge it than to pretend it didn't exist. Now that TPC-E is an industry standard benchmark, these vendors owe it to the customers to make their positions/plans on TPC-E known publicly and officially. I think it is probably to their own benefit to do so. Otherwise, you can't blame people for suggesting that these vendors didn't publish any TPC-E numbers simply because they had tried it in their labs but their numbers sucked big time.

January 7, 2012 4:47 PM
 

Linchi Shea said:

Scott;

That blog post was from 2008, and I'd love to see an up-to-date follow-up by the same author.

January 7, 2012 5:09 PM
 

Bill Anton said:

Hi Scott,

that blog post, while informative, is over 3 yrs old and it appears Microsoft is still the only db vendor allowing TPC-E results to be published (http://www.tpc.org/tpce/default.asp)

In the comments, I think the post by Charles Levine nailed it... "Customers hold onto TPC-C because it is familiar and available, not because it is better. Database vendors need to exercise leadership by embracing a superior benchmark that will drive customer-relevant engineering innovation."

January 7, 2012 8:04 PM
 

Greg Linwood said:

I agree - they shouldn't agree the benchmarks & then fail to allow results.

Although Microsoft is still the only vendor to publish under TPC-E, the lack of competition shows in the kit quality. They have TOP n used without ORDER BY (inconsistent results) and also poor index choices (we were easily able to run it faster with better indexes).

I hope Microsoft improves kit quality & the other vendors jump in with some more competition rather than just bitching from the sidelines.

January 8, 2012 7:20 AM
 

Linchi Shea said:

Just read the following at http://www.microsoft.com/sqlserver/en/us/product-info/benchmarks.aspx:

"Leader in TPC-E

SQL Server is the uncontested leader in TPC-E"

Technically true. But when you are the only one doing it, it still feels funny.

January 13, 2012 11:54 AM
 

Ty Yosh said:

Greg Linwood,

Can you share details about the missing indexes you discovered?  I have not been able to spot anything just by doing various things with the kit database, and would love to make it do better.

Thank you

January 18, 2012 10:35 PM
 

Michael Morris said:

If you run Database Engine Tuning Advisor you will see that it suggests creating a number of additional indexes.

Just open up the Stored Procedures and take a look at how its running to see some problems.

January 27, 2012 10:05 AM
 

Paul Wehland said:

There is finally a new TPC-E published with Windows 2012 ability to address 4TB of RAM!

Performance jumps another 18.3%. to 5,457.20 TpsE

March 18, 2013 2:33 AM

Leave a Comment

(required) 
(required) 
Submit

About Linchi Shea

Checking out SQL Server via empirical data points

This Blog

Syndication

Powered by Community Server (Commercial Edition), by Telligent Systems
  Privacy Statement