Benchmark Update - Astounding Fujitsu RX900 8-way Xeon 7560 TPC-E Scaling

I am only concerned with differences that impact performance. Normally I do not care about differences in price performance, as it is usually a silly game. The TPC price-configuration is also an artificiality of the benchmark. By rule, the database size is scaled with performance, and the active data is distributed across the entire database. In real life, a transactional database might be 10+TB, but usually the active data is highly concentrated. With proper cluster key design, it is possible to bring most of the active data into memory, greatly reducing disk IO.

In the few production environments that have active that cannot be concentrated, well then, massive disk IO is the only answer. Also, in a live transaction server that will be driving 80K RPC per sec, the total cost of the environment will be in the $100M-2B range. No one will give a damn about $1M. If the system we to fail, it will be your @$$. So people tend to look for the $10M solution when a $1M solution works. Of course, spending tens of millions on storage did not help the state of virginia in their outage.

TPC-E IO is mostly read, so RAID level has negligible impact on storage cost. For HDDs, it is the number of spindles necessary to support the IO load. For SSD, it is capacity. I think in the live system, it might be SSD for the most active data, HDD for less active and archival data, plus a few backups, plus a restored backup, plus snapshots.

sure, but TPC benchmark system configuration cost and real world system costs have no relation. Almost all TPC configurations are direct-attach storage. Very very few use SAN. Almost all large production OLTP systems use SAN storage. Direct-attach storage with 146G 15K HD might be $600 per disk on the low-end, $750 per disk on the NEC. A SAN with the 15K 146G drive might run $2000-4000 amortized per disk. The SSDs in the Fujitsu are $1100 each, plus $200 per disk for the enclosure. It still bothers me that the Crucial 64GB SSD is only $150. SSDs for a SAN might be hugely more. Yet people have no problem paying the 4-6X premium for SAN storage.

This is why I say TPC system costs are irrelevent. It may seem obvious to you that people would want to save money. The reality is people buy the really really expensive storage system (on a per disk basis). Then when they need to save money, they sacrifice performance by cutting the number of disks, then by going to even fewer big capacity disks to reduce the enclosure cost, to guarantee crappy performance. Then people interested in saving money go with RAID 5 and sharing the storage system, as if performance was not already bad enough,

Anyways, the TPC configuration is really just what is needed for the benchmark specification. I would configure a hybrid SSD/HDD. The price difference between the Fujitsu and NEC is not hugely relevent, a real system is going to be much more expensive. Right now, no direct-attach storage system supports clustering, even though SAS should be able to do this. Since we have to use SAN for clustering, yet no commercial SAN system is properly architected for SSD. All the main SAN systems use 3.5in disks with too few bays per enclosure, and daisy chain too many enclosures sharing the same IO channel.

The right way to do a storage system is 8-12 bays per IO channel, 8 for SSD, and 4 for HDD. And we may as well use 2.5in disks, not 3.5.

I do not see a technical problem with the way TPC-E stresses CPU-memory because they configured the storage system correctly. The problem with real world is people who don't know what they are doing configure storage incorrectly, and end up IO crippled. Sure we could build a benchmark that is required to be IO crippled, but all this would prove is we can be just as stupid as real world IT pretenders/dilettantes. They are why 30% of major IT projects are outright failures and another 30% seriously under achieve key objectives, These people do not belong in this profession, giving the rest of us a bad name.

Ok, lets try this yet one more time.

The benchmark performance characteristics have meaning, especially the scaling from 2->4->8P, the difference between Xeon and Opteron and average response time between low queue HDD, high queue HDD and SSD, while the system cost does not. That why I focus on the parts that have meaning, and ignore the parts that do not.

All large (4-way+) and almost all medium (2-way) OLTP system I have seen are on SAN storage, and all are clustered too (a significant reason for SAN). The critical ones have dedicated SAN, usually this is because the DB team doesn't trust the SAN team. Potentially an OLTP system could use mirroring, but the ones I have seen cluster locally (with SAN), mirror to remote site. If your customers with large OLTP system are not on SAN, then we have different customers.

I didn't say people have unlimited budget. I said people put large OLTP systems on SAN storage, which has a totally different cost structure between HDD and SSD than the direct-attach storage in TPC benchmark reports. So the cost savings in going from 1872x147G 15K HDD to 336x64G SSD in a direct attach storage system has no relation to the cost structure of a similar switch in a SAN system. Hence, I don't pay attention to this.

If people ran TPC-E benchmarks with the typical TPC-E configuration, I would pay attention to its cost structure. But people run their own OLTP applications on SAN system, so I pay some attention to SAN cost structure.

ps. I strongly recommend direct-attach storage for DW because should have no firm clustering requirement, hence no requirement for SAN.

pps, oh yeah, one more thing. The financial decision point between HDD and SSD is really one of capacity versus IOPS. On HDD, capacity is effectively free (direct-attach $500 per 146GB, no quite on SAN ~$2K per 146G) so the cost is in the IOPS - 200 at low latency. For SAN, the IOPS are nearly free at typically 25K IOPS per SATA/SAS device (Fusion-IO might be 1M, but it is equivalent to an arrays of SSDs) but capacity is "expensive". So in most situations, we can just compare the HDD IOPS cost to the SSD GB cost with the type of storage we are interested in, direct-attach or SAN.

For each customer application, there will be a specific IOPS/GB ratio, to be equated to the storage cost structure. The TPC-E and C have specific IOPS/GB ratio per benchmark rule. Again, the TPC-E IOPS/GB ratio is driven the benchmark rule, has absolutely no direct mapping to every different production environment's IOPS/GB ratio. Hence the cross-over point in TPC-E and a specific production environment is different, even if they use direct-attach storage. So why you waste time looking at TPC-E total system price. (the component prices is helpful because you can use it to calculate the cross-over point for your environment)

ppps. It is not necessary for a benchmark to be exactly like your application to be useful. In fact it is impossible for there to be benchmarks like every application. This is why it is important to break down a result in to components that can be mapped to your application. Too many people get lost in the superficial metrics specific to the benchmark, but with no meaning in their application.

Great spot Rob, actually this is valid, alteast for many well- designed server apps, HT does help, by 30% or more, more cache helps alittle too. I wonder how many people are running server apps on their Gateway SX2841-09e. Of course, uou are already paying the modest premium for HT and 24M LLC on the Xeon 7560. Still, I don't think this is a good idea. Its really better to have two SKUs. Asking customers to do something never works. Or it should be mandatory that this upgrade be done at time of purchase by the Geek Squad techs co-located with Best Buy.

Also, it will be xmas soon, we could ask Santa for the magic button to make our servers run faster.

My understanding is the R2 limit is actually 1024 logical processors, but there may be a split between Enterprise Edition, for which CAL is allowed, and Datacenter Edition, for which only per processor licensing is available. MS policy is not announce support for more than they have tested. So the enumeration scheme may work for more. 1024 lp could be 512 cores with HT, which would be more than a Superdome2 with 64 sockets, quad-core = 256 cores. I suppose HP has a 16 or 32-way Xeon 7x00 in development. I also assume Westmere-EX has taped out + first silicon. So 32 x 10 x 2 = 640?

My advice to Intel is to increase threads per core to 4, meaning 1280, because proper coding and continued work should make better use of HT. Of course, I do not know if the Intel cpu people read my blog and followed my advice verbatim.

TPC-E was put codeveloped by all the major software and hardware vendors, Microsoft, Oracle, IBM, Sybase etc. Everyone recognized that TPC-C specifications was distorting system configuration as time progressed due in part to the different rate of performance growth in the components of the complete server system. The other factor was that as processors becames immensely powerful, people threw in lots of read queries, while TPC-C remain pure transaction processing. This generates an even mix of read/write (a read forces the eviction of a dirty page).

The TPC-C disk IO pattern has become so difficult that neither the Oracle/Sun 7.6M and IBM 10M tpm-C result could use HDD, both used SSD storage, which is also very challenging for such a large database.

I think MS want the switch to TPC-E with SQL Server 2008. But HP still wanted to put out TPC-C for thier own reasons. So MS allowed this for SQL Server 2005 (which will mean that no > 64 logical procs). Another reason their will not be higher SQL Server TPC-C on a single database is that TPC-C generates intensive log writes, that is pushing the limit of what can be written in serialized fashion given current write latencies and the SQL Server log write in-flight limits. I do not think MS will introduce multiple concurrent log file system.

I do think it is possible to distribute the TPC-C load across multiple databases on a single server and instance to get around the log write issue. Perhaps we can ask HP if they would try this on their DL980G7.

Anyways, Oracle was a codeveloper of TPC-E. They have no complaint on the technical aspects of TPC-E. As I said probably in the Big Iron III blog, Oracle in fact runs TPC-E just fine, and can probably put out a very impressive big system scale-up result, possibly better than SQL Server. The reason as I understand it that Oracle will not publish a TPC-E is that then they would be obligated to put out a Oracle RAC TPC-E, which is the problem. TPC-C naturally partitions into localizes activity, with only a small fraction of cross district  transactions, ie, almost everything can stay on a RAC node. This is not the case with TPC-E. Oracle has made it company policy to push scale-out via RAC, not the agnostic scale up or out. But Oracle just changed the ODM to include 8-way, so they may put out a 8-way TPC-E.

Have you ever heard the saying: don't argue with the village idiot? people may not be able to tell which one is the idiot. I suggest not arguing with Oracle fanboys. Oracle is indeed a very sophisticated product (that used to require a great deal of knowledge to implement). Many simpletons think if they use a sophisticated product, then that makes them sophisticated too. The idiot can leave the village, but somethings won't change.