Search results matching tags 'testing' and 'SAN'

Performance impact: file fragmentation and SAN – Part IV

Linchi Shea — Mon, 22 Dec 2008 16:34:00 GMT

Lies, damned lies, and statistics!

If you have read my three previous posts (1, 2, 3), you may walk away with an impression that on a drive presented from a high-end enterprise class disk array, Windows file fragmentation does not have a significant performance impact. And I’ve given you empirical data—oh yeah, statistics—to support that impression.

But that is not the whole story! No, I didn’t lie to you. The numbers I presented were solid. It’s just that the story is not yet finished.

In these previous posts on the performance impact of file fragmentation, I presented the I/O throughput data as the evidence. The arguments were valid, especially we did see file fragmentation causing the I/O throughput to degrade in a directly attached storage. But I/O throughput is but one I/O performance metric, and it is not enough to look at the I/O throughput alone.

Let me start with an analogy. So suppose we have an eight-lane super highway going from New York City to Los Angles. As we pumping (okay, driving) more cars from NYC to LA, we take measure at a checkpoint in LA to find out how many cars are crossing that checkpoint every hour, i.e. we are measuring the throughput of the super highway. Now, instead of building the eight-lane super highway straight from NYC to LA, we have it take a detour via Boston. At that same checkpoint in LA, we again measure the throughput. Everything else being equal, we should get the same throughput.

However, for a given car, the trip from NYC to LA would take a lot longer on this detoured highway.

An I/O path is similar to a super highway. While its throughput is an important measure, how long it takes for an I/O request to complete—I/O latency or response time—is also an important measure. The question is, will file fragmentation take your I/O traffic for a detour?

Indeed, empirical test data show that when a file is severely fragmented, the maximum I/O latency of large sequential reads and writes (e.g. 256KB reads and writes) can suffer significantly. The following chart shows the impact of file fragmentation on the maximum I/O latency. The data were obtained from the same tests whose throughputs were reported in Part III of this series of posts.

Clearly, when the test file was fragmented into numerous 128KB disconnected pieces, some of the 256KB reads suffered terribly latency degradation. And if your applications happen to be issuing these I/Os, you would most likely experience a performance degradation regardless whether the I/O path can maintain the same I/O throughput.

Having some valid statistics may seem to add force to an argument, which makes it so much easier to be misleading if the whole story is not told, and technically everything is valid, and nobody is lying. By the way, this is a trick often employed by the vendors, who tend to conveniently ignore the bad news, or intentionally bury it with statistics on the good news. In my book, that would be lies, damned lies, and statistics.

Performance Impact: file fragmentation and SAN – Part III

Linchi Shea — Wed, 10 Dec 2008 16:55:00 GMT

256KB Sequential Reads

In my two previous posts (1, 2), I highlighted the fact that while file fragmentation had a huge adverse performance impact on directly attached storage (DAS), it did not have much, if any, impact on the drive presented from a high end enterprise class disk array. That observation was derived from running disk I/O tests with 1KB sequential writes.

What about other disk I/O workloads? In this post, let look at the test results from running 256KB sequential reads on the same DAS and SAN drives.

To see the behavior in even more extreme, I also ran the tests with the test file fragmented into 60,000 fragments. Each fragment was 128KB in size. So I checked three test scenarios:

The 10GB file was created on a freshly formatted empty drive, thus without any fragmentation at all.
The freshly formatted drive was first filled to the full capacity with 2MB files, and then some of these 2MB files were randomly deleted to make sufficient room for the 10GB test file. In this case, the 10GB test file was fragmented into more than 3000 non-contiguous fragments.
The freshly formatted drive was first filled to the full capacity with 128KB files, and then some of these 128KB files were randomly deleted to make sufficient room for the 10GB test file. In this case the 10GB test file was fragmented into more than 60,000 non-contiguous fragments.

The following chart shows the results of many repeated tests, applying 256KB sequential reads at various load levels.

Again and clearly, as is the case with the 1KB sequential write tests, severe file fragmentation had no impact on the disk I/O performance of 256KB sequential reads on this drive presented from a high end enterprise class fibre channel disk array.

Note that the disk array had a cache that was much larger than 10GB. You may say, that’s cheating and not a fair comparison with a directly attached storage. Well, maybe, but that’s how high end disk arrays work.

How did Random I/Os Outperform Sequential I/Os?

Linchi Shea — Wed, 04 Apr 2007 19:14:00 GMT

Recently, when I was doing some I/O performance tests on an I/O path, I found that 8K random reads (and writes) significantly and consistently outperformed 8K sequential reads (and writes) in terms of I/O throughput (megabytes per second). I was puzzled.

With a traditional hard disk that is made up of a stack of magnetic platters held by a spindle, an electro-mechanic access arm, a printed circuit board, and a hard-case enclosure, the average seek latency is always significantly higher than the average rotational latency (e.g. 9.5ms vs. 4.2ms on a 7200 rpm 500GB disk). And therefore, the random I/O throughput is always expected to be significantly lower than the sequential I/O throughput.

In fact, sequential I/Os have such a huge performance advantage over random I/Os that the computer industry has labored over the past few decades trying very hard to reduce random I/Os and convert them to sequential I/Os with such techniques as caching, transaction logging, sorting, and log-structure file systems.

Granted, the I/O path I was working with was not a traditional hard disk. It was a LUN presented from a SAN with a large amount of cache, and to simplify to some extent, the LUN was a RAID 0 stripe set across 12 virtualized drives with a rather large stripe unit size (960K). But how should I explain why 8K random I/Os could outperform 8K sequential I/Os?

After some discussions with a storage professional, we came up with a theory consisting of the following three key factors:

Random I/Os were able to effectively hash I/Os across multiple drives that make up the RAID 0 device.
Relatively large RAID 0 stripe unit size of 960K caused 8K sequential I/Os to cluster around the same drives. Note that it would take 120 sequential I/Os to fill a single 960K stripe.
A base amount of cache was assigned to each drive in RAID 0. And when random I/Os were hashed across 12 drives, the I/Os benefited from larger amount of cache.

Do I have solid proof that these three factors were the root cause of 8K random I/Os outperforming 8K sequential I/Os? No, I don't. But I do have some circumstantial evidence supporting the theory.

First of all, if the theory is correct, I should see the same behavior with smaller I/Os such as 1K reads and writes. Indeed, 1K random I/Os outperformed 1K sequential I/Os on the same I/O path.

Secondly, if the theory is correct, I should not see the same behavior with larger I/Os, especially I/O block size that is not much smaller than 960K. Indeed, 128K random I/Os did not outperform 128K sequential I/Os on the same I/O path.

Thirdly, if the theory is correct, I should not see the same behavior on an I/O path that has fewer drives. Indeed, on a RAID 0 device with three drives in the same SAN, 8K random I/Os did not outperform 8K sequential I/Os.

Finally, if the theory is correct, I should not see the same behavior on a RAID 0 device with much smaller amount of cache. Indeed, on a directly attached RAID 0 device, 8K random I/Os did not outperform 8K sequential I/Os.

Now as mentioned, I'm not 100% confident about this theory. I can't prove it beyond reasonable doubt. Hopefully, some of you reading this blog post know exactly what caused or could have caused 8K random I/Os to outperform 8K sequential I/Os. And if my explanation doesn't match up with yours, I'd love to hear your comments.

Should I Use a Windows Striped Volume?

Linchi Shea — Mon, 12 Mar 2007 16:25:00 GMT

In Windows Server 2003, you can use the Disk Management console to create a striped volume over multiple dynamic disks (well, you can also create a mirrored, a RAID-5 volume, etc). If these disks (or LUNs) are presented from a SAN, most likely you can stripe across the same storage devices--making up these LUNs--inside the SAN to present a single LUN to Windows.

The question is: if you can create a striped set either inside SAN or at the Windows level, is there any performance advantage of doing it at the Windows level?

From what I have seen so far, you are probably better off using a SAN-based striped set rather than create a Windows striped volume. Although in no way I'm recommending this in all cases, I do have some empirical data to back myself up.

Recently, I had three 32GB LUNs presented to a server. I first used the Windows disk management tool to create a 96GB Windows striped volume on top of these three LUNs, and conducted a series of I/O performance tests on the Windows striped volume. Then, I returned the three LUNs to the SAN, and a 96GB LUN was created to stripe across the same devices that used to make up the three 32GB LUNs. Finally, the same I/O tests were conducted on the 96GB SAN striped LUN.

The I/O tests included 8K random reads, 8K sequential reads, 8K random writes, and 8K sequential writes. There was no difference in I/O throughput between the two configurations when it came to 8K random/sequential reads and 8K sequential writes. However, the SAN striped LUN significantly outperformed the Windows striped volume for 8K random writes, as shown in the following chart.

Now, how much of this 8K random write difference may translate into the performance difference in SQL Serer transaction throughput? Probably not as much or not a lot. But unless there are the numbers to show some performance advantage of using a Windows striped volume over a SAN striped LUN, why use it?

By the way, note that to create a Windows striped volume, you need to convert the LUNs to dynamic disks which are not supported in a Microsoft Cluster Server.

Is RAID 5 Really That Bad?

Linchi Shea — Wed, 07 Feb 2007 14:31:00 GMT

RAID 5 is a dirty word in the DBA community and beyond. There are websites devoted to trash RAID 5. I've seen DBAs declaring performance root cause found the very moment they found out that some database files were placed on RAID 5 volumes. You'd be ridiculed and run out of town if you dare to suggest putting the transaction log file on RAID 5. Is this knee jerk reaction to blaming RAID 5 for a database's performance failings really justified?

Why am I even asking this question? Isn't it already settled that RAID 5 is bad for writes? Well, only if life is that simple.

Below are two charts showing the performance of the same 8K sequential writes at various I/O queue depths applied to two RAID volumes presented to the same server: one is RAID 5 and the other is RAID 10. Both of them are presented from SANs. For all the data points in the charts, sqlio.exe--which is a free download from Microsoft--is configured with 32 threads to generate the I/O loads.

In the charts, the RAID 5 volume sustains much higher I/O throughput than does the RAID 10 volume, and it has significantly lower I/O latency than does the RAID 10 volume. Before you question the validity of the data, let me assure you that the performance difference illustrated in the charts are no fluke. The pattern is consistently reproducible.

How can RAID 5 outperform RAID 10 on 8K sequential writes?

Let's note that behind each of the two volumes is an I/O path consisting of many hardware/software components. Some of them are shared between the volumes, others are not. Many of the components have a significant impact on the I/O throughput and latency of the volume; they include HBAs and the software that manages the I/O paths on the host, SAN architecture, model of the SAN frames, cache on the SAN, RAID configuration, specifications of the physical disks inside the SAN frames, configuration of hardware hardware replication, number of spindles used by the volume, and so on.

So the RAID configuration is but one of many factors that influence the I/O performance of a volume. Differences in some of these other factors can result in a RAID 5 volume outperforming a RAID 10 volume on 8K sequential writes. The easiest example is that if the RAID 10 volume is enabled for synchronous replication in the SAN.

Okay, you can accuse me of comparing apples and oranges--i.e. it's not strictly RAID 5 vs. RAID 10 in the charts. Guilty as charged. But when you automatically declare the performance root cause found at the first sight of RAID 5, you have just committed the same fallacy of comparing apples and oranges.

I'm not disputing that RAID 5 is not as good as RAID 10 for writes with everything else being equal. However, the fact of life in a real enterprise SAN environment is that a RAID 5 volume and a RAID 10 volume almost never differ only in their RAID configuration. In such an environment, to be certain about the performance of a volume, you need to actually evaluate/test it. The information on the RAID configuration alone is far from being sufficient in making a recommendation.

If you have been asked to decide on which of these two volumes to place the transaction log file of a database, how would you proceed?

Beware of Shifting SAN

Linchi Shea — Thu, 04 Jan 2007 02:18:00 GMT

Let’s say you are trying to determine the performance impact of a neat database design change you have just devised on an application. So you run some tests with the existing design and the tests run for several hours. Coming back the next day, you make the change and re-run the same tests. The test results look fantastic. Now, before you jump up and down announcing to the world how great your new design trick is, double check whether your change is the only variable responsible for the performance improvement.

If you are using Storage Area Network (SAN) and storage is a significant factor in your tests, there is a danger that your conclusion may be built on shifting sand because the performance of the drives provisioned from SAN may have changed between your tests. Unbeknown to you, the test results reflect primarily, not your design change, but some uncontrolled change inside SAN.

I ran into a similar situation when I was checking the performance impact of disk partition alignment (or misalignment). Good thing that the SAN change didn’t happen between tests of different configurations, but took place when I was repeating the same tests. So I caught the change right away. The following chart shows that when I ran the exactly same SQLIO benchmark tests for the 3rd and 4th time, the I/O performance profile of the drive changed dramatically.

Your SAN performance may shift underneath you for many reasons, some good and some not so good. You can try to cozy up to your SAN folks. But that would not guarantee you the inside knowledge of all the changes. Nor would you want to know all the nitty-gritties happening inside SAN.

The best way to guard against this situation from misleading you to an embarrassingly wrong conclusion is to randomize your test schedule. You may want to alternate between the different test configurations as you carry out your tests.

Say, you are testing the performance difference between config 1 and config 2. So instead of doing all the tests on config 1 on day one and all the tests on config 2 on day two, schedule your tests so that config 1 tests are interleaved with config 2 tests throughout the two days. If that’s not possible, conduct all your config 1 tests followed by all your config 2 tests as usual. But before you draw any conclusion, repeat some of your config 1 tests to verify that the results you obtained previously are still repeatable.

Now, don’t get me wrong. I think SAN has been a boon to our industry. It pushes storage down the infrastructure stack and abstracts us away from the nasty details of its management. The adoption of SAN moves us closer to the grand vision of utility computing (okay, not that we are ral close to that vision).

Today, many enterprises have deployed SAN of one kind or another, and some large enterprises are exclusively relying on SAN for storage provisioning. You just have to learn to live with its idiosyncrasies.