Your laptop may be ready for SSDs, but are your SQL Servers?

I'm hearing back that it's a driver problem, and they're testing a new build of the drivers as we speak.  Fingers crossed.

This is sad scare-mongering, based on half-baked tests.

Even if he was testing in an environment completely provisioned by the vendor, a simple software bug doesn't mean the whole technology isn't ready.

If we ran away from Microsoft software every time a driver problem got in the way, no-body would be using Windows or SQL Server.

FusionIO drives have been used on a wide-spread basis for the past couple of years & in my experience (real world vs Brent's Lab tests) they work very well & I've not seen a single failure in a production environment yet.

Me too - rolling to prod with some SSD's next weekend...

This post was mentioned on Twitter by aaronbertrand: [Blog] : Your laptop may be ready for SSDs, but are your SQL Servers? : http://bit.ly/9QObSV : quick post reflecting on @BrentO's SSD tests

I think your point about due diligence is a good one, but I’m not sure that the premise is fair.  Making a general assumption (SSD’s may not be ready for primetime) based on a single test of one particular brand is a bit of a stretch I think.  Fusion IO is a fairly new player in the SSD market and last I knew the only offering they had was specific to SUN and Oracle.  There are other players in the market with more mature products that I’ve stress tested with SQLIO and never once had an issue. The key differentiator of any SSD product is really the low level IO logic.  It requires quite a bit of intellectual capital to produce and therefore product maturity (in an immature market) does make a difference.

Thanks Pete, two points, one is that FusionIO has certainly been around for some time and that they certainly do have the ability to support Windows and SQL Server (that's precisely what Bremt was testing, and I had a conversation with a PM about that very topic over a year ago); the other is that my intention was purely to make sure people are aware that they should perform adequate tests before making the general assumption that any specific SSD *is* ready for prime time.  I'll repeat what I told Greg, that I am not scare-mongering or trying to get people to stay away from SSDs... just wants to point out that there are cases where they have failed and that, like with all hardware changes, testing is required and that you shouldn't just trust any configuration that happens to say SSD on it.  

I just want to clarify that I wasn't referring to our own environment (as this would be a limited perspective). As you're aware I run a remoteDBA business since 2002 & we've had a growing number of customers running FusionIO on extremely high volume (millions of Tx /min) production systems since 2008.

Despite the huge transaction volumes that have been processed on these systems, we've not seen a single live failure on any of them other than one faulty device that was returned to the vendor prior to production release.

It should also be pointed out that these customers have saved HUGE amounts of money by using SSD b/c achieving the same throughput on HDDs would have been enormously expensive in comparison.

Whilst I agree that all devices need to be tested before released, this is as true of any hardware & there's no reason to single FusionIO out just because somebody runs a single lab experient that didn't work out

Greg, nobody said to forget about FusionIO or even SSDs in general.  Again, just pointing out that FusionIO has *acknowledged* the problem that Brent discovered in their own lab.

I suppose you are suggesting that what happened to Brent -three times- couldn't possibly happen to anyone else.  Maybe there is some I/O characteristic of his test that all of your million tx/sec customers haven't hit yet?

Let's switch the argument to tires.  A couple of tears ago Goodyear had a real problem with their tires failing on certain Ford SUVs.  The failure rates were relatively low but obviously in those few cases the consequences were obviously disastrous.  Now let's say you owned a Ford at that time, and had a friend who was considering buying one, but hadn't heard about the failures.  Do you think it would be fair of you to keep that information to yourself, and let your friend buy the Ford unaware, just because you haven't (yet) experienced the problem?

I would definitely agree with your Ford example, but neither you or Brent seem to have actually "owned" or used FusionIO in any real way & one failed lab test is no reason to "raise the alarm".

As for FusionIO acknowledgements - did they actually say that three devices failed in a row, or just that the same bad driver was used in all three tests? If the problem was simply that someone installed a wrong or out-of-date driver, it wouldn't matter how many devices were tested..

Fair enough, but I don't think Brent's lab test was very valuable, even as a negative testimonial as there are too many unanswered questions.

It's not at all clear that three devices actually failed, as it could easily have been the same software driver failing each time, with no problems in the hardware (who knows if the drivers were re-installed between tests?). I also asked Brent about which specific devices were used (as FusionIO have a few products) but haven't received an answer.

It would be good to know more about what actually happened before reading too much into his experience.

We have conducted our own testing into FusionIO devices, by actually obtaining devices & installing them into our own lab. We haven't had any issues with any of the devices used in our lab (we've had 4 so far).

We have three ioDrives in production, and I *have* run into an issue that sounds similar.  While doing a reload of a 60GB MySQL database, performance slowed to a crawl.  The drive didn't fail, but IOPS dropped to about 5% of what we had been achieving, and didn't recover.  We swapped the card for a spare and restored from backup.

It wasn't until some months later that we discovered the reason behind this.  The ioDrive keeps a pool of spare blocks free that it can write to rapidly without having to do an erase cycle first.  If you hit it with random I/Os fast enough and for long enough, the pool will run out and performance will nosedive.

You can reformat the card to reserve more blocks for this pool, and we've done that with good results, but it's not something that's exactly obvious.

On Linux the drives didn't fail out, though performance was terrible.  Could be different on Windows servers.

Oh, and we're currently alos using large numbers of the Intel X25-E SSDs (up to 16 per array).  While not as fast as the ioDrive, they deliver more consistent performance.

There are a lot of machines that have never been tested with devices as fast as Fusion-io drives... and it's them that's breaking, not the drives.  Even some of the high-end OEM machines can't handle the speed.

Pixy, degradation in I/O performance on FusionIO devices is known to occur if memory is not correctly configured. Unlike HDDs, FusionIO devices require a significant allocation of RAM for normal operation & if both SQL Server & the FusionIO device are competing for RAM, they end up using the page file heavily, with serious performance degradation. Did you check this out?

We generally leave at least 4Gb RAM for the smaller devices (80Gb / 160Gb devices) and more for the larger devices. It's also something that you need to keep monitoring, but this is generally part of wider memory monitoring requirements for busy mission critical servers.

Intel SSDs aren't in the same performance universe as FusionIO yet.

Just to add my 2 pence.  We've just bought 8 of these drives off the back of testing and speaking to a current high end user of these drives.  We've seen no such problems and we've heard back from our contact at Fusion confirming what Greg mentioned about memory allocation as recommended when you purchase the drives.  2GB memory for every 80GB of FusionIO

Hi,

By way of disclosure, I'm one of the team members at Fusion-io.

Let me start by thanking Brent for his interest in Fusion-io. I think

the readers of this blog may want to know that Brent was logged  

into a test machine in Fusion's very own labs. Unfortunately, an overzealous sales engineer was so excited to showcase our performance that he put more ioDrives in the server than the power supply was rated to  

handle.

Let me explain, Fusion's ioDrives are designed to go offline in the

event of a power cut rather than losing the in-flight data. In  

Brent's case, the ioDrives were detecting signs of what they thought might be an abrupt power cut, and so the cards made sure that all the in-

flight data got saved ahead of the potential power loss before safely

powering down.

Thanks again for the feedback. Brent, we understand you may have more

questions, so please feel free to reach out to me and I'll make sure

we have the right people on it. That said, we're surprised that this

is getting so much attention as the cards were working exactly how

they were  designed to work.

Jodi

Fusion-io

Thanks for clarifying this Jodi. It seems your sales engineer wasn't the only person who got overzealous in this exercise.

Perhaps you might want to consider using NDAs when you provide testing environments to stop incidents like this in future.

Greg - if you had questions or complaints about the way I tested the gear with FusionIO, I wish you would have talked to me.  I would have been more than happy to elaborate on it.

In my testing, I had three successive failures.  They were able to reproduce the failures, and the responses I got via email do not line up with Jodi's explanation of too many cards in the server.  I'm waiting to hear back from FusionIO before I go live with a post about the incident.  Frankly, if the cards were really working as designed, I would have expected that the engineers involved would have recognized that rather than having me go through three successive drives, each time telling me the problem was fixed.

I'm mainly disappointed at Brent Ozar's original post & my responses are inteded to focus on his mis-information. Not only did he slap FusionIO around amongst an influential group, he also failed to answer questions (not good)..

You simply referred to his post in a public forum & I felt it was important to respond because I routinely recommend this product and this kind of mis-information can make my life harder when dealing with my own customers and staff (some of who read this blog).

I have a lot of respect for you & highly regard your blog, which is one of only a handful that I read routinely and actually benefit from. You usually provide great information which is useful for real world DBAs, unlike some of the SQL Server MVPs who rant on about useless stuff such splitting IP addresses with single SQL statements.

Your posts are nearly always useful (eg insights into CUs & SPs as well as many other things) On this occassion, I think you missed the mark but hopefully we can move on now.

I defintely apologise if I've come across in any negative way toward you personally - that certainly wasn't intended.

Brent - I asked you for clarification on the devices you were using in the forum thread but you failed to respond.

Not a SQL/DB admin, and no association with Fusion-IO whatsoever. I was directed to this page as an example of what to-do and what not-to-do when using social media.

From reading this entire comment thread, I'd have to say that Greg Linwood's comments are doing the most to damage credibility, because of his insistence that there are no problems.

Here's how I read it.

Brent posts that he had some problems with the drives.  Aaron indicates that this is kind of scary, and bears watching.  Greg cries "scare-mongering" and "half-baked tests" without even knowing what the exact tests were, and cites only anecdotal evidence that there are no problems because he hasn't seen "a single failure in a production environment".

Seems to me that Brent and Aaron were taking a wait-and-see attitude, but noting that there were problems, while Greg is denying problems altogether. That damages credibility.

I'd much rather hear about problems, so that I can follow the process of the resolution make a final decision.

But hey, that's just me.

Steve, I'm not trying to win a popularity contest here, so you might do well to look elsewhere for social media tips.

You said I "insisted that there are no problems" but I actually described a device fault that we encountered, maybe you didn't read the whole thread?

Brent & Aaron's posts go a lot further than saying that problems are "possible" (of course they are). The blog post actually questions whether FusionIO & SSDs are ready for SQL Server generally (refer to the title of this blog post) and I feel it is reasonable to respond to that question. Others have also agreed with me on this in the thread.

Anyway, the information posted by FusionIO rep here seems to confirm that the devices didn't fail after all.

Greg - I don't keep coming back to this blog post to look for questions.  I didn't even know there were comments about what was going on.  Aaron's blog doesn't have a "subscribe to comments" feature, so I have no way of knowing that people are asking me questions here unless I come visit.

If you have questions for someone, leaving comments for them on somebody else's blog is not the way to get them answered.  You will be disappointed with the lag time in results, just as you've been here.

As Steven pointed out, I was taking a wait-and-see attitude.  I waited days for FusionIO to get back to me, and they haven't been able to clarify the problem for me.  I've got a blog post coming out tomorrow to summarize my findings.

Brent, I'm not referring to this blog - I questioned your post to the private NG, to which you still haven't responded.

Greg - what newsgroup are you talking about?

I have been watching this thread with great interest. Among many things, I must say that I was hugely surprised that FusionIO did not cover Brent's tests with an NDA. I used to do a lot of tests on various vendor products and often ran into interesting results. But in almost all cases, I could not talk about them publicly even though I was dying to share the results.

Linchi - yeah, I didn't have to sign an NDA or anything like that.  We set the whole thing up pretty casually over email.

PingBack from http://www.pythian.com/news/8255/log-buffer-177-a-carnival-of-the-vanities-for-dbas/

PingBack from http://topsy.com/tb/bit.ly/9QObSV

Greg Linwood has asked that I publicly clarify that I wasn't thinking about the MVP newsgroups.  Microsoft maintains a private newsgroup server where MVPs can communicate, and because it's private, I don't usually think about it.  A few minutes after my comment here, I realized that he meant those newsgroups, so I went in to check the messages there.  We continued the discussion in the newsgroup.