A cautionary tale about grandfathering CAL licenses in SQL Server 2012 Enterprise

AaronBertrand — Fri, 27 Apr 2012 16:05:00 GMT

I've tried to keep this post to a minimum, but there's a lot to say. Please bear with me.

I know there has been a lot of griping about the new licensing model in SQL Server 2012. I understand the outcry; folks on high-end processors are going to end up paying much more for the same licenses, and some are even bypassing the upgrade because of it. My feeling is that we've had a relatively easy ride since hyper-threading was first stable enough to trust with SQL Server. I also feel that it is fair for us to be paying based on the power we're utilizing, regardless of how many cores are bundled together in each physical CPU. Essentially we're only now starting to pay for the computing power we've been taking advantage of for several years, and many of the other vendors have been much quicker about making their customers pay for this power.

That said, there is a lot of undue panic as well. I've heard many people relay presumptions that their licensing costs will now quadruple (or worse), because they thought that Microsoft was keeping the $27K Enterprise processor price tag and just applying it to each core instead. As I explain in my "What's New in SQL Server 2012" presentations, the new core licensing model doesn't hurt you if you've already been paying for Enterprise processor licenses on previous versions, unless you are deploying to high-end processors (more than 4 cores) or if you're continuing to use a processor with one or two cores (since you need to buy the licensing in pairs, and 4 cores is the minimum IIRC). The cost per core is almost exactly 25% of the previous per-processor license cost; meaning that if you have quad-core processors, you're paying about the same as you did before. And folks using certain AMD processors even get a bit of a break, if those CPUs are a fit for their environment, as Glenn Alan Berry outlines.

The Real Problem

The real problem with this release, in many eyes, is that Microsoft took away CAL licensing for Enterprise Edition. CAL obviously made it much cheaper, if your situation warranted it, since you paid by the user/device rather than by the number of cores. So anybody who has been using CAL licensing up to this point has a choice to make with SQL Server 2012: pony up the much higher licensing costs, or move down to Standard Edition (CAL or core-based) or Business Intelligence Edition (CAL only). The latter choices mean they give up all of those Enterprise features they've already been using - never mind the new ones that they probably want, like AlwaysOn + Availability Groups.

The Exception to the Rule

Now, there is an exception: if you have current software assurance (SA) for SQL Server 2008 R2, this allows you to slide into SQL Server 2012 while maintaining CAL licensing (which you can't do if you are buying new SQL Server 2012 licenses - those have to be core-based, always). The term I've heard most often for this is "grandfathering." Theoretically, grandfathering allows you to enjoy all the Enterprise Edition benefits of SQL Server 2012, while pushing off the much higher core-based licensing costs until the next version that you upgrade to. But there's an important catch in the way this exception works, that many customers seem to overlook. From http://www.microsoft.com/sqlserver/en/us/get-sql-server/licensing.aspx (hidden by default under the "Licensing by users - Server + CAL licensing"):

Existing Enterprise Edition licenses in the Server + CAL licensing model that are upgraded to SQL Server 2012 and beyond will be limited to server deployments with 20 cores or less. This 20 core limit only applies to SQL Server 2012 Enterprise Edition Server licenses in the Server + CAL model and will still require the appropriate number/versions of SQL Server CALs for access.

(Emphasis mine.) I know of at least one customer who rushed out to buy Software Assurance for SQL Server 2008 R2 CAL licensing for a handful of servers, with the assumption that he would be able to upgrade later to 2012 and keep the CAL licensing. He missed the footnote above and now realizes that his 48-core servers are actually not eligible for CAL licensing. SQL Server will install, of course, but it will only see 20 cores.

But it Gets Worse

Multiple customers *did* upgrade their servers through Software Assurance, and all have been surprised by the 20-core limit. One customer's server, in particular, showed serious performance degradation after the upgrade. Because they didn't know about the 20-core limit, they didn't realize that their performance was worse after upgrading simply because of it. The limit is not just a line item in your licensing agreement, SQL Server actually puts a hard limit on how many logical processors it will identify and use, which meant in this case that their 2008 R2 instance was using all 40 cores, but when they upgraded, it was only using 20. This was not the upgrade experience they expected.

So, why didn't they know about the 20-core limit?

Their licensing rep didn't tell them. It's surprising to me that a licensing rep for a company would sell them upgrade licensing under a very specific grandfathering clause, without knowing anything about their environment (or asking), or at the very least warning them about the limit. This seems to be a pretty important facet of the licensing exception.
Setup doesn't prompt them for anything. With Volume Licensing, you get pre-pidded installation binaries. You don't choose the licensing model or the number of processors/cores like you did in the old days. Here is an example of the edition choice screen for a VL version of setup (click to enlarge):

At no point does the visible portion of setup mention or offer in any way that the server will be limited to 20 cores or that this product key does, in fact, represent CAL licensing. The person installing the software may have no idea what the licensing should be, and may be putting it on servers where it wasn't intended. I think Microsoft can do better in preventing these things from happening by putting some very obvious indication about what type of licensing is about to be installed.
The footnote on the web site is not highly visible. I think that the grandfathering exception for CAL-based licensing should be much more visible. As if licensing isn't a complex enough topic already, now customers have to deal with licensing restrictions that aren't well-publicized and that aren't disclosed when they're actually purchasing the licenses.
The EULA does have some information, but it's way down, not trivial to decipher, and you have to actively scroll to find it. Here is what it says ("OSE" means "operating system environment"):

MICROSOFT SOFTWARE LICENSE TERMS
MICROSOFT SQL SERVER 2012 ENTERPRISE SERVER/CAL EDITION

...about 70 lines of typical EULA mumbo-jumbo, then item 2.2 states (in part):

Running Instances of the Server Software. Once you have assigned the license to the server, you may run any number of instances of the server software in up to four OSEs (physical and/or virtual) on the licensed server at a time, provided that:
(a) if you are running the software in a physical OSE, the OSE may access up to 20 physical cores at any time

Which means that, even if you install multiple instances on the same OS, you won't be able to split your actual cores between instances - they can all see only the same 20 cores (and the rest will sit idle). It seems you can get around this somewhat by having four virtual machines, provided you can set affinity correctly, but any single VM still cannot make use of more than 20 cores.
I'm of course not going to suggest that it's a valid excuse that they didn't read the EULA. But this is the only place where they get a warning when working with the product itself or the licensing reps, who are supposed to ensure compliance?

There were two clear indicators for this customer that they weren't utilizing all 40 cores. But they had to dig deeper than they should have to find this out.

Performance Advisor v7 (now in beta) breaks out Windows and SQL Server memory by NUMA node on its dashboard. The customer upgraded to v7 and immediately noticed that only two of the four nodes were showing up; SQL Server had automatically taken the other two nodes offline (click to enlarge):

The server was also suffering strange performance issues, which led to...
...checking the SQL Server error log. In this case it actually tells you that the logical cores it is going to identify and use has been limited to 20:

04/24/2012 18:04:23,Server,Unknown,Detected 262133 MB of RAM. This is an informational message; no user action is required.
04/24/2012 18:04:23,Server,Unknown,SQL Server detected 4 sockets with 10 cores per socket and 10 logical processors per socket 40 total logical processors; using 20 logical processors based on SQL Server licensing. This is an informational message; no user action is required.

I wonder how long, without the odd symptoms in Performance Advisor, the customer would have spent assuming that SQL Server 2012 was simply not ready for production? How long before they would have spotted that message in the error log? How long would it take to get this information out of their licensing rep, if they hadn't already been armed with it due to their own digging?

The issue I see with all of this is that most customers won't see this issue until they move to production. Why?

Many customers have budgets that are spread thin enough (especially with the increased costs of core-based licensing) that they can't always afford a test or QA machine with the same number of processors as their production machines. So when they run their tests against a 16-core box, they expect the production box to perform better, and to not have flaky symptoms due to SQL Server thinking that some NUMA nodes are only partially online or completely offline. And there is no way for them to come across this 20-core limitation until they deploy to production. This is not an excuse for having insufficient hardware in test and QA environments, but the reality is that most customers can't do it.
Even for those customers that *do* have big enough hardware in their QA/test environments, I haven't verified, but I can only assume that the MSDN versions of the software (which most customers use to test their deployments in non-production environments) do not have these built-in limits of 20 cores. If they do, that's a real problem for all of the other customers. If you test on the MSDN version (which also doesn't let you specify whether you're using CAL- or core-based licensing), you're not testing the exact functionality you'll get in the real production systems, and again you'll have a surprise waiting for you when you do.

What Am I Expecting Here?

Now I don't think it's reasonable to ask for any of the actual licensing policies to change, and I'm sure they anticipated certain objections when they decided on this path in the first place. So I don't think that would do us any good. What I would like to see, however, is better disclosure. Those licensing reps and account managers should never let a customer buy software assurance or upgrade directly to SQL Server 2012 under CAL grandfathering without verifying that they will be eligible based on the number of cores in their server(s). Assuming that they have read some footnote on a web page and are completely aware of the issue is the opposite of customer service.

More importantly, my expectation here is that I hope to prevent other customers from rushing into Software Assurance contracts for SQL Server 2008 R2, with the expectation that they can upgrade to SQL Server 2012 and keep their CAL-based licensing regardless of the size of their servers. When you get into conversations with your account manager, *please* be sure that you discuss this issue, and get their promises in writing.

As some of you already know, I've provided some feedback to Microsoft about this, and gave them a 24-hour grace period to respond. In fact I gave them more than 36 hours before hitting publish, but I have yet to receive a response. I'll update this space when I hear anything.

Search results matching tags 'licensing', 'CAL', and 'processor'

A cautionary tale about grandfathering CAL licenses in SQL Server 2012 Enterprise