This week was a bit of a milestone for me in my current job. We took delivery of six shiny new Nehalem-based HP DL380’s. These are very nice little servers (not so little anymore, truth be told, if you look at the workload they can handle) but that’s not really the fun part for me. The servers, together with one pair of existing previous generation 380’s will form my company’s first set of geographically distributed, mirrored SQL Server 2008 clusters. Sweet! I might be almost last to the 2008 + mirroring party, but we’re getting there!
This is exciting for me personally because this is a strategic project I’ve been quietly working on since starting in my current position about two years ago – when I joined the organization, there was already a team of skillful, hardworking people who built an impressive set of services and applications. There didn’t appear to be the time or focus, however, for someone to take on SQL Server itself as a long-term, strategic piece of infrastructure. Instead I faced a sprawling collection of separate clusters and standalone SQL Server instances, in various stages of heath or not, built on the “Oh, SQL Server is a prereq. for this new application, so let’s install it” model. It’s not that the team was lax; they were just, rightfully, focused on other things. It was my mandate not to be sidetracked by other work, and instead be that person with exclusive focus on the DBA role.
Between then and now, I’ve had a few challenges that I tried to make into opportunities. First, I had to ask the team if they realized how much they were spending to run all these separate SQL Servers, which inevitably led, of course, to a consolidation project. Second, due to outside pressures, we also had to perform a data center move. Finally, all of this had to happen during this marvelous recession – which is to say, if not on a shoestring, then as close as possible, because the cash wasn’t exactly raining down on our little team. The idea of replacing the many SQL Servers with a few, thankfully, dovetailed nicely with the need to cut back financially. (Oh, and also “fix” the B.I. warehouse, but that’s a whole other story.)
Tactics > Strategy
At a high level what I’ve tried to do is, while taking on those three challenges, make each tactical decision into progress toward a clear, maintainable SQL environment that makes strategic sense. I have a theory that the main challenge any sysadmin faces in most IT jobs is entropy: that is, IT systems have this strong tendency to deteriorate over time as people make small, incremental decisions and changes with a short time horizon. The trick to being a good admin, I hope, is to be able to meet or exceed the immediate business objectives you have while not allowing your systems to backslide. That is, you have to stay agile but at the same time make sure that your 2-5 year strategy actually takes shape as you make day to day modifications.
So, my master plan is not complete, but the arrival of these servers has certainly put a smile on my face. We’re making real progress. Before long, mission critical stuff will be on modern hardware and software with state-of-the-art HA and DR. The small moves and tactical decisions seem to be adding up.
The design we’re building is a common one, consisting of two-node clusters with automatic failover in each data center, overlaid with database mirroring between our data centers both for HA and DR. (Paul Randal has a really excellent, concise description of HA/DR architectures in a recent white paper here.) And while the existing systems are safe and do a reasonable job now, this new architecture should take us to the next level, and last a good five years into the future.