Big Data and the Cloud - More Hype or a Real Workload?

BuckWoody — Tue, 18 Oct 2011 13:57:36 GMT

Last week Microsoft announced several new offerings for “Big Data” - and since I’m a stickler for definitions, I wanted to make sure I understood what that really means. What is “Big Data”? What size hard drive is that? After all, my laptop has 1TB of storage - is my laptop “Big Data”?

There are actually a few definitions for this term, most notably those involving the “Four V’s” Volume, Velocity, Variety and Variability. Others disagree with this definition. I tend to try and get things into their simplest form, so I’m using this definition for myself:

Big data is defined as a large set of computationally expensive data that is worked on simultaneously.

Let me flesh that out a little. To be sure, “Big Data” has a larger size than say a few megabytes. The reason this is important is that it takes special hardware to be able to move large sets of data around, store it, process it and so on. (large set)

If you store a LOT of data, but only use a small portion of it at a time, that really isn’t super-hard to do. It’s mainly a storage issue at that point. But, if you do need to work with a large portion of the data at one time, then the memory, CPU and transfer components of the system have to adapt to be responsive - new ways to work with that data (game theory, knot-algorithms, map-reduce, etc.) need to be brought into play. (computationally expensive)

Once that data is loaded into the processing area (memory or whatever other mechanism is used) it must be worked on in parallel to come back in a reasonable time. You have two options here - you can scale the system up with more internal hardware (CPU’s, memory and so on) or you can scale it out to have multiple systems work on it at the same time using paradigms such as map/reduce and so on. Actually, when you lay this out in an architecture diagram, scale up or out doesn’t actually change the logical structure of the process - in scale out the network becomes the bus, and the nodes become more RAM and computing power. Of course, there are changes in code for how you stitch the workload back together. (worked on simultaneously)

So back to the original question. Is Big Data, as I have defined it here, a workload for Windows and SQL Azure? Absolutely! In fact, it’s probably one of the main workloads, and I believe it represents the latest, and perhaps also the earliest frontier of computing. Jim Gray, a former researcher here at Microsoft and a hero of mine, was working on this very topic. I believe as he did - all computing is simply an interface over data.

Microsoft has multiple offerings on the topic of Big Data. In posts that follow from myself and my co-workers, we’ll explore when and where you use each one. Whether you are a data professional or a developer, this is the new frontier - don’t wait to educate yourself on how to leverage Big Data for your organization.

Hadoop on Windows Azure and SQL Server - Microsoft’s partnership to include Hadoop workloads on Windows Azure and SQL Server/Parallel Data Warehouse (PDW)

LINQ to HPC - Microsoft’s High-Performance Computing SKU of HPC is now in Azure

Windows Azure Table Storage - A key/value pair type storage with full partitioning that is immediately consistent, able to handle huge loads of data and works with any REST-compatible language

Other offerings - Including the new Data Explorer, Project Daytona (with a Big Data Toolkit for Scientists and researchers), Power View and more.

The era of Big Data is here. And you can use Windows and SQL Azure to bring it to your organization.

The New World of Work: Unique Tasks

BuckWoody — Thu, 26 Aug 2010 11:26:59 GMT

In a previous post I mentioned that I’ve seen some Gartner Inc. studies, among others, that state that we’re in a “new world of work”. Since I’ve been working in technology for quite some time, and working around the world for longer, I agree that things are not what they were in the 9-5, Monday through Friday, pre-defined, single-employer world of yesterday. Things are far more dynamic, and include emerging work methods like the “swarms” I mentioned, and another interesting by-product: unique tasks.

What this means is that we’ve pretty much “drained the swamp” for many industries in the area of automation of repeatable tasks. Machines now perform tasks that years ago were done by people – but even more so than just performing obvious tasks like making pasta or painting cars, machines perform very complex work that many thought would never be taken over by electronics. In software, this trend is even more prevalent, in everything from scripting to 4GL programming languages and visual programming, where non-professional developers write code. In fact, Microsoft is one of the key vendors catering to this market. We just released project “Lightswitch”, which lets non-developers write fairly sophisticated code. In effect, we’ve pushed the coding tasks to the end user, much like what happened when word-processing software pushed typing out to the executives, removing the need for a secretary to do that work.

You can even see this in the next computing wave: the “cloud”. It remains to be seen what the full impact of having servers somewhere else is, but one scenario might be that developers and business folks write code using data objects, not tables and so on, using simple programming techniques that bypass many of the roles that we data professionals currently hold. If this scenario plays out, they would simply buy a cloud services, open Visual Studio, and begin to write code. No design, no planning for server layouts, HADR, scalability – all that would be handled by the cloud. They won’t need (or they’ll think they won’t need) a data professional at all.

I’m not saying this is a good outcome – one can look at the state of business communications to see that the removal of professional secretaries may not be a great thing. But it did happen. Business folks and non-trained developers won’t write code properly, or plan for disasters the way they should. Things will go wrong, and the cloud vendors (like Microsoft) will react to that to fix those issues. But what of the data professional? How should we react?

My personal opinion is to begin now to work with the organization to find out where the business needs for new technologies for data lies. I work closely with the Business Analysts, or, if there aren’t any, with the business people, to educate them on the value, placement, retirement and so on of data within the organization. I’m helping the organizations I manage to find where data “in the cloud” fits, and when it should stay “on-premise”. In fact, I had this discussion just a couple of weeks ago, when I migrated two applications to the cloud for a small church I work with. I laid out the pro’s and con’s of the decision quite clearly, and we decided to implement two solutions on the cloud. I did the work, and they were up and running in an afternoon. I simply couldn’t have ordered a server, installed and configured the software, and written the code that quickly. It’s maintained, versioned and so on, and they can now hit the application from anywhere. In that case, it was the right thing to do.

But I still have applications “on-premise”, even at that small church. For those, I’ve automated just about everything I can think of. My role as the data professional there is what I call a “System’s Shepherd” – I just ensure the automation is running. So my new role is twofold: I do a lot of design and decision work to enable this new world of automation. From there I automate heavily. And after that, it’s a monitoring and reacting environment. Even the reacting is largely automated. And in that way, I can support more systems from a remote location than I ever could before.

My advice? Learn the cloud. Learn scripting. Learn automation. Or be dominated by all three.

Search results matching tags 'SQL Server', 'Cloud', and 'Career'

Big Data and the Cloud - More Hype or a Real Workload?

The New World of Work: Unique Tasks