Search results matching tags 'SQL Server' and 'Windows Azure'

SQL Server in Windows Azure Infrastructure Services – Updated Documentation and Best Practices for GA, Upcoming Blogs

SQLOS Team — Wed, 24 Apr 2013 22:04:16 GMT

It’s been just over a week since Windows Azure announced the GA of Infrastructure Services, marking the beginning of a fully supported Infrastructure as a Service in Windows Azure, with SQL Server as a major component.

Pre-installed SQL Server VMs are available for pay-per-hour usage in the Windows Azure gallery. Currently Enterprise, Standard and Web edition VMs running on Windows Server 2008 R2 SP1 are available, with more SQL Server editions coming soon. SQL Server editions running on Windows Server 2012 images are also on the way. For more details on the scenarios and benefits of running SQL Server workloads on Windows Azure Virtual Machines, please visit the SQL Server blog post here.

We are very happy to announce that the updated technical documentation for deploying and running SQL Server in Windows Azure Infrastructures Services is now available online. When deploying SQL Server in Windows Azure Virtual Machines, we recommend that you follow the detailed guidance given in the new SQL Server in Windows Azure Virtual Machines documentation in the library. This documentation includes a series of articles and tutorials that provide detailed guidance on:

Over the next few weeks we are planning a series of blog posts to provide more detailed information on specific SQL Server topics. Subjects in the pipeline include: high availability, disaster recovery, performance, application migration and security. Let us know what topics you would like to see covered in this series by adding comments to this post.

SQL Server Team

Originally posted at http://blogs.msdn.com/b/sqlosteam/

How Does the Cloud Change a Database Administrator’s Job?

BuckWoody — Tue, 29 Jan 2013 15:08:32 GMT

I recently posted a blog entry on how cloud computing would change the Systems Architect’s role in an organization. In a way, the Systems Architect has the easiest transition to a new way of using computing technologies. In fact, that’s actually part of the job description. I mentioned that a Systems Architect has three primary vectors to think about for cloud computing, as it applies to what they should do:

Knowledge - Which options are available to solve problems, and what are their strengths and weaknesses.
Experience - What has the System Architect seen and worked with in the past.
Coordination - A system design is based on multiple factors, and one person can't make all the choices. There will need to be others involved at every level of the solution, and the Systems Architect will need to know who those people are and how to work with them.

The Database Administrator Role

But a Database Administrator (DBA) is probably one of the harder roles to think about when it comes to cloud computing. First, let’s define what a Database Administrator usually thinks about as part of their job:

Planning, Installing and Configuring a Database Platform
Planning, designing and creating databases
Planning, designing and implementing High Availability and Disaster Recovery for each database (HADR) based on requirements for its workload
Maintaining and monitoring the database platform
Implementing performance tuning on the databases based on monitoring
Re-balancing workloads across database servers based on monitoring
Securing databases platforms and individual databases based on requirements and implementation

That’s just a short list, and each of those unpacks into a larger set of tasks.

The issue is that I’ve never actually met a DBA that does all of those things, or just all of those things. Many times they do much more, sometimes the systems are so large they specialize on just a few of them.

And as you can see from the list, some of these areas are shared with other roles. For instance, in some shops, the DBA plans, purchases, sets up and configures the hardware for database servers. In others that’s done
by the Infrastructure Team. In some shops the DBA designs databases from software requirements, and in others the developers do that – or perhaps it’s done as a joint effort. The same holds true for database code – sometimes the
DBA does it, other times the developer, and still others it’s a shared task.

In fact, you could argue that there are few other roles in IT where the roles are so intermixed. Also, the DBA works with software the company develops, and software the company buys. They work with hardware, networking, security and software. There are certain aspects of design and tuning that are outside the purview of some of those things, and inside the others.

With all of these variables, simply telling a DBA that they should “use the cloud” is not the proper approach.

How the Cloud Changes Things

To be sure, the DBA has the same vectors as the Systems Architect. They need to educate themselves on the options within this new option (Knowledge), try a few test solutions out (Experience) and of course work with others on various parts of the implementation (Coordination). But it goes beyond that.

There are three big buckets of cloud computing, dealing with simply using a Virtual Machine (IaaS) to writing code without worrying about the virtualization or even the operating system (PaaS) and using software that’s already written and being delivered via an Application Programming Interface (API). Each of these has so many options and configurations that it’s often better to think about the problem you’re trying to solve rather than all of the technology within a given area - although some of that is certainly necessary anyway.

Database Platform Architecture

I’ll start with when the DBA should even consider cloud computing for a solution. Once again, it’s not an “all or nothing” paradigm, where you either run something on premises or in the cloud – it’s often a matter of selecting the right components to solve a problem. In my design sessions with DBA’s I break these down into three big areas where they might want to consider the cloud –and then we talk about how to implement each one:

Audiences
HADR
Data Services

Audiences

If the users of your database systems all sit in the same facility, you own the servers and networking, and the application servers are separate from the database server, it doesn’t usually make sense to take that database workload and place it on Windows Azure – or any other cloud provider. The latency alone prevents a satisfactory performance profile, and in some cases won’t work at all. It doesn’t matter if the cloud solution is cheaper or easier – if you’re moving a lot of data every second between an on-premises system and the cloud it won’t work well.

However – if your users are in multiple locations, especially globally, or you have a mix of company and external customer users, it might make sense to evaluate a shared data location. You still need to consider the implications of how much data the application server pushes back and forth, but you may be able to locate both the application server and SQL Server in an IaaS role. Assuming the data sent to the final client will work across public Internet channels, there may be a fit. There are security implications, but unless you have point-to-point connections for your current solution you’re faced with the same security questions on both options.

Your audience might also be developers looking for a way to quickly spin up a server and then turn it down when they are done, paying for the time and not the hardware or licenses. This is also a prime case for evaluating IaaS. And there are others that you'll find in your own organization as you work through the requirements you have.

Resources: Windows Azure Virtual Machines: http://www.windowsazure.com/en-us/manage/windows/tutorials/virtual-machine-from-gallery/ and Windows Azure SQL Server Virtual Machines: http://www.windowsazure.com/en-us/manage/windows/common-tasks/install-sql-server/

HADR

The next possible place to consider using cloud computing with SQL Server is as a part of your High Availability and Disaster Recovery plans. In fact, this is the most common use I see for cloud computing and the Database Administrator. The key is the Recovery Point Objective (RPO) and Recovery Time Objective (RTO). Based on each application’s requirements, you may find that using Windows Azure or even supplementing your current plan is
the right place to evaluate options. I’ve covered this use-case in more detail in another article.

References: SQL Server High Availability and Disaster Recovery options with Windows Azure: http://blogs.msdn.com/b/buckwoody/archive/2013/01/08/microsoft-windows-azure-disaster-recovery-options-for-on-premises-sql-server.aspx

Data Services

Windows Azure, along with other cloud providers, offers another way to design, create and consume data. In this use-case, however, the tasks DBA’s normally perform for sizing, ordering and configuring a system don’t apply.

With Windows Azure SQL Databases (the artist formerly known as SQL Azure), you can simply create a database and begin using it. There are places where this fits and others where it doesn’t, and there are differences, limitations and enhancements, so it isn’t meant as replacement for what you could do with “Full-up” SQL Server on a Windows Azure Virtual Machine or an on-premises Instance. If a developer needs an Relational Database Management
(RDBMS) data store for a web-based application, then this might be a perfect fit.

But there is more to data services than Windows Azure SQL Databases. Windows Azure also offers MySQL as a service, RIAK and MongoDB (among others) and even Hadoop for larger distributed data sets. In addition you can use Windows Azure Reporting Services, and also tap into datasets and data functions in the Windows Azure Marketplace.

The key for the DBA with this option is that you will have to do a little investigation this time, and potentially without a specific workload in mind this time. I think that’s acceptable thing to ask – DBA’s constantly keep up with data processing trends, and most will consider different ways to solve a problem.

References:

Windows Azure SQL Databases: http://www.windowsazure.com/en-us/home/features/data-management/

Windows Azure Reporting Services: http://www.windowsazure.com/en-us/manage/services/other/sql-reporting/

HDInsight Service (Hadoop on Azure): https://www.hadooponazure.com/

MongoDB Offerings on Windows Azure: http://www.windowsazure.com/en-us/manage/linux/common-tasks/mongodb-on-a-linux-vm/

Windows Azure Marketplace: http://www.windowsazure.com/en-us/store/overview/

Microsoft Windows Azure Disaster Recovery Options for On-Premises SQL Server

BuckWoody — Tue, 08 Jan 2013 14:40:00 GMT

One of the use-cases for a cloud solution is to serve as a Disaster Recovery option for your on-premises servers. I’ll explain one particular use-case in this entry, specifically using Windows Azure “IaaS” or Virtual Machines as a Recovery Solution for SQL Server (more detail here: http://www.windowsazure.com/en-us/home/features/virtual-machines/). In future installments I’ll explain options for other workloads such as Linux and Windows Servers, SharePoint and other solutions. Some architectures also allow for using Windows Azure SQL Database (Formerly SQL Azure) in recovery scenarios; I’ll cover that separately.

Using Azure as a Disaster Recovery site gives you a range of options, uses world-wide datacenters that you can pick from, and does not require traditional licensing and maintenance paths. You can also integrate the offsite data into other uses, such as reporting (in some cases) or to leverage within other applications. However, the cost-model is different, so make sure you do your homework to ensure that it makes sense to use a cloud provider for safety. You may find that it is cheaper, more expensive, or that you require a mix of technologies and options to get the best solution.

NOTE: The Microsoft Windows Azure platform evolves constantly. That means new features and capabilities, as well as security, optimizations and more improve on a frequent basis. As with any cloud provider, ensure that you check the date of this post to ensure you are within six months or so. If the date is longer than that, then check each of the “Details” links to ensure you are working with the latest information.

The options you have range from simple off-site storage for database backups to systems that your users can access when your primary options are offline. To select which options to use, evaluate the databases you want to protect, and then create your Recovery Point Objectives (RPO) and Recovery Time Objectives (RTO) for each workload. Those two vectors will provide the starting point for each choice you make.

NOTE: If you’re not familiar with RPO and RTO on a database system, learn those terms carefully before designing a recovery solution – on any platform. RPO and RTO are business/technology terms, and are not vendor or platform-specific. http://wikibon.org/wiki/v/Recovery_point_objective_-_recovery_time_objective_strategy

The range of protection you have is very similar to the on-premises options for SQL Server (on-premises details here: http://msdn.microsoft.com/en-us/library/ms190202.aspx), with the primary limitation being bandwidth. While Microsoft has the largest connections we can get into our datacenters, depending on where your systems are and their connection to the Internet, you will need to consider how much data you transfer, and how often. For backup files, a single, larger transfer is acceptable, using Log Shipping or Database Mirroring, smaller, more frequent transfers are preferable.

Another limitation is controlling the hardware on the Windows Azure Virtual Machine. That means hardware-based clustering isn’t possible, as of this writing. You’re also limited to the size of the Virtual Machines that Windows Azure (or any other cloud provider) offers. It’s important to keep in mind that you’re building a Disaster Recovery solution, not necessarily a full Highly-Available system. The difference is that in this case DR provides a means to recover and operate at a more limited fashion than a full on-premises HA (with matching hardware and licenses) involves. Storage, however, isn’t as affected. You can mount large amounts of storage on a Windows Azure Virtual Machine, so it’s more memory and CPU that you need to consider for your solution.

The final consideration is security. There are two aspects in security that you need to consider: data security and authentication and access. For the first consideration, the Windows Azure system does hold multiple certifications and attestations that you can find here: . In some cases those certifications are agreements on the part of security each party will hold liability for; so it’s important to carefully read and understand what the agreement states. There are also methods of encrypting data (such as the backups) using your own certificates or hardware devices and then storing them externally. This means no one can easily un-encrypt your data.

For the authentication portion, you can create a secure “tunnel” between your network and Windows Azure. This involves a certificate that is installed on your hardware firewall at your facility, and an agent that is enabled with the same certificate on Windows Azure. This gives you a “point to point” connection, encrypted but over a public connection. From there you can use Active Directory to connect the authentication for the systems involved in the DR solution.

Backups

The First and most simple DR solution using Windows Azure is to store your backup files (*.bak) in Windows Azure storage. Windows Azure Storage is triple-redundant across multiple fault-domains within a single datacenter, and then all three copies are replicated to a geographically separate (although data-sovereignty same) location. That translates to six copies of data stored remotely. In case of a disaster, you connect to storage, download the images, and restore them to a new server. The server can have the same name or different, and unless you’re using contained databases, you’ll need to re-create and re-authorize the security accounts needed for the database.

Note that you also have the option of using an “appliance”, which is a piece of hardware you install at your facility which will act as a backup device or share location (or both). The device handles the encryption, de-duplication and compression for the files, and then stores those files on Windows Azure. More information on that option is here: http://www.storsimple.com/

RPO: As of last backup

RTO: (Time of transfer from Windows Azure + Time of Restore to New System + Bringing System Online with User Accounts) - Time of Backup

References:

More detail on storing files on Windows Azure: http://sqlblog.com/blogs/sqlos_team/archive/2013/01/24/backup-and-restore-to-cloud-simplified-in-sql-server-2012-sp1-cu2.aspx

Free Client: http://azurestorageexplorer.codeplex.com/

Database Mirroring

Database Mirroring is a deprecated feature in SQL Server, which means it will be removed in a future release. It is, however, still supported in SQL Server 2012, and it can be used between on-premises SQL Server Instances and Windows Azure VM’s. Using connection strings and .NET languages, clients can actually point to the partner server automatically.

The granularity of this solution is at the individual database level. Machines can retain their individual identities. You can use certificates to connect the systems, or you can use the point-to-point solution and Active Directory.

There are limitations, however. You won’t use a Listener in this configuration, and you’ll be using Asynchronous mode. If you are not running in the same Active Directory, you’ll also need to factor in the time to re-create and tie out those accounts when calculating the RTO value.

RPO: As of last good synchronization

RTO: (Time of failure + Time of client redirect to New System ) - Time of last good synchronization

References:

A complete tutorial on setting up this configuration is here: http://msdn.microsoft.com/en-us/library/jj870964.aspx

Log Shipping

Another feature available for DR in a Hybrid fashion is using Log Shipping, which also protects your system at a database level. Log shipping involves an automated log backup of your database, and the log is copied and then applied at the secondary server. Because the log file is copied to a Windows share, this solution requires both networking access and an Active Directory integration.

RPO: As of last good log backup application to the secondary system

RTO: (Time of failure + Time of manual client redirect to New System + Time of Manual Failover ) - Time of last good log backup

References:

Log Shipping information is here: http://technet.microsoft.com/en-us/library/ms187103.aspx

AlwaysOn Availability Groups

SQL Server 2012 introduces a new set of features called “AlwaysOn” that encompass many of the HA/DR features in previous releases. One feature within that set is called “Availability Groups”, and with certain caveats that feature is available for a Hybrid on-premises to Windows Azure VM solution.

AlwaysOn requires a Windows Cluster (WFSC), which is where the caveats come into play. You’re able to set up a multi-subnet WSFC cluster, but you won’t have access to the Availability Group Listener function, so you need to consider the client reconnection.

RPO: As of last good synchronization

RTO: (Time of failure + Time of manual client redirect to New System + Time of Manual Failover ) - Time of last good log backup

References:

A complete tutorial on setting up this configuration is here: http://msdn.microsoft.com/en-us/library/jj870959.aspx

Using Hadooop (HDInsight) with Microsoft - Two (OK, Three) Options

BuckWoody — Tue, 04 Dec 2012 15:28:23 GMT

Microsoft has many tools for “Big Data”. In fact, you need many tools – there’s no product called “Big Data Solution” in a shrink-wrapped box – if you find one, you probably shouldn’t buy it. It’s tempting to want a single tool that handles everything in a problem domain, but with large, complex data, that isn’t a reality. You’ll mix and match several systems, open and closed source, to solve a given problem.

But there are tools that help with handling data at large, complex scales. Normally the best way to do this is to break up the data into parts, and then put the calculation engines for that chunk of data right on the node where the data is stored. These systems are in a family called “Distributed File and Compute”. Microsoft has a couple of these, including the High Performance Computing edition of Windows Server. Recently we partnered with Hortonworks to bring the Apache Foundation’s release of Hadoop to Windows. And as it turns out, there are actually two (technically three) ways you can use it.

(There’s a more detailed set of information here: http://www.microsoft.com/sqlserver/en/us/solutions-technologies/business-intelligence/big-data.aspx, I’ll cover the options at a general level below)

First Option: Windows Azure HDInsight Service

Your first option is that you can simply log on to a Hadoop control node and begin to run Pig or Hive statements against data that you have stored in Windows Azure. There’s nothing to set up (although you can configure things where needed), and you can send the commands, get the output of the job(s), and stop using the service when you are done – and repeat the process later if you wish.

(There are also connectors to run jobs from Microsoft Excel, but that’s another post)

This option is useful when you have a periodic burst of work for a Hadoop workload, or the data collection has been happening into Windows Azure storage anyway. That might be from a web application, the logs from a web application, telemetrics (remote sensor input), and other modes of constant collection.

Second Option: Microsoft HDInsight Server

Your second option is to use the Hadoop Distribution for on-premises Windows called Microsoft HDInsight Server. You set up the Name Node(s), Job Tracker(s), and Data Node(s), among other components, and you have control over the entire ecostructure.

This option is useful if you want to have complete control over the system, leave it running all the time, or you have a huge quantity of data that you have to bulk-load constantly – something that isn’t going to be practical with a network transfer or disk-mailing scheme.

Third Option (unsupported): Installation on Windows Azure Virtual Machines

Although unsupported, you could simply use a Windows Azure Virtual Machine (we support both Windows and Linux servers) and install Hadoop yourself – it’s open-source, so there’s nothing preventing you from doing that.

Aside from being unsupported, there are other issues you’ll run into with this approach – primarily involving performance and the amount of configuration you’ll need to do to access the data nodes properly. But for a single-node installation (where all components run on one system) such as learning, demos, training and the like, this isn’t a bad option.

Did I mention that’s unsupported? :)

You can learn more about Windows Azure Virtual Machines here: http://www.windowsazure.com/en-us/home/scenarios/virtual-machines/

And more about Hadoop and the installation/configuration (on Linux) here: http://en.wikipedia.org/wiki/Apache_Hadoop

And more about the HDInsight installation here: http://www.microsoft.com/web/gallery/install.aspx?appid=HDINSIGHT-PREVIEW

Choosing the right option

Since you have two or three routes you can go, the best thing to do is evaluate the need you have, and place the workload where it makes the most sense. My suggestion is to install the HDInsight Server locally on a test system, and play around with it. Read up on the best ways to use Hadoop for a given workload, understand the parts, write a little Pig and Hive, and get your feet wet. Then sign up for a test account on HDInsight Service, and see how that leverages what you know. If you're a true tinkerer, go ahead and try the VM route as well.

Oh - there’s another great reference on the Windows Azure HDInsight that just came out, here: http://blogs.msdn.com/b/brunoterkaly/archive/2012/11/16/hadoop-on-azure-introduction.aspx

The Data Scientist

BuckWoody — Tue, 15 Nov 2011 15:00:18 GMT

A new term - well, perhaps not that new - has come up and I’m actually very excited about it. The term is Data Scientist, and since it’s new, it’s fairly undefined. I’ll explain what I think it means, and why I’m excited about it.

In general, I’ve found the term deals at its most basic with analyzing data. Of course, we all do that, and the term itself in that definition is redundant. There is no science that I know of that does not work with analyzing lots of data. But the term seems to refer to more than the common practices of looking at data visually, putting it in a spreadsheet or report, or even using simple coding to examine data sets.

The term Data Scientist (as far as I can make out this early in it’s use) is someone who has a strong understanding of data sources, relevance (statistical and otherwise) and processing methods as well as front-end displays of large sets of complicated data. Some - but not all - Business Intelligence professionals have these skills. In other cases, senior developers, database architects or others fill these needs, but in my experience, many lack the strong mathematical skills needed to make these choices properly.

I’ve divided the knowledge base for someone that would wear this title into three large segments. It remains to be seen if a given Data Scientist would be responsible for knowing all these areas or would specialize. There are pretty high requirements on the math side, specifically in graduate-degree level statistics, but in my experience a company will only have a few of these folks, so they are expected to know quite a bit in each of these areas.

Persistence

The first area is finding, cleaning and storing the data. In some cases, no cleaning is done prior to storage - it’s just identified and the cleansing is done in a later step. This area is where the professional would be able to tell if a particular data set should be stored in a Relational Database Management System (RDBMS), across a set of key/value pair storage (NoSQL) or in a file system like HDFS (part of the Hadoop landscape) or other methods. Or do you examine the stream of data without storing it in another system at all?

This is an important decision - it’s a foundation choice that deals not only with a lot of expense of purchasing systems or even using Cloud Computing (PaaS, SaaS or IaaS) to source it, but also the skillsets and other resources needed to care and feed the system for a long time. The Data Scientist sets something into motion that will probably outlast his or her career at a company or organization.

Often these choices are made by senior developers, database administrators or architects in a company. But sometimes each of these has a certain bias towards making a decision one way or another. The Data Scientist would examine these choices in light of the data itself, starting perhaps even before the business requirements are created. The business may not even be aware of all the strategic and tactical data sources that they have access to.

Processing

Once the decision is made to store the data, the next set of decisions are based around how to process the data. An RDBMS scales well to a certain level, and provides a high degree of ACID compliance as well as offering a well-known set-based language to work with this data. In other cases, scale should be spread among multiple nodes (as in the case of Hadoop landscapes or NoSQL offerings) or even across a Cloud provider like Windows Azure Table Storage. In fact, in many cases - most of the ones I’m dealing with lately - the data should be split among multiple types of processing environments. This is a newer idea. Many data professionals simply pick a methodology (RDBMS with Star Schemas, NoSQL, etc.) and put all data there, regardless of its shape, processing needs and so on.

A Data Scientist is familiar not only with the various processing methods, but how they work, so that they can choose the right one for a given need. This is a huge time commitment, hence the need for a dedicated title like this one.

Presentation

This is where the need for a Data Scientist is most often already being filled, sometimes with more or less success. The latest Business Intelligence systems are quite good at allowing you to create amazing graphics - but it’s the data behind the graphics that are the most important component of truly effective displays.

This is where the mathematics requirement of the Data Scientist title is the most unforgiving. In fact, someone without a good foundation in statistics is not a good candidate for creating reports. Even a basic level of statistics can be dangerous. Anyone who works in analyzing data will tell you that there are multiple errors possible when data just seems right - and basic statistics bears out that you’re on the right track - that are only solvable when you understanding why the statistical formula works the way it does.

And there are lots of ways of presenting data. Sometimes all you need is a “yes” or “no” answer that can only come after heavy analysis work. In that case, a simple e-mail might be all the reporting you need. In others, complex relationships and multiple components require a deep understanding of the various graphical methods of presenting data. Knowing which kind of chart, color, graphic or shape conveys a particular datum best is essential knowledge for the Data Scientist.

Why I’m excited

I love this area of study. I like math, stats, and computing technologies, but it goes beyond that. I love what data can do - how it can help an organization. I’ve been fortunate enough in my professional career these past two decades to work with lots of folks who perform this role at companies from aerospace to medical firms, from manufacturing to retail.

Interestingly, the size of the company really isn’t germane here. I worked with one very small bio-tech (cryogenics) company that worked deeply with analysis of complex interrelated data.

So watch this space. No, I’m not leaving Azure or distributed computing or Microsoft. In fact, I think I’m perfectly situated to investigate this role further. We have a huge set of tools, from RDBMS to Hadoop to allow me to explore. And I’m happy to share what I learn along the way.

Big Data and the Cloud - More Hype or a Real Workload?

BuckWoody — Tue, 18 Oct 2011 13:57:36 GMT

Last week Microsoft announced several new offerings for “Big Data” - and since I’m a stickler for definitions, I wanted to make sure I understood what that really means. What is “Big Data”? What size hard drive is that? After all, my laptop has 1TB of storage - is my laptop “Big Data”?

There are actually a few definitions for this term, most notably those involving the “Four V’s” Volume, Velocity, Variety and Variability. Others disagree with this definition. I tend to try and get things into their simplest form, so I’m using this definition for myself:

Big data is defined as a large set of computationally expensive data that is worked on simultaneously.

Let me flesh that out a little. To be sure, “Big Data” has a larger size than say a few megabytes. The reason this is important is that it takes special hardware to be able to move large sets of data around, store it, process it and so on. (large set)

If you store a LOT of data, but only use a small portion of it at a time, that really isn’t super-hard to do. It’s mainly a storage issue at that point. But, if you do need to work with a large portion of the data at one time, then the memory, CPU and transfer components of the system have to adapt to be responsive - new ways to work with that data (game theory, knot-algorithms, map-reduce, etc.) need to be brought into play. (computationally expensive)

Once that data is loaded into the processing area (memory or whatever other mechanism is used) it must be worked on in parallel to come back in a reasonable time. You have two options here - you can scale the system up with more internal hardware (CPU’s, memory and so on) or you can scale it out to have multiple systems work on it at the same time using paradigms such as map/reduce and so on. Actually, when you lay this out in an architecture diagram, scale up or out doesn’t actually change the logical structure of the process - in scale out the network becomes the bus, and the nodes become more RAM and computing power. Of course, there are changes in code for how you stitch the workload back together. (worked on simultaneously)

So back to the original question. Is Big Data, as I have defined it here, a workload for Windows and SQL Azure? Absolutely! In fact, it’s probably one of the main workloads, and I believe it represents the latest, and perhaps also the earliest frontier of computing. Jim Gray, a former researcher here at Microsoft and a hero of mine, was working on this very topic. I believe as he did - all computing is simply an interface over data.

Microsoft has multiple offerings on the topic of Big Data. In posts that follow from myself and my co-workers, we’ll explore when and where you use each one. Whether you are a data professional or a developer, this is the new frontier - don’t wait to educate yourself on how to leverage Big Data for your organization.

Hadoop on Windows Azure and SQL Server - Microsoft’s partnership to include Hadoop workloads on Windows Azure and SQL Server/Parallel Data Warehouse (PDW)

LINQ to HPC - Microsoft’s High-Performance Computing SKU of HPC is now in Azure

Windows Azure Table Storage - A key/value pair type storage with full partitioning that is immediately consistent, able to handle huge loads of data and works with any REST-compatible language

Other offerings - Including the new Data Explorer, Project Daytona (with a Big Data Toolkit for Scientists and researchers), Power View and more.

The era of Big Data is here. And you can use Windows and SQL Azure to bring it to your organization.

SQL Azure Use Case: Shared Storage Application

BuckWoody — Tue, 26 Apr 2011 13:33:50 GMT

This is one in a series of posts on when and where to use a distributed architecture design in your organization's computing needs. You can find the main post here: http://blogs.msdn.com/b/buckwoody/archive/2011/01/18/windows-azure-and-sql-azure-use-cases.aspx

Description:

On-premise data will be a part of computing for quite some time – perhaps permanently. Bandwidth requirements, security, or even financial considerations for large data sets often dictate that relational (on non-relational) systems will be maintained locally in many organizations, especially in enterprise computing.

But distributed data systems are useful in many situations. Organizations may wish to store a portion of data off-site, either for sharing the data with other applications (including web-based applications) or as a supplement to a High-Availability and Disaster Recovery (HADR) strategy.

Implementation:

SQL Azure can be used to add an additional option to an HADR strategy by copying off portions (or all) of an on-premise database system.

In this arrangement, on-premise systems remain as they are. Data is replicated using many technologies, such as SQL Server Integration Services (SSIS), scripts, or Microsoft’s Sync Framework to a SQL Azure database. This data can be kept “cold”, meaning that a manual process is required to bring the data back, or as a “warm” standby using connection string management in the application.

Recently we architected a solution where a company kept a rolling two-week window of data replicated to SQL Azure using the Sync Framework. The application, a compiled EXE running on user’s systems, had a “switch connections” button, that allowed the users to take a laptop to another location, select that option, and continue working from anywhere they had Internet connectivity. This required forethought and planning, and did not replace their primary HADR systems, but it did allow them to continue operations in the case of a severe outage at multiple sites. Since they are an emergency services provider, this gave them the highest redundancy.

Another option is to amalgamate data from disparate sources.

In this arrangement, two or more data services (one of which is SQL Azure) are accessed by a single program. The program queries each system independently, and using LINQ a single query can work across all of the data, assuming there is some sort of natural or artificial “key” that can join the data sets together. The user programs simply view this single data set as a single data source, unaware of the underlying data sets. This allows great flexibility and agility in the downstream program. The upstream data sources can change as long as the elements are kept consistent.

There are performance and security implications to amalgamated data systems, but if architected carefully they provide multiple benefits. A few of of these are that other systems can access the individual data sources, reporting is simplified and standardized, and multiple copies of data are eliminated.

Resources:

You can read more about the Sync Framework and SQL Azure here: http://social.technet.microsoft.com/wiki/contents/articles/sync-framework-sql-server-to-sql-azure-synchronization.aspx

If you are new to LINQ, you can find more resources on it here: http://msdn.microsoft.com/en-us/library/bb308959.aspx

Search results matching tags 'SQL Server' and 'Windows Azure'

SQL Server in Windows Azure Infrastructure Services – Updated Documentation and Best Practices for GA, Upcoming Blogs

How Does the Cloud Change a Database Administrator’s Job?

The Database Administrator Role

How the Cloud Changes Things

Database Platform Architecture

Audiences

HADR

Data Services

Microsoft Windows Azure Disaster Recovery Options for On-Premises SQL Server

Backups

Database Mirroring

Log Shipping

AlwaysOn Availability Groups

Other Solution Options

Using Hadooop (HDInsight) with Microsoft - Two (OK, Three) Options

First Option: Windows Azure HDInsight Service

Second Option: Microsoft HDInsight Server

Third Option (unsupported): Installation on Windows Azure Virtual Machines

Choosing the right option

The Data Scientist

Big Data and the Cloud - More Hype or a Real Workload?

SQL Azure Use Case: Shared Storage Application