Search results matching tag 'Azure Use Cases'

Keeping your options open in a cloud solution

BuckWoody — Tue, 25 Sep 2012 14:20:00 GMT

In on-premises solutions we have the full range of options open for a given computing solution – but we don’t always take advantage of them, for multiple reasons. Data goes in a Relational Database Management System, files go on a share, and e-mail goes to the Exchange server.

Over time, vendors (including ourselves) add in functionality to one product that allow non-standard use of the platform. For example, SQL Server (and Oracle, and others) allow large binary storage in or through the system – something not originally intended for an RDBMS to handle. There are certainly times when this makes sense, of course, but often these platform hammers turn every problem into a nail. It can make us “lazy” in our design – we sometimes don’t take the time to learn another architecture because the one we’ve spent so much time with can handle what we want to do.

But there’s a distinct danger here. In nature, when a population shares too many of the same traits, it can cause a complete collapse if a situation exploits a weakness shared by that population. The same is true with not using the right
tool for the job in a computing environment. Your company or organization depends on your knowledge as a professional to select the best mix of supportable, flexible, cost-effective technologies to solve their problems, whether you’re in an architect role or not.

So take some time today to learn something new. The way I do this is to select a given problem, and try to solve it with a technology I’m not familiar with. For instance – create a Purchase Order system in Excel, then in Hadoop or MongoDB, or even in flat-files using PowerShell as an interface. No, I’m not suggesting any of these architectures are the proper way to solve the PO problem, but taking something concrete that you know well and applying that meta-knowledge to another platform will assist you in exercising the “little grey cells” and help you and your organization understand what is open to you.

And of course you can do all of this on-premises – but my recommendation is to check out a cloud platform (my suggestion would of course be Windows Azure :) ) and try it there. Most providers (including Microsoft) provide free time to do that.

Creating a Corporate Data Hub

BuckWoody — Tue, 26 Jun 2012 14:36:41 GMT

The Windows Azure Marketplace has a rich assortment of data and software offerings for you to use – a type of Software as a Service (SaaS) for IT workers, not necessarily for end-users. Among those offerings is the “Data Hub” – a codename for a project that ironically actually does what the codename says.

In many of our organizations, we have multiple data quality issues. Finding data is one problem, but finding it just once is often a bigger problem. Lots of departments and even individuals have stored the same data more than once, and in some cases, made changes to one of the copies. It’s difficult to know which location or version of the data is authoritative.

Then there’s the problem of accessing the data. It’s fairly straightforward to publish a database, share or other location internally to store the data. But then you have to figure out who owns it, how it is controlled, and pass out the various connection strings to those who want to use it. And then you need to figure out how to let folks access the internal data externally – bringing up all kinds of security issues. Finally, in many cases our user community wants us to combine data from the internally sources with external data, bringing up the security, strings, and exploration features up all over again.

Enter the Data Hub. This is an online offering, where you assign an administrator and data stewards. You import the data into the service, and it’s available to you - and only you and your organization if you wish.

The basic steps for this service are to set up the portal for your company, assign administrators and permissions, and then you assign data areas and import data into them. From there you make them discoverable, and then you have multiple options that you or your users can access that data. You’re then able, if you wish, to combine that data with other data in one location.

So how does all that work? What about security? Is it really that easy? And can you really move the data definition off to the Subject Matter Experts (SME’s) that know the particular data stack better than the IT team does?

Well, nothing good is easy – but using the Data Hub is actually pretty simple. I’ll give you a link in a moment where you can sign up and try this yourself. Once you sign up, you assign an administrator. From there you’ll create data areas, and then use a simple interface to bring the data in. All of this is done in a portal interface – nothing to install, configure, update or manage.

After the data is entered in, and you’ve assigned meta-data to describe it, your users have multiple options to access it. They can simply use the portal – which actually has powerful visualizations you can use on any platform, even mobile phones or tablets.

Your users can also hit the data with Excel – which gives them ultimate flexibility for display, all while using an authoritative, single reference for the data. Since the service is online, they can do this wherever they are – given the proper authentication and permissions. You can also hit the service with simple API calls, like this one from C#: http://msdn.microsoft.com/en-us/library/hh921924

You can make HTTP calls instead of code, and the data can even be exposed as an OData Feed. As you can see, there are a lot of options.

You can check out the offering here: http://www.microsoft.com/en-us/sqlazurelabs/labs/data-hub.aspx and you can read the documentation here: http://msdn.microsoft.com/en-us/library/hh921938

BlobShare: Corporate Filesharing with Security and Control

BuckWoody — Tue, 08 May 2012 12:49:21 GMT

We’ve all done it. We have a file we need to transfer to someone - but it’s big. Really big. “Big”, in this case, is defined as “bigger than my corporate e-mail will let me send.” So of course we scout around, and find a free file share location. We plop it in, and send a link to the other person. Problem solved!

No, problem created. You see, many of these file-sharing sites have some…interesting…language in their terms. Sure, for a picture of aunt Sally riding that llama on the vacation, who cares if someone else steals it and puts it on an advertisement for a shipping company from Elbonia? But most of these offerings were not designed for a corporation with private or secure information - sometimes your private or secure information.

So what’s a company to do? You have to let people send files - they are just going to work around IT if you don’t - but you need a way to keep it secure, and maintain a chain of custody in some cases. There is a solution.

There’s a free Codeplex project called “BlobShare” that we have for you here. You pull it down, follow the instructions, and deploy it to Windows and SQL Azure. From there, you have a clean, safe, secure, controlled interface to let your folks share files.

There’s a full post on the inner-workings and the design of this code - and it’s just code. You can change it, use your own logo, restrict it further, loosen it up, whatever you like. That’s the point of Platform as a Service - control and ease.

http://blogs.msdn.com/b/vbertocci/archive/2011/10/31/blobshare-sample-acs-protected-file-sharing.aspx

Windows Azure Use Case: Supplementing Infrastructure

BuckWoody — Tue, 01 Nov 2011 13:22:01 GMT

I’ve explained before that Windows Azure is a Platform as a Service - at its simplest, that means that you write software and Azure runs it for you. But what if you are a shop that normally buys “off the shelf” software, and the only software you write is an internal utility here and there - can you still use Windows Azure?

Absolutely. Windows Azure is made up of several components, such as computing, storage and other objects you can call in code. And as such, some companies have extended software and hardware you can use that is backed by Windows Azure. In some cases that’s simply software you can use like you would any other - making Azure a “Software as a Service”, which I’ll write about later.

But in other cases the software a vendor provides a utility you can use for your infrastructure. For instance - “BlobShare”, a free codeplex offering, allows you to create a place where you can upload and download files securely. It allows you to hook in your local Windows accounts (without sending names and passwords over the Internet) or even Google, LiveID or Yahoo logins to access the files. You can think of it as a “corporate DropBox”. Download it and read more about it here: http://blobshare.codeplex.com/ and learn how to hook in Active Directory here: http://blogs.msdn.com/b/vbertocci/archive/2011/10/31/blobshare-sample-acs-protected-file-sharing.aspx

Another option along these lines is using a hardware appliance to store data in Azure. This has a particular attraction, since some of these appliances (such as the offerings from StorSimple) do more than just act as a storage target. The device plugs into the network as storage, moves the data to Windows Azure, de-duplicates (thus saving storage costs), encrypts, acts as a backup device, works with SharePoint and more. No coding is required to use their solution, and it can even act as a Disaster Recovery site.

Storage isn’t the only way to use Windows Azure in place of infrastructure. Using the Application Fabric Service Bus, you can perform some functions of data access and transfer between companies that used to take a VPN setup - which is no longer required. the caching function in the Service Bus can even relieve the need to upgrade a server for performance. And of course SQL Azure allows you to access a SQL Server database without any server at all.

So if you’re thinking that Windows and SQL Azure aren’t for you, think again. Start with the problem you have, and see what options you have for solving it.

Big Data and the Cloud - More Hype or a Real Workload?

BuckWoody — Tue, 18 Oct 2011 13:57:36 GMT

Last week Microsoft announced several new offerings for “Big Data” - and since I’m a stickler for definitions, I wanted to make sure I understood what that really means. What is “Big Data”? What size hard drive is that? After all, my laptop has 1TB of storage - is my laptop “Big Data”?

There are actually a few definitions for this term, most notably those involving the “Four V’s” Volume, Velocity, Variety and Variability. Others disagree with this definition. I tend to try and get things into their simplest form, so I’m using this definition for myself:

Big data is defined as a large set of computationally expensive data that is worked on simultaneously.

Let me flesh that out a little. To be sure, “Big Data” has a larger size than say a few megabytes. The reason this is important is that it takes special hardware to be able to move large sets of data around, store it, process it and so on. (large set)

If you store a LOT of data, but only use a small portion of it at a time, that really isn’t super-hard to do. It’s mainly a storage issue at that point. But, if you do need to work with a large portion of the data at one time, then the memory, CPU and transfer components of the system have to adapt to be responsive - new ways to work with that data (game theory, knot-algorithms, map-reduce, etc.) need to be brought into play. (computationally expensive)

Once that data is loaded into the processing area (memory or whatever other mechanism is used) it must be worked on in parallel to come back in a reasonable time. You have two options here - you can scale the system up with more internal hardware (CPU’s, memory and so on) or you can scale it out to have multiple systems work on it at the same time using paradigms such as map/reduce and so on. Actually, when you lay this out in an architecture diagram, scale up or out doesn’t actually change the logical structure of the process - in scale out the network becomes the bus, and the nodes become more RAM and computing power. Of course, there are changes in code for how you stitch the workload back together. (worked on simultaneously)

So back to the original question. Is Big Data, as I have defined it here, a workload for Windows and SQL Azure? Absolutely! In fact, it’s probably one of the main workloads, and I believe it represents the latest, and perhaps also the earliest frontier of computing. Jim Gray, a former researcher here at Microsoft and a hero of mine, was working on this very topic. I believe as he did - all computing is simply an interface over data.

Microsoft has multiple offerings on the topic of Big Data. In posts that follow from myself and my co-workers, we’ll explore when and where you use each one. Whether you are a data professional or a developer, this is the new frontier - don’t wait to educate yourself on how to leverage Big Data for your organization.

Hadoop on Windows Azure and SQL Server - Microsoft’s partnership to include Hadoop workloads on Windows Azure and SQL Server/Parallel Data Warehouse (PDW)

LINQ to HPC - Microsoft’s High-Performance Computing SKU of HPC is now in Azure

Windows Azure Table Storage - A key/value pair type storage with full partitioning that is immediately consistent, able to handle huge loads of data and works with any REST-compatible language

Other offerings - Including the new Data Explorer, Project Daytona (with a Big Data Toolkit for Scientists and researchers), Power View and more.

The era of Big Data is here. And you can use Windows and SQL Azure to bring it to your organization.

Creating a Distributed Computing System Using a Windows Azure Queue

BuckWoody — Tue, 11 Oct 2011 13:12:42 GMT

The Windows Azure Queue component, like all Windows Azure components (Roles, Storage, App Fabric, SQL Azure) can be used by itself or with other Windows Azure components. That’s why I refer to Windows Azure as “Distributed Computing” rather than “cloud”.

Having a distributed off premise queue has a lot of use-cases. An interesting use-case is a company that wanted to harness the power of all of the PC’s and laptops in the company when they were not being used throughout the day. A developer wrote a screen-saver program that connected to an Azure Queue, pulling work off of the queue and placing an entry when it was done. In essence he had a partially connected distributed work relay system, and since he used a Windows Azure Queue, the system worked from anywhere in the world.

He uses an on-site central server (which was actually only a workstation-level system) that holds the computations in a scatter/gather paradigm. The computations are broken into less-than-8K chunks, so that it fits within a message. The server connects to a Windows Azure Queue, and places the message marked for computation. It also scrubs the Queue for completed work, and as part of the process puts that kind of message into a mapping function (queues are not guaranteed a message order).

The workstations that are not being used (even those systems at remote workers and travelers) connect to the same Windows Azure Queue when the system is not being used for a period of time, when the screen saver kicks in. It then takes one message from the queue, computes the information, and then sets a new message for the server to pick up with the answer. The workstation then deletes the message.

The Server picks up the completed work, processes it and then deletes that queue message. He also added logic to process messages for computation on the server as well, when the server function of adding work is not required.

There are a few caveats here. This works because of the mapping function on the head server. Order is not guaranteed, so he includes a number for the function step as part of the message body, which cuts the size a bit. Also, he’s careful to watch the encoding, since Azure will hand binary back in Base64 format.

He’s found that there are enough systems to ensure that the messages are cleared every few days – important, since the Windows Azure Queue ages out after seven days. Also, he’s careful to use the CloudQueue.PeekMessage function when he wants to monitor the system – that function ensures that the message status doesn’t reset as “read” when he accesses it.

This is a great example of using the “cloud” as what it is intended to be – a distributed architecture you can use as needed to solve a business problem. It’s not an “all or nothing” proposition, but instead it is simply another set of components to use where you need them.

Rip and Replace or Extend and Embrace?

BuckWoody — Tue, 13 Sep 2011 11:20:05 GMT

As most of you know, I don’t like the term “cloud” very
much. It isn’t defined, which means it can be anything. I prefer “distributed
computing”, which is more technically accurate and describes what you’re doing
in more concrete terms.

So when you think about Windows and SQL Azure, you don’t
have to think about an entire product – you can use parts of the system
together or independently to accomplish what you need to do. You can use the
computing functions, storage, and more and more I see folks leverage the
Service Bus to enable current applications to expose things to the web.

And that brings up the point of this post. Once you decide
that a distributed architecture works to solve a problem, you’re faced with a
decision: should you completely re-write your architecture to take advantage of
the current systems or should you just fold in new code that makes the data or
function available to the web?

Of course, the answer is always “it depends” on the situation
– and it does. But unless you’re fixing a problem with current code, I usually
advocate a migration approach. That means at the very least retaining the
business logic (again, unless it’s not currently working) and as much of the
code as you can. In fact, if you follow this paradigm, you’re on your way to
making a Service Bus out of the functions you currently have. You can expose
the results of a system rather than opening the system up. Let’s take an
example.

Assume for a moment that you have an order-taking system
on-premise. That system performs many functions, one of which might creating a
Purchase Order. Your system might be enclosed, meaning that it has an
application that talks to a middle-tier, and then from there to a database
system. A query is generated from a screen, and passed along to eventually
compute, store and return a Purchase Order Number, along with other
information. Imagine now that you wire up the code not only to return the PO
number to the client, but to make that number available on an endpoint –
actually really not that hard to do.

Now you can make that PO number available to the web using
Azure. You could restrict who can make that call to the system, or open it up
to a broader audience. Or instead of the PO Number, you could make a product
list available. And you can go further than that – EBay, for instance, uses the
OData protocol (which is very cool in and of itself) which you can query from
the web. You could compare your company’s product catalog to what is on EBay,
and list the items you have there if there are no competitors in that space.
And on and on it goes.

So the point is this – where you can, retain what works.
Fold in systems like Azure where they make sense. Extend and Embrace.

Demand Capacity Real World Example on Windows Azure

BuckWoody — Tue, 30 Aug 2011 14:08:48 GMT

I recently had a fascinating conversation with CoreConnex, a company that has a Financial Analysis dashboard called Corelytics. Corelytics is an Azure based SaaS product with about 1,000 customers supported in 11 countries. Last week Corelytics was announced as the Intuit Application Showcase Grand Prize winner. Intuit has a huge customer base for their QuickBooks accounting system and this award creates an interesting dilemma for Corelytics: like, how do you support a flood of new customers that you were not expecting?

Corelytics uses Azure and they were able to connect to Intuit QuickBooks through the Azure App Fabric. This allows a tight, fast and highly secure integration between the accounting system and the dashboard. But the platform decision wasn’t expecting such a big spike in new customers.

But as we all know, whenever you introduce any new product, you have to be ready to service the need – especially if it happens to grow explosively. There’s always a curve of introduction, ramp-up, maturity and so on. Sometimes that curve is incredibly quick, and other times it has a different shape – and that’s the difficulty. You can’t always predict how many people will use your product and when. Not only that, you’re not certain when that need will change, or even at what frequency.

As we discussed sales and adoption cycles, they pointed out that one of the reasons they chose to use a platform like Azure was specifically for the elasticity to accommodate an unknown growth rate. With stateless code, the system can handle massive changes in workload automatically – something that is far more difficult with an on-premise or even an Infrastructure as a Service system where you have to build, configure and patch operating systems.

There’s a subtle but important point here - the point is you need to be prepared for all possibilities, great and small, before you decide where it needs to live. This is another real-world example of that.

The Conundrum of Cloud Computing Evidence

BuckWoody — Tue, 26 Jul 2011 12:56:08 GMT

“Who is using Windows Azure? How are they doing it?”

I get asked this all the time as I speak to our clients when we mention Windows or SQL Azure as a possible solutions to an architectural problem the company has. I completely understand the question. I’ve worked far longer outside of Microsoft than here, and one of my roles as a Systems Architect was to select solutions from a range of possibilities. When you’re faced with a decision like that, it saves a ton of time if you can find out where others have done the same thing and what problems and successes they had with this or that approach. I’m not a marketing person, so I work with clients directly all the time, and so folks know I’ve seen the way others do things. And they want to know what someone else has done before they try it.

But therein lies the rub.

Most folks aren’t willing to talk about their internal infrastructure much. We talk about how we do things here at Microsoft quite frequently, and if you’re not using this resource, definitely go research what we’ve done. In fact, I don’t think I’ve seen many other technology companies with this depth of information. For instance, here’s a quick list of the areas we talk about on our internal systems:

Microsoft Desktop Optimization Pack

Office System

System Center

Windows

And yes, Azure as you can see is part of that list - we’ve implemented it everywhere. But when I point out that we’re using IT resources just like any other company, I hear “yeah, but you’re Microsoft. You have lots of people to handle the systems, and you can just have the developer who wrote the software come over and fix it if it breaks.” Well, that’s not the way it works at all. When I got here, I met with some folks from Microsoft IT, and I was absolutely shocked at how few people manage our systems. From Systems Administrators all the way to the DBA’s, the ratio of technical resource to machines is really extraordinary, and I had far more folks managing systems for me in my other companies than we have here. And no, we don’t send developers to fix production systems “just for us”, although we will do that when we are using a beta of something that we haven’t even released to CTP yet.

But even with this evidence of ourselves, clients want to “see how others are doing things with Azure”. We do have the customer evidence site, and we have lots of case-studies there. But many of these lack technical details, again, because folks aren’t always willing to share that. That’s understandable - for instance, even though Microsoft does it, I’m not sure I would have the Visio diagrams for my company’s applications posted in a public place. There may even be legal or competitive reasons not to do that. So as you peruse that list, keep in mind these are only the companies that are willing to talk in public about their experience - we have far more on Azure than this list.

J.D. Meiers has a good blog post entry of a rundown of the companies that have released case studies on Azure. And be assured that we have a patterns and practices group that works with the teams here at Azure to make sure we apply what we learn as we help each client. And many of our partners that you use to help you implement technology are here at Microsoft this week, sharing customer stories (with permission, of course), which allows us to learn from them on how you’re doing things - what works and what doesn’t.

As time goes on I think you’ll see more patterns emerge from the engagements we’ve done. Right now those are sometimes a competitive advantage, so you’ll see us generalize the patterns into content we can release. For now, check out those case studies, and do make use of the Internal Microsoft IT sites. They really are quite detailed, and have a lot of information you can use.

SQL Azure Use Case: Shared Data Hub

BuckWoody — Tue, 05 Apr 2011 14:10:50 GMT

This is one in a series of posts on when and where to use a distributed architecture design in your organization's computing needs. You can find the main post here: http://blogs.msdn.com/b/buckwoody/archive/2011/01/18/windows-azure-and-sql-azure-use-cases.aspx

Description:

Organizations often need to share all or part of a data set, which is consumed by other systems. These systems can be on-premise or at another location, or even at a different organization.

Many times these systems use a well-defined data interchange system, such as EDI or other standards. In the case of a trusted system, simply using a direct connection into another database is the process used to transfer data. This process might be one-way or bi-directional.

But there are systems that transfer data back and forth in stages using intermediate systems. A typical data flow in this case looks similar to the following:

In this example, the owning system contains data set A. This is set to a staging system or server, where the receiving system collects it. The receiving system contains data set B and works with data set A to create a new data set, C. This new data is consumed by the original system to complete the cycle. A concrete example is an inventory control system. Data set A is the original inventory list, shipped to a manufacturer. The manufacturer consumes the inventory available, orders and components, and returns the ordering bid with any changes to the staging server as data set C. The data is consumed by the originating system and components are noted in the overall flow of data set A.

Note: Normally this is solved with a full EDI implementation, but this process is still a common practice.

There are other examples, but the general concept is one where the need is for two, possibly untrusted systems to share a common source of data.

Implementation:

One possible solution is to segregate the data that is being transferred into an agree-upon set of entities that can be added or edited real-time, where both systems (or many) feed from the same data set instead of shipping the data. This removes latency, improves data quality, and shares the cost of the data. Also, security is increased because there are no shared logins - each firm gets its own.

One consideration with this layout is that the source systems must be altered to use a shared data set. If this is not possible, this is still a possibility as the system can be used as it was before - a data transfer - but the data can be cleansed real-time by both systems. It’s also a more secure and shared-cost system even if used in the original manner.

Resources:

Security is a concern in this arrangement, so it’s best to understand exactly how the security works in SQL Azure: http://msdn.microsoft.com/en-us/library/ff394108.aspx

Another possibility to solve this pattern is to use Data Sync, in many different arrangements that involve SQL Azure. You can learn more about it here: http://blogs.msdn.com/b/sync/archive/2010/10/07/windows-azure-sync-service-demo-available-for-download.aspx