Buck Woody : Career, Cloud, SQL Azure

“I could use a little help here” or “I can do it myself, thank you” for Cloud Projects

BuckWoody — Tue, 03 Apr 2012 13:20:55 GMT

Windows Azure allows you to write code in languages within the .NET stack, you can use Java, C++, PHP, NodeJS and others. Code is code - other than keeping things stateless, using a Web or Worker Role in Azure is not all that different from working with an on-premises system.

However….

Working in a scalable, component-based stateless architecture that can use federated security is not all that common for many developers. Some are used to owning the server, scaling up, and state-full paradigms that have a single security domain. Making the transition whilst trying to create a new software application or even port a previous one can be daunting. Sure, we have absolutely tons of free training, kits, videos, online books and more to learn on your own, but some things like architecture can be pivotal as you move along.

So the question is, should you just strike out on your own for a Cloud project, or get Microsoft Consulting Services or another partner to work with you on your first one? I use a few decision points to help guide the projects I assist in.

Note: I’m a huge fan of having help that ends up giving you training and leaves you in charge. If you do engage with someone to help you, make sure you keep this clear and take more and more ownership yourself as the project progresses.

How much time do you have?

Usually the first thing I ask is about the timeline for the project. It doesn’t matter how skilled you are, if you have a short window to get things done it’s better to get help - especially if this is your first cloud project. Having someone that knows the platform well can save you amazing amounts of time. If you have longer, then start with the training in the link above and once you feel confident, jump in.

How complex is the project?

If there are a lot of moving parts, it’s best to engage a partner. The reason is that certain interactions - particularly things like Service Bus or Data Integration - can be quite different than what you may have encountered before.

How many people do you have?

I have a “pizza rule” about projects I’ve used in my career - if it takes over two pizzas to feed everyone on the project, it’s too big and will fail. That being said, one developer and a one-week deadline does not a good project make, usually. It’s best to have at least one architect (or someone in that role) guiding the project along, and at least two developers to work on a cloud project. That’s a generalization of course, since I’ve seen great software on Azure with one developer writing code all by herself, but for more complex projects, more (to a point) is better. The nice thing about bringing on a partner is that you don’t have to hire them full time - they help you and then they go away.

How critical is the project?

There’s no shame in using some help. If the platform is new, if the project is large and complex, and if it is critical to the business, you should engage a partner. That’s regardless of Cloud or anything else - get some help. You don’t want to hit your company’s bottom line in a negative way, but you have to innovate and get them a competitive advantage. Do your research, make sure the partner is qualified to help you, and get it done.

Don’t let these questions scare you off. There are lots of projects you can implement on Windows and SQL Azure with nothing other than the Software Development Kit (SDK) that you get for free with Windows Azure. And assistance comes in many forms - sometimes just phone support, a friend you can ask. Microsoft Consulting Services or any of our great partners. You can get help on just the architecture piece or have them show you how to write the code. They’ll get involved as little or as much as you like.

Big Data and the Cloud - More Hype or a Real Workload?

BuckWoody — Tue, 18 Oct 2011 13:57:36 GMT

Last week Microsoft announced several new offerings for “Big Data” - and since I’m a stickler for definitions, I wanted to make sure I understood what that really means. What is “Big Data”? What size hard drive is that? After all, my laptop has 1TB of storage - is my laptop “Big Data”?

There are actually a few definitions for this term, most notably those involving the “Four V’s” Volume, Velocity, Variety and Variability. Others disagree with this definition. I tend to try and get things into their simplest form, so I’m using this definition for myself:

Big data is defined as a large set of computationally expensive data that is worked on simultaneously.

Let me flesh that out a little. To be sure, “Big Data” has a larger size than say a few megabytes. The reason this is important is that it takes special hardware to be able to move large sets of data around, store it, process it and so on. (large set)

If you store a LOT of data, but only use a small portion of it at a time, that really isn’t super-hard to do. It’s mainly a storage issue at that point. But, if you do need to work with a large portion of the data at one time, then the memory, CPU and transfer components of the system have to adapt to be responsive - new ways to work with that data (game theory, knot-algorithms, map-reduce, etc.) need to be brought into play. (computationally expensive)

Once that data is loaded into the processing area (memory or whatever other mechanism is used) it must be worked on in parallel to come back in a reasonable time. You have two options here - you can scale the system up with more internal hardware (CPU’s, memory and so on) or you can scale it out to have multiple systems work on it at the same time using paradigms such as map/reduce and so on. Actually, when you lay this out in an architecture diagram, scale up or out doesn’t actually change the logical structure of the process - in scale out the network becomes the bus, and the nodes become more RAM and computing power. Of course, there are changes in code for how you stitch the workload back together. (worked on simultaneously)

So back to the original question. Is Big Data, as I have defined it here, a workload for Windows and SQL Azure? Absolutely! In fact, it’s probably one of the main workloads, and I believe it represents the latest, and perhaps also the earliest frontier of computing. Jim Gray, a former researcher here at Microsoft and a hero of mine, was working on this very topic. I believe as he did - all computing is simply an interface over data.

Microsoft has multiple offerings on the topic of Big Data. In posts that follow from myself and my co-workers, we’ll explore when and where you use each one. Whether you are a data professional or a developer, this is the new frontier - don’t wait to educate yourself on how to leverage Big Data for your organization.

Hadoop on Windows Azure and SQL Server - Microsoft’s partnership to include Hadoop workloads on Windows Azure and SQL Server/Parallel Data Warehouse (PDW)

LINQ to HPC - Microsoft’s High-Performance Computing SKU of HPC is now in Azure

Windows Azure Table Storage - A key/value pair type storage with full partitioning that is immediately consistent, able to handle huge loads of data and works with any REST-compatible language

Other offerings - Including the new Data Explorer, Project Daytona (with a Big Data Toolkit for Scientists and researchers), Power View and more.

The era of Big Data is here. And you can use Windows and SQL Azure to bring it to your organization.

Book Review (Book 1) - Programming Windows Azure by Siriram Krishnan

BuckWoody — Tue, 28 Jun 2011 17:18:00 GMT

As part of my professional development, I’ve created a list of books to read throughout the year, starting in June of 2011. This a review of the first one, called Programming Windows Azure by Siriram Krishnan. You can find my entire list of books I’m reading for my career here: http://blogs.msdn.com/b/buckwoody/archive/2011/06/07/head-in-the-clouds-eyes-on-the-books.aspx

Why I Chose This Book:

As part of my learning style, I try to read multiple books about a single subject. I’ve found that at least 3 books are necessary to get the right amount of information to me. This is a “technical” work, meaning that it deals with technology and not business, writing or other facets of my career. I’ll have a mix of all of those as I read along.

I chose this work in addition to others I’ve read since it covers everything from an introduction to more advanced topics in a single book. It also has some practical examples of actually working with the product, particularly on storage. Although it’s dated, many examples normally translate. I also saw that it had pretty good reviews.

What I learned:

I learned a great deal about storage, and many useful code snippets. I do think that there could have been more of a focus on the application fabric - but of course that wasn’t as mature a feature when this book was written. I learned some great architecture examples, and in one section I learned more about encryption. In that example, however, I would rather have seen the examples go the other way - the book focused on moving data from on-premise to Azure storage in an encrypted fashion. Using the Application Fabric I would rather see sensitive data left in a hybrid fashion on premise, and connect to for the Azure application. Even so, the examples were very useful.

If you’re looking for a good “starter” Azure book, this is a good choice. I also recommend the last chapter as a quick read for a DBA, or Database Administrator. It’s not very long, but useful. Note that the limits described are incorrect - which is one of the dangers of reading a book about any cloud offering. The services offered are updated so quickly that the information is in constant danger of being “stale”. Even so, I found this a useful book, which I believe will help me work with Azure better.

Raw Notes:

I take notes as I read, calling that process “reading with a pencil”. I find that when I do that I pay attention better, and record some things that I need to know later. I’ll take these notes, categorize them into a OneNote notebook that I synchronize in my Live.com account, and that way I can search them from anywhere. I can even read them on the web, since the Live.com has a OneNote program built in. Note that these are the raw notes, so they might not make a lot of sense out of context - I include them here so you can watch my though process.

Programming Windows Azure by Siriram Krishnan: Learning about how to select applications suitable for Distributed Technology.

Application Fabric gets the least attention; probably because it was newer at the time.
Very clear (Chapter One)
Good foundation
Background and history, but not too much
I normally arrange my descriptions differently, starting with the use-cases and moving to physicality, but this difference helps me.
Interesting that I am reading this using Safari Books Online, which uses many of these concepts.
Taught me some new aspects of a Hypervisor – very low-level information about the Azure Fabric (not to be confused with the Application Fabric feature) (Chapter Two)
Good detail of what is included in the SDK. Even more is available now. CS = Cloud Service (Chapter 3)
Place Storage info in the configuration file, since it can be streamed in-line with a running app. Ditto for logging, and keep separated configs for staging and testing. Easy-switch in and switch out. (Chapter 4)
There are two Runtime API’s, one of external and one for internal.
Realizing how powerful this paradigm really is.
Some places seem light, and to drop off but perhaps that’s best.
Managing API is not charged, which is nice. I don’t often think about the price, until it comes to an actual deployment (Chapter 5)
Csmanage is something I want to dig into deeper. API requires package moves to Blob storage first, so it needs a URL.
Csmanage equivalent can be written in Unix scripting using openssl.
Upgrades are possible, and you use the upgradeDomainCount attribute in the Service-Definition.csdef file
Always use a low-privileged account to test on the dev fabric, since Windows Azure runs in partial trust. Full trust is available, but can be dangerous and must be well-thought out. (Chapter 6)
Learned how to run full CMD commands in a web window – not that you would ever do that, but it was an interesting view into those links.
This leads to a discussion on hosting other runtimes (such as Java or PHP) in Windows Azure. I got an expanded view on this process, although this is where the book shows its age a little. Books can be a problem for Cloud Computing for this reason – things just change too quickly.
Windows Azure storage is not eventually consistent – it is instantly consistent with multi-phase commit. Plumbing for this is internal, not required to code that. (Chapter 7)
REST API makes the service interoperable, hybrid, and consistent across code architectures. Nicely done.
Use affinity groups to keep data and code together.
Side note: e-book readers need a common “notes” feature.
There’s a decent quick description of REST in this chapter.
Learned about CloudDrive code – PowerShell sample that mounts Blob storage as a local provider. Works against Dev fabric by default, can be switched to Account.
Good treatment in the storage chapters on the differences between using Dev storage and Azure storage. These can be mitigated.
No, blobs are not of any size or number. Not a good statement (Chapter 8)
Blob storage is probably Azure’s closest play to Infrastructure as a Service (Iaas).
Blob change operations must be authenticated, even when public.
Chapters on storage are pretty in-depth.
Queue Messages are base-64 encoded (Chapter 9)
The visibility timeout ensures processing of message in a disconnected system.
Order is not guaranteed for a message, so if you need that set an increasing number in the queue mechanism.
While Queues are accessible via REST, they are not public and are secured by default.
Interesting – the header for a queue request includes an estimated count. This can be useful to create more worker roles in a dynamic system.
Each Entity (row) in the Azure Table service is atomic – all or nothing. (Chapter 10)
An entity can have up to 255 Properties
Use “ID” for the class to indicate the key value, or use the [DataServiceKey] Attribute.
LINQ makes working with the Azure Table Service much easier, although Interop is certainly possible.
Good description on the process of selecting the Partition and Row Key.
When checking for continuation tokens for pagination, include logic that falls out of the check in case you are at the last page.
On deleting a storage object, it is instantly unavailable, however a background process is dispatched to perform the physical deletion. So if you want to re-create a storage object with the same name, add retry logic into the code.
Interesting approach to deleting an index entity without having to read it first – create a local entity with the same keys and apply it to the Azure system regardless of change-state.
Although the “Indexes” description is a little vague, it’s interesting to see a Folding and Stemming discussion a-la the Porter Stemming Algorithm. (Chapter 11)
Presents a better discussion of indexes (at least inverted indexes) later in the chapter.
Great treatment for DBA’s in Chapter 11.
We need to work on getting secondary indexes in Table storage.
There is a limited form of transactions called “Entity Group Transactions” that, although they have conditions, makes a transactional system more possible.
Concurrency also becomes an issue, but is handled well if you’re using Data Services in .NET. It watches the Etag and allows you to take action appropriately.
I do not recommend using Azure as a location for secure backups. In fact, I would rather have seen the examples in (Chapter 12) go the other way, showing how data could be brought back to a local store as a DR or HA strategy. Good information on cryptography and so on even so.
Chapter seems out of place, and should be combined with the Blob chapter.
(Chapter 13) on SQL Azure is dated, although the base concepts are OK.
Nice example of simple ADO.NET access to a SQL Azure (or any SQL Server Really) database.

Computer books are dead. Well, some of them, anyway.

BuckWoody — Tue, 10 May 2011 13:58:23 GMT

I read a lot. I mean a LOT. It seems that computer professionals have much in common with medical professionals – we have to read in order to stay on top of our game. For me, this used to mean web sites, magazines, and other print medium, and of course lots of books. I’ve even written several computer books myself and had them published.

Whenever I teach a class, do a presentation, or hold an architectural design session on a new (or new to that person) technology, they usually follow up with “what’s a good book for learning X technology?” This happens so often that I have a list I keep of the titles I like for a particular subject – you probably have similar book lists.

Windows, SQL Server, and other Microsoft products change on an average of around three or four year cycles. That’s enough time to play with a beta product, wait until it releases, and write a solid book about it, and have that in a decent market for sales, and allow people to read and recommend it.

Enter “the Cloud” – Distributed Computing.

Windows Azure and SQL Azure don’t release every three years. Changes – some of them dramatic – release every three or four months. You can’t even write a book that fast, much less update it that quickly and re-sell it. So what is a technical professional to do?

Well, although I really like a couple of books I’ve read so far (especially this one, print and e-book version here), they are out of date almost by the time they publish. Instead, I rely on blogs, the web, documentation from the vendor and how-to articles published online. Many of these, ironically, are stored, hosted or delivered using – wait for it – Windows Azure. That’s interesting because it’s a medium that describes itself – “reflection”, anyone?

This brings up an interesting conundrum. Books have a version, are arranged, thought-out and categorized. Since I’m now getting information off of the web, it’s difficult to figure out whether that material is correct at the time, what level it’s aimed at – and forget about any coherent structure. It’s topic-by-topic.

So, like most of you, I use links and favorites to arrange things. And I found myself making “virtual books” by essentially creating my own Table-Of-Contents. I’ve shared some of those, such as my Windows and SQL Azure Learning Plan. The key is that I have to update that to ensure that the latest information is there – otherwise it becomes an organized list that is not authoritative.

Don’t get me wrong – I still have tons of (e-book format) books, especially on “conceptual” topics like development paradigms and so on. But when it comes to specifics and how-to’s – electronic medium is best for me. It’s more current, adaptable, searchable, interactive and immersive than books. But how long will I retain regular print-type books? We’ll see. Times, they are a changing – fast.