Buck Woody : Developer, Career, Concepts

The Data Scientist

BuckWoody — Tue, 15 Nov 2011 15:00:18 GMT

A new term - well, perhaps not that new - has come up and I’m actually very excited about it. The term is Data Scientist, and since it’s new, it’s fairly undefined. I’ll explain what I think it means, and why I’m excited about it.

In general, I’ve found the term deals at its most basic with analyzing data. Of course, we all do that, and the term itself in that definition is redundant. There is no science that I know of that does not work with analyzing lots of data. But the term seems to refer to more than the common practices of looking at data visually, putting it in a spreadsheet or report, or even using simple coding to examine data sets.

The term Data Scientist (as far as I can make out this early in it’s use) is someone who has a strong understanding of data sources, relevance (statistical and otherwise) and processing methods as well as front-end displays of large sets of complicated data. Some - but not all - Business Intelligence professionals have these skills. In other cases, senior developers, database architects or others fill these needs, but in my experience, many lack the strong mathematical skills needed to make these choices properly.

I’ve divided the knowledge base for someone that would wear this title into three large segments. It remains to be seen if a given Data Scientist would be responsible for knowing all these areas or would specialize. There are pretty high requirements on the math side, specifically in graduate-degree level statistics, but in my experience a company will only have a few of these folks, so they are expected to know quite a bit in each of these areas.

Persistence

The first area is finding, cleaning and storing the data. In some cases, no cleaning is done prior to storage - it’s just identified and the cleansing is done in a later step. This area is where the professional would be able to tell if a particular data set should be stored in a Relational Database Management System (RDBMS), across a set of key/value pair storage (NoSQL) or in a file system like HDFS (part of the Hadoop landscape) or other methods. Or do you examine the stream of data without storing it in another system at all?

This is an important decision - it’s a foundation choice that deals not only with a lot of expense of purchasing systems or even using Cloud Computing (PaaS, SaaS or IaaS) to source it, but also the skillsets and other resources needed to care and feed the system for a long time. The Data Scientist sets something into motion that will probably outlast his or her career at a company or organization.

Often these choices are made by senior developers, database administrators or architects in a company. But sometimes each of these has a certain bias towards making a decision one way or another. The Data Scientist would examine these choices in light of the data itself, starting perhaps even before the business requirements are created. The business may not even be aware of all the strategic and tactical data sources that they have access to.

Processing

Once the decision is made to store the data, the next set of decisions are based around how to process the data. An RDBMS scales well to a certain level, and provides a high degree of ACID compliance as well as offering a well-known set-based language to work with this data. In other cases, scale should be spread among multiple nodes (as in the case of Hadoop landscapes or NoSQL offerings) or even across a Cloud provider like Windows Azure Table Storage. In fact, in many cases - most of the ones I’m dealing with lately - the data should be split among multiple types of processing environments. This is a newer idea. Many data professionals simply pick a methodology (RDBMS with Star Schemas, NoSQL, etc.) and put all data there, regardless of its shape, processing needs and so on.

A Data Scientist is familiar not only with the various processing methods, but how they work, so that they can choose the right one for a given need. This is a huge time commitment, hence the need for a dedicated title like this one.

Presentation

This is where the need for a Data Scientist is most often already being filled, sometimes with more or less success. The latest Business Intelligence systems are quite good at allowing you to create amazing graphics - but it’s the data behind the graphics that are the most important component of truly effective displays.

This is where the mathematics requirement of the Data Scientist title is the most unforgiving. In fact, someone without a good foundation in statistics is not a good candidate for creating reports. Even a basic level of statistics can be dangerous. Anyone who works in analyzing data will tell you that there are multiple errors possible when data just seems right - and basic statistics bears out that you’re on the right track - that are only solvable when you understanding why the statistical formula works the way it does.

And there are lots of ways of presenting data. Sometimes all you need is a “yes” or “no” answer that can only come after heavy analysis work. In that case, a simple e-mail might be all the reporting you need. In others, complex relationships and multiple components require a deep understanding of the various graphical methods of presenting data. Knowing which kind of chart, color, graphic or shape conveys a particular datum best is essential knowledge for the Data Scientist.

Why I’m excited

I love this area of study. I like math, stats, and computing technologies, but it goes beyond that. I love what data can do - how it can help an organization. I’ve been fortunate enough in my professional career these past two decades to work with lots of folks who perform this role at companies from aerospace to medical firms, from manufacturing to retail.

Interestingly, the size of the company really isn’t germane here. I worked with one very small bio-tech (cryogenics) company that worked deeply with analysis of complex interrelated data.

So watch this space. No, I’m not leaving Azure or distributed computing or Microsoft. In fact, I think I’m perfectly situated to investigate this role further. We have a huge set of tools, from RDBMS to Hadoop to allow me to explore. And I’m happy to share what I learn along the way.

Big Data and the Cloud - More Hype or a Real Workload?

BuckWoody — Tue, 18 Oct 2011 13:57:36 GMT

Last week Microsoft announced several new offerings for “Big Data” - and since I’m a stickler for definitions, I wanted to make sure I understood what that really means. What is “Big Data”? What size hard drive is that? After all, my laptop has 1TB of storage - is my laptop “Big Data”?

There are actually a few definitions for this term, most notably those involving the “Four V’s” Volume, Velocity, Variety and Variability. Others disagree with this definition. I tend to try and get things into their simplest form, so I’m using this definition for myself:

Big data is defined as a large set of computationally expensive data that is worked on simultaneously.

Let me flesh that out a little. To be sure, “Big Data” has a larger size than say a few megabytes. The reason this is important is that it takes special hardware to be able to move large sets of data around, store it, process it and so on. (large set)

If you store a LOT of data, but only use a small portion of it at a time, that really isn’t super-hard to do. It’s mainly a storage issue at that point. But, if you do need to work with a large portion of the data at one time, then the memory, CPU and transfer components of the system have to adapt to be responsive - new ways to work with that data (game theory, knot-algorithms, map-reduce, etc.) need to be brought into play. (computationally expensive)

Once that data is loaded into the processing area (memory or whatever other mechanism is used) it must be worked on in parallel to come back in a reasonable time. You have two options here - you can scale the system up with more internal hardware (CPU’s, memory and so on) or you can scale it out to have multiple systems work on it at the same time using paradigms such as map/reduce and so on. Actually, when you lay this out in an architecture diagram, scale up or out doesn’t actually change the logical structure of the process - in scale out the network becomes the bus, and the nodes become more RAM and computing power. Of course, there are changes in code for how you stitch the workload back together. (worked on simultaneously)

So back to the original question. Is Big Data, as I have defined it here, a workload for Windows and SQL Azure? Absolutely! In fact, it’s probably one of the main workloads, and I believe it represents the latest, and perhaps also the earliest frontier of computing. Jim Gray, a former researcher here at Microsoft and a hero of mine, was working on this very topic. I believe as he did - all computing is simply an interface over data.

Microsoft has multiple offerings on the topic of Big Data. In posts that follow from myself and my co-workers, we’ll explore when and where you use each one. Whether you are a data professional or a developer, this is the new frontier - don’t wait to educate yourself on how to leverage Big Data for your organization.

Hadoop on Windows Azure and SQL Server - Microsoft’s partnership to include Hadoop workloads on Windows Azure and SQL Server/Parallel Data Warehouse (PDW)

LINQ to HPC - Microsoft’s High-Performance Computing SKU of HPC is now in Azure

Windows Azure Table Storage - A key/value pair type storage with full partitioning that is immediately consistent, able to handle huge loads of data and works with any REST-compatible language

Other offerings - Including the new Data Explorer, Project Daytona (with a Big Data Toolkit for Scientists and researchers), Power View and more.

The era of Big Data is here. And you can use Windows and SQL Azure to bring it to your organization.

The Top-Five Most Valuable Development Team Positions

BuckWoody — Tue, 30 Nov 2010 14:34:00 GMT

In many organizations, there’s a push to save cost. “Cost” is an integral part of a profit-and-loss statement in the organization. Even if you work in a non-profit organization, cost control is central. Of course, you can take cost-control too far. You could, for instance, remove the shipping costs from your company, but if you happen to be an online retailer, removing the ability to ship your product would eventually doom your company. So there’s obviously a spectrum of necessary cost and wasteful costs.

People are a large part of the cost of any company - in some cases the highest cost. So it stands to reason that there’s a natural tension between paying well to find good people, and the need to lower the cost of those resources. In the rush to save costs, we’ve all seen people let go that really shouldn’t have been – and in other cases some folks stick around that need to go. How do you make those choices? Well, there are a lot of factors to consider, but if your firm has an IT focus, meaning that you sell software or develop software that is core to your organization’s business strategy, then certain positions can actually save or make you money – even when they are very expensive.

There are a few caveats here -

The first is that I’m assuming that every resource here is the best they can possibly be. By “best”, I mean that they know their craft inside and out, are recognized as an expert, and that they are continually improving themselves. Also, I realize there are lots of definitions for these roles, but I’m taking the minimalist approach. The role is what the title says.

The second caveat is that you actually have a team with these positions. You might have a smaller or different team that doesn’t include the position. And keep in mind, I’m talking development teams here. No mention of operations, like sysadmins or platform administrators, networking professionals, nothing like that.

The third caveat is that this is *my* list. You’ll probably have a different one, and good on you if you do. I’m not saying that you should make more or less money, that you’re not important, or that you’re not a good person. I’m sure you are. This list is strictly confined to making and saving money for an organization or company. If you disagree with my list, post your comments – with one requirement: include your logic.

On with the list. In descending order, the top five money makers for your IT development shop are:

Number Five: Developer

It stands to reason that a good developer is faster, makes less mistakes and is generally more efficient than a poor one. But the developer will also save money for the entire organization when the code they write runs faster, and with fewer errors. That means even after a software project is complete, the good developer saves you money every time their code runs.

Number Four: Tester

But code has errors. All code, from everyone. And studies show that every error that gets out with deployed code costs up to four times more to fix than having a tester catch it before it hits the field. So it follows that a tester – a really good one – has a bigger cost/profit impact than a good developer alone.

Number Three: Lead Developer

Software is complicated. Keeping a development team on track, and keeping the work “pipelined” is also a key to utilizing those developer and tester resources at as close to %100 as possible. Every dev or tester that sits around, or writes code that isn’t needed or eventually is thrown away, is a waste of resources. A good Lead Developer keeps that from happening. Oh, and don’t forget who hires those developers and testers and assembles them into a team. Done properly, the Lead Developer handles that, so they have a direct impact on the prior two positions, making them more valuable. When there’s no lead developer, a power vacuum forms and the loudest voices win. And heaven help you if you have the misfortune of an overbearing, micro-manager in one of the so-called “Agile” environments. You’ll lose good people faster than the U.S. Government spends money.

Number Two: Project Manager

Really? In some companies I’ve worked at, the Project Manager is not viewed as a more valuable resource than a lead developer, but if you think about it, they have an incredible impact on cost (and subsequently, profit). How many projects have you seen that were scoped incorrectly? If the organization had a good PM, they would have had a better estimate or work and schedule – and perhaps have made a choice to implement or not implement a feature. Both of those choices have vast cost implications. Make something you shouldn’t, and the whole thing is wasted, don’t make something you should, and you lose opportunity costs.

Add to the scope question the ability to properly lay out time and resources, and the PM can make or break the budget on not just one but multiple projects.

Number One: Architect

In many firms, the title “Architect” is sorely mis-used, handed out with little definition. What an architect should do is understand the business and mission of the organization, and thoroughly understand multiple technologies. I’ve seen lead devs play this role, and in my opinion that’s hard to pull off successfully. In some companies, the IT Director, CTO or even CIO act as the Architect, and in my mind those roles have too many HR and Budget duties to be effective as an Architect.

An Architect is a single role that is allowed to focus on the marriage between business and technology strategies. Even if they are a “Data Architect” or a “Software Architect”, they need to know security, hardware, networking, the Cloud, whatever. That’s their job – to “know”. All areas inter-mingle at this level.

So why does this knowledge make them the most important resource in an IT-centric company? Because innovation, market applications, and general business strategy are where the architect can have the most impact. Decisions for technology strategies affect every single person not only at the company, but shareholders and the public as well. If – and these are a big deal – they are valued, competent, can communicate that strategy and are listened to by leadership.