Depends on the data of course.

I can't see myself having TB databases in the cloud or pushing 50GB once we recalculate a product back to inception, how long would it take to push that out over the wire? Then you also need to make sure it is encrypted so that nobody can see it which slows the transfer down.

You can't do BULK INSERT/BCP

For most companies it is probably fine but for others it is not practical

For example I need all the production data on staging so I can test some stuff..... hold on let me download it from SkyNet/Azure/EC2 over the weekend, you will have it by Monday morning hopefully

Also another question will be

do we buy several machines or do we use virtual SQL Servers on a big box

from there people might go to the sky (a cloud is just a piece of the sky, if one cloud fails you would fail over to another cloud)

It would have been more interesting if you cited examples of why you made this prediction.  Or was it simply pulled out of thin air?  I see no evidence that there will be any kind of major shift towards outsourced data storage in as little as five years, especially in slow moving industries such as finance, banking, and insurance. Of course, it's difficult to see evidence when none is given!

Not to sound like a luddite, but the cloud is never going to host the majority of data, databases, or users. At least not in the 20 years I have left in my career. If you doubt that, take a look at the predictions pundits were making 20 years ago. In the 80's, networks and PC's were going to revolutionize business in the next 5 years. In 1995, I couldn't convince a client to replace his 10 mbit hub for a *switch* for his 10 PCs because the switch would cost $10k, even though it would let his sales people get results from their contact management app about 25% faster.

It has taken 20 years to move from dial up to high-speed internet, and cable/dsl coverage in the U.S. is still much, much less than 100%. It took cell phones more than 20 years to become the omnipresent device they are now. It's been roughly 20 years since we started seeing electric cars, and it took about 20 years for airplanes to compete with trains for long distance travel. A large number of people were still traveling across the Atlantic in boats as late as the early 1950's. Look at every major technological cycle in the last 150 years, and you'll see 20 years is much closer to the average time between introduction and adoption by 25-50% of the target population. I would also like to point out that we still use trains, boats, wired telephones, gas-powered cars, paper-based mail, and even dial-up internet. I'm not sure how you can think strategically about new technology when you ignore the history of all the technologies that have been developed since the steam engine.

I have seen the cloud work for small inception ideas such as when Brent Ozar tried to go in the t-shirt selling business. I've also seen some use of cloud resources with products like Twitter using Amazon Web Services, but I haven't heard of any large scale deployments to the cloud of sensitive data as of yet. Anyone aware of any?

I vote also against the "prediction".

Would you as a bank put your highly sensitive data in "the cloud" ?

Would you as a pharmaceutical company put your HIPAA regulated data in "the cloud ?

Would you as a research firm would put you trading secrets in "the cloud" ?

....

The examples are *MANY*.

Keep in mind there are two types of “clouds” right now. Utility computing which virtualizes the hardware like Amazon EC2, GoGrid and the company I work for, Terremark. There are also cloud databases like SDS, Google big table and Amazon SimpleDB. The cloud databases are the ones we have to worry about. The are still beta and unproven. Utility computing is heavily used from small to large companies and even http://usa.gov. Utility computing will require admin and SQL devs.

It is hard to imagine that the cloud databases will fail with all the resources that MS, Amazon and Google are throwing at it. However, I think 5 years might be a little soon.

I am by nature incredibly conservative and I suppose most of us in the large Enterprise space tend to be.  Currently we have one major client looking seriously at the cloud for part of their production set up and I have been resistant to that.  However there is an immense impetus to move to a cloud model and all the big players are either there or on the way.  In addition I believe the main push toward a cloud model will come from business managers and at our technology level we may have no choice but to move in that direction. As Paul says above "plan your career wisely".

Maybe. But I doubt those companies still running SQL 6.5 will be moving towards a cloud anytime soon.

This is a marketing video which is a long way from reality but MS clearly wants to shift to services even to the point a utility company. The ma' bell of 2020.

http://video.msn.com/video.aspx/?mkt=en-us&vid=b4d189d3-19bd-42b3-85d7-6ca46d97fe40

Look at the bright side Paul, if the prediction holds true you can start writing fiction because there won't be any Cloud DB Bibles needed since part of the sell is their ease of use, the lessened need for DBAs and SQL Developers, etc.

The technical books can be less technical and smaller  ;-)

Hi Paul,

What's the lawyers' perspective on it? How does SOX, HIPAA, and stuff like that tolerate Clouds?

I disagree; I do think that the cloud will become a viable option for data mining and for storage of small unregulated stockpiles of data.  As others have said, in areas where regulation and security are paramount, internal storage isn't going away anytime soon.  Besides, as hardware gets cheaper and more "green", I think the trend toward cloud computing may actually reverse.  Why pay storage fees when you can own the hardware you store your data on?

Hot off the presses and relevant to this discussion.

http://www.worldprivacyforum.org/cloudprivacy.html

But why Iceland and not Alaska?

Well, there's different levels of security, and a cloud computing environment would need to account for all of them; the obvious one to question would be physical security.  How do I know that my Islandic vendor meets my physical security requirements?  Sure, they can have a contract, but when I get audited, I better be able to prove that the physical requirements are met.

Having been in that type of situation, usually as long as your service vendor can show meeting the same sort of applicable audit standard by an accredited audit firm will generally be accepted by the firm doing your audit. For instance, we had an app that used to be co-located with a different vendor. We were required to maintain SAS70 on that app. Our auditors were satisfied with SAS80 attestation from the auditors used by that facility.

You spell those kinds of things in the support contract. The question is, however, what do you do once you've got your stuff out there and they fail to meet the terms? What's your recourse?

A negative review of the azure beta. Happy for my job but innovation is always good or we would be living in caves.

http://www.ytechie.com/2009/02/azure-performance-ioc-and-instances.html

For me, the biggest issue with Cloud computing is legal, not technical.  Given the massive lawsuit abuse we deal with in the US, can you imagine any company placing their entire data set where any judge in any podunk jurisdiction anywhere in the country can order it opened and shared with anyone.  Not happening.  Since , by definition, the "cloud" is everywhere, someone will assert universal jurisdiction and go after an unpopular company.  

Don't laugh, an Alabama prosecutor once sued a satellite company since the "pornographic" images in their signals "impinged" (landed) on their territory.  Until this abuse is curbed, no responsible owners will allow their core proprietary data to be subjected to that level of disclosure.

The other day Paul Neilson wrote a simple post making predictions about SQL Server in the Cloud . That

Paul,

"

A year ago, I didn’t like the idea of cloud databases (not for any reason given here.) But, I’ve seen the future and I like it.

"

Where do you fit in the future. A .NET dev? The way these cloud databases are shaping up, it is like ORM code exposed as a non relational database that Microsoft\Amazon\Google manages.

"

A year ago, I didn’t like the idea of cloud databases (not for any reason given here.) But, I’ve seen the future and I like it.

"

Future? Are you really seeing it?

I could go the other way: Have you ever heard about IT Outsourcing eliminating *most* IT employment in the US? What has happened?

In spite of the "cheapness", etc we are still here, aren't we.

"A virtual db in the cloud, with legal assurances of privacy, uptime, disaster recovery, instant scalability, that requires fewer admin employees, at a fraction of the cost as an expense not a capital purchase is an option the VP will buy to make his budget"

As others have commented, herein lies the rub with a "cloud db" - legal assurances (not necessarily strictly legal assurances, just trust) of privacy, security, availability and recoverability are also becoming a harder and harder sell for just a hosting provider. How do you assure a VP of privacy of data if you cannot say "your data is here..."? To me this sounds like a bank saying "We guarantee the safety and privacy of your deposit box, but it won't necessarily be at this bank. Well, it might be, or it could be over there, or at our bank in Iceland, etc.".

Maybe VPs will fall for that line - I don't think so. I think there is an inevitability of consolidation of resources, but data must be "owned" by someone. When you own something, you tend to like to know where it is...like your car.

Time will tell, but my prediction is the concept of a cloud for data services will clear up into a nice blue sky.

Prices were released for AppEngine today. I would expect Amazon and MS to at least stay near this.

http://googleappengine.blogspot.com/2009/02/new-grow-your-app-beyond-free-quotas.html

Paul,

I think your missing the point I'm trying to make - and I yours. *If* the cloud db idea allows the provider to be able to identify which server in which data center pod holds a client's data, I retract my objection. I don't think it can, nor is intending too.

I disagree with your translation of my example of commodities and serializable items -but that takes us off topic and I'll argue that another day.. :)

Here's where we're disconnecting Paul. I agree with that - a client doesn't care where it would be hosted. Absolutely. However, they would care about the business/legal issues involved, and that would require that the hosting provider would be able to - at any point - be able to identify "where" the data is...

Here's an example - Say I start a business and uses Amazon's service. I read something about a security breach at Amazon that, in my crazy mind, might compromise my information, and my business (as small as it might be). When I call my contact with the "cloud" hosting provider, they better be able to tell me that my information was not involved with the systems compromised. If they can't tell me "where" my data is in this case (and it can be somewhat abstract, I'm talking about having them knowing the exact physical storage device(s) involved), I'm pretty nervous. To me, it seems seperating the "where" from the legal/business/technically issues with outsourcing your data is begging the questions of privacy, security, reliability, etc.

Now, if the cloud db movement does make it possible to determine the "where", then it is really just a much more sophisticated hosting system. If that's the case, then I urge ditching the word "cloud".

My problem with "cloud" is that words carry much more weight than most realize. The Internet is usually represented on any network diagram as a cloud. The thing with the internet is that I can not guarantee my packets will take any specific path from point A to point B. Right now, my connection (via tracert) takes me out to chicago, then the east coast to get to Microsoft. Last night - it went to Chicago, west coast, east cost, then westcoast to get to Microsoft.

So...we may be arguing about nothing - I may be fully misunderstanding the intent, but it is definitely due to the baggage associated with the word "cloud" in computer systems. I'll stop talking now...

Hey Paul,

I think there's a lot to this.  I'll go further and say that the traditional relational database vendors will implement some kind of cloud-compliant storage in order to be in the market.  There is certainly a need for this.  Oracle RAC doesn't quite cut it.  MySql scale-out replication is not even close.  

But I will also say that even though lawyers are satisfied by contracts, a lot of companies will not want to put any data which represents a strategic advantage in anyone else's hands.  There will always be some representation of locally-controlled storage because of this.  The advent of SSD technologies (eventually making storage footprints cheaper and faster) will also give an advantage to locally-controlled storage.

But it would be wrong to say that this means I think that cloud database systems will lose out because of locally-controlled storage.  What I really think may happen is that our data center may host a database cloud (or a set of clouds) that *may* be based on new cloud-compliant versions of the same software we are currently using (<plug value="mostly SQL Server in our case, at GoDaddy.com" />).  In other words, for reasons of scalability and uptime, locally-controlled clouds will replace SANs for companies large enough to support them.  Cloud service providers will be able to take some of this market, and at first having your own cloud(s) will be as unusual as it used to be for companies to have their own SAN(s).

Another aspect of this is the sheer scarcity of bandwidth.  It would absolutely kill us, either with respect to performance if we use commodity bandwidth technologies, or with respect to finances if we used pricey-fast technologies, to move our databases to a site that is topologically remote from our app servers, no matter what technology is used.  Even though bandwidth will increase, it will always be cheaper to have the data "nearby."  This will force somebody to figure out that a cloud in our own data center may give the best of both worlds.

Thoughts?  Great post.

Cheers,

Chris

PS.  I did only skim the other posts, so sorry if everything I've said has already been covered.  

As a master dba once said of the cloud:

"Life creates it, makes it grow. Its energy surrounds us and binds us. Luminous beings are we, not this crude matter. You must feel the (cloud) around you; here, between you, me, the tree, the rock, everywhere, yes. Even between the land and the ship.”

A couple of reasons why I believe Paul is spot on with his prediction(s):

1. The ease of use.

Manager asks "What do we need to put this thing into production?"

Employee answers "I need either a server or we can put it into the cloud".

2. Cost

Manager asks "How much is a server vs how much is the cloud?"

Employee answers "$6,000 for a server, free if it's in the cloud".

3. Physical vs. virtual

How many VMs can be manufactured per hour vs how many physical servers can be manufactured per hour?  Remember we are building upon existing technology.  The reason why fax machines became ubiquitous so quickly was because everyone had phones installed already.  Now that everyone has high speed connectivity, changes such as the this one will happen very quickly.

4. Predictions about the future are always a little scary.  Let's predict using the benefit of hindsight.  Two years ago what was your email client?  How connected were you to the Internet in 2007?  How connected are you today?

PingBack from http://www.brentozar.com/archive/2009/02/sql-server-and-cloud-links-for-the-week-14/

Paul,

I bet this has something to do with your predictions.

http://www.theregister.co.uk/2009/02/23/sql_server_azure/

JM

Philip, I hear what you're saying, but it's not free in the cloud. They will charge for utilization, etc. And you can build the same model locally using virtualization. And while ordering and installing physical hardware may take time, with proper practices you could roll out servers (physical or virtual) in about 30-45 minutes from statement of need to ready for use if the infrastructure is kept up with and planned properly.

Yes, I've been waiting for Azure's SQL implementation to see the light of day for months.  Before that, I'd been waiting for over a decade.  :)

I think that SDS will just be getting the most heinous kinks worked out in two years time.  And that in five year times, enterprises whose business relies on the database may even consider using SDS for small and insignificant databases.

-Kev

Normal 0 false false false MicrosoftInternetExplorer4 /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table

After scanning what turned out to be a very healthy discussion, I have a few comments.

1) Legal contracts - A contract is only as good as its enforceability. A VP who steps out onto the bleeding edge of technology to save a few bucks is going to find out why they call it the "bleeding edge" when the legal safeguards fail to protect the company's data. Additionally, the CEO and CFO are legally responsible for complying with SOX regulations, and I doubt that any CEO is going to risk getting fitted for an orange jumpsuit and filling out a dance card in federal prison just because someone in Iceland promises to protect his corporate accounting database from bad guys. Once his/her Chief Information Officer explains the risks, the VP will get overruled. Economically, the marginal cost of additional in-house capacity is small compared to the risk of losing their income and prestige. Nothing makes a CEO, CFO, or CIO sleep better than having their data sitting under lock and key guarded by someone they can get their hands on if something goes wrong.

High speed connectivity - When I have 1 gigabit/sec internet connectivity, I'll be willing to discuss the cloud as a technologically viable option for large databases. At 40 mbit/sec, it takes roughly 200 seconds to move 1 GB. That works out to about 55.5 hours to move a terabyte. When that time drops to a few hours, we can have meaningful discussions about the cloud's usefulness to business.

Local vs. third party - this is the only worthwhile discussion on this topic. If the technology makes cloud computing better than current technology for robustness, disaster recovery, availability, response time, and other performance metrics that make users happy, my prediction is that Paul's prediction might come true for *locally* managed clouds. If clouds can run effectively on VM farms, even better. The big question then becomes how and at what cost will Microsoft license SQL Server Cloud Edition?

Availability, Disaster Recovery, Performance - I have 7 production servers/clusters that support about 6000 users each day. The cluster that supports about 1500 users has 99.99985% up time for 1.25 years, assuming only 8 hour days instead of 24. My least available server has 99.99% up time. The technology to do that gets cheaper every day, and the ability to do that involves common sense and a conservative approach to administration more than any special set of skills. How is cloud computing with its new paradigms superior enough to well understood technology and methodology to warrant my considering fixing something that obviously isn't broken?

Data Security - This is the only concern that I see is mitigated by existing, reliable technology - encryption and digital signatures. Any company that puts data in the cloud without encrypting it first deserves whatever nasty thing happens to it. Maybe someone will break the current best of breed encryption algorithms, but at least encrypting data heads off the charge of gross negligence. And it covers physical security and protection from lawsuits too.

I'm in agreement with Kevin Kline: It'll take a couple of years at least to work out just the worst of the kinks. It'll take even longer to educate the programming community on how to use it. It will take even longer still to convince the decision makers that the technology is viable enough to use for a major product. It will take even longer than that for the product to get to market.

I'll stick to my prediction that it'll be 20 years until we see 50% adoption, but I can see that locally managed clouds might be common in the Fortune 100 before that.

Paul Nielsen posted about a rather provocative statement a few days back regarding SQL in the cloud.&#160;

Paul, security is based on trust. I don't really care how much someone tells me their solution is secure because their jobs are not on the line if there is a security breach. When laws like SOX and HIPAA contain criminal and civil penalties for those who do not protect the information properly, it is in my best interest, and the company management's best interest, to control every facet of the data's care and safety. I'd rather show that we had followed best practices used by other well-respected companies than try to explain how a third party's security measures were sufficient enough to meet the law's standards. It is a matter of what *does* work, not what *should* work. Trying to explain that a cloud operator's security policies meet or exceed our own is a much more difficult discussion with management, lawyers, and even the press if it comes to it, than simply telling them we're using physical, network, and database security with more than a decade of proven success managed by our employees. And I don't want even to begin the discussion of the legal ramifications of having our data hosted in a foreign country.

What I think is the basic problem here is that you place a higher value on cost than anyone I have ever worked for. Every boss I've had is wary of the words "free" and "cheap", usually because they've been burned by them. I work in an environment where downtime can jeopardize people's lives and cost the organization huge amounts of money. For example, I have one department that would lose more money in one 8 hour day than it would cost to completely duplicate the clustered server environment where their app's database resides. Why should they consider using something that isn't under our control when the risks of losing it for even an hour are so great? The VP who chooses to risk patients and revenues to save money is one with a very short career. Cost just isn't the single greatest consideration in something like this.

As for Steve Ballmer, I just don't see his single-handedly forcing a paridigm shift in less than 20 years. Ford, Rockefeller, AT&T, Edison, Scott Ellis, Bill Gates, and a whole host of other visionaries couldn't do it, so why should Ballmer be any different. He may convince the companies that don't have the resources to do their own data centers, but the Fortune 500 moves cautiously and slowly.

A man convinced against his will is a man of the same opinion still. I'm patient enough to wait for my prediction to be proven by time. Maybe I'll be wrong. I have a very successful track record of predicting trends, but it's always possible I could be wrong on this one.

Btw, I don't think I've complimented you on coming up with a topic that generated such diverse discussion. That's why I read blogs. Keep on making it interesting. :-)

And the cloud went poof....

http://www.theregister.co.uk/2009/03/16/azure_cloud_crash/

also interesting conversation here  http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/6c1cd8a2-8d9d-43e9-a1d8-928e0ca4de78

I understand that it is still early and they will work out the kinks but if they are already down with minimal usage what will happen when they have tons of users

And why is the whole cloud down and it takes 5 hours to resolve this?

Forgive me for being dumb but what does BBS stand for here?

I'll be back here on the 24th Feb 2011/2014 :)

Bulletin Board System

Let's beat this horse to death once again shall we, Paul Nielsen wrote about SQL in the cloud already

PingBack from http://209.251.180.21/site/archive/2009/02/capt-varcharmax-and-the-pagelatch-posse-vol-21/

Agree... transformation to Cloud based database is inevitable.

...when a major news magazine that speaks to a large number of CEOs and gray-haired types whose closest contact to email is having their admin send a message for them, it's important to the world at large. Really important.

PingBack from http://kevinekline.com/2009/11/15/clash-of-the-clouds/

For those that doubt the intent or current state of play, I can attest that large financial services firms are moving mission critical applications and databases to cloud services. There is a logical progression to establish comfort in doing so, but it's already well underway. I am currently engaged in a project for a large fin svcs firm that is seeking to move 50-75% of the enterprise application portfolio and associated databases to external cloud service providers within two years.