THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

Kevin Kline

In-Memory Databases (IMDBs)

Moore's Law tells us that CPU's get a LOT faster over time.  Unfortunately for the database professional, all of the secondary elements of our databases DO NOT get a lot faster over time.  Overall, the main methods of storing data since the 1960's, magnetic tape and hard disks, have improved only in the single percentiles year over year.  Even those of us who were never good at math can tell that the CPU is outpacing the other system components.

A portable computer, circa 1985
An Osborne Executive portable computer, from 1982, and an iPhone, released 2007. The Executive weighs 100 times as much, has nearly 500 times the volume, cost 10 times as much, and has a 100th the processing power of the iPhone.

Two recent developments are helping to change that equation.  First, solid state drives (SSDs) are having a dramatic impact many IT scenarios. My friends, Brent Ozar and Paul Randall, have each written about SSDs here and here, respectively.

Second, database vendors are supporting relational database systems that run entirely in system RAM.  If you'd like to learn more about in-memory databases (IMDB), read more in my new article in Data Management Magazine.  As we look to the future, I expect to see a lot more of both technologies in the data center.

And give me your feedback here! Thanks,

-Kev

Published Monday, June 28, 2010 9:10 AM by KKline

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

Bart Czernicki said:

IMDbs are nothing new...especially for read-only or business intelligence scenarios.

June 28, 2010 10:46 PM
 

GrumpyOldDBA said:

I think we all try to keep as much of our database in memory and SSDs are really just such an extension, servers now support much more memory, a few years a go asking for 512GB of ram on a basic server was almost impossible, but not now. Large databases struggle of course, a 100TB database would be tricky to put in memory, I'm sure I've heard that microsofty is working on improving in memory processing for the next version of SQL Server.

June 29, 2010 6:02 AM
 

Adam Machanic said:

Bart: Yes, they're nothing new, but hardly mainstream in enterprise data centers.

June 29, 2010 5:42 PM
 

Jimmy May @aspiringgeek said:

I recently had an opportunity to recommend that a pharmaceutical enterprise put their sufficiently small yet mission critical database in memory/on SSD (by quadrupling memory & introducing SSD to host the entire database & tempdb).  Alas, they could not be persuaded to adopt such a "radical" solution--& they suffer still from egregious I/O challenges--very high latencies.  The solution would have been a game changer for them.  And it won't be long before such solutions are routine.  One can lead an equine beastie to water, but can't force it to imbibe...

June 30, 2010 12:50 PM
 

Bart Czernicki said:

Adam,

With products like Microsoft Velocity (Distributed Cache server) it is now coming to the Microsoft "community".

MySpace,Twitter, Facebook architecture have been using large scale distributed in-memory databases for 5+ years now. Is that enterprise enough? :)

July 2, 2010 6:05 PM
 

KKline said:

Thanks for all of these great comments, guys!

Bart, I've never heard of Velocity before.  Thanks for the tip.  I'm looking in to it now.  However, to say that IMDBs are nothing new might be reaching.  There have long been "memory-resident databases", but that's basically just a regular database loaded in memory.  IMDBs are cutting some new ground in that they are optimized specifically for memory, not a disk-based DB that's been upsized into memory.

Great anecdote about horse + water <> drink, Jimmy! As you say, it'll soon be routine but those days are still in the making.

-Kev

July 5, 2010 10:40 PM
 

Russ Monson said:

For several years I served as DBA for three large Government

databases and was frustrated by the excessive time consumed by

Business Management analyses (hours). In Memory processing looks

attractive but I foresee problems.

History shows that there will always be more data to analyze than will fit in the available RAM. Thus the core problem will always be: how does one get the right chunk of data into RAM at the right time?  The 'right' chunk would be all the data that the in-memory analysis will need. The scientists at Hilbert Technology have showed me an appliance that runs the Hilbert Engine native on the CPU (no operating system at all) in which they can, for instance, find all 'Jones' entries as a first filtering step on a disk with 5TB of data. They deliver all the data for all attributes related to 'Jones' to RAM in about 2ms. Since there is absolutely no cycle-stealing operating system, the remainder of in-memory analysis

takes only a few microseconds. Is anyone aware of anything similar

in the market?

July 12, 2010 12:03 PM
 

KKline said:

Great feedback, Russ.  I'd like to see what's going on with Hilbert Technologies.  I've not heard of anything similar to that.

August 25, 2010 11:40 AM

Leave a Comment

(required) 
(required) 
Submit

About KKline

Kevin Kline is a well-known database industry expert, author, and speaker. Kevin is a long-time Microsoft MVP and was one of the founders of PASS, www.sqlpass.org.

This Blog

Syndication

Powered by Community Server (Commercial Edition), by Telligent Systems
  Privacy Statement