THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

Andy Leonard

Andy Leonard is CSO of Linchpin People and SQLPeople, an SSIS Trainer, Consultant, and developer; a Business Intelligence Markup Language (Biml) developer; SQL Server database and data warehouse developer, community mentor, engineer, and farmer. He is a co-author of SQL Server 2012 Integration Services Design Patterns. His background includes web application architecture and development, VB, and ASP. Andy loves the SQL Server Community!
Note: Comments are moderated. Spam shall not pass! </GandalfVoice>

Everything Scales

The tune to a Bush song is running through my head as I type this... the band, not the president - although imagining the President singing the song is an interesting brain-stretch.

It's a fact of IT life that everything scales. Some successfully, even. Problems start when things do not scale successfully (or well). It happens in business. It happens with software systems.

When it happens with businesses, you hear things like "They grew too fast." When it happens with software systems, you browse to a website and receive an HTTP 500 or 404 error.

Can this be avoided (in business or software)? I think that's an excellent question - one well worth examining.


The answer, I believe, lies with how predictable the scalability is.

Consider a database application: If you know which tables are going to grow, how, and how much, you can plan for said growth. How would you plan? You could partition the tables using one or a combination of partitioning techniques. You could appropriate filegroups, snapshots, and a host of other functionality. If you only you knew where to apply these techniques.

That's the key.


Achieving scalability starts with capturing metrics. If you know how your database is growing from the beginning - if you can chart the growth of individual tables, access patterns, and internal performance data - you can predict growth and manage scalability.

So the key is measurement.

Measurement is an engineering discipline in its own right. The field of applied measurement is called Instrumentation. Applying measurement to a process is referred to as "instrumenting the process."

How do you instrument a database process? Iteration 1 would include creating an internal table to house and maintain process metadata:

CREATE TABLE dbo.ProcessData
(ProcessDataID int IDENTITY(1,1) NOT NULL,
ProcessDataDateTime datetime NULL CONSTRAINT DF_ProcessDataDateTime DEFAULT (getdate()),
ProcessDataIndicatorName varchar(50) NULL,
ProcessDataIndicatorValue varchar(50) NULL,
ProcessDataIndicatorStatus char(1) NULL CONSTRAINT DF_ProcessDataIndicatorStatus DEFAULT ('C'),
CONSTRAINT PK_ProcessData PRIMARY KEY CLUSTERED (ProcessDataID)

If your instrumented process is stored-procedure-based, you could add INSERT statements to your existing stored procedures. Consider instrumenting a parent stored procedure that calls child stored procedures. The instrumented proc could look like the following (instrumentation emphasized):

CREATE PROCEDURE dbo.SomeProcess
AS

begin

INSERT INTO dbo.ProcessData
(ProcessDataIndicatorName,
ProcessDataIndicatorValue)
VALUES('ChildProc1','Starting');


EXEC dbo.ChildProc1

INSERT INTO dbo.ProcessData
(ProcessDataIndicatorName,
ProcessDataIndicatorValue)
VALUES('ChildProc1','Ending');

INSERT INTO dbo.ProcessData
(ProcessDataIndicatorName,
ProcessDataIndicatorValue)
('ChildProc2','Starting');


EXEC dbo.ChildProc2

INSERT INTO dbo.ProcessData
(ProcessDataIndicatorName,
ProcessDataIndicatorValue)
VALUES('ChildProc2','Ending');

INSERT INTO dbo.ProcessData
(ProcessDataIndicatorName,
ProcessDataIndicatorValue)
('ChildProc3','Starting');


EXEC dbo.ChildProc3

INSERT INTO dbo.ProcessData
(ProcessDataIndicatorName,
ProcessDataIndicatorValue)
VALUES('ChildProc3','Ending');


end

Before moving forward, removing code duplication would be a worthwhile effort. In application development, this is one of many processes generally referred to as Refactoring.

The INSERT statements are a prime candidate for refactoring and we can address this with a stored procedure:

CREATE PROCEDURE dbo.AddProcessData
@ProcessDataIndicatorName varchar(50),
@ProcessDataIndicatorValue varchar(50)
AS

begin

INSERT INTO dbo.ProcessData
(ProcessDataIndicatorName, ProcessDataIndicatorValue)
VALUES(@ProcessDataIndicatorName, @ProcessDataIndicatorValue);

end

Now the parent stored procedure instrumentation above can be modified to look like this:

CREATE PROCEDURE dbo.SomeProcess
AS

begin

EXEC dbo.AddProcessData @ProcessDataIndicatorName='ChildProc1', @ProcessDataIndicatorValue='Starting';

EXEC dbo.ChildProc1

EXEC dbo.AddProcessData @ProcessDataIndicatorName='ChildProc1', @ProcessDataIndicatorValue='Ending';

EXEC dbo.AddProcessData @ProcessDataIndicatorName='ChildProc2', @ProcessDataIndicatorValue='Starting';


EXEC dbo.ChildProc2

EXEC dbo.AddProcessData @ProcessDataIndicatorName='ChildProc2', @ProcessDataIndicatorValue='Ending';

EXEC dbo.AddProcessData @ProcessDataIndicatorName='ChildProc3', @ProcessDataIndicatorValue='Starting';


EXEC dbo.ChildProc3

EXEC dbo.AddProcessData @ProcessDataIndicatorName='ChildProc3', @ProcessDataIndicatorValue='Ending';

end

Much better.

Measuring the current process provides a baseline - the first step in a continuous improvement process that will provides dynamic design changes, performance monitoring, and - eventually - a dynamically-scalable system. It also supplies the current performance status against which we can benchmark future improvements and modification.

:{> Andy

Published Sunday, August 19, 2007 9:51 AM by andyleonard

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

No Comments

Leave a Comment

(required) 
(required) 
Submit

This Blog

Syndication

My Company


Other Blog

Check out my personal blog...
http://andyleonard.me

Contact Me

Twitter: @AndyLeonard
Email: andy.leonard@gmail.com

Powered by Community Server (Commercial Edition), by Telligent Systems
  Privacy Statement