Buck Woody : Design, Developer, SQL Server

SQL Azure Use Case: Shared Storage Application

BuckWoody — Tue, 26 Apr 2011 13:33:50 GMT

This is one in a series of posts on when and where to use a distributed architecture design in your organization's computing needs. You can find the main post here: http://blogs.msdn.com/b/buckwoody/archive/2011/01/18/windows-azure-and-sql-azure-use-cases.aspx

Description:

On-premise data will be a part of computing for quite some time – perhaps permanently. Bandwidth requirements, security, or even financial considerations for large data sets often dictate that relational (on non-relational) systems will be maintained locally in many organizations, especially in enterprise computing.

But distributed data systems are useful in many situations. Organizations may wish to store a portion of data off-site, either for sharing the data with other applications (including web-based applications) or as a supplement to a High-Availability and Disaster Recovery (HADR) strategy.

Implementation:

SQL Azure can be used to add an additional option to an HADR strategy by copying off portions (or all) of an on-premise database system.

In this arrangement, on-premise systems remain as they are. Data is replicated using many technologies, such as SQL Server Integration Services (SSIS), scripts, or Microsoft’s Sync Framework to a SQL Azure database. This data can be kept “cold”, meaning that a manual process is required to bring the data back, or as a “warm” standby using connection string management in the application.

Recently we architected a solution where a company kept a rolling two-week window of data replicated to SQL Azure using the Sync Framework. The application, a compiled EXE running on user’s systems, had a “switch connections” button, that allowed the users to take a laptop to another location, select that option, and continue working from anywhere they had Internet connectivity. This required forethought and planning, and did not replace their primary HADR systems, but it did allow them to continue operations in the case of a severe outage at multiple sites. Since they are an emergency services provider, this gave them the highest redundancy.

Another option is to amalgamate data from disparate sources.

In this arrangement, two or more data services (one of which is SQL Azure) are accessed by a single program. The program queries each system independently, and using LINQ a single query can work across all of the data, assuming there is some sort of natural or artificial “key” that can join the data sets together. The user programs simply view this single data set as a single data source, unaware of the underlying data sets. This allows great flexibility and agility in the downstream program. The upstream data sources can change as long as the elements are kept consistent.

There are performance and security implications to amalgamated data systems, but if architected carefully they provide multiple benefits. A few of of these are that other systems can access the individual data sources, reporting is simplified and standardized, and multiple copies of data are eliminated.

Resources:

You can read more about the Sync Framework and SQL Azure here: http://social.technet.microsoft.com/wiki/contents/articles/sync-framework-sql-server-to-sql-azure-synchronization.aspx

If you are new to LINQ, you can find more resources on it here: http://msdn.microsoft.com/en-us/library/bb308959.aspx

More Tables or More Databases?

BuckWoody — Wed, 19 May 2010 11:37:22 GMT

I got an e-mail from someone that has an interesting situation. He has 15,000 customers, and he asks if he should have a database for their data per customer. Without a LOT more data it’s impossible to say, of course, but there are some general concepts to keep in mind.

Whenever you’re segmenting data, it’s all about boundary choices. You have not only boundaries around how big the data will get, but things like how many objects (tables, stored procedures and so on) that will be involved, if there are any cross-sections of data (do they share location or product information) and – very important – what are the security requirements?

From the answer to these types of questions, you now have the choice of making multiple tables in a single database, or using multiple databases. A database carries some overhead – it needs a certain amount of memory for locking and so on. But it has a very clean boundary – everything from objects to security can be kept apart. Having multiple users in the same database is possible as well, using things like a Schema. But keeping 15,000 schemas can be challenging as well.

My recommendation in complex situations like this is similar to a post on decisions that I did earlier – I lay out the choices on a spreadsheet in rows, and then my requirements at the top in the columns. I give each choice a number based on how well it meets each requirement. At the end, the highest number wins.

And many times it’s a mix – perhaps this person could segment customers into larger regions or districts or products, in a database. Within that database might be multiple schemas for the customers. Of course, he needs to query across all customers, that becomes another requirement.

Use Those Schemas, People!

BuckWoody — Tue, 18 May 2010 12:39:35 GMT

Database Schemas are just containers – they aren’t users or anything else – think of a sub-directory on the hard drive. In early versions of SQL Server we “hid” schemas, placing all objects under “dbo”, which gave the erroneous perception that Schemas are users.

In SQL Server 2005, we “un-hid” or re-introduced schemas within the database. Users can have a default schema (a place where their new objects go), you can add new schemas and transfer objects between them, and they have many other benefits.

But I still see a lot of applications, developed by shops I know as well as vendors, that don’t make use of a Schema. Everything is piled under dbo. I completely understand this – since permissions can be granted to a schema, they feel a lot like a user, so it’s just easier not to worry about both users and schemas when you create a database. But if you’ll use them properly you can make your application more understandable and portable.

You should at least take a few minutes and read more about them – you owe it to your users: http://msdn.microsoft.com/en-us/library/ms190387.aspx

Using linked servers, OPENROWSET and OPENQUERY

BuckWoody — Tue, 16 Mar 2010 12:41:33 GMT

SQL Server has a few mechanisms to reach out to another server (even another server type) and query data from within a Transact-SQL statement. Among them are a set of stored credentials and information (called a Linked Server), a statement that uses a linked server called called OPENQUERY, another called OPENROWSET, and one called OPENDATASOURCE. This post isn’t about those particular functions or statements – hit the links for more if you’re new to those topics.

I’m actually more concerned about where I see these used than the particular method. In many cases, a Linked server isn’t another Relational Database Management System (RDMBS) like Oracle or DB2 (which is possible with a linked server), but another SQL Server. My concern is that linked servers are the new Data Transformation Services (DTS) from SQL Server 2000 – something that was designed for one purpose but which is being morphed into something much more.

In the case of DTS, most of us turned that feature into a full-fledged job system. What was designed as a simple data import and export system has been pressed into service doing logic, routing and timing. And of course we all know how painful it was to move off of a complex DTS system onto SQL Server Integration Services.

In the case of linked servers, what should be used as a method of running a simple query or two on another server where you have occasional connection or need a quick import of a small data set is morphing into a full federation strategy. In some cases I’ve seen a complex web of linked servers, and when credentials, names or anything else changes there are huge problems.

Now don’t get me wrong – linked servers and other forms of distributing queries is a fantastic set of tools that we have to move data around. I’m just saying that when you start having lots of workarounds and when things get really complicated, you might want to step back a little and ask if there’s a better way. Are you able to tolerate some latency? Perhaps you’re able to use Service Broker. Would you like to be platform-independent on the data source? Perhaps a middle-tier might make more sense, abstracting the queries there and sending them to the proper server. Designed properly, I’ve seen these systems scale further and be more resilient than loading up on linked servers.

It’s OK to take a Shortcut Sometimes

BuckWoody — Mon, 08 Mar 2010 14:06:12 GMT

I was working this weekend with a fairly simple Excel spreadsheet, and I had to decompose one cell in it out to three columns in a SQL Server table. There are tools within SQL Server Integration Services (SSIS) that should be able to do that, but I just couldn’t find my way around them properly.

I’m not as familiar with SSIS as I would like to be – it’s just not my day-to-day tool. I was struggling with the split of the cells, since it had several different types and kinds of delimeters for the data. I knew how to do it with regular expressions, I knew how to do it with Transact-SQL, I knew how to do it with lots of programs, but I just couldn’t figure it out in SSIS. I’m sure that’s my fault – not the fault of the tool, by the way.

I spoke with the person that created the spreadsheet, and it turns out that he had combined those fields from his source to begin with! In other words, he could split them out for me. What a relief! Yes, I need to come back some time and figure out how to beat this, but right now I just needed to get the work done. He split the cells out for me in the spreadsheet (took him about 3 minutes) and then I had the three discreet cells I was looking for.

Sometimes it’s OK to take a shortcut. As long as you think it through, you can safely do things the “easy way”, when you’re careful and understand what the ramifications are. The key here is that this was a one-time import, I understood both sides of the data, and had done my research. The data itself didn’t change; just it’s cell location. And that was all OK.

So don’t make it harder on yourself than it needs to be. Consider the alternatives – and take the help where you safely can.

Code that Writes Code - A Good Idea or Not?

BuckWoody — Tue, 16 Feb 2010 14:14:00 GMT

I’m a big fan of code that writes code – most of the time. For instance, whenever you use the “templates” feature in SQL Server Management Studio (SSMS) or the Maintenance Wizard, you’re using code that writes other code. There’s even a trick of writing Transact-SQL (T-SQL) code that in turn creates other code.

But there is a class of code that writes code that I’m more cautious about. Whenever a program “automatically” generates a database schema, I begin to get nervous. No, I’m not talking about Entity Relationship Diagram (ERD) tools such as those from Quest and Embarcadero, I’m talking about things like NHibernate and other coding paradigms that “abstract” the database layer away from the developer.

I have two reasons that I’m not impressed with these programs and paradigms.

First, they do not take the entire solution into account. As data professionals we learn our platform (whether that’s XML, flat files, SQL Server, Oracle, IBM, whatever) and we study how each of the features maps to a complete solution. I might choose to use Replication, Service Broker, Clustering, FileStream, or any number of features to completely remove the need for code in that area. And by making those choices, I change the design of the database accordingly, based on the solution. The abstraction tools don’t – they just spit out the same Data Definition Language (DDL) statements they know how to create, without thinking about maintenance, speed, reliability or anything else.

When I mention this to the developer, they say “just put that in later” – and that’s the beginning of woes for the data professional. Most of the time you can’t put things of a fundamental nature in later. In some cases, it’s a complete tear-down and re-write of the entire database. Very painful, and something you never want to experience.

The second main reason that I am skeptical about tools that automatically create DDL statements to create databases is that they don’t do a good job. Once again, this is because of a holistic view – the tool doesn’t have the capability to take everything into account, including the data pattern, so it has no idea how far to normalize, where to put files and filegroups, what kind (if any) indexes are needed, or when to choose between a natural or surrogate key.

Is all this just because I’m a DBA, and anal about my databases? No. It’s a business problem, because these tools continue to separate the DBA from the Developer, and both from the Business Analyst. And it's the business that suffers when developers (or DBA’s) take shortcuts. I wonder how much time and money are wasted in business re-writing databases because of a shortcut using poor tools?

So let’s do the hard work – let’s let the business requirements dictate the solution, rather than the other way around. Even for the “little” databases, which of course never stay that way.

Now, does this mean I hate NHibernate or the Entity Framework? Not at all! Just work with your data professionals instead of without them. You’ll find that by bringing the best practices of these ORM tools together with a well-designed database, you’ll deliver a solution to the business that is fast, reliable and safe. Isn’t that what we all want?

Data Design

BuckWoody — Tue, 02 Feb 2010 14:31:00 GMT

I give series of classes and presentations on Data Design. I say “data” design instead of “database” design because we should consider more than just the database. Data might actually be stored in non-relational stores, such as Excel or XML files, and it might also be located in remote data stores like “cloud” technologies.

Here are the links I use in that presentation. Although this isn't a comprehensive list of Data Design topics, I’ll visit this topic from time to time so you may want to bookmark this page in your favorites:

Evidence and Comments:

http://tinyurl.com/ygvbadc

http://tinyurl.com/ydeh9ut

http://tinyurl.com/ycu9apq

http://tinyurl.com/ya7a9xc

http://tinyurl.com/ydlumyp

Importance:

Yes, it matters: http://tinyurl.com/yb5ucvn

Classic software development mistakes: http://tinyurl.com/y9x9vn8

Necessity of Good Design: http://tinyurl.com/yac3um3

Importance of Database Design Ranked high for BA/Dev interaction: http://tinyurl.com/yex4sjf

Structured Methods:

Test-Driven Development: http://tinyurl.com/maus78

From the BA’s Perspective: http://tinyurl.com/yea6pbe

Data-Map Examples: http://tinyurl.com/2d2nhr

Universal Patterns for Data Modelling: http://tinyurl.com/ybj7tb3

Tools:

EF Framework Design Example: http://tinyurl.com/yeuda49

Using EF and LINQ can cause divisions: http://tinyurl.com/yaljmhn

Tools and Processes for “Fitting it all in”

BuckWoody — Mon, 18 Jan 2010 14:42:05 GMT

Most data professionals I’ve met work in two modes: we plan for our day, and we react to the situations around us. I’m staring at my list of things that I need to do today right now, which is my planned work. Of course, I have no idea how much of that will really get done – it’s optimistic to be sure. On the other hand I have several systems I manage, and at any moment one of them or the people that interface with them may “change state” such that I need to give them some attention.

So how do I meld the two? Sometimes it can be quite difficult. I’m constantly working through my list in my mind, re-arranging what I’m focusing on based on what I perceive as the highest need. There are, however, some tools that I use each day to help me manage the workflow.

I use Outlook for tracking everything, since it has a task list (my primary tracking), a calendar, mail and so on. Also I can share the information, it’s on-line so I can see it anywhere, and I can even take it offline onto the plane this week when I fly out of town.

For the “ad-hoc” work, I rely on a script library, which I keep as SQL Server Management Studio projects. I keep those scripts and projects backed using Microsoft Live Mesh, which synchronizes those files (along with a few other critical files and my IE Favorites) across not only my laptop and primary systems, but even with my Virtual Machines.

Also for my SQL Server systems I use the Standard Reports I’ve blogged about here. I also use Greg Larsen’s Database Dashboard, and a series of PowerShell scripts that work across my systems, alerting me to any problems. Of course I’m using SQL Server Agent Jobs quite a bit, and I also use Alerts and some Perfmon automation for my monthly baselining.

So – is this your experience as well? Do you get driven by both planned and unplanned work? What tools and processes do you use to keep it all straight with your SQL Server Instances?

Know Your Product Specifications

BuckWoody — Wed, 13 Jan 2010 14:57:01 GMT

As the Data Professional in your organization, the rest of the org looks to you to ensure that the system can handle what the business requires. To do that, you need to know two things: what the business requires, and what SQL Server can do.

But of course there’s a bit more to it than that. Knowing the business side of the requirements – well, I teach an entire course on that. But knowing what SQL Server can do is something you can find out on your own.

SQL Server comes in versions, which are released based on date, and editions, which are based on features and capabilities. It’s that last part that I want to focus on today.

As Microsoft SQL Server matures, you’re going to see even more separation between what each edition of SQL Server can do and where it should be used. In the past, most folks have only focused on three editions – Express (the “free” one), Standard, and Enterprise. The rule of thumb was that if Standard was good enough at the moment, put it in. And it is true (and a good thing) that you can upgrade from one edition to another fairly easily.

But as time goes on, we should spend a little more time understanding what each edition does, what it’s features and capabilities are, and where and when we should put them in. As I study this information, I’ll throw in my 2 cents and you can as well based on what you see. One thing I’ve found so far is that once I have the business requirements, there’s a mix of what I can write in code and what might already be included in a different edition. It’s important to look long and hard at that choice – writing a feature on my own is certainly cheaper in the short term than moving to a “higher” edition, but in some cases it makes sense to let Microsoft handle that lifting.

These links are ones that you should bookmark and take a peek at periodically. They are the “header” links for more information on those features and capabilities:

SQL Server 2008: http://msdn.microsoft.com/en-us/library/ms143287.aspx

SQL Server 2008 R2: http://msdn.microsoft.com/en-us/library/ms143287(SQL.105).aspx

In addition, you might start learning a little more about SQL Azure. I’ll talk more about that later.

It’s Data Tier Application and Data Application Component

BuckWoody — Wed, 23 Dec 2009 15:52:35 GMT

OK – In SQL Server 2008 R2 we did “re-use” an acronym or two (DAC and DTA), but it’s important to remember there are actually two parts to this new feature. One is the Data Application Component (DAC) and the other is the Data Tier Application (DTA). The DAC is the file created for a DTA.

In SQL Server 2008R2 and Visual Studio you’ll find there is a new way to write and transfer database code. I’ll blog about it more as I finish my testing, but the process works kind of like this…

You can “birth” a Data Tier Application in two places. You can create a new project type in Visual Studio where the developer can create the database structure, put all of the policies that they want to enforce and so on there. The DBA can also right-click a database and make a Data Tier Application out of a current structure.

In both cases, something called a DAC – or Data Application Component – is created. It’s a file with the payload of the the major parts of the structure of the database and so on. that’s the “package” you use to transfer the DTA around.

From there, you right-click in the “Data Tier Application” node in SQL Server Management Studio on another Instance and “Deploy” the Data Tier Application. It will build the database for you and keep it together as a DTA. You can make changes in the “originating” system or code, and then “upgrade” the Data Tier Application.

So there you have it. It’s DTA and DAC – but I think you’ll know the difference when the time comes to use one…

The Top 20 Questions in Database Design

BuckWoody — Thu, 17 Dec 2009 14:18:00 GMT

I'm still re-reading the "Fourth Paradigm" book by Microsoft Research, and one section continues to intrigues me. There's a part where the book explains database design, and puts forth that the most important thing when you're designing large data sets is to find out the "Top Twenty Questions" the database has to answer. The quote is this:

"Most selections involving human choices follow a 'long tail,' or so-called 1/f distribution, it is clear that the relative information in the queries ranked by importance is logarithmic, so the gain realized by going from approximately 20 (24.5) to 100 (26.5) is quite modest."

I find this facinating - it just doesn't seem to make "common" sense. Surely you have to ask a lot more questions than that to "get" the shape of the data? I researched the mathematical concept he's describing (http://www.scholarpedia.org/article/1/f_noise), and I'll try some experiments here. I'll let you know what I uncover!

Here's the link for the book if you want to read it:

http://research.microsoft.com/en-us/collaboration/fourthparadigm/4th_paradigm_book_complete_lr.pdf

The Coming Database Design Wave

BuckWoody — Wed, 16 Dec 2009 13:50:00 GMT

Database design has been on my mind as of late – I’ve been teaching it in class, and I have a friend from Twitter that has a couple of questions. In fact, I think I’ll actually do a submission to PASS this year on this topic.

Don’t think it has to do with you? Well, I’m seeing a new “wave” of design options coming at the data professional. Do these ring a bell?

Traditional Logical/Physical Database Design Methodologies

Object-Role Modeling

EDMX

Entity Framework Code - .NET Classes

If any of these are new to you, get ready….

Aren’t DBA’s Just System Admins for Databases?

BuckWoody — Mon, 30 Nov 2009 16:37:32 GMT

Last week I ran into an argument I’ve had since I left the mainframe space decades ago. A developer told me “DBA’s don’t design databases.” The inference was that DBA’s (i.e., Database Administrators) only worry about hardware, security, OS, database backups, things like that. He seemed amazed that a DBA would ever do “data” work.

It may be the name. Perhaps the “admin” part confuses developers. Also, it is true that in some shops, a systems admin does double duty with Windows, SQL Server, and perhaps even mail and web admins.

But there are a LOT of DBA’s, or as the term I like to use, “Data Professionals”, that actually DO get down in the trenches and design databases, write Transact-SQL code and stored procedures, and do almost everything in the database other than write middle-tier or User Interface (UI) code. Some I know even do that.

So what if there is a miscommunication on this? Well, the ramifications can be huge. For one thing, there’s a lack of respect. That’s not called for ever, no matter what anyone’s role is. Also, one of the most impactful areas in a database is the design. When a DBA is asked to export data, tune a process, or troubleshoot an issue, it invariably involves the design. So when a DBA doesn’t get to do the design, they have to live with the results. And anything you’re responsible for when you don’t have the authority over is a recipe for stress.

Another issue is that DBA’s “inherit” all kinds of data structures form around the company. From Microsoft Access to Excel, to amateur Business Analysts creating databases in the Express Editions, they deal with bad design day after day. The newer “modeling” languages that are coming into vogue will make this problem much worse. These languages do not take scale, extensibility, security or performance into account – they just make sure that the data ends up in the right place for that particular design, which is a recipe for data disaster when the “small application” the developer writes becomes a “mission critical” system the DBA has to troubleshoot at 2:00 A.M.

So in case you’re a developer, and in case you think DBA’s “just do admin” – think again. DBA’s spend their whole day in this world, and can be a valuable asset to your development efforts. Bring them in early, bring them in often, and whatever you do – don’t design alone. Business Analysts, Developers and Data Professionals are needed to make a good, sustainable, secure, well-performing database.

Code that Writes Code

BuckWoody — Wed, 25 Nov 2009 14:01:40 GMT

I have scripts that re-create my databases for testing and development purposes. But sometimes I want to take the data from a set of tables and move it as well – I could use SSIS, or a SELECT INTO statement, but what if I want to “re-set” the data to a point in time? In other words, load it with some “base data”?

I thought this might be a good place to demonstrate “code that writes code”. No, it isn’t that big of a deal – most DBAs know how to do this, but in the interest of those who don’t, I thought I would share.

So the end result I want in this case is to have a bunch of INSERT statements that contain my base data from a table where that data already exists. I could script out the table from SSMS and use the replacement parameter feature to fill out each record individually:

I have a table in one of my databases that I right-clicked to show the INSERT statement:

-- Person

INSERT

INTO [WAVS].[dbo].[Person]

([PersonName]

,[Street]

,[CityStateZip]

,[PersonType])

VALUES

(<PersonName, varchar(150),>

,<PersonStreet, varchar(100),>

,<PersonCityStateZip, varchar(255),>

,<PersonPersonType, varchar(100),>)

I can use the CTRL-SHIFT-M keys to replace the VALUES part with the values I want.

But I already had a copy of that “base” data before I turned the database over to the testers. Before I give it to them, I run code like this on each table – this one is for the Person table above (note that there should only be two lines here):

SELECT

'INSERT INTO [WAVS].[dbo].[Person] ([PersonID], [PersonName], [Street], [CityStateZip], [PersonType]) VALUES (' + CAST([PersonID] AS VARCHAR(5))+ ', ''' + [PersonName]+ ''', ''' + [Street]+ ''', ''' + [CityStateZip] + ''', ''' + [PersonType] + ''')'

FROM

[WAVS].[dbo].[Person]

When I run this code, I get these statements:

INSERT INTO [WAVS].[dbo].[Person] ([PersonID], [PersonName], [Street], [CityStateZip], [PersonType]) VALUES (1, 'Buck Woody', '123 Here Street', 'Covington, WA 98042', 'Vet')
INSERT INTO [WAVS].[dbo].[Person] ([PersonID], [PersonName], [Street], [CityStateZip], [PersonType]) VALUES (2, 'Jane Doe', '231 Function Ave', 'Redmond, WA 98052', 'Vet')
INSERT INTO [WAVS].[dbo].[Person] ([PersonID], [PersonName], [Street], [CityStateZip], [PersonType]) VALUES (3, 'Diane Wilson', '34251 Appt 3 7th Street', 'Seattle, WA 98061', 'Vet')
INSERT INTO [WAVS].[dbo].[Person] ([PersonID], [PersonName], [Street], [CityStateZip], [PersonType]) VALUES (4, 'John Kelso', '89734 Country Lane', 'Covington, WA 98042', 'Farmer')
INSERT INTO [WAVS].[dbo].[Person] ([PersonID], [PersonName], [Street], [CityStateZip], [PersonType]) VALUES (5, 'Marjorie Christian', '9893452 Changein Lane', 'Maple Valley, WA 98072', 'Farmer')
INSERT INTO [WAVS].[dbo].[Person] ([PersonID], [PersonName], [Street], [CityStateZip], [PersonType]) VALUES (6, 'Joanne Lister', '98904 Mapelwood Drive', 'Spokanne, WA 98045', 'Zoo Worker')

All I have to do is copy those statements into my “initialization” scripts just after the table creations, and then I can run them at will to create my new database (I know I can take a backup – I have reasons for not doing that).

There are two keys for writing code that writes code: The first is to simply use the single tick (') to write out what the end code should say (INSERT INTO, in my case) and then to use multiple ticks (''') when you want the code to contain the ticks themselves. This “escapes” the characters so that you can embed them. If you use SSMS to write your queries, the color coding will look as mine does above. You’ll also need to remember to CAST or CONVERT any data types that don’t concatenate (+) together well. In my case I had to CAST an integer value – but that’s OK, since it is only printing to the screen and I don’t include the ticks, the INSERT works fine.

You can use this method not only with INSERTS, but with lots of operations. And you can also even fire off the code once you create it – but that’s another post :)

Changing the Primary Key After You Have Data

BuckWoody — Tue, 24 Nov 2009 13:25:22 GMT

Yesterday I blogged about changing a Primary Key (PK) during the design phase, and before you have data in the database. Even then, it’s not trivial to change the data type or column(s) that make(s) up the PK. When you have data in that Primary Key and/or you have Foreign Keys (FK) that point to a PK field, this becomes a much more involved process.

First, you MUST take a complete backup of the system, and you MUST do this work on a development system. You’re going to be manipulating base data, and in the worst kind of way. Why do I say that?

There are three kinds of data problems. One is physical data corruption, and once you restore it (I don’t trust repair with data loss) then you know where the “good” data stops and starts. You know that the FKs are pointing to the right PKs, and that the data is readable.

The second kind of data problem is “incorrect” entry, meaning that you put in “Buck Woodie” when you meant “Buck Woody”. This is also often correctable, as your Declarative Referential Integrity (DRI) will still keep that record together. It’s a simple matter to locate and correct.

But the third kind of data problem is the worst. In this kind, the FKs point to the wrong PK, or not at all. Or perhaps some of the FK data is missing. The reason this is the worst is that the data cannot be trusted – did Buck Woody really pay for that purchase or was that payment from a different record? If the links are incorrect, everything will look fine, but it won’t be correct.

So in the situation where you want to change the PKs when you already have data, one of the steps is to re-point the FKs to the proper PKs. Get this wrong and you’ll have “dirty data”. The only recourse there is to restore the backup and start over.

I’m assuming that you have a REALLY GOOD reason to make this change, and that you’ve taken that backup and you’re on a test system. As Hannibal Lector would say: “Okeedokee – here we go.”

I’ll describe two possible choices. In the first, understand and document the PK and FK relationships for each table. Then create queries that tie out the INSERTS for that data in the correct order, and include all fields for both tables, including the PKs in the parent table and the FKs in the child tables.

With those INSERT statements made, drop the PK and FK constraints on all of the tables using the script I mentioned yesterday. Make your change to the data type (or field) in the tables. Then edit the INSERT statements to have the new value types, ensuring that the “sets” or records are kept together by keeping the FK’s pointing to the right PKs. Then re-apply the PK and FK constraints, watching for errors. With backups and scripts, you can make corrections along the way.

The second method works much as the first, with the exception of using the INSERT statements. You can use SSIS or a third-party transfer process to actually move the data and change the data type or values “in flight”, although this requires more thought and planning in my experience.

In any case, you should test the results thoroughly and ensure that you’re getting the data you expect. Communicate what you’re doing, and be ready to fall back to your backups at the first sign of trouble.

Oh – since SQL Server 2005, you’ve been able to “relax” constraints to insert data. That won’t help you here, since you’re changing the entire key for whatever reason. That’s only meant to allow you to get data in when the key doesn’t change.

OK – feel free to comment on this post, since there are other ways I didn’t cover. But if you’re reading my material or any of the comments, make sure you test it yourself – I might have missed something, no one is perfect, your situation may vary. Once again, you can see why I value a good design and ERD so highly – that way you never have to get here to begin with!