Yesterday I received a notification that one of my submissions to http://connect.microsoft.com had received a comment. Nothing unusual about that, I receive those sorts of emails every day, on this occasion however the comment was actually worth reading.
The comment was from SSIS development guru Jeff Bernhardt (he has changed job titles recently but I don’t know the new one so “dev guru” will have to do) and was posted in reply to my request for a single type system in SSIS (which I also blogged about last month at Have SSIS' differing type systems ever caused you problems?). Jeff’s comment provided a rather unique insight into some of the machinations within the SSIS product team as they go about triaging our many Connect submissions and also gave an interesting potted history of the SSIS product itself. For these reasons I thought it would be worth highlighting the comment to the larger SSIS community so with Jeff’s permissions I have copied the comment below (you can see the original at [SSIS] Consolidate three type systems into one)
The three type systems make me crazy too. They make all of the developers working on the guts of SSIS crazy. The history is interesting so I will share it:
The first part of SSIS that was built in the proto-days of 1999 was the replacement for the DTS import export wizard. I know, hard to believe. This was the first ‘host’ for the dataflow pipeline. The pipeline was built to use the OLEDB type system to make reading and writing from OLEDB really fast. Our internal buffers are really just OLEDB bound memory layouts. The DT_Foo type system is an exact copy of the OLEDB type system with some extra types added for SSIS specific use. That type system is built in deep to the dataflow and all transforms.
Next to arrive was the Runtime, we needed a real host for pipelines and a way to coordinate activities. In those days, the way software was built and extended here at Microsoft was to use COM and OLE Automation friendly interfaces. We expected that most folks that embedded and extended SSIS (then still called DTS) was to use native code and COM interfaces. Naturally, that is how the Runtime was designed and built; the COM type system is pervasive and runs deep inside the native Runtime. This is why we see the VARIANT types with BSTRs, SAFEARRAYS, etc. (as a fun note, the VARIANT type was design to copy the internal type system of Visual Basic, a VARIANT is a Var )
When it came time to build a designer for these interfaces (and believe it or not, the designer started a year or so after the internals were working) we took a bet on managed code and c#. This was a bet at the time; managed code was new and un-proven. There was a lot of anxiety. The UI is all managed code and so its chock full of the new and fancy CLR type system. We expected that some people might use managed code to host or extend SSIS so a lot of ‘wrappers’ were put together to make that work. Of course the seams on the type system show through.
So here we are with three type systems. Getting rid of the DT_Foo system would mean re-writing the pipeline (probably in managed code) and putting in a big shunting system to maintain backward compatibility with older transforms.
Getting rid of the VARIANT type system would mean re-writing the Runtime (probably managed) and providing some shunting and upgrade system for old packages and tasks.
Re-writing the UI is crazy talk, and would still need all of the back-compat layers put in.
So as much as we hate where we are, fixing this issue is a HUGE undertaking. For instance, it may have been the only work we took on for Denali. Certainly we thought about it, but the value delivered to customers of this one change is low when compared to all of the other things we could do for the same effort.
What does the future hold? It is hard to imagine us ever investing the effort to fix the SSIS type problem, mostly because of all of the backward compatibility issues that it would create. As we look to new problem spaces and new ways of solving problem in the cloud world, I expect the CLR type system will become the common language of data movement for us.
We will remember the pain that our short sighted decision making in 2001 unleashed on our devoted developer friends and we will strive to avoid making these mistakes again.