Search results matching tags 'ssis' and 'XML'

Outputting data to an XML file : SSIS video nugget

jamiet — Sat, 27 Feb 2010 16:53:00 GMT

The last video nugget that I produced seemed to be fairly well received so I thought I would attempt a few more; and indeed, here is the next one. In this video I demonstrate a simple technique for outputting data to an XML file using SSIS’s Script Task.

The video is 8m22s long and is embedded below however I’m aware of some problems that we are having, at the time of writing, with embedding of videos here on SQLBlog so for the time being head over to http://vimeo.com/9781950 where it can be viewed.

If you are so inclined I have made the package that I built whilst recording this demonstration available on my SkyDrive for your reference.

I look forward to your comments.

@Jamiet

SSIS Package Stats Collector version 2

jamiet — Sat, 07 Nov 2009 01:58:24 GMT

A few weeks ago I published a blog entitled Collecting information about your SSIS packages which demonstrated a way that you could use T-SQL and XQuery to derive information about your SSIS packages. Since then I’ve made a few amendments to the script that accompanied that blog post as follows:

New columns

The collected data now includes the following information about your packages:
    PackageId
    PackageXML (The XML that forms the .dtsx file)
    NumberOfContainers
    NumberOfDataflows
    NumberOfConnectionManagers
    NumberOfVariablesInEntirePackage

New feature

Returns a list of non-unique GUIDs and the packages that they appear in. If a GUID appears more than once in the same package it tells us how many times

Bug fixes

If there were no tasks in the package no data would be returned. That is now no longer the case.
Previous version didn’t work on case-sensitive collations, should be OK now.

There are also more comments liberally sprinkled around it now as well!

Here’s a screenshot showing some of those new columns:

Returning [PackageId] as a field is an important addition because we can now very easily see if any of our packages share the same Package Id. Package IDs are supposed to be unique but if they are not then they are a problem waiting to happen and unfortunately Business Intelligence Design Studio (BIDS) doesn’t really protect you against it very well, in fact if you use package templates then its very likely that you have non unique GUIDs somewhere. (“Non unique globally unique identifiers” – that should set the alarm bells ringing if nothing else).

As alluded to above the script now returns a brand new dataset showing all the GUIDs that occur more than once across all of your packages. Now, I’m not just talking about PackageId GUIDs here, every single object in a SSIS package be it a task, container, component, variable, connection manager, column, whatever… has a GUID attached to it and given the healthy amount of copy-and-paste that goes on when developing SSIS packages its a sure bet that some of those GUIDs are going to get repeated somewhere. Of course, its up to you whether you want to do anything about it or not – if you’re a purist then the idea of repeated GUIDs might keep you awake at night although I suspect most people simply won’t care. Still, its there if you need it:

You may wonder why I have the [NumberOfTimeshisGUIDOccursInThisPackage] column in there. Well, its perfectly plausible that the a GUID can reoccur in the same package and in fact when I was testing this earlier on today I had a package that had 6 occurrences of a GUID because it contained 6 near-identical connection managers that had obviously been copied-and-pasted.

That’s it for now. Let me know if this script proves useful to you and let me know if there’s anything you’d like to see added to it. Download the latest version from: http://cid-550f681dad532637.skydrive.live.com/self.aspx/Public/BlogShare/20091105/SSISPackageStatsCollector%5E_v2.sql. I have also dropped it into SQLMesh.

Looking forward to some feedback (hint hint).

@Jamiet

Collecting information about your SSIS packages [SSIS Nugget]

jamiet — Sun, 18 Oct 2009 21:08:00 GMT

Did you know that is is possible to read the contents of a SSIS package (i.e. a .dtsx file) from within SQL Server Management Studio (SSMS) using T-SQL? For example, take the following T-SQL snippet:

select    cast(BulkColumn as XML)
from    openrowset(bulk 'C:\tmp\MyPkg.dtsx',
                        single_blob) as pkgColumn;

It uses OPENROWSET to return the contents of a specified package (C:\tmp\MyPkg.dtsx) as an XML document. Here is a screenshot showing what this returns:

and clicking on that result opens up the following:

That’s what the inards of a .dtsx file look like. In other words we now have a queryable XML document representing a package, thereafter you’re limited only to what you can do with XQuery which is quite a lot. Now, I’m no XQuery expert by any means but I did manage to find a few uses for this. Here are some sample queries:

All properties of a package

SELECT    Props.Prop.query('.')                                                        as PropXml
,        Props.Prop.value('declare namespace p1="www.microsoft.com/SqlServer/Dts";
                            string(./@p1:Name)','nvarchar(max)')                    as PropName
,        Props.Prop.value('.', 'nvarchar(max)')                                        as PropValue
FROM    (
        SELECT    CAST(pkgblob.BulkColumn AS XML) pkgXML
        FROM    OPENROWSET(bulk 'C:\tmp\MyPkg.dtsx',single_blob) AS pkgblob
        ) t
CROSS    APPLY pkgXML.nodes('declare namespace DTS="www.microsoft.com/SqlServer/Dts";
                        /DTS:Executable/DTS:Property') Props(Prop)

I had a lot of help from @rbarryyoung with this one!

Name and type of every task in a package

SELECT    Pkg.props.value('declare namespace p1="www.microsoft.com/SqlServer/Dts";
                            ./p1:Property[@p1:Name=''ObjectName''][1]','nvarchar(max)') as TaskName
,        Pkg.props.value('declare namespace p1="www.microsoft.com/SqlServer/Dts";
                            ./@p1:ExecutableType','nvarchar(max)') as TaskType
FROM    (
        select    cast(pkgblob.BulkColumn as XML) pkgXML
        from    openrowset(bulk 'C:\tmp\Package.dtsx',single_blob) as pkgblob
        ) t
CROSS    APPLY pkgXML.nodes('declare namespace DTS="www.microsoft.com/SqlServer/Dts";
                            //DTS:Executable[@DTS:ExecutableType!=''STOCK:SEQUENCE''
                        and    @DTS:ExecutableType!=''STOCK:FORLOOP''
                        and    @DTS:ExecutableType!=''STOCK:FOREACHLOOP''
                        and not(contains(@DTS:ExecutableType,''.Package.''))]') Pkg(props)

Note that this one will also return all tasks that exist in eventhandlers and it ignores how “deep” a task is in the container hierarchy.

Putting it all together

Ok, that’s all pretty cool but it would be nice to combine it all together and get a summary of many packages, perhaps all of the packages on your machine. Hence I’ve put together a T-SQL script that will display summary information about all of the packages in a folder and its subfolders. Here’s the results of running that script on my dev machine:

217 packages – I’ve collected lot of them over the years! This shows a sampling of some of the information that it is possible to collect:

Package name
Original creator of the package
Package Type (signifies which version of BIDS was used to originally build it)
Version numbers
Number of tasks in the package

Want to know what your most complex package might be? Simply order this dataset in descending order of [NumberOfTasks]. Or maybe you want to know which of your developers has built the most packages – the answers are right here!

Of course, this could be extended to capture much much more information than what I have captured here. You may want to know how many eventhandlers each of your packages has, when the packages were created, or perhaps how many components are in your dataflows. The (SSIS) world is your oyster!!!

One thing that I thought would be very useful would be to stick this script into a SQL Agent job, run it on a daily basis, and insert the results into a history table thereby giving you a running history of all the packages in your system and when they changed. If anyone does do that let me know how it goes!

To execute the script simply open it and change the following line as appropriate:

DECLARE    @Path    VARCHAR(2000) = 'C:\*.dtsx';

Note that you will need to enable [xp_cmdshell] in order to run the script which is available on my SkyDrive at:

http://cid-550f681dad532637.skydrive.live.com/self.aspx/Public/BlogShare/20091018/SSISPackageStatsCollector.sql

If you experience any problems with it let me know. If you adapt it any way let me know that too because writing these XQuery statements is no easy task (believe me!!!) and it would be great to share that stuff with other people!

@JamieT

Update: The original version of this script only worked on SQL Server 2008. I have now updated it so that it works on SQL2005 also! Thanks to Bruce in the comments for alerting me to this fact/

Update 2: Found a few more issues and hence have uploaded another new version (same link still works though). Changes:

Script will now work on a server with a case-sensitive collation
xp_cmdshell is turned on at the top of the script
An error message that could get returned by the command-line call under certain circumstances wasn't getting handled. It is now!

Update 3: An updated version with bug fixes and new features is now available at SSIS Package Stats Collector version 2

XML Data Type Methods and SSIS Package Variables

Peter DeBetta — Mon, 18 Dec 2006 19:31:00 GMT

This is a follow-up to a posting from several months ago (Using XML Data Type Methods to query SSIS Packages). I included a temp table (based on information I found in the Microsoft.SqlServer.Dts.Runtime.VarTypeConvert class's constructor) which gives the conversion of the numeric data type value to the appropriate TypeCode (System.TypeCode). Thank goodness for Lutz Roeder's Reflector. This code iterates through all the executables in the package and for all variables returns the following properties: Executable Scope, Name, Namespace, Value, DataType (and data type description), EvaluateAsExpression, Expression, ReadOnly, RaiseChangedEvent, DTSID, Description, and CreationName. I am waiting on some confirmation about the property data types themselves (string lengths in particular) and will follow up as soon as I know more. In the meantime, here is the code. Be sure to change the package path to the location of your package file before running...

DECLARE @x xml

CREATE TABLE #t (PackageName varchar(100), PackageCode xml)

SET @X = (SELECT * FROM OPENROWSET
(BULK N'C:\Projects\MyPackage.dtsx',
SINGLE_BLOB) AS Document)

INSERT INTO #t VALUES ('MyPackage.dtsx', @x)

CREATE TABLE #VarTypeCode (TypeID int, TypeName varchar(20))

INSERT INTO #VarTypeCode VALUES(0, 'Empty')
INSERT INTO #VarTypeCode VALUES(1, 'DBNull')
INSERT INTO #VarTypeCode VALUES(2, 'Int16')
INSERT INTO #VarTypeCode VALUES(3, 'Int32')
INSERT INTO #VarTypeCode VALUES(4, 'Single')
INSERT INTO #VarTypeCode VALUES(5, 'Double')
INSERT INTO #VarTypeCode VALUES(6, 'Decimal')
INSERT INTO #VarTypeCode VALUES(7, 'DateTime')
INSERT INTO #VarTypeCode VALUES(8, 'String')
INSERT INTO #VarTypeCode VALUES(9, 'Object')
INSERT INTO #VarTypeCode VALUES(10, 'Object')
INSERT INTO #VarTypeCode VALUES(11, 'Boolean')
INSERT INTO #VarTypeCode VALUES(12, 'Object')
INSERT INTO #VarTypeCode VALUES(13, 'Object')
INSERT INTO #VarTypeCode VALUES(14, 'Decimal')
INSERT INTO #VarTypeCode VALUES(16, 'SByte')
INSERT INTO #VarTypeCode VALUES(17, 'Byte')
INSERT INTO #VarTypeCode VALUES(18, 'UInt16')
INSERT INTO #VarTypeCode VALUES(19, 'UInt32')
INSERT INTO #VarTypeCode VALUES(20, 'Int64')
INSERT INTO #VarTypeCode VALUES(21, 'UInt64')
INSERT INTO #VarTypeCode VALUES(22, 'Int16')
INSERT INTO #VarTypeCode VALUES(23, 'UInt16')
INSERT INTO #VarTypeCode VALUES(24, 'Empty')

;WITH XMLNAMESPACES ('www.microsoft.com/SqlServer/Dts' as p1,
    'www.microsoft.com/SqlServer/Dts' as DTS)
, Executables AS
(   SELECT CAST('' as varchar(100)) AS ParentNodeName,
      P.Vars.value('./p1:Property[@p1:Name="ObjectName"][1]', 'varchar(100)') AS NodeName,
      P.Vars.query('.') AS CurrentNode,
      P.Vars.query('./p1:Executable') AS ChildNodes,
      1 AS Level
   FROM #t
      CROSS APPLY #t.PackageCode.nodes('/DTS:Executable') AS P(Vars)

UNION ALL

   SELECT Executables.NodeName AS ParentNodeName,
      P.Vars.value('./p1:Property[@p1:Name="ObjectName"][1]', 'varchar(100)') AS NodeName,
      P.Vars.query('.') AS CurrentNode,
      P.Vars.query('./p1:Executable') AS ChildNodes,
      Executables.Level + 1
   FROM Executables
      CROSS APPLY Executables.ChildNodes.nodes('/DTS:Executable') AS P(Vars)
)
,Variables AS
(   SELECT Executables.NodeName, Executables.Level,
      V.Vars.value('./p1:Property[@p1:Name="ObjectName"][1]', 'varchar(255)') AS [Name],
      V.Vars.value('./p1:Property[@p1:Name="Namespace"][1]', 'varchar(255)') AS Namespace,
      V.Vars.value('./p1:VariableValue[1]', 'varchar(1024)') AS [Value],
      V.Vars.value('./p1:VariableValue[1]/@p1:DataType', 'int') AS DataType,
      V.Vars.value('./p1:Property[@p1:Name="Expression"][1]', 'varchar(1024)') AS Expression,
      V.Vars.value('./p1:Property[@p1:Name="EvaluateAsExpression"][1]', 'bit') AS EvaluateAsExpression,
      V.Vars.value('./p1:Property[@p1:Name="ReadOnly"][1]', 'bit') AS ReadOnly,
      V.Vars.value('./p1:Property[@p1:Name="RaiseChangedEvent"][1]', 'bit') AS RaiseChangedEvent,
      V.Vars.value('./p1:Property[@p1:Name="DTSID"][1]', 'uniqueidentifier') AS DTSID,
      V.Vars.value('./p1:Property[@p1:Name="Description"][1]', 'varchar(255)') AS Description,
      V.Vars.value('./p1:Property[@p1:Name="CreationName"][1]', 'varchar(255)') AS CreationName
   FROM Executables
      CROSS APPLY Executables.CurrentNode.nodes('/p1:Executable/p1:Variable') AS V(Vars)
)
SELECT V.NodeName AS ExecutableName, V.Level, V.[Name],
   V.Namespace, V.[Value], V.DataType, TCC.TypeName,
V.Expression, V.EvaluateAsExpression, V.ReadOnly,
   V.RaiseChangedEvent, V.DTSID, V.Description, V.CreationName
FROM Variables AS V
   INNER JOIN #VarTypeCode AS TCC ON V.DataType = TCC.TypeID
ORDER BY Level, NodeName, [Name]

DROP TABLE #t

DROP TABLE #VarTypeCode

--Peter

Search results matching tags 'ssis' and 'XML'

Outputting data to an XML file : SSIS video nugget

SSIS Package Stats Collector version 2

New columns

New feature

Bug fixes

Collecting information about your SSIS packages [SSIS Nugget]

All properties of a package

Name and type of every task in a package

Putting it all together

Recommended readings: 2008 update

Business Intelligence

General data and programming topics

GIS and spatial topics

SQL Server

XML Data Type Methods and SSIS Package Variables