Davide Mauri : 2008 R2

Extended Events did it again: monitoring Stored Procedures performance

Davide Mauri — Tue, 19 Mar 2013 16:14:29 GMT

In the last days I had to work quite a lot with extended events in order to deeply monitor SQL Server performance. One interesting request that came out while implementing the monitoring infrastructure, was the possibility to monitor the performance of a set of stored procedures, vital for the correct handling of an online booking procedure.

The challenge was to give a sort of real-time monitor of procedure performances so that one can then create alert and/or do some stream-analysis to keep response time always under the desired amount of time.

Here’s how you can do it using Extended Events, monitoring, for example, the execution of procedure uspGetManagerEmployees in AdventureWorks2012:

CREATE EVENT SESSION [monitor_procedure_performance] ON SERVER
ADD EVENT sqlserver.rpc_completed
    (
        ACTION    (package0.collect_system_time,package0.process_id,sqlserver.client_app_name,sqlserver.client_connection_id,sqlserver.client_hostname,sqlserver.client_pid,sqlserver.database_id,sqlserver.database_name,sqlserver.nt_username,sqlserver.request_id,sqlserver.session_id,sqlserver.session_nt_username,sqlserver.sql_text,sqlserver.transaction_id,sqlserver.transaction_sequence,sqlserver.username)
        WHERE    ([object_name]=N'uspGetManagerEmployees')
    ),
ADD EVENT sqlserver.module_end
    (
        ACTION    (package0.collect_system_time,package0.process_id,sqlserver.client_app_name,sqlserver.client_connection_id,sqlserver.client_hostname,sqlserver.client_pid,sqlserver.database_id,sqlserver.database_name,sqlserver.nt_username,sqlserver.request_id,sqlserver.session_id,sqlserver.session_nt_username,sqlserver.sql_text,sqlserver.transaction_id,sqlserver.transaction_sequence,sqlserver.username)
        WHERE    ([object_name]=N'uspGetManagerEmployees')
    )
ADD TARGET package0.ring_buffer
WITH
    (
        STARTUP_STATE=OFF
    )
GO

Once done, it’s all about decoding the XML that Extended Events returns:

Analyze XEvent data
*/

IF (OBJECT_ID('tempdb..#t') IS NOT NULL) DROP TABLE #t
IF (OBJECT_ID('tempdb..#r') IS NOT NULL) DROP TABLE #r
go

select
    cast(target_data as xml) xdoc
into
    #t
from
    sys.dm_xe_sessions s
inner join
    sys.dm_xe_session_targets t on t.event_session_address = s.address
where
    s.name = 'monitor_procedure_performance'
;

with cte as
(
    select
        event_number = ROW_NUMBER() over (order by T.x),
        event_timestamp = T.x.value('@timestamp', 'datetimeoffset'),
        T.x.query('.') as event_data
    from
        #t
    cross apply
        xdoc.nodes('/RingBufferTarget/event') T(x)
),
cte2 as (
    select
        c.event_number,
        c.event_timestamp,
        --data_field = T2.x.value('local-name(.)', 'varchar(100)'),
        data_name = T2.x.value('@name', 'varchar(100)'),
        data_value = T2.x.value('value[1]', 'varchar(100)'),
        data_text = T2.x.value('text[1]', 'varchar(max)')
    from
        cte c
    cross apply
        c.event_data.nodes('event/*') T2(x)
),
cte3 as (
    select
        *
    from
        cte2
    where
        data_name in ('collect_system_time', 'object_name', 'cpu_time', 'duration', 'logical_reads', 'row_count', 'database_name', 'database_id')
)
select
    *
into
    #r
from
    cte3
pivot
    (max(data_value) for data_name in (database_id, database_name, object_name, cpu_time, duration, logical_reads, row_count)) T
go

--SELECT * FROM #t
SELECT * FROM #r
go

select
    execution_date = cast(event_timestamp as date),
    execution_hour = datepart(hour, event_timestamp),
    execution_minute = datepart(minute, event_timestamp),
    [object_name],
    duration_msec = avg(cast(duration as int)) / 1000.
from
    #r
group by
    cast(event_timestamp as date), datepart(hour, event_timestamp), datepart(minute, event_timestamp), [object_name]

That’s it. You can now keep monitored execution times of your procedure.

Trapping SQL Server Errors with Extended Events

Davide Mauri — Sun, 17 Mar 2013 16:45:06 GMT

One very useful usage of Extended Events is the ability to trap SQL Server error without the need to have a server trace running (which, btw, is deprecated), with the additional feature of being able to query the data as soon as it comes in. This means that we a solution to monitor and trap errors as soon as they happen can be easily created, in order to help developers to fix problems as soon as they are detected. This is really, really, really helpful especially in very big applications, where the code base is quite old and there is no-one really knowing everything of the solution.

To start a Extended Events sessions in order to trap SQL Server errors with severity greater than 10, just run the following script:

CREATE EVENT SESSION [error_trap] ON SERVER
ADD EVENT sqlserver.error_reported
    (
        ACTION    (package0.collect_system_time,package0.last_error,sqlserver.client_app_name,sqlserver.client_hostname,sqlserver.database_id,sqlserver.database_name,sqlserver.nt_username,sqlserver.plan_handle,sqlserver.query_hash,sqlserver.session_id,sqlserver.sql_text,sqlserver.tsql_frame,sqlserver.tsql_stack,sqlserver.username)
        WHERE    ([severity]>10)
    )
ADD TARGET package0.event_file
    (
        SET filename=N'D:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\XEvents\error_trap.xel'
    )
WITH
    (
        STARTUP_STATE=OFF
    )
GO

ALTER EVENT SESSION [error_trap] ON SERVER
STATE = START;
GO

The problem with Exended Events is that they only talks XML which is surely flexible and extensible, but not at all confortable to be queried. That’s why I prefer to turn it into something relational. Using the xml nodes function and pivoting data can make the trick:

IF (OBJECT_ID('tempdb..#e') IS NOT NULL) DROP TABLE #e
go

WITH cte AS
(
    SELECT
        CAST(event_data AS XML) AS event_data
    FROM
        sys.fn_xe_file_target_read_file('D:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\XEvents\error_trap*.xel', NULL, NULL, NULL)
),
cte2 AS
(
    SELECT
        event_number = ROW_NUMBER() OVER (ORDER BY T.x)
    ,    event_name = T.x.value('@name', 'varchar(100)')
    ,    event_timestamp = T.x.value('@timestamp', 'datetimeoffset')
    ,    event_data
    FROM
        cte
    CROSS APPLY
        event_data.nodes('/event') T(x)
)
SELECT * INTO #e FROM cte2
go

WITH cte3 AS
(
    SELECT
        c.event_number,
        c.event_timestamp,
        --data_field = T2.x.value('local-name(.)', 'varchar(100)'),
        data_name = T2.x.value('@name', 'varchar(100)'),
        data_value = T2.x.value('value[1]', 'varchar(max)'),
        data_text = T2.x.value('text[1]', 'varchar(max)')
    FROM
        #e c
    CROSS APPLY
        c.event_data.nodes('event/*') T2(x)
),
cte4 AS
(
    SELECT
        *
    FROM
        cte3
    WHERE
        data_name IN ('error_number', 'severity', 'message', 'database_name', 'database_id', 'client_hostname', 'client_app_name', 'collect_system_time', 'username')
)
SELECT
    *
FROM
    cte4
PIVOT
    (MAX(data_value) FOR data_name IN ([error_number], [severity], [message], database_name, database_id, username, client_hostname, client_app_name, collect_system_time)) T
WHERE
    [severity] > 10
ORDER BY
    event_timestamp DESC
go

and voilà, now all errors can be easily identified:

It’s now very easy to create reports and dashboards to monitor the system in (near) real time. And this is vital when you run a 24x7 online business.

Forcing “Custom Properties” of a Data Flow Transformation to support DTS Expression

Davide Mauri — Wed, 30 Jan 2013 16:48:22 GMT

Today I was using a 3rd Party Data Flow component that has several Custom Properties for which I need to change some of their values at runtime using a DTS Expression.

To mimic the situation let’s use the “Percentage Sampling” that has two Custom Properties:

Of the two Custom Properties only some (one in this case) are available also outside that dataflow, so that they can be targeted by a DTS Expression. Such properties are listed under the “Misc.” section of Data Flow properties

and also in the Property Expression Editor window:

Now, what if you need to make also the “hidden” custom properties available for DTS Expression usage? As you may have noticed, the SamplingSeed is not exposed outside the Data Flow. I tried to search the web, but after several minutes I gave up since I wasn’t able to find anything that could help. I then started to look into the SSIS object model and I found a nice property named expressionType in the IDTSCustomProperty interface that tells to the engine if the property value can be specified using DTS Expression or not:

http://msdn.microsoft.com/en-us/library/bb510794.aspx

If the value is set to Notify than the usage of DTS Expression is possible otherwise, if the property is set to None, as the name implies, DTS Expression cannot be used.

So all you need to do is to open the .dtsx file, look for the component you want to touch and its properties

and add (if not exists) or change the expressionType attribute to Notify:

Save the file and voilà, the property is now available for DTS Expression usage!

Now, just be aware that is a sort of an hack, so double check it works for you. On the 3rd party components we’re using it works like a charm, and it just saved that day since without the ability to change some properties at run time, the deployment of our package in production could have been a nightmare.

I tested this approach both on SQL Server 2012 and SQL Server 2008 and in both cases I hadn’t had any problems.

Hope this helps someone , enjoy!

Power Pivot SQL Agent Job Monitor

Davide Mauri — Mon, 07 Jan 2013 21:23:29 GMT

In the last days I needed a quick and efficient way to monitor the performance of a job and of each step over time. There’s a lot of interesting data in MSDB database and the various sysjob tables, but the stored data is not really easy to use outside the Management Studio. Date and Time are not stored as date/time data type, there is no easy way to correlate the performance of a step with the related job execution (for example is not that easy to return all the steps of a job executed, say, yesterday at 10:00).

So I decided to write some views to make the querying easier and then the next move was to create a Tabular Model with Power Pivot on Excel to make also analysis easier.

Since this is a somehow recurring task that I find myself doing during my job – not often, but when I have to do it now I have something much more powerful in my hands – I decided to put it also con CodePlex so that that little work can be shared with the community.

You can find the scripts with the objects (3 UDFs and 4 Views) that must be created inside MSDB and the Power Pivot Excel 2010 model here:

http://jobmonitor.codeplex.com/

Enjoy!

SSAS DMVs: useful links

Davide Mauri — Fri, 18 May 2012 14:47:42 GMT

From time to time happens that I need to extract metadata informations from Analysis Services DMVS in order to quickly get an overview of the entire situation and/or drill down to detail level. As a memo I post the link I use most when need to get documentation on SSAS Objects Data DMVs:

SSAS: Using DMV Queries to get Cube Metadata
http://bennyaustin.wordpress.com/2011/03/01/ssas-dmv-queries-cube-metadata/

SSAS DMV (Dynamic Management View)
http://dwbi1.wordpress.com/2010/01/01/ssas-dmv-dynamic-management-view/

Use Dynamic Management Views (DMVs) to Monitor Analysis Services
http://msdn.microsoft.com/en-us/library/hh230820.aspx

DTLoggedExec 1.1.2008.4 SP2 released!

Davide Mauri — Wed, 01 Feb 2012 17:15:44 GMT

I’ve released couple of hours ago the SP2 of my DTLoggedExec tool.

For those who don’t know it, it’s a DTEXEC replacement, useful to execute SSIS and having logging provided right from the engine and not from the package itself.

More info can be found here:

http://dtloggedexec.codeplex.com/

This SP2 release add an important feature to the CSV Log Provider. It's now possible to store a personalized label into each log, in order to make it easy to identify or group logs.

Let's say, for example, that you have 10 packages in your ETL solution, and each time you have to load your data, you need to execute all those 10 packages. In other words , you have a batch made of 10 packages. It would be nice if all logs - one for each package - can be identified as a whole and grouped together, in order to quickly identify all the log of a articular batch.

This will make things easier when you want to know the overal time consumed by each batch execution.

The new "ExecutionLabel" attribute will help to achieve this. A useful ExecutionLabel can be obtained using SQL Server Agents Token. For example:

LABEL=$(ESCAPE_SQUOTE(JOBID))_$(ESCAPE_SQUOTE(STRTDT))_$(ESCAPE_SQUOTE(STRTTM))

ChangeLog

Updated the CSV Log Provider in order to write the status of the log file in the header.
- An OPEN status means that the log file is being written.
- A CLOSED status means that the log files has been written correctly.
- A file can be loaded into the database log only if is in the CLOSED state.
CSV Log files have 2 additional rows in the header: one for the FileStatus and one for the ExecutionLabel values.
- File format has been update to 4 from 3. (Only the header section of the file has been changed).
CSV Log Provider will now display EndTime value to the Console
Updated the import-log-data.sql to correctly load file with format 3 (the old one) and 4 (the new one).
Updated database schema to version 19 in order to store the new ExecutionLabel value
Update samples in order to show how to use the new ExecuteLabel option

As usual the download is available for free here:

http://dtloggedexec.codeplex.com/releases/81526/download/336490

SYS2 scripts updated (December 2012)

Davide Mauri — Mon, 12 Dec 2011 22:26:43 GMT

I’ve updated my SYS2 scripts:

Added a new script to see how much buffer cache memory is used by each database
Updated the sys2.stats script in order to have only one row per statistics
Updated the sys2.query_stats script to use the sys.dm_exec_plan_attributes dmv to get better information on which database was used by the cached plans

As usual they are available from CodePlex:

http://sys2dmvs.codeplex.com/

Enjoy!

Spotlight session at PASS 2011 - Temporal Snapshot Fact Table

Davide Mauri — Sun, 29 May 2011 13:44:59 GMT

I’m very happy to announce that my proposal for the Spotlight session I’ve been invited to deliver at PASS 2011 has been accepted!

Temporal Snapshot Fact Table

You are designing a BI Solution and your customer ask you to keep a snapshot of the status of all their documents (orders, insurances, contracts, bills...whatever the word "document" may mean) for all the days of the year. They have millions of documents and they want to have in their Data Warehouse all the data they have gathered right from the very first operating day.

If you have 1 million of documents (on average) and you have to keep a snapshot of them for each one of the 365 days in a year, and you have 10 year of history, you're going to have a 3 billions table just to start with. That's a very big and challenging number, and you may have not the option to buy a Parallel Data Warehouse.

In this session, we'll see how we can turn the usual snapshot tables into temporal table so that we can store time intervals in order to avoid data duplication, while keeping the Data Warehouse design usable by Analysis Services (that doesn't know what an interval is) and optimizing it to have very good performance even on standard hardware.

The explained technique is a result of several month of research and has been applied to the Data Warehouse of an insurance company where we had to deal with two times the number said before.

The topic is very interesting and – I bet – very intriguing for many people working in BI and specially with Analysis Services, since it lacks the support of “time intervals” to define the validity period of a fact row. With my SolidQ Italian collegues we’ve been able to find a way to overcome this limitation, allowing the storage of daily snapshots of data with a very high efficency and performance.

In this session I’m going to share everything we discovered with you. It will be really interesting, I can tell you! Probably one of the most advanced – yes simple - usage of SSAS and Many-To-Many relationship you’re going to see.

SYS2 Scripts Updated – Scripts to monitor database backup, database space usage and memory grants now available

Davide Mauri — Sun, 13 Feb 2011 17:30:22 GMT

I’ve just released three new scripts of my “sys2” script collection that can be found on CodePlex:

Project Page: http://sys2dmvs.codeplex.com/

Source Code Download: http://sys2dmvs.codeplex.com/SourceControl/changeset/view/57732

The three new scripts are the following

sys2.database_backup_info.sql
sys2.query_memory_grants.sql
sys2.stp_get_databases_space_used_info.sql

Here’s some more details:

database_backup_info

This script has been made to quickly check if and when backup was done. It will report the last full, differential and log backup date and time for each database. Along with these information you’ll also get some additional metadata that shows if a database is a read-only database and its recovery model:

By default it will check only the last seven days, but you can change this value just specifying how many days back you want to check.

To analyze the last seven days, and list only the database with FULL recovery model without a log backup

select * from sys2.databases_backup_info(default)
where recovery_model = 3 and log_backup = 0

To analyze the last fifteen days, and list only the database with FULL recovery model with a differential backup

select * from sys2.databases_backup_info(15)
where recovery_model = 3 and diff_backup = 1

I just love this script, I use it every time I need to check that backups are not too old and that t-log backup are correctly scheduled.

query_memory_grants

This is just a wrapper around sys.dm_exec_query_memory_grants that enriches the default result set with the text of the query for which memory has been granted or is waiting for a memory grant and, optionally, its execution plan

stp_get_databases_space_used_info

This is a stored procedure that list all the available databases and for each one the overall size, the used space within that size, the maximum size it may reach and the auto grow options. This is another script I use every day in order to be able to monitor, track and forecast database space usage.

As usual feedbacks and suggestions are more than welcome!

BI Virtual Chapter: “Adaptive BI: Engineering a BI Solution”

Davide Mauri — Thu, 09 Sep 2010 09:54:27 GMT

On the next 20th Semptember I’ll deliver a session on a topic that is IMHO really important but is not yet covered enough:

Adaptive BI: Engineering a BI Solution

Are you starting to create a BI solution....but where do you begin with? How to setup everything correctly so that you'll be able to handle new features required by the customers easily and without too much effort? Which standard do you put in place so that everyone who will join the project will be immediately operative? How do you define your BI architecture so that it can be sound and flexible and the same time? All these questions come out every time someone has to start a BI project. In this session I'll present a set of standard rules we use to create our BI project, consolidated in more than five years of development (since SQL 2005 was released) and that allow to put some technical rules that helps you to start in the correct way right from the beginning: naming convention, architecture decisions, database unit testing, layering and everything you need to know to build an "Adaptive" Business Intelligence solution. The idea is to be flexible in terms of architectural decision but have some well-known rules in the solution to make it “engineerable”: in one word..."Adaptive"!

If you’re into BI this session can give you some interesting ideas on how to structure your next project. See you in cyberspace here

https://www.livemeeting.com/cc/usergroups/join?id=C97PWB&role=attend&pw=Z%40%2B%5Ew%2B%40h4

on Sept 20th at 12pm EST.

At the end of the session I’ll show also how the rules can be applied automatically with the right tools, creating an entire package to load a dimension in less then 1 minute, with all best pratices and standard in place!

PowerPivot not visibile in Excel

Davide Mauri — Fri, 30 Jul 2010 07:49:44 GMT

Yesterday I had to install PowerPivot on my new machine since I would have liked to use it to process some data, but after having succesfully run the installation package, the PowerPivot menu wasn’t visibile in Excel .

Looking in the Event Log for errors I found the following one:

Exception: Customization could not be loaded because the application domain could not be created.

Microsoft.VisualStudio.Tools.Applications.Runtime.CannotCreateCustomizationDomainException: Customization could not be loaded because the application domain could not be created. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Office.Interop.Excel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.Office.Interop.Excel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c'
at Microsoft.AnalysisServices.Modeler.FieldList.ThisAddIn..ctor()

Google (nor Bing ) was able to help me…and after some failed attempt to fix the problem I remembered then when I installed Office 2010 I’ve choose not to install the .NET Programmability Support.

After having installed it (Just run the Office Installation program)

I was finally able to install PowerPivot

So, if you find yourself in the same situation, now you know what to do

PASS Summit 2010 – BI Workshop

Davide Mauri — Wed, 30 Jun 2010 12:01:10 GMT

PASS Summit 2010 Pre & Post conference are out! This year I’ll deliver the “Creating BI Solution from A to Z” seminar in which, as the title implies, attendees will see how to create a BI solution starting from scratch. Going through the dimensional modeling and the creation of the Datawarehouse, the implementation of the ETL process with SSIS, the creation of cube with Analysis Services and reports with Reporting Services with, if time permits, also a glance at PowerPivot, attendees will get a solid ground on the whole process that drives the creation of a BI solution.

The workshop will also show all the best practices and the best methodological approach matured in more than 5 years of working in the BI field.

If you’ll be working on BI or you already have a created an initial BI solution and you want to be sure you’re following the right path, this is workshop is for you!

Don’t miss it!

DTLoggedExec 1.0 Stable Released!

Davide Mauri — Sun, 09 May 2010 08:51:43 GMT

After serveral years of development I’ve finally released the first non-beta version of DTLoggedExec! I’m now very confident that the product is stable and solid and has all the feature that are important to have (at least for me).

DTLoggedExec 1.0
http://dtloggedexec.codeplex.com/releases/view/44689

Here’s the release notes:

FIRST NON-BETA RELEASE! :)
Code cleaned up
Added SetPackageInfo method to ILogProvider interface to make easier future improvements
Deprecated the arguments 'ProfileDataFlow', 'ProfilePath', 'ProfileFileName'
Added the new argument 'ProfileDataFlowFileName' that replaces the old 'ProfileDataFlow', 'ProfilePath', 'ProfileFileName' arguments
Updated database scripts to support new reports
Split releases in three different packages for easier maintenance and updates: DTLoggedExec Executable, Samples & Reports
Fixed Issue #25738 (http://dtloggedexec.codeplex.com/WorkItem/View.aspx?WorkItemId=25738)
Fixed Issue #26479 (http://dtloggedexec.codeplex.com/WorkItem/View.aspx?WorkItemId=26479)

To make things easier to maintain I’ve divided the original package in three different releases. One is the DTLoggedExec executable; samples and reports are now available in separate packages so that I can update them more frequently without having to touch the engine.

Source code of everything is available through Source Code Control:

http://dtloggedexec.codeplex.com/SourceControl/list/changesets

As usual, comments and feebacks are more than welcome! (Just use Codeplex, please, so it will be easier for me to keep track of requests and issues)

SQL Server 2008 R2 Installation and the Phantom of SQL Server 2005 Express

Davide Mauri — Tue, 04 May 2010 07:44:36 GMT

Today I’ve happy started to install SQL Server 2008R2 on my development machine, which has this software installed

Windows Server 2008 R2 Standard
SQL Server 2008 SP1 CU5
Visual Studio 2008 SP1
BOL October 2009
AdventuresWorks2008 Databases SR4
Visual Studio 2010 RTM

So, all the basic standard stuff.

SQL Server 2008 R2 installation went smooth ‘till somewhere in the middle, where the rule engine checks that software pre-requisite are satisfied before starting to copy files.

Here I had this @][@@[?!?! error:

“The SQL Server 2005 Express Tools are installed. To continue, remove the SQL Server 2005 Express Tools.”

Fun enough, I don’t have and I’ve never had SQL Server 2005 Express on my machine.

Armed with patience I analyzed the install log here

C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\Log\yyyymmdd_hhmmss\Detail.txt

and I’ve found that the rule “Sql2005SsmsExpressFacet” is the one in charge of this check and it look for existance of the registry key

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\Tools\ShellSEM (on x86)

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Microsoft SQL Server\90\Tools\ShellSEM (on x64)

In my registry I’ve found that key existsing, due to the installation of the uber-cool Red-Gate SQL Search.

I removed the registry key and here it is! SQL Server 2008 R2 is installing while I’m writing this post.

A note to Microsoft: can you please add more detailed information on the setup while such error happens. Just saying “you have SQL Server 2005 Express installed” is not enough. Please show us what the rule look for and why it has failed directly in the Detailed Report, so that we don’t have to spend time to look for the needle in the logs. Thanks! :)

I did a side-by-side installation with the existing SQL Server 2008 instance.

Master Data Services Employees Sample Model

Davide Mauri — Mon, 03 May 2010 09:17:40 GMT

I’ve been playing with Master Data Services quite a lot in those last days and I’m also monitoring the web for all available resources on it.

Today I’ve found this freshly released sample available on MSDN Code Gallery:

SQL Server Master Data Services Employee Sample Model
http://code.msdn.microsoft.com/SSMDSEmployeeSample

This sample shows how Recursive Hierarchies can be modeled in order to represent a typical organizational chart scenario where a self-relationship exists on the Employee entity.