THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

Jamie Thomson

This is the blog of Jamie Thomson, a freelance data mangler in London

Analysing SQLBits Feedback

Earlier this week I received all the feedback that people offered on my session at SQLBits 7 in York – “SSIS Dataflow Performance Tuning” (the video is available online if you wish to see it). As you may have gathered from previous posts on this blog and my less-SQLy-focused Wordpress blog I am a big fan of collecting and tracking both personal and public data and session feedback lends itself very well to tracking because it is quantitative rather than qualitative; by that I mean attendees are invited to provide marks out of ten rather than (or, in the case of SQLBits, as well as) written comments.

The SQLBits feedback is also useful because they use a consistent format – the same questions are asked each time – this means it is particularly easy to to track whether the scores that people give are trending up or down. I suspect that somewhere the SQLBits organisers have a big Analysis Services cube (ok, perhaps its an Excel pivot table) that allows them to analyse these scores per conference, speaker, track etc.… and there’s no reason that we as session speakers cannot do the same thing. To that end I have started to store my feedback in an Excel spreadsheet of my own which in the interests of transparency is available for public viewing (only a web browser required) on SkyDrive at http://cid-550f681dad532637.office.live.com/view.aspx/Public/Misc/Personal%20SQLBits%20Session%20Feedback.xlsx. I have used a pivot table to aggregate all that feedback and here is a screenshot:

SQLBits feedback pivot table

I am hereby making a public plea to the SQLBits organisers (on the off-chance that they are reading) to please continue to keep the feedback format consistent in the future and I encourage them to publish all of the feedback in an anonymised form.

I would also encourage anyone doing conference speaking to track their conference feedback in the same way that I am doing so that you get an insight into whether or not you are improving over time. It is not difficult to setup and maintaining it as you do more sessions takes very little effort.


Storing feedback data like this leads me to wider thoughts about well-known conventions and data format standardisation. Let’s imagine a utopia where there were a standard set of questions for capturing session feedback that were leveraged at every conference regardless of subject matter, location or culture; that would give rise to immense cross-conference and cross-discipline analysis – the data analyst in me goes giddy at the thought.

It is scenarios like this that drive my interest both in data formats such as iCalendar, microformats and RDF, and in emerging movements such as the semantic web and linked data, all things which I have written about in the past. I don’t know whether we will ever reach the stage where every piece of data has structured, descriptive and agreed metadata associated with it but I live in hope.

@Jamiet


UPDATE

Putting this data in the public domain is already paying rich dividends for me. Jen Stirrup (twitter), a whizz at data visualisation (data viz wiz????) here in London, has kindly done some further analysis on my feedback scores by downloading my spreadsheet and analysing it using a software package called Tableau. Here is the heatmap that Jen produced for me:

Jamie Thomson Split Session Feedback

I’m not trying to boast about my scores here, truly I’m not (but thank you to all those that were generous with their scoring – I am rather humbled), what I want to try and show is the power of data visualisation. In this one image Jen has managed to convey much more information than I did by merely providing averages; I can see the range of all the votes, how many votes there were, outliers and highest/lowest votes. Most of all I can see why my averages are changing, be they up or down. I now find myself asking questions and making observations like:

  • Ok, my scores improved in for “SSIS Dataflow performance Tuning” over “SSIS Nuggets Live”, but clearly more people scored “SSIS Nuggets Live” so which is more valuable?
  • Clearly my averages for “SSIS Nuggets Live” are negatively affected by some very low scores. That’s something that could be investigated further.
  • Which areas have more scope for improvement?
  • In both sessions highest marks were attained for “Speaker’s knowledge of the subject area”. This was not something I particularly noticed when looking at the raw averages. Visualisation makes it stand out so much more.

This is all really valuable information and will become more so as I hopefully do more presentations in the future. Thank you for doing this for me Jen, I am incredibly grateful for your work on my behalf.

If you would like to read more about the academic theory behind heatmaps like this be sure to check Jen’s recent post SQLBits Feedback Post-Mortem where she dissects her own SQLBits feedback in intimate detail, really interesting stuff that gives a flavour of Jen’s past and future presentation material.

Published Saturday, January 15, 2011 1:07 PM by jamiet
Filed under: , , ,

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

Gavin Payne said:

I found my feedback from SQLBits really useful and was careful not to over-analyse it, I wanted to get a good feel for whether I did bad, ok or well, although the comments also gave me constructive feedback for the future.  Having numbers and comments is really useful.

January 15, 2011 7:48 AM
 

Jen Stirrup said:

Hello Jamie,

I've produced a simple heatmap of my feedback data in Tableau, and if you like, I can make a heatmap of your data and send you an image across?

As you say, the format is identical, so it's no problem for me to wrap up your online data and put it into Tableau.

If you don't like heatmaps, that's ok, lots of people don't like them :) I'm fine either way. Here's the link so you can take a look if you are interested. http://goo.gl/BNYOr

And yes, I do spend my Saturday night doing this sort of stuff!

Jen

January 15, 2011 6:34 PM
 

Jen Stirrup said:

Hello Jamie,

I've produced a simple heatmap of my feedback data in Tableau, and if you like, I can make a heatmap of your data and send you an image across?

As you say, the format is identical, so it's no problem for me to wrap up your online data and put it into Tableau.

If you don't like heatmaps, that's ok, lots of people don't like them :) I'm fine either way. Here's the link so you can take a look if you are interested. http://goo.gl/BNYOr

And yes, I do spend my Saturday night doing this sort of stuff!

Jen

January 15, 2011 6:34 PM
 

Brent Ozar said:

Jamie - WOW, dude, I have to congratulate you on improving on every single measurement. That is one hell of an achievement.  Nice work.

I agree about capturing consistent questions from conference to conference - I'd really like that myself.  It would also help the audience, too.

January 16, 2011 10:33 AM
 

jamiet said:

Thanks Brent. I guess the more you practise something the better at it you become.

January 16, 2011 11:54 AM
 

Jen Stirrup said:

Hi, back again.

I've had a look at the data, which is fantastic for you as a presenter; all 8 and 9 ratings, which is great news!

As a thought, since you have so many 8/9 ratings, I wondered if you'd thought about using the median rather than the mean? Generally, if data is skewed, it is better represented by a median rather than the mean, because the mean is more sensitive to the value of extreme scores.

In other words, the couple of lower ratings you have (*not many*) pulls the overall mean down, but the median would be less sensitive to the outliers. I suppose what I'm saying is that maybe think about the median, might pull the score up even higher so you can see the 8/9s :)

Hopefully more feedback data available after you've done SQLBits 8 (I hope you get picked!)

Jen

January 16, 2011 5:26 PM
 

SSIS Junkie said:

Over the last year or so it will not have escaped my regular readers’ attention that much of my blogging

January 15, 2013 4:44 PM

Leave a Comment

(required) 
(required) 
Submit

This Blog

Syndication

Powered by Community Server (Commercial Edition), by Telligent Systems
  Privacy Statement