THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

Buck Woody

Carpe Datum!

Session Evaluations

I do a lot of public speaking. I write, teach, present and communicate at many levels. I love to do those things. And I love to get better at them. And one of the ways you get better at something is to get feedback on how you did.

That being said, I have to confess that I really despise the “evaluations” I get at most venues. From college to technical events to other locations, at Microsoft and points in between, I find these things to be just shy of damaging, and most certainly useless. And it’s not always your fault.

Ouch. That seems harsh.

But let me ask you one question – and be as honest as you can with the answer – think about it first:

“What is the point of a session evaluation?”

I’m not saying there isn’t one. In fact, I think there’s a really important reason for them. In my mind, it’s really this: To make the speaker / next session better.

Now, if you look at that, you can see right away that most session evals don’t accomplish this goal – not even a little. No, the way that they are worded and the way you (and I) fill them out, it’s more like the implied goal is this: Tell us how you liked this speaker / session. The current ones are for you, not for the speaker or the next person. It’s a popularity contest.

Don’t get me wrong. I want to you have a good time. I want you to learn. I want (desperately, oh, please oh please) for you to like me. But in fact, that’s probably not why you went to the session / took the class / read that post. No, you want to learn, and to learn for a particular reason. Remember, I’m talking about college classes, sessions and other class environments here, not a general public event.

Most – OK, all – session evaluations make you answer the second goal, not the first. Let’s see how:

First, they don’t ask you why you’re there. They don’t ask you if you’re even qualified to evaluate the session or speaker. They don’t ask you how to make it better or keep it great. They use odd numeric scales that are meaningless. For instance, can someone really tell me the difference between a 100-level session and a 200-level one? Between a 400-level and a 500? Is it “internals” (whatever that means) or detail, or length or code, or what? I once heard a great description: A 100-level session makes me say, “wow - I’m smart.” A 500-level session makes me say “wow – that presenter is smart.”

And just what is the difference between a 6 and a 7 answer on this question:

How well did the speaker know the material? 1  2  3  4  5  6  7  8  9  10

Oh. My. Gosh. How does that make the next session better, or the speaker? And what criteria did you use to answer? And is a “10” better than a “1” (not always clear, and various cultures answer this differently). When it’s all said and done, a speaker basically finds out one thing from the current session evals:


“They liked me. They really really liked me.”


Or, “Wow. I think I may need to schedule some counseling for the depression I’m about to go into.”

You may not think that’s what the speaker hears, but trust me, they do. Those are the only two reactions to the current feedback sheets they get. Either they keep doing what they are doing, or they get their feelings hurt. They just can’t use the information provided to do better. Sorry, but there it is.

Keep in mind I do want your feedback. I want to get better. I want you to get your money and time’s worth, probably as much as any speaker alive. But I want those evaluations to be accurate, specific and actionable. I want to know if you had a good time, sure, but I also want to know if I did the right things, and if not, if I can do something different or better.

And so, for your consideration, here is the evaluation form I would LOVE for you to use. Feel free to copy it and mail it to me any time. I’m going to put some questions here, and then I’ll even include why they are there. Notice that the form asks you a subjective question right away, and then makes you explain why. That’s work on your part.

Notice also that it separates the room and the coffee and the lights and the LiveMeeting from the presenter. So many presenters are faced with circumstances beyond their control, and yet are rated high or low personally on those things. This form helps tease those apart.

It’s not numeric. Numbers are easier for the scoring committees but are useless for you and me. So I don’t have any numbers. We’re actually going to have to read these things, not put them in a machine. Hey, if you put in the work to write stuff down, the least we could do is take the time to read it.

It’s not anonymous. If you’ve got something to say, say it, and own up to it. People are not “more honest” when they are anonymous, they are less honest. So put your name on it. In fact – this is radical – I posit that these evaluations should be publicly available. Forever. Just like replies to a blog post. Hey, if I’m an organizer, I would LOVE to be able to have access to specific, actionable information on the attendees and the speakers. So if you want mine to be public, go for it. I’ll take the good and the bad.

Enjoy.

-------------------------------------------------------------------------------------------------------------------------------------------

Session Evaluation – Date, Time, Location, Topic

Thanks for giving us your time today. We know that’s valuable, and we hope you learned something you can use from the session. If you can answer these questions as completely as you can, it will help the next person who attends a session here.

Your Name:

What you do for a living:
(We Need your background to evaluate your evaluation)

How long you have been doing that:
(Again, we need your background to evaluate your evaluation)

Paste Session Description Here:
(This is what I said I would talk about)

Did you like the session?                     No        Meh        Yes
(General subjective question – overall “feeling”. You’ll tell us why in a minute.) 

Tell us about the venue. Temperature, lights, coffee, or the online sound, performance, anything other than the speaker and the material.
(Helps the logistics to be better or as good for the next person)

1. What did you expect to learn in this session?
(How did you interpret that extract – did you have expectations that I should work towards for the next person?)

2. Did you learn what you expected to learn? Why? Be very specific.
(This is the most important question there is. It tells us how to make the session better for someone like you.)

3. If you were giving this presentation, would you have done anything differently? What?
(Helps us to gauge you, the listener, and might give us a great idea on how to do something better. Thanks!)

4. What will you do with the information you got?
(Every presenter wants you to learn, and learn something useful. This will help us do that as well or better)

 

Published Wednesday, June 15, 2011 12:42 PM by BuckWoody
Filed under:

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

Aaron Bertrand said:

Buck Woody ( blog | twitter ) just published a great post on session evaluations , and a lot of his points

June 15, 2011 3:58 PM
 

merrillaldrich said:

I love it. Great ideas!

June 15, 2011 10:25 PM
 

David Wimbush said:

I like it. Just one thing: I'm not sure what 'Meh' means. Is it 'yes and no'? Not really? (Maybe it's just because I'm English!)

June 16, 2011 7:16 AM
 

Adam Machanic said:

I disagree about people being "less honest" when they're anonymous. I think anonymity gives people the freedom they need to be harsh when necessary, and to not be nice if the situation doesn't warrant it.

For example, if I know a speaker and my name is on the eval I'm probably more likely to say "great job, Buck!" But if my name is not on the eval and the speaker flops I'll feel much more comfortable saying "as usual, Buck totally messed up this talk and he really should never again be given the chance to stand in front of an audience."

Anonymity has been well established as a vehicle for honesty in the world of food criticism. I see nothing different about presentation criticism.

June 16, 2011 10:19 AM
 

Adam Machanic said:

I'm also not sure it matters whether people are "qualified" to judge. I'm not even sure what that means. Everyone has a right to an opinion, and as a member of an audience you're either going to like the talk or not. No one, aside from me, can tell me what I'm going to like or dislike. And this means that I am the ONLY person qualified to give my own opinion. The same is equally true of everyone else. If you were in the audience at all, you're automatically qualified to give feedback.

June 16, 2011 10:32 AM
 

Buck Woody said:

Adam - try this little experiment for me: take two employees who are angry at each other, one at a time, to explain the issue. Record the responses. Then, repeat that same experiment, only this time they are both in the room at the same time. Record the responses. Then tell me which is closer to the truth. Sorry, there is just too much evidence that when people are allowed to hide behind anonymity, they are more likely to exagerate. If you're willing to make the comment, put on the "big boy pants" and sign the name. Big and bold, like on the Constitution.

June 16, 2011 10:33 AM
 

Buck Woody said:

Oh - Deavid - yeah, "Meh" is the sound you make when you neither love something nor hate it. It's just "there."

June 16, 2011 10:33 AM
 

Adam Machanic said:

Buck: You've found an extreme counter-example. And I've already mentioned one for you (a speaker with whom you're friends, who doesn't do a very good job).

The reality is somewhere in the middle, and in most cases anonymity probably neither helps nor hurts -- most attendees at any given talk don't personally know the speaker. But anonymity gives people an added bit of comfort knowing that they can be as honest as necessary without any repercussion. Even if a small number will take it as an opportunity to exaggerate.

It sucks to get harsh criticism--especially undue harsh criticism--from someone whose name you don't have. We've all been on the receiving end. But I'd rather get that bitter pill from time to time than force people into submission by making them identify themselves.

June 16, 2011 10:57 AM
 

Buck Woody said:

Adam - completely understand. And I'll take whatever criticism is dished out - justified or not. No problem there at all.

Again, I just have to point back to: "what is the point of the eval?" If it doesn't make things better, you're just whining or praising. Good. We all feel better now. To what end? To me, there has to be a point.

Anyway, it really doesn't matter. Because this approach would require work from organizers, speakers and attendees, there's little danger it would *ever* get put into place. The ranting, pointless evals we have now are safe. :)

June 16, 2011 11:08 AM
 

Joe Fleming said:

I think I agree with Buck here.  If you're friends with the speaker and can't bring yourself to offer a productive critique of their style, then you probably need to re-evaluate that friendship.  You don't have to offer *harsh* criticism, just something productive.  

By putting your name on there it also allows the speaker to seek you out to clarify your feedback.  It allows the speaker to clarify vagaries in the evaluation.  It would also help identify the grumpy old man who is rating every presentation low because his underwear is bunching up on him.

While I agree that there are people who are unqualified to rate sessions, I think that there is no useful way of identifying the unqualified people (short of comments that out them as incompetent).  Just because I'm a .NET developer doesn't mean I don't have deep SQL knowledge to rate a session with, and similarly just because I'm a DBA that doesn't mean I am unqualified to rate a .NET session.

I think an "I liked it/I didn't like it" eval has some use from the organizer's perspective.  If a speaker consistentley gets poor ratings, it's time for hm/her to be dropped.  Similarly, if a speaker consistently gets good ratings, obviously he/she can be a draw for future events.

Unfortunately, I think that trying to make people put in effort to evaluate sessions is a lost cause.  If they are willing to take the time to do it, they will, and if they aren't, you're going to get a crappy evaluation with more words on it.  That doesn't mean it isn't worth the effort to try to make evaluations a little more valuable, because you will covnert some that way.  But overall, you have to allow for the people who are just as likely to toss the eval in the trash.

June 16, 2011 11:28 AM
 

AaronBertrand said:

Joe, your last point is huge. I agree that some people are just not going to be helpful, but I think we can enable some attendees to be A LOT more helpful.

June 16, 2011 11:55 AM
 

Marco Russo said:

I have a long experience of evaluating session evaluation scores. Always anonymous. Sometime I disagree on feedback for a specific session (not considering mine but others I attended). But I respect the numbers. They never lies. You know whether a session was good or not reading the numbers. But to improbe your session you definitely need the feedback of other speakers Who attended your session. I don't think a number (or a form) can really help you to improve if not for the motivation.

Marco

June 16, 2011 12:00 PM
 

andyleonard said:

I agree with Buck. I also agree with Adam.

With Buck: Evaluation forms need to be better-designed and their purpose explained to those providing presenter / experience / environment information. Raw numbers in a vacuum stink. Context counts. I also like the idea of public evaluations and utilize speakerrate.com for this reason.

With Adam: Anonymous feedback is useful for the reasons Adam mentions, but it is not useful for evaluation forms in my opinion. Anonymity belongs to forums for community / event feedback. I think having people provide their name is good for reasons Buck mentioned, but if someone had a really poor experience Adam is correct: they're more likely to tell you about (and you'd rather know than not know, wouldn't you?) if identification is opt-in.

:{>

June 17, 2011 12:46 PM
 

Bobby D said:

Two quick things to note:

1. As an event organizer and attendee (and perhaps speaker down the road) it is essential that the speaker evals are quick and easy.  That having been said, number systems just do not work for me.  I would prefer the more standard, "Greatly Disliked to Greatly Liked" scale.  For analysis purposes, there is nothing wrong with converting the answers to numbers.  I rarely have time at the end of a session to write what I really think on the forms that are handed out.  I do my best, but honestly, how can I truly express my experience in a sentence within a minute?  Which leads me to my second point:

2. Speaker evals are a fantastic opportunity to network!  When I attend a conference, I keep private notes on each talk that I see and what I thought about their presentation (including questions, compliments and criticisms).  I make it a point to shake the speaker's hand and thank them for taking the time to make the presentation.  After the event, I open up gmail and take the time to send a note to each speaker that I saw including all of my comments/questions/etc.  10 times out of 10 I get a response from the speaker, a new follower in twitter, a new professional resource, and best of all, a new friend.

June 20, 2011 9:23 AM
 

Sal Young said:

I agree with Buck’s ultimate reason for a session evaluation which should be “to make the speaker / next session better”.   The evaluations with number ratings tell nothing, they are a failed process from the get go.  

If you want the truth and able to handle the truth, I have a couple of suggestions:

-Nobody is more critical of you than yourself, so film your presentation and review it later when you get your session evaluations -you may experience an “aha!” moment.

-Create an eForm with the questions (Buck’s list is pretty good) and give something away like a hard drive or $25 gift cert. to the 25th evaluator.  For sure you’ll get a name if that is what you are looking for (Buck).

As an evaluator, you do not need to be harsh to be honest. If you explain why you feel he should keep his day job and stop the nonsense of public speaking, he may realize that basket weaving was his calling in life.

Bobby D is a nice approach when giving feedback to a speaker.

June 23, 2011 5:03 PM
 

Jason Strate said:

Nice, going to talk to my chapter about adopting some of this.

July 23, 2011 2:14 AM

Leave a Comment

(required) 
(required) 
Submit

About BuckWoody

http://buckwoody.com/BResume.html

This Blog

Syndication

Powered by Community Server (Commercial Edition), by Telligent Systems
  Privacy Statement