5 Common Quiz Question Mistakes (And How to Avoid Them)

Kia ora Tom!

It’s like I said to
Tony Karrer: When it comes to learning, the diversity can be as wide. Present day examination (standards) techniques tend to accept in a narrow band of interpretation/regurgitation and so this diversity is often looked on as ineffective learning. In fact, I believe that a lot of it can be put down to diverse interpretation.

Ka kite
from Middle-earth

[…] 5 Common Quiz Question Mistakes (And How to Avoid Them) | The Rapid E-Learning Blog | Tom Kuhlmann | 28 October 2008 […]

Thanks for a great post, as always, Tom! I work in a regulatory field and we are often forced to add questions to our training, regardless of whether it makes sense to test on the material. Grrr! I’ve learned many of the things you mention along the way, based on mistakes my colleagues and I have made. Nothing makes a trainee shout louder than a double negative in a test question!

As instructional experts, we have academic “recipes” for test question & answer formatting but unfortunately it often leads to overly complicated questions and tricky answers. The end result is that the trainee has learned nothing and is even more resentful of training. My motto is to be thoughtful and keep it simple!

Ooh, pick me, pick me!

My all time favourite was a list of questions submitted by the subject matter for inclusion in an assessment quiz, which included the question “Do you know where to find the first aid box?” The answer options were “yes” and “no”.

I kid you not!

Love your posts. I have one comment about using True/False questions. My high school valedictorian, one of the smartest men I’ve known, would ace every part of tests . . . except the T/F questions. I didn’t think much of it until in grad school when one of my professors discouraged us from ever using True/False questions. Why? He believed they penalized the brightest students, who read more into the questions than intended. Also, he believed few things in life are 100% true or 100% false, which makes it difficult to write an effective T/F question. I haven’t seen much hard data in this area, but how he explained it has convinced me never to use T/F questions. I’ll present users with two reposes, but use more specific options, allowing them to choose the “best” option instead of the “true” or “false” option.

Keep up the great work!

How about the “instructor” who has only 3 questions at the end of a course with six lessons! It goes back to setting your learning objectives, making sure your lessons “teach” the content effectively, and then developing a quiz that really measures something.

And, something I learned recently (after 20 years in the computer-based training field – you can teach an old dog new tricks:) is that end of course quizzes don’t measure anything other than what they’ve retained in the last hour. According to David Sousa in “How The Brain Learns”, research on retention indicates that 70 to 90% of new learning is forgotten within 18 to 24 hours after the lesson. If we’re not testing for retention why are we testing at all? However, the real value of course quizzes may be that they are just another way for the learner to practice or rehearse new information and skills, that might help them retain information after they’ve turned off their computer.

Good overview article. The one point on which I would disagree is your recommendation that True/False items be included. True/False items tend to be either too confusing, or too easy, and in most cases they don’t relate to the desired performance and behavioral outcomes of the course. In most workplace scenarios, the learner will not need to make a “true-false” decision when presented with information; instead, the learner is likely to need to either compose a solution independently (a parallel to a short answer response) or to choose from several available options (a parallel to a multiple choice response).

I totally agree with the comments in the article. One thing I would add is that it is always a good idea to test run your quiz by a few folks who have reviewed the content but are NOT subject matter experts. It’s hard for us to unlearn what we know and questions that may make sense to us, don’t always make sense to our audience. Things like company speak, acronyms and other baddies invariably creep in and a beta run will always flush out the duds.

I disagree with you logic of the cutting of the bread. You want to know how many slices you can get from a loaf that is 25 cm long, if each slice is .5 cm. If you sliced the bread into the number of correct slices and put it back together again, you would still have approx 25cm (OK allowing for crumbs). Of course you are changing the size of the bread everytime you make a cut, but that was not your question. JMHO.

Nifty, shared with my Nursing faculty.

>>the real value of course quizzes may be that they are just another way for the learner to practice or rehearse new information and skills. . . .

Good point, Lisa. This, and more realism, is why branched scenerios are often better choices than multiple choice, T/F, etc. quizzes. They reinforce the learning in a fashion that is more similar to how the learner will use it.

As to T/F questions, I was taught to avoid them when possible because all too often the only thing they measure is the learner’s ability to play the odds, especially if they can use POE (process of elimination) to tip the odds in their favor.

–Allen

[…] article on typical mistakes creating e-learning questions, which appeared on Tom’s Rapid Development E-Learning Blog, included a link to this nifty […]

MSome of my test pet peeves: One SME submitted material followed by a set of T/F questions in which every answer was true! She simply didn’t seem to be able to phrase any false statement about her material. Another SME submitted a test with T/F and multiple choice questions that were almost verbatim out of a policy manual. As “feedback,” she requested that we restate the policy verbatim, even if it was 3-4 paragraphs long. One more: An external vendor we sometimes work with includes tests in which any time the choice “All of the above” appears as a multiple choice option, that is always the correct response. It never appears otherwise.

In my undergrad classes I remembered studying about designing educational assessments and measures, and one of the big ideas that has remained in my head from those many years ago is: “Begin With The End In Mind” (right about the time Covey’s 7 Habits came out).

The point was this: after you’ve identified the expected outcome(s) from the course or lesson, you carefully design your measurement for success, whether it is a quiz, an essay, or some other type of authentic assessment. Only after completing these steps would you begin to design the learning lesson, using the expected outcomes and the evaluation as your guide. Then, simply design lessons which lead the learner toward success.

It may seem overly simplistic: “teach to the test”, but instructional design masters (and proponents of Mastery Learning) will understand that there is a lot of power in this simplicity. I go could on and on about this, but I’ll stop here for now. Thanks Tom!

My comment relates to True/False questions encountered on a recent self-paced lesson on Violence in the Workplace that all employees are required to take. The lesson included a narrator reading all the text that was on the screen, except the T/F questions. You had to read the T/F question, select an answer, and click the “next” or “forward” button. The next screen immediately had the return of the narrator reviewing the answer of the previous T/F question: “This is false.” the narrator would say and proceed to explain the reasons why the correct answer to the T/F question was “false”. But all the while, I was thinking the narrator was telling me my response was false or wrong. I could no longer see the question or how I answered it, and so I wasn’t sure how I had answered while I’m listening to a long-winded explanation from the narrator. The True/False questions in this lesson, combined with the narrator and the explanation of the answer, no matter how you responded, was not effective in helping the learner recognize whether they had understood the material. It was a very poor self-check design.

I have to agree with Allen. There are methods one can use to measure the validity of an assessment question. A highly valid question would be one where, statistically, the Learner would have the best chance to score it correctly only if he or she actually knows the correct answer. Even better are questions structured in such a way that they simulate the actual work the Learner will be doing!
I am certainly not able to assess my Learners’ understanding of the material if they have a 50/50 chance of getting the question right!
As for reducing the number of possible answers, that only raises the statistical odds of guessing correctly. More possible answers are better – though only if those choices make sense. (It goes back to your point about getting more raw data from your SME). Don’t you just love questions like:

How many eggs are in a package of one dozen?
A. 4
B. 12
C. Red
D. None of the above
E. All the above

Thanx, as always, for facilitating these great discussions
=Chuck

Hello! Great post, I do like the Elvis and Snail questions, as a wordy man, those are real pitfalls for me.

I am just wondering what you made the quiz in? Maybe I’ve not plumbed the depths of Articulate Quizmaker, but it doesn’t look like what I get out of my application….

Kind regards,
Kev

Interesting points coming up in a few of these responses that put me in mind of a 9 year old kid I knew who was furious when she only got 19/20 on a school test.

The test consisted of statements that she had to assess as being certain, likely, unlikely or impossible.

The question she got wrong was “I will die someday.” She answered “likely”. The teacher marked this wrong, saying that the correct answer was “certain.” The child was adamant that “likely” was the correct answer because, as a practising Christian, she believed in the Second Coming and had to allow for the possibility, however small, that this might happen during her lifetime.

Tough one.

As someone has already indicated, some of the most facile questions can be interpreted more deeply than was the assessor’s intention.

IDespite the fact that most people are educated, I still see the occasional double negative, which particulaly confuses the learner who is taking a true/false quiz.

Know your audience is also still important when designing a test. I was once delivering training and an assessment (I was not the designer) to a group of adult learners who would not have been getting this particular job if they were brilliant test takers in the first place. Every time the correct multiple choice answer was “All of the above” they got it wrong. Since this training was regulatory, wrong answers had to be remediated. During discussions, I realized that they read the question, saw that the first answer was right and selected it. They’d never learned to read all the choices before making their selection. This is not a skill the course was designed to teach, but it caused lots of unecessary remediation. I now try to avoid “all” or “none” types of choices.

Thanks for the input on “trick” questions. they annoy the heck out of me! I just took a training (for certification even) on a LMS and the quizzes were full of trick questions.

Tom,

I’m new to eLearning and this type of development. What tools did you use to create the Snail graphic?

Flash, Captivate . . . ?

Thanks

I was frustrated by the example “who was the first U.S. President.” After playing the videos, there was no indication on how to choose the answer. However, selecting Submit told me I needed to select. So that was a good indication of the importance of providing clear instructions.

Thanks for the post. I’ll share with my team. The new quizmaker looks good.

Every now and then I see what you send and think that is great, but I have no idea how to do that, the common mistakes material was eye opening to say the least. Can the next post be a how you did it or at least a few clues (Please)

I like the Einstein drag & drop question. But that’s survey question right? So no way to score or give feedback on right/wrong answer?

[…] Since the blank slide is empty, you can pretty much do what you want. Here are three ideas that work for issues common to a lot of courses. You can see all of the ideas in action in this demo that I did for the Rapid E-Learning Blog on 5 Common Question Mistakes (And How to Avoid Them). […]

thank you . iam waiting for your new letter

[…] 5 Common Quiz Question Mistakes (And How to Avoid Them) […]

[…] 5 Common Quiz Question Mistakes (And How to Avoid Them) […]

[…] There are probably some favorite blog posts you like to save too, like this one on getting past click and read elearning, this one on creating perfect screenshots to use in your course, or this one on common quiz mistakes. […]

[…] […]

I am loving this Tom.

[…] This is a great example of how you can use Articulate Quizmaker's slide view feature to break out of that typical looking form-based quiz.  You can learn more about this quiz from the blog post, 5 Common Quiz Question Mistakes (And How to Avoid Them).  […]