MC exams

Zen and the Art of Multiple Choice Exams

Multiple choice exams are universal on college campuses. But how do they stack up against other kinds of assessment, especially in courses where higher-level thinking (application, analysis, evaluation), and not mere memorization (recognition and recall), are emphasized?

Crafting good multiple choice (MC) stems (questions) and conceiving plausible distractors (wrong answers) are skills in themselves. Yet, when artfully done, MC exams can be effective in testing higher-level cognitive abilities – it just takes time and effort in the creation process.

A few years ago it dawned on me that students could shoulder some of these question-crafting responsibilities to good pedagogical effect.

Coming up with plausible – dare I say artful? – distractors is part of the practice I so thoroughly enjoy about writing MC exams. I try to imagine the reasons why a student may choose a incorrect answer: is X conceptually close to Y?, is it easy to confuse X for Y?, is X actually the opposite of Y?, does X actually negate Y?, is X spelled similarly as Y?, and so forth. These questions develop the pool of candidates from which I select distractors that are, for the most part, plausible and rationally derived.

Indeed, through this very exercise of crafting questions with good distractors I’ve built a better understanding of the relationships between concepts and a better appreciation of where conceptual pitfalls might appear. In other words, this activity helped me build a network of reinforced meanings which allowed me to better understand the materials at hand. This is precisely what I want students to do!

Thus, I decided to unveil a new extra-credit option for my Zen Buddhism class: students can craft five of their own MC questions following the criteria I set out for them. Some may be used on our midterm and final exams. From my perspective, this exercise has the following benefits:

  1. Students create their own webs of meaning between conceptually similar ideas or confusing terms. Creating plausible distractors causes students to study in a new manner.
  2. Students post their MC questions on an online forum and thus have the opportunity to take student-created mini practice tests.
  3. By selecting a handful of the best questions for class MC exams, students feel they had a sense of agency in the learning process.
  4. It can save time on my end in crafting MC questions.

Multiple choice exams will never replace the value of writing sustained, well-reasoned prose, but having students write well-reasoned multiple choice questions is a step in the same direction.

Review

Overall, about 50% of my class took me up on the extra-credit offer. I provided a hand-out outlining the requirements of a good question and an “artful” set of distractors. I offered 5 extra credit points for five well-crafted questions and sets of answers. I retained the option to withhold extra credit points should a question-answer set not be handled with sufficient attention to the directions. All submissions met my basic criteria and all students received full extra-credit.

Many of the MC questions did not match my highest standards – but, honestly, not all of my questions are perfect! Several were thoughtful and well-crafted and thus were incorporated into our exams.

Would I do anything different in the future? I would consider having students explain why they selected each distractor, laying bare the relationships I noted above or other critical logic. If I wanted to take this creative exercise more seriously, I could require students to produce MC questions, perhaps first as drafts, and practice with them during class review sessions before exams.

Online Quizzes: Recap

I want to quickly review my use of online quizzes this summer, returning to a post I started here. As I noted, I use “low-stakes retrieval practices,” i.e. quizzes, regularly in my courses so students can assess their efforts and make any micro-adjustments to their study habits they deem necessary. Instead of diverting time in class to taking quizzes I decided to have students take quizzes at home online as part of their homework. Here are some quick thoughts.

  1. Practice Syllabus Quiz: Not too many students are familiar with taking quizzes at home, nor was I familiar with the mechanics of making them. I decided to give my students a syllabus quiz after the first day of class, making sure they understood the requirements of the course as well as trying to troubleshoot any issues that may arise when the real quizzes start. As enticement, I offered minimal extra credit for completing the quiz with 100%.
  2. Multiple Choice (MC) Only: Clearly, this will depend on an individual instructor’s educational goals, but I shifted from the norms of my paper quizzes. I would often add a few short answer questions to the end of in-class quizzes, but found that grading these online took a bit too much time. Clicking, loading, scrolling, typing, and saving took far longer than flipping a page. After including short answer questions to my first online quiz I decided to make all subsequent online quizzes just multiple choice so everything could be graded automatically. If this was not a time-condensed summer class I may have kept the short answers, but I looking to save some time overall with online quizzes.
  3. Open-Book/Timed: Honestly, I just don’t trust anyone to not use their notes or book when taking a quiz at home, so I just decided to make them open-book. To offset this a little, I made all the quizzes timed meaning that students would not be able to casually get 100% on every test just by flipping through their notes or my slides. I told them they need to study before starting the quiz. I gave them just over 1 minute per multiple choice question.
  4. MC Strategy I: Crafting good multiple choice questions can be a skill in itself, especially ones that test higher-level thinking (application, analysis, evaluation) as opposed just memory (recognition and recall).  The first question below is an example of simple recognition and the correct response (“B”) would (or should) be found word-for-word in students’ notes.
    Screen Shot 2017-09-16 at 11.57.40.png
    The second question combines recognition with application, asking the students to apply their knowledge (in this case, to real world examples). In addition to testing higher-level thinking, the correct answer could not be found word-for-word in students’ notes.
    Screen Shot 2017-09-16 at 11.57.48.png
    Because the main focus of my quizzes is to test simple recognition and recall (midterms and finals are different), I mix in only a handful of MC questions to test higher-level thinking. Yet, it is possible in some cases to transform lower-level cognitive MC questions into higher-level ones.
  5. MC Strategy II: While generally not regarded as a good strategy for crafting MC questions, I’ve found that using some “exception” questions also requires to students to know (recognize) more about a concept.
    Screen Shot 2017-09-16 at 13.00.18.png
  6. Testing Recall: Another method I did not employ this summer is to use “fill in the blank” questions that are automatically graded.

Overall, I will continue to develop my use of online quizzes. For me, the benefits of saving valuable class time and automatic grading offset the issues of limiting (or avoiding) short answer questions. 

   Relevant Pedagogy Posts

A Question of Quizzes & Low-Stake Assessment

Tayuan Temple Chanting.jpg
Buddhist monks chanting at Tayuan Temple 塔院寺, Mt. Wutai, China, summer 2016. Photograph Peter Romaskiewicz.
Introduction

Grading is one of the toughest parts of teaching. It also gives you immediate feedback on how well students are grasping the materials. Thus I’ve come to find that frequent, low stakes assessment is helpful in preparing students for larger and more complicated tasks.

In my summer course I’m having quizzes each week which review the material from the previous week. I’ve decided that quizzes will mostly be multiple-choice for a few reasons. First, these assessments are making sure students are familiar with basic terms and themes, mostly focused on recognition and recall. These ideas form the basis for more analytical and critical writing assignments in the coming weeks. Second, because of the pace of the course (we meet four times a week), I cannot spend too much time grading. Perfect for multiple choice.

The new angle I am trying this summer is online quizzes. Thus, I am also attempting to make this class hybrid, as I expect to teach a form of it fully online in the near future. The students have to take the quiz before they attend Monday’s class. I allow them a 36 hour window to take the test, opening it Sunday morning. To prepare them, and myself, for this new experience, I offered an online quiz (for minimal extra credit) on the syllabus after the first day of class.

I decided on the online quiz after much thought. The main concern I had was missing 10-15 minutes of class for these quizzes. I found them to be useful, even necessary, for low-level learning, but they also ate into lecture and discussion time. In addition, I was  hoping that automatic grading would save me time throughout the summer session.

I decided that I would make the quizzes open book and open note. Perhaps it is poor judgement on my part, but I just do not fully trust my students to not use notes for an assessment like this (!). This this is a concern I have for online quizzes, especially of the multiple choice variety.

To counterbalance this leverage, I decided to make the quizzes timed, instructing my students that they would need to study beforehand in order to answer all of the questions. My hope with this set-up is that the students would get used to the type of questions I would ask and potentially become familiar with the adequacy of the notes they are taking. (The midterm and final are in-class.) The first quiz was 10 multiple choice questions with two short answer questions. The total time I allowed for the quiz was 15 minutes, just about the time I would allow in class.

I will review the use of online quizzes at the end of the course.

  Relevant Pedagogy Posts