MC exams

Zen and the Art of Multiple Choice Exams

Scantron scoring machines are fairly common, if not universal, on university campuses. Thus one can infer that multiple choice exams are equally widespread. But how do they stack up against other kinds of assessment, especially in classes where higher-level thinking (application, analysis, evaluation), and not mere memorization (recognition and recall), are emphasized?

Crafting good multiple choice (MC) stems (questions) and conceiving of plausible distractors (wrong answers) are skills in themselves. Yet, when artfully done, MC exams can be effective in testing higher-level cognitive abilities – it just takes time and effort.

It had dawned on me a few years earlier that it would benefit students to shoulder some of these question making responsibilities. Coming up with plausible (dare I say artful?) distractors is part of the practice I so enjoy about writing MC exams. I try to think of the reasons why a student may choose a wrong answer (is X conceptually close to Y?, is it easy to confuse X for Y?, is X actually the opposite of Y?, does X actually negate Y?, is X spelled similarly as Y?, and so forth), and include those as the distractors.

Indeed, through this very process of crafting good questions with good distractors I come to have a better understanding of the relationships between concepts, in other words, I built a strong network of reinforced meanings. This is precisely what I want students to do!

Thus, I decided for my Zen class to unveil a new extra-credit option: for student to craft five of their own MC questions following the criteria I set out for them. From my perspective this had the following benefits:

  1. Students would create their own web of meaning between conceptually similar or confusing terms. This was a coercive way to get them to study in a new manner.
  2. Students wold post their MC questions on an online forum, and thus would get the opportunity to take several practice tests.
  3. By selecting several of the best MC exams, students would feel they had a sense of agency in the learning process.
  4. It might save time on my end from having to craft so many MC questions.

Overall, about 50% of my class took me up on the offer. Many of the MC questions did not match my highest standards (but, honestly, not all of my questions are perfect!), but several were thoughtful and well-crafted and were incorporated into our exams.

Something I would consider doing in the future is to have students explain why they selected each distractor (laying out the relationships I noted above).

Taking multiple choice exams will never replace the value of writing well-reasoned prose, but having students write well-reasoned multiple choice questions is a step in the same direction.

Online Quizzes: Recap

I want to quickly review my use of online quizzes this summer, returning to a post I started here. As I noted, I use “low-stakes retrieval practices,” i.e. quizzes, regularly in my courses so students can assess their efforts and make any micro-adjustments to their study habits they deem necessary. Instead of diverting time in class to taking quizzes I decided to have students take quizzes at home online as part of their homework. Here are some quick thoughts.

  1. Practice Syllabus Quiz: Not too many students are familiar with taking quizzes at home, nor was I familiar with the mechanics of making them. I decided to give my students a syllabus quiz after the first day of class, making sure they understood the requirements of the course as well as trying to troubleshoot any issues that may arise when the real quizzes start. As enticement, I offered minimal extra credit for completing the quiz with 100%.
  2. Multiple Choice (MC) Only: Clearly, this will depend on an individual instructor’s educational goals, but I shifted from the norms of my paper quizzes. I would often add a few short answer questions to the end of in-class quizzes, but found that grading these online took a bit too much time. Clicking, loading, scrolling, typing, and saving took far longer than flipping a page. After including short answer questions to my first online quiz I decided to make all subsequent online quizzes just multiple choice so everything could be graded automatically. If this was not a time-condensed summer class I may have kept the short answers, but I looking to save some time overall with online quizzes.
  3. Open-Book/Timed: Honestly, I just don’t trust anyone to not use their notes or book when taking a quiz at home, so I just decided to make them open-book. To offset this a little, I made all the quizzes timed meaning that students would not be able to casually get 100% on every test just by flipping through their notes or my slides. I told them they need to study before starting the quiz. I gave them just over 1 minute per multiple choice question.
  4. MC Strategy I: Crafting good multiple choice questions can be a skill in itself, especially ones that test higher-level thinking (application, analysis, evaluation) as opposed just memory (recognition and recall).  The first question below is an example of simple recognition and the correct response (“B”) would (or should) be found word-for-word in students’ notes.Screen Shot 2017-09-16 at 11.57.40.pngThe second question combines recognition with application, asking the students to apply their knowledge (in this case, to real world examples). In addition to testing higher-level thinking, the correct answer could not be found word-for-word in students’ notes.Screen Shot 2017-09-16 at 11.57.48.png Because the main focus of my quizzes is to test simple recognition and recall (midterms and finals are different), I mix in only a handful of MC questions to test higher-level thinking. Yet, it is possible in some cases to transform lower-level cognitive MC questions into higher-level ones.
  5. MC Strategy II: While generally not regarded as a good strategy for crafting MC questions, I’ve found that using some “exception” questions also requires to students to know (recognize) more about a concept. Screen Shot 2017-09-16 at 13.00.18.png
  6. Testing Recall: Another method I did not employ this summer is to use “fill in the blank” questions that are automatically graded.

Overall, I will continue to develop my use of online quizzes. For me, the benefits of saving valuable class time and automatic grading offset the issues of limiting (or avoiding) short answer questions. 

A Question of Quizzes

Tayuan Temple Chanting.jpg

Buddhist monks chanting at Tayuan Temple 塔院寺, Mt. Wutai, China, summer 2016. Photograph Peter Romaskiewicz.

Grading is one of the toughest parts of teaching. It also gives you immediate feedback on how well students are grasping the materials. Thus I’ve come to find that frequent, low stakes assessment is helpful in preparing students for larger and more complicated tasks.

In my summer course I’m having quizzes each week which review the material from the previous week. I’ve decided that quizzes will mostly be multiple-choice for a few reasons. First, these assessments are making sure students are familiar with basic terms and themes, mostly focused on recognition and recall. These ideas form the basis for more analytical and critical writing assignments in the coming weeks. Second, because of the pace of the course (we meet four time a week), I cannot spend too much time grading. Perfect for multiple choice.

The new angle I am trying this summer is online quizzes. Thus, I am also attempting to make this class hybrid, as I expect to teach a form of it fully online in the near future. The students have to take the quiz before they attend Monday’s class. I allow them a 36 hour window to take the test, opening it Sunday morning. To prepare them, and myself, for this new experience, I offered an online quiz (for minimal extra credit) on the syllabus after the first day of class.

I decided on the online quiz after much thought. The main concern I had was missing 10-15 minutes of class for these quizzes. I found them to be useful, even necessary, for low-level learning, but they also ate into lecture and discussion time. In addition, I was  hoping that automatic grading would save me time throughout the summer session.

I decided that I would make the quizzes open book and open note. Perhaps it is poor judgement on my part, but I just do not fully trust my students to not use notes for an assessment like this (!). This this is a concern I have for online quizzes, especially of the multiple choice variety. To counterbalance this leverage, I decided to make the quizzes timed, instructing my students that they would need to study beforehand in order to answer all of the questions. My hope with this set-up is that the students would get used to the type of questions I would ask and potentially become familiar with the adequacy of the notes they are taking. (The midterm and final are in-class.) The first quiz was 10 multiple choice questions with two short answer questions. The total time I allowed for the quiz was 15 minutes, just about the time I would allow in class.

I will review the use of online quizzes at the end of the course.