Multiple choice « PD4CS - Python programming, common student misconceptions, student projects

For many instructors, multiple choice questions are a popular option. They are easy to grade, and can often be easily aggregated for data analysis. This assumes that the questions are valid and reliable, which cannot be assumed if they are generated by individual instructors and used solely within their own classrooms. It simply isn’t something an instructor can guarantee, but one can work on writing questions that are of a high enough quality and that target an important component of the learning process.

A good practice before writing any multiple choice questions is to identify the concepts that you wish your students to demonstrate on the quiz or exam. Try to limit each question to a single concept. To give an example from a mathematics course, many teachers will include questions about solving equations of one variable that include fractions. When they evaluate the question, can they be convinced that the student had difficulty with solving the equation itself, or was it a lack of familiarity with the relationships expressed by fractions? If two concepts cannot be isolated, efforts should be made to vary the concepts that are being combined so that the overall picture can suggest meaningful information about how the student has mastered the material.

Once you know which concepts you want to test, you should focus on writing several questions for each concept. The more questions a student can answer on the same topic, the more confident you can be that their incorrect answers are the result of a lack of understanding and not a random mistake. There is no perfect amount of questions, but replication should provide more validity to your results. When constructing exams, it is best practice for the number of questions about particular concept to be in proportion to the importance of that concept (often based on the amount of class time devoted to it) in the course. In other words, if you spend a week on topic A but only two days on topic B, you should ask more questions about topic A than topic B.

A multiple choice question consists of a stem (the question itself) and the leaves (choices). One leaf will be the correct answer, and the others are known as distractors. It is important to make sure that each distractor is connected to a common mistake (i.e. a reason or misconception) made by students. Many educators aren’t sure what kinds of answers students will come up with, and so the alternate answers to their multiple choice items are simply randomly generated. This practice allows students to game the system without demonstrating understanding. If you don’t feel confident that you have fully understood how students are confused by a particular concept, a multiple choice item is probably not the best way for the concept to be addressed. Instead, use short answer questions to help you start to analyze common misconceptions among your students.

A poor example of a multiple choice item regarding conditional execution.

The concept tested by this question is conditional statements with multiple test clauses. The instructor wants to know whether a student understands that the conditional statements are evaluated in order, and that the result depends on the first Boolean relation that evaluates to true. Given the following code segment, what will be printed out to the screen when the code has been executed?

x = 11
if x % 3 == 0:
    print &quot;Alan Turing&quot;
elif x % 3 == 1:
    print &quot;Bill Gates&quot;
elif x % 3 == 2:
    print &quot;Claude Shannon&quot;
else:
    print &quot;Dennis Ritchie&quot;

A) Alan Turing
B) Bill Gates
C) Claude Shannon
D) Dennis Ritchie
E) Nothing will be output, the program has a syntax error

What should be noted about this question is that it doesn’t test the intended concept at all. This question really appears to test student understanding of the modulus operator, and maybe even their arithmetic skills. The mistakes made were in the choices of Boolean expressions to be evaluated, and then the way the leaves are displayed. If the intent was to check whether or not a student understood how the if-elif-else statements work, the leaves should reflect some of the common misunderstandings. One of these, as an example, is that every true statement in a conditional set is executed. A better presentation of the same question is shown below. Given the following code segment, what will be printed out to the screen when the code has been executed?

x = 11
if x == 11:
    print &quot;Alan Turing&quot;
elif x + 1 == 12:
    print &quot;Bill Gates&quot;
elif x + 2 == 13:
    print &quot;Claude Shannon&quot;
else:
    print &quot;Dennis Ritchie&quot;

A)	Alan Turing
B)	Alan Turing
	Bill Gates
C)	Alan Turing
	Bill Gates
	Claude Shannon
D)	Dennis Ritchie
E)	Nothing will be output, the program has a syntax error

This may not be a perfect set of leaves, as answers B and C both depict the same common misconception of students. Nonetheless, it gets closer to identifying whether or not a student understands how a conditional statement with multiple conditions executes. One common choice many instructors make with multiple choice is to insist that all stems have the same number of choices. More choices reduces the likelihood of a random guess being correct, but leaves without purpose may even give a student a context clue about an incorrect leaf or set of leaves that they can eliminate without demonstrating understanding. For instance, in the case above, a clever student that does not understand conditionals may recognize that B and C both represent situations that would occur if all correct statements were executed in the conditional, and thus eliminate them both (they both cannot be true). A better set of responses would select only one of the two (B or C) for use.

A typical computer science question asked in multiple choice format.

Here the intended concept to test is how a for-in structure works in Python, particularly at the boundaries.

How many times will the statement inside the following loop execute?

for i in range(0,5):
    print &quot;This many!&quot;

A) 0
B) 4
C) 5
D) 6
E) This loop will execute an infinite number of times

Here the stem and the leaves appear to be okay, but it can be argued that a clever student may simply select the middle value based on their knowledge that an instructor might be worried that they were “off-by-one”. That still requires some domain knowledge, so perhaps it is a reasonable question. That being said, it is just as easy for a student to be asked to answer this question in short answer form without worrying about selecting a bad set of leaves.

When composing multiple choice questions, here are a few good rules of thumb to follow to make sure that your questions are well-constructed:

Ask questions that get at one main idea or concept, and provide only one correct or “best” answer.
If the stem is not a complete sentence, put as much of the question as you can in the stem. Do not repeat words in the leaves that could be in the stem.
Use negatives sparingly, and if you do use a negative in the stem highlight it (underline or bold) to make sure students notice it. Do not use double negatives, i.e. a negative in the stem and in one or more of the answers.
When you can, keep alternate responses about the same length, and be sure that all responses are grammatically consistent with the stem.
Use consistent formatting when writing questions, i.e. indent answers the same amount, start each alternative on a new line, etc.

The ultimate takeaway of this section should be that multiple choice should not be used lightly, in spite of the advantages gained from grading ease. Assessment needs to be a thoughtful process, and without the ends to the means the amount of information gained will be limited.