Post-Examination Analysis

Multiple-choice examinations are typically marked by computer (example: MARKEX Program). This means that other types of computed data about your examination are also readily available (see attached example). Two important pieces of information are ‘item difficulty' and ‘point biserial'.

Item Difficulty

This is simply a question by ques­tion listing of the per­cent of students who got each question correct. All exams should have a range of item diffi­culties. Some ques­tions should be answered correctly by 90% of the students (or better). Others should be answered cor­rectly by only 35%. If all questions are answered correctly by 70% of the students the average for the examination will be 70% but very likely the same group of students will have answered each of the questions correctly, and the other 30% will be batting zero; probably not an accurate meas­ure of their knowledge.

If students have done more poorly than you anticipated or much better than other years, the item difficulty data can often assist you in determining whether the reason lies with the examination itself or a difference in students. If sim­ilar questions have been used other years you can compare their respective item difficulties. The item difficulty printout also tells you how many students selected each of the alternatives. If a distracter is never selected you may want to revise it or replace it with one which is more attractive. If a distracter is chosen almost as often (or more often) than the answer you should look at the question critically to see if it is misleading (or you may want to look at your lecture notes to see whether there is a more effective way of teaching this material). Knowing the item difficulty for a question, can often help when analyzing an individual student's examin­ation because it indicates how far off the average s/he is.

 

Point Biserial

This number represents a comparison between how often the top portion of the class answered a given question correctly and how often the bottom portion of the class answered it correctly. The higher the point biserial number, the better the question discriminates between the top and bottom of the class. When 80% or more of the students have answered a question correct­ly, the point biserial is often low because most of both the top and bottom of the class have got the question correct. The point biserial is not very important for these questions. For more diffi­cult questions, however, it becomes very import­ant. If the ques­tion is very difficult a low point biserial (i.e., the top half of the class does no better than the bot­tom half) is an indication that something may be wrong with the question. If all alternatives are chosen more or less equally, it probably means that the whole class was "just guess­ing". A review of the question may reveal ambiguous wording, emphasis on a very "picky" point, contradictions with other course material, or you didn't teach the material very well, or that they thought a crucial piece of information was "picky". When the item is difficult, the point biserial low and a distracter is favored over the answer there are very likely serious problems with the question or the teaching of the concept. Ambiguously worded questions, or content which conflicts with other information usually are to blame in these cases.

 

Use of Multiple-choice Completions to Analyze Student Problems

Analysis of a student's multiple-choice examinations can reveal some helpful indicators of the nature of their difficulties. When students do very much better on the "easier" format questions (one-choice completions, quantitative and functional relationships) than on the multiple-choice completions, this is often an indication that they are studying the material as facts without doing any analysis or synthesis. They know what you said but they haven't made sense of it for themselves. An analysis of the types of errors which are made on the multiple-choice completions format itself can also reveal the nature of the student's problems. For example if the student most frequently gets this type of MCQ wrong be­cause s/he includes distracters as answers (eg. says l, 2 and 3 are correct when only 1 and 3 are correct, or says 2 and 4 are correct when only 4 is correct, etc.) then the basic problem may be more one of confidence than lack of understand­ing. They may simply be willing to believe that they don't know all the information and hence willing to believe an incor­rect statement is right. When they fail to recognize correct answers (eg. they say 1 and 3 are correct when 1, 2 and 3 are correct, or they say only 1, 2 and 3 are correct when all are correct, etc.), however, the problem is one of lack of depth of knowledge. The student who says 1 and 3 are correct when 1, 2 and 3 are correct is, however, in a much better position than a student who says 1 and 3 are correct when actually 2 and 4 are correct. Teaching students how to analyze their own exams is well worth the effort. It will save you time and the act of examination-analysis is a very effective teaching tool and they will learn a great deal from it.