Written assessments

Chapter 37


Written assessments





Question format


Often a distinction is made between open-ended and closed question formats.



image Response formats: Closed questions require candidates to select answers from a list of options. Open-ended questions require candidates to generate answers spontaneously.


Multiple-choice items are often deemed unfit for testing higher-order cognitive skills (e.g. medical problem solving) because the correct answer can be produced by mere recognition of the correct option (the so-called cueing effect). From the literature comparing response formats, however, it has become clear that the response format (open-ended or closed) is not so important, but that the stimulus (what you ask) is essential.





Consider, for example, the following two questions regarding the response format:




and





In these two questions there is very little difference in what is asked, whereas the way the response is recorded differs.


Now consider these two questions regarding stimulus format:




and




In these questions the response format is similar, but the content of the question is completely different. In the second question the expected thought processes are completely different from those evoked by the first.



image Stimulus formats: The content of a question will determine the thought processes required to answer it.


For the rest of this chapter we will therefore use the distinction between response formats and stimulus formats.



Quality control of items


No matter which assessment format is used, the quality of the examination is always related to the quality of the individual items. Therefore, it is of no use discussing strengths, weaknesses and uses of various item types if we cannot assume that optimal care has been taken to ensure their quality. One aspect of the quality of an item is its ability to discriminate clearly between those candidates who have sufficient knowledge and those who do not. So they are a ‘diagnostic for medical competence’. This implies that when a student who has not mastered the subject matter answers a question correctly, this can be regarded as a false-positive result, and the opposite would signify a false-negative result. In quality control procedures, prior to test administration, the diagnosis and elimination of sources of false-positive and false-negative results are essential. In addition, the relevance of the items, the congruence between curricular goals and test content, and the use of item analysis and student criticisms are other important factors in quality control of assessment, but in this chapter we will focus on quality aspects of the individual items.



Response formats



Short-answer open-ended questions




When to use and when not to use


Open-ended questions generally take some time to answer. Therefore, fewer items can be asked per hour of testing time. There is a relationship between the reliability of test scores and the number of items, so open-ended questions can lead to less reliable scores per hour of testing time. Also, they need to be marked by a content expert, which makes them logistically less efficient and more costly. For these reasons the advice is to use short-answer open-ended questions preferably only if closed formats would not do.


A question such as:




is not very sensible. Theoretically, there are only two possible answers, so the content prescribes a two-option multiple choice.


On the other hand, a question like:




would be odd in a multiple-choice format.


In general, the advice concerning the use of short-answer open-ended questions is not to use them unless the content of the question requires it.



Tips for item construction


Use clearly phrased questions. Reading errors may not be what you want to test, and may thus be a source of error. Therefore, use short sentences and avoid double negatives. If in doubt as to whether to use more words for clarity, or fewer words for brevity it is best to opt for more clarity.


Make sure a well-defined answer key is written. Correct and incorrect answers must be clarified and possible alternative answers must be identified (perhaps by a panel of examiners) beforehand. This is especially necessary if more than one person will be correcting the test papers.


Make sure that it is clear to the candidates what kind of answer is expected. A disadvantage of open-ended questions may be that it is unclear to the candidates what kind of answer is expected, for example, about the required level of detail. As an illustration, for a question on chest infection, the expected answer could be pneumonia, bacterial pneumonia or even pneumococcal pneumonia. There may also be a lack of clarity about what type of answer is expected, as for example in:




Is the expected answer here that one is a mirror image of the other, the number of lobes, the angle of the main bronchus, the surface or the contribution to gas exchange?


Indicate a maximum length for the answer. Students often apply a ‘blunderbuss’ approach, which means that they will write down as much as they can in the hope that part of their response will be the correct answer. By limiting the length of the answer this can be prevented.


Use multiple correctors effectively. If more than one teacher is involved in correcting the test papers, it is better to have one corrector score the same item(s) for all students and another corrector score another set of items for all students than to have one corrector score the whole test for one group of students and another the whole test for another group of students. In the latter case a student can be advantaged by a mild corrector or be disadvantaged by a harsh one; in the former the same group of examiners will be used to produce the scores for all students, which leads to a better reliability with the same effort.



image For short-answer open-ended questions, it should be clear beforehand what would be correct answers and what would be incorrect answers.



Essay questions



Description


Essay questions are open-ended types of questions that require a longer answer. Ideally, they are used to ask the candidate to set up a reasoning process, to evaluate a given situation or specifically to apply learnt concepts to a new situation. An example is:




Of course, it is important that the situation of John and Jim is new to the students, and that it has not already been explained during the lectures or practicals. A question such as:




is therefore less suitable as an essay type of question. Although the knowledge asked may be relevant, the question asks for reproduction of factual knowledge, which is more efficiently done with other formats.



When to use and when not to use


It is essential to use essay questions only for specific purposes. The main reason for this is the lower reliability and the need for hand-scoring. Essay questions are therefore best used when the answer requires spontaneous generation of information and when more than a short text is required. Examples are:



• Evaluating a certain action or situation, for example:


    



• Application of learnt concepts on a new situation or problem, for example:


    


Related posts:

  1. Clinical skills centre teaching
  2. Postgraduate training
  3. Instructional design
  4. Basic sciences and curriculum outcomes

Stay updated, free articles. Join our Telegram channel
Tags:
Dec 9, 2016 | Posted by in GENERAL & FAMILY MEDICINE | Comments Off on Written assessments

Full access? Get Clinical Tree
Get Clinical Tree app for offline access