Assessment



img Key Features of Assessment Tools

Reliability: reflects the reproducibility of the assessment tool and the accuracy with which a score is being measured. It is higher in written assessments such as multiple choice and extended matching question formats, and lower in clinical competency-based assessments where there are more uncontrolled variables. Reliability is quantitative and reflected by the statistic known as Cronbach’s alpha. Evaluation using generalisability theory can be performed to account for complex variables.

Validity: reflects the accuracy with which a test measures what it is purported to measure. It is a qualitative factor that evaluates the authenticity of an assessment and its fitness for purpose. A number of categories of validity are described; for example, the content validity reflects the way in which the test items relate to the curriculum content being assessed, face validity refers to the ‘real life’ nature of the assessment and high construct validity suggests that the test discriminates well between the abilities of candidates.

Educational impact: assessment is an important driver of learning; appropriate assessment tools encourage learners to acquire the desired knowledge, skills and attitudes.

Cost-effectiveness: reflects the practical aspects of assessment and helps determine the choice of assessment tool.

Acceptability: successful assessment formats must be acceptable to the teaching faculty and the learners.

Blueprinting: ensures the assessment tool samples content across the full range of learning objectives for the curriculum.

Standard Setting


Numerous methods to determine pass-marks for different assessment formats are available.


Norm-referencing: in norm-referenced assessments the pass mark is determined by examiners using comparison within the cohort of examinees and thus the pass-mark varies at each sitting. A percentage of candidates will pass the assessment on each occasion (Fixed Percentage Method). Norm-referencing does not take account of the content of the assessment or the competence of the candidates.


Criterion-referencing: in criterion-referenced assessments the pass-mark is set in advance by a team of experienced examiners using their judgement about the degree of difficulty of the assessment and the minimum score expected of a candidate who just reaches the acceptable standard. A number of criterion- referenced standard setting methods are described including the Angoff and Ebel procedures.


Good practice for summative assessments in medical education demands that a minimum competence (safety) level should be set – the assessment should identify the Pass/Fail border and all candidates who reach the required standard should pass the examination. Assessments should thus be criterion-referenced by experienced examiners who recognise the standard required of the candidates at whatever level of undergraduate or postgraduate experience. Norm-referencing is not acceptable for high-stakes professional examinations.


Borderline group methods: these methods have been developed specifically for use in OSCE and similar formats where an experienced, trained clinician examiner is present at every station to score each candidate. In essence, each examiner scores the candidate using the station checklist – this constitutes the candidate’s score for that station. In addition, the examiner awards the candidate a global score, based on an overall judgement of performance. Global rating scales include a spread of judgements such as ‘fail – borderline fail – borderline pass – clear pass – outstanding’. The mean score of all candidates marked borderline becomes the pass-mark for that station and the mean of all the stations’ borderline scores becomes the pass-mark for the assessment. These methods have gained credibility as they allow experienced clinicians to make judgements about professional competence and they are currently the gold-standard methods for assessments of clinical competence.


Assessments in medical education fall into three main categories – those that measure knowledge, competence and performance. ‘Miller’s Triangle’ (Fig. 9.1) illustrates the relationship between these categories.



Figure 9.1 Miller’s Triangle: categories of assessment methods. Adapted from Miller GE (1990) The assessment of clinical skills/competence/performance. Academic Medicine 65 (Suppl): S63–S7.

img

Only gold members can continue reading. Log In or Register to continue

Stay updated, free articles. Join our Telegram channel

Mar 14, 2017 | Posted by in PHARMACY | Comments Off on Assessment

Full access? Get Clinical Tree

Get Clinical Tree app for offline access