Standard Setting
Numerous methods to determine pass-marks for different assessment formats are available.
Norm-referencing: in norm-referenced assessments the pass mark is determined by examiners using comparison within the cohort of examinees and thus the pass-mark varies at each sitting. A percentage of candidates will pass the assessment on each occasion (Fixed Percentage Method). Norm-referencing does not take account of the content of the assessment or the competence of the candidates.
Criterion-referencing: in criterion-referenced assessments the pass-mark is set in advance by a team of experienced examiners using their judgement about the degree of difficulty of the assessment and the minimum score expected of a candidate who just reaches the acceptable standard. A number of criterion- referenced standard setting methods are described including the Angoff and Ebel procedures.
Good practice for summative assessments in medical education demands that a minimum competence (safety) level should be set – the assessment should identify the Pass/Fail border and all candidates who reach the required standard should pass the examination. Assessments should thus be criterion-referenced by experienced examiners who recognise the standard required of the candidates at whatever level of undergraduate or postgraduate experience. Norm-referencing is not acceptable for high-stakes professional examinations.
Borderline group methods: these methods have been developed specifically for use in OSCE and similar formats where an experienced, trained clinician examiner is present at every station to score each candidate. In essence, each examiner scores the candidate using the station checklist – this constitutes the candidate’s score for that station. In addition, the examiner awards the candidate a global score, based on an overall judgement of performance. Global rating scales include a spread of judgements such as ‘fail – borderline fail – borderline pass – clear pass – outstanding’. The mean score of all candidates marked borderline becomes the pass-mark for that station and the mean of all the stations’ borderline scores becomes the pass-mark for the assessment. These methods have gained credibility as they allow experienced clinicians to make judgements about professional competence and they are currently the gold-standard methods for assessments of clinical competence.
Assessments in medical education fall into three main categories – those that measure knowledge, competence and performance. ‘Miller’s Triangle’ (Fig. 9.1) illustrates the relationship between these categories.