Improving workplace based assessments
Authors: Bill Burr, Winnie Wade, Hannah Watts
Publication date: 31 Jul 2012
Bill Burr and colleagues outline their 12 month pilot of a new system for workplace based assessments, starting this month
A major pilot is starting this month to test a range of new workplace based assessments for core medical trainees and higher specialty trainees.
Workplace based assessments were originally intended as formative educational assessment tools to provide structured feedback and to aid the development of trainees, but in practice they are often not used for this purpose and instead are sometimes used summatively (that is, to inform the assessment of trainees at their annual review of competence progression (ARCP)). Trainees and supervisors say that there are too many mandatory assessments; for example, between August 2008 and August 2009 foundation trainees at 24 of the 25 UK foundation schools did a total of 249 564 clinical assessments and 193 338 multisource feedback appraisals. In his review of the foundation programme John Collins, visiting professor in the Nuffield Department of Surgery at Oxford University, commented, “The number of assessments required is formidable.” In addition, the current system is often described as a tick box exercise that does not reflect trainees’ performance.
The revised system will result in fewer assessments, at appropriate times through the training year, with emphasis on feedback and reflection, as recommended by the General Medical Council, the Academy of Medical Royal Colleges, and the Conference of Postgraduate Medical Deans.
The Joint Royal Colleges of Physicians Training Board, in partnership with the education department of the Royal College of Physicians of London, is to run the 12 month pilot of this new system (see box). The board sets and maintains the standards for medical specialist training in the United Kingdom on behalf of the three Royal Colleges of Physicians in the United Kingdom, including writing the curriculum and developing assessment methods for core medical training and each of the medical specialties.
About the pilot
The pilot is taking place in 10 specialties and in three deaneries: East Midlands North, Northern, and Wales. The General Medical Council has agreed that participants will undertake only the new assessments and will not be required to use the existing approach to workplace based assessment.
In these three deaneries all trainees, trainers, and training programme directors in the following specialties will be taking part:
Acute internal medicine
Core medical training
General internal medicine (for trainees who are doing a dual certificate of completion of training with one of the specialties above).
The main features of the new system are as follows:
Workplace based assessments will be redesignated as supervised learning events (SLEs). These will be formative educational tools—that is, assessments for learning—which will be confidential to trainees and their supervisors and will not be seen by the ARCP panels.
The use of assessments of performance (AoPs) will be examined. These are summative assessments—that is, assessments of learning—and will be few in number but will count towards progression and will be considered by the ARCP panels.
AoPs will sample from a set of “key competencies” within each specialty curriculum.
The tick box appearance of the current workplace based assessment forms has been replaced with free text boxes that we hope will encourage more feedback from supervisors.
A more robust system for sign off of practical procedures has been developed. For potentially life threatening procedures, trainees will now require sign off by two different consultants.
A system to capture the opinions of several clinical supervisors has been developed that will feed into the end of placement educational supervisor’s report and be available to ARCP panels. We hope that having access to the opinions of several supervisors will enable educational supervisors to form balanced judgments of trainees’ performance on the basis of observation and evidence of performance in the workplace and engagement in the educational process. Such an approach should help prevent any one person having undue influence over a doctor’s progression.
There are two particularly controversial aspects of the new system. The first is that we believe that the formative nature of SLEs can be sustained only by insisting that they be absolutely confidential to the trainee and his or her educational and clinical supervisors. In particular, the outcomes of SLEs will not be available to ARCP panels. This means that it will not be possible to sign off multiple curriculum competencies through the SLE system, as has become the practice in many physician specialties, so we will also be emphasising the place of sampling from key competencies to assess trainees’ performance.
The second controversial aspect concerns the AoPs, whose use in training and assessment has been proposed by the General Medical Council. As mentioned, these are summative, will count for progression, and will be trainer led. We have an open mind as to what their eventual role will be. They may not be suitable for all specialties and all years and could ultimately have a role in supporting the assessment of trainees whose progress is causing concern.
Evaluation will be carried out throughout the pilot. Quantitative and qualitative data will be collected from questionnaires to trainees, clinical and educational supervisors, and training programme directors, with a focus on feasibility and acceptability. Focus groups will also be convened to provide feedback, and data will be gathered from e-portfolios.
In the longer term, the impact of this new approach on the numbers and outcomes of assessments, the quality of feedback, and overall educational outcomes will be assessed. Depending on the pilot’s outcomes, the intention will be to roll out the new system for all Joint Royal Colleges of Physicians Training Board specialties in August 2013. An interim report on progress will be produced in April 2013.
In making and piloting these changes to the workplace based assessment system, we hope that we will have dealt successfully with the major criticisms of workplace based assessments while preserving the important educational benefits that followed their introduction.
For more information about the Joint Royal Colleges of Physicians Training Board and the workplace based assessment pilot please visit www.jrcptb.org.uk/assessment. If you have a question about the pilot please email firstname.lastname@example.org.
Competing interests: None declared.
Bill Burr medical director
Joint Royal Colleges of Physicians Training Board, London, UK
Winnie Wade director of education Royal College of Physicians, London
Hannah Watts curriculum and assessment manager Joint Royal Colleges of Physicians Training Board, London, UK