Situational judgment tests: the future of medical selection?

Authors: Fiona Patterson, Vicki Ashworth 

Publication date:  27 Oct 2011


Fiona Patterson and Vicki Ashworth of the Work Psychology Group ( [Link] ) look at the growing use of situational judgment tests

Throughout their training doctors face a variety of methods of assessment and selection. One approach that is becoming increasingly popular is situational judgment tests (SJTs), a selection method designed to assess individuals’ judgment regarding situations encountered in the workplace. A good understanding of the background and evidence behind SJTs is therefore important for doctors at all stages of the training pathway.

What are SJTs?

In an SJT candidates are presented with a set of hypothetical work based scenarios and asked to make judgments about possible responses. A variety of response formats, such as ranking possibilities or multiple choice, can be used, and candidates’ responses are evaluated against a predetermined scoring key so that the outputs of the test are standardised. Each candidate’s responses are assessed in exactly the same way, and it is therefore possible to compare candidates.

SJT scenarios are normally based on extensive analysis of the job role, to ensure that test content reflects the most important situations that a candidate may face. They tend to be concerned with testing attitudes and ethical values rather than knowledge or clinical skills and are often used in combination with knowledge based tests to give a better overall picture of an applicant’s aptitude for a particular job.

SJTs have been used extensively in other occupational groups outside medicine and have been introduced as a way of obtaining detailed information on interpersonal oriented skills and various professional attributes such as empathy and integrity.

SJTs are a measurement methodology rather than a single type of test, and so their content and format can be altered to fit the job specification. They are becoming increasingly popular in large scale selection processes and are used by the police and the Department for Work and Pensions, for example, as well as in many public and private sector graduate recruitment processes and training schemes.

Do SJTs work?

Evidence has consistently shown that, as a selection tool, SJTs show good reliability and validity.[1] [2] The research literature indicates that SJTs are able to predict job performance and training criteria across a range of occupations.[2] [3] [4] A number of studies have shown that SJTs demonstrate validity beyond structured interviews, tests of IQ, and personality questionnaires.[5] [6] [7]

In dealing with issues of fairness, selection researchers generally examine adverse effects—that is, the extent to which particular groups (such as black people or women) systematically perform worse than other groups in a particular selection method. In relation to ethnicity, differences in mean scores on SJTs between ethnic groups tend to be smaller than for tests of cognitive ability.[8]

Applicants’ perceptions relating to SJTs are generally positive, perhaps because of the relevance of the scenarios to the target role—more so than, for example, cognitive ability tests.[9]

SJTs in selection for postgraduate training

Role analyses of numerous specialties in the medical arena have led to wider acknowledgment that non-cognitive or professional attributes (such as communication and team working) are essential requirements for a doctor. In 2006, after research showed that SJTs are especially effective in measuring a range of non-cognitive, professional attributes, an SJT was piloted as a shortlisting tool for selection into training for general practice.

Since the successful live implementation in 2007 of the general practitioner SJT, the Work Psychology Group has collaborated with a number of specialties, including surgery, medicine, histopathology, and the anaesthesia and acute care common stem to pilot SJTs. SJTs have also been used as part of the selection methods for public health since 2010. Most recently an SJT has been developed for selection to the foundation programme.

Currently two types of items are used in medical selection SJTs: ranking and multiple choice. All items start with a scenario, but the two types of items have different response formats. Ranking asks candidates to rank a series of response options, usually from most appropriate to least appropriate. Multiple response items ask candidates to select the three most appropriate options from a larger number of options.

Although the correct answer is always predetermined, the scoring is not “all or nothing.” For ranking, candidates get points for “near misses”; therefore a candidate does not need to get every option in exactly the correct order to obtain a good score on the SJT. For multiple response items, points are received for each correct answer provided; no negative marking is used.

In general practitioner selection, ongoing evaluation consistently shows that the SJT, used with a clinical problem solving test, is reliable, effectively predicts performance in the subsequent assessment centre, and receives positive reactions from applicants.[2] Long term predictive validity studies have shown that the SJT effectively predicts performance in training, in terms of supervisors’ ratings.[10]

SJT example 1

You are looking after Sandra Jones, who is being investigated in hospital. You are asked by her family not to inform Mrs Jones if the results confirm cancer.

Choose the THREE most appropriate actions to take in this situation.

A. Ignore the family’s wishes.

B. Agree not to tell Mrs Jones.

C. Explain to the family that it is Mrs Jones’s decision.

D. Ask Mrs Jones whether she wishes to know the test results.

E. Ask Mrs Jones whether she wishes you to inform the family.

F. Inform Mrs Jones that her family do not wish her to have the results.

G. Give the results to the family first.

H. Give the results to the next of kin first.

SJTs in foundation year 1

In 2009, a review of the method by which final year medical students are selected into foundation training was recommended. Although the current system works well, several concerns needed to be dealt with, including the risk of plagiarism and costs in terms of time to score competency based questions.

The recommendations were for an SJT to assess professional attributes, judgment, and employability for a foundation programme post (to replace the application form). This was recommended to be used with a measure of educational performance to assess clinical knowledge and skills (evolved from the academic quartiles).

Pilots of the SJT involving more than 1000 UK students in the past 12 months have shown it to be a valid and reliable method of selection in this context. The SJT was based on a job analysis of the foundation year 1 role and was developed in consultation with clinicians who work with foundation year 1s and foundation doctors themselves. This was to ensure that the scenarios are relevant, realistic, and fair. The SJT targets five professional attributes: commitment to professionalism; coping with pressure; effective communication; patient focus; and working effectively as part of team.

After recommendations to the Department of Health, an SJT is likely to be used in live recruitment from foundation programme 2013 onwards. More information about the forthcoming changes is at [Link] .

A parallel recruitment exercise is due to be run later this year, with the aim being to pilot the SJT alongside the current selection methods. Thus the parallel recruitment exercise is the final step in ensuring that the selection methods can be consistently and robustly applied before implementation for foundation programme 2013. All final year medical students are encouraged to participate, in addition to completing their foundation programme application system forms.

SJT example 2

Mr Johnson is admitted with a minor groin abscess requiring surgical drainage, although he is otherwise well and has full mental capacity. Four hours before surgery Mr Johnson informs a nurse that he wishes to discharge himself, as he says that he is due in court. Mr Johnson’s next of kin are aware that he has been admitted for surgery and that he is due in court. The nurse asks you to speak to him.

Rank in order the following actions in response to this situation (1=most appropriate; 5=least appropriate).

A. Allow Mr Johnson to leave but advise him to see his general practitioner if there are further problems.

B. Prevent Mr Johnson from leaving by telephoning security.

C. Explain to Mr Johnson the risks of leaving without treatment.

D. Telephone Mr Johnson’s next of kin to ask them to try to persuade him not to leave.

E. Allow Mr Johnson to leave but ask him to return to the hospital as soon as possible.

SJTs are being used successfully in a number of areas within medical selection. With their proved reliability and validity, it is likely that their use will increase. In addition to selection, there may be further uses of SJTs in the medical context. SJTs can target non-cognitive abilities at various levels in the training pathway; and although they have been more commonly used as tests of aptitude, they could also be used for education, training, and development.

Competing interests: The Work Psychology Group collaborates with and acts as advisers for the GP National Recruitment Office and the Medical Schools Council in the development and delivery of SJTs.

References

  1. McDaniel MA, Morgeson FP, Finnegan EB, Campion MA, Braverman EP. Use of situational judgment tests to predict job performance: a clarification of the literature. J Appl Psychol  2001;86:730-40.
  2. Patterson F, Baron H, Carr V, Plint S, Lane P. Evaluation of three short-listing methodologies for selection into postgraduate training in general practice. Med Educ  2009;43:50-7.
  3. Chan D, Schmitt N. Situational judgment and job performance. Human Performance  2002;15:233-54.
  4. Borman WC, Hanson MA, Oppler SH, Pulakos ED, White LA. Role of early supervisory experience in supervisor performance. J Appl Psychol   1993;78:443-9.
  5. Lievens F, Buyse T, Sackett PR. The operational validity of a video-based situational judgment test for medical college admissions: illustrating the importance of matching predictor and criterion construct domains. J Appl Psychol   2005;90:443-52.
  6. McDaniel MA, Hartman NS, Whetzel DL, Grubb WL. Situational judgment tests, response instructions, and validity: a meta-analysis. Personnel Psychol  2007;60:63-91.
  7. O’Connell MS, Hartman NS, McDaniel MA, Grubb W, Lee I, Lawrence A. Incremental validity of situational judgment tests for task and contextual job performance. Int J Selection Assess  2007;15:19-29.
  8. Mototwidlo SJ, Tippins N. Further studies of the low-fidelity simulation in the form of a situational inventory. J Occup Organiz Psychol   1993;66:337-44.
  9. Lievens F, Sackett PR. Situational judgment tests in high-stakes settings: issues and strategies with generating alternate forms. J Appl Psychol   2007;92:1043-55.
  10. Patterson F, Carr V, Zibarras L, Burr B, Berkin L, Plint S, et al. New machine-marked tests for selection into core medical training: evidence from two validation studies. Clin Med  2009;9:417-20.

Fiona Patterson professor of psychology
Vicki Ashworth senior consultant, Work Psychology Group, Ashbourne, Derbyshire, UK

 v.ashworth@workpsychologygroup.com

Cite this as BMJ Careers ; doi: