Board Scores

Years – sometimes a lifetime – of work to get to this point, and a single three-digit score from one day of testing determines your future.  If it’s better than expected, you can dream big; if it’s good enough, you can feel some measure of confidence; if it’s “fair to middling” you have some work to do; and if it’s low, then you need some soul searching.  

Sadly, the United States Medical Licensing Examination (USMLE) was not developed to serve this purpose.  It was initially administered by the National Board of Medical Examiners (NBME) in 1992 after many years of attempting to find a way to unify the complicated, erratic interstate physician licensure process; it’s official purpose is to aid authorities granting medical licenses (and also assure stake holders that licensed physicians have “attained a minimum standard of medical knowledge”)  I hate to be redundant, but to drive it home:  the psychomotor validity of USMLE scores is as a pass/fail measure for decisions related to physician licensure.  

It was not and is not meant to serve as a “Residency Aptitude Test”, and at one point, the NBME had to issue a disclaimer to that effect.

Why, then are USMLE scores used in the residency selection process?  Because they provide a nationwide standard for comparison.  If a measure is uniform, validated, easy to interpret and seemingly objective, we are all over it.  “The USMLE scores are currently the only nondemographic continuous variable by which applicants can be rapidly screened.”  Given the perpetual increase in applications programs are receiving, anything that allows applications to be “rapidly screened” is undeniably going to be emphasized.

But, stepping back … what are we screening?  What do the USMLEs reliably predict?  On its face, Step 1 is a multiple-choice test of basic science, little of which has much direct relevance to the practice of ophthalmology (to no one’s surprise).  Perhaps even less shocking is that very little of the material tested is retained by most students – there is significant decline in examinee performance after just one or two years.  There is little correlation between Step 1 scores and patient care or clinical outcomes. The strongest link is between Step 1 and performance on other standardized tests.  

I’m going to throw down this awesome quote – let it stew for a while:  “Such associations raise a question of whether these instruments are truly independent measures of knowledge – or whether we are simply repeatedly assessing a skill in test taking.  While success on standardized tests is a skill prized in our society, it is not necessarily one that adds value to patient care.  In an era when medical knowledge is more accessible than ever before, it seems curious that we have chosen to prioritize a measure of basic science memorization over higher-level analysis and critical thinking.”

Ok, but let’s be realistic – even if that’s all this test measures, standardized test performance is a big deal.  We can’t forget the summative standardized test in medicine – the specialty boards.  Since board certification is of obvious importance on an individual and program level, surely this test is an appropriate predictor of that?  Well, let’s look at the data.

I’m only aware of three studies in our literature (please inform me if you find others), and first a bit of alphabet soup that will become second nature if you matriculate through ophthalmology residency: 

• OKAP = Ophthalmic Knowledge Assessment Program, a 260 multiple choice test provided nationwide to each ophthalmology resident annually throughout the three years of training.  “It is designed to measure the ophthalmic knowledge of residents, relative to their peers, in order to facilitate the ongoing assessment of resident progress and program effectiveness”
• WQE = Written Qualifying Exam, the written boards for the American Board of Ophthalmology (ABO).  It is the first of two tests one must pass to become board certified (the second being an oral exam).  Not coincidentally, it is 250 questions long and similar in format to the OKAP.

So, first – the most recent publication that just came out this year.  It is an online survey sent to program directors and then disseminated to residents. It was anonymous, self-reported and only 19 programs (15.7%) passed it on to residents, for a completion rate of 13.8% of all ophthalmology residents (read:  major limitations).  Respondents selected their USMLE scores in increments of 10 (210-220, 220-230, etc.) and similarly reported their OKAP scores.  The authors found that in this sample, a 9-point increase in OKAP percentile and a 2.5 higher odds of scoring about the 75^thpercentile on the OKAPs when USMLE scores moved up by every 10-point category.  Take home – major limitations, but suggests that a higher USMLE score correlates to a better OKAP performance. 

Second, a study of 76 residents from 15 consecutive training classes (1991-2006) at 1 ophthalmology residency training program found that OKAP scores were significantly associated with WQE pass rate, and that passing or failing the OKAP exam all three years of residency was associated with a significant odds of passing or failing the WQE, respectively (“passing” on OKAPs was considered above the 30^thpercentile in this study).  Interestingly, the authors did not find an association with USMLE Step 1 scores and WQE performance.  Take home – in this single institutional longitudinal study, passing OKAPs was correlated with passing the boards (and vice versa), but USMLE scores were not. 

Lastly, a study of 15 residency programs for a total of 339 residents graduating between 2003-2007 were evaluated to determine whether five variables (USMLE scores, OKAP scores years 1, 2, and 3 and maximum OKAP scores) were predictors of passing or failing the WQE.  The authors found that OKAP scores during the final year of residency was the best and USMLE scores were the poorest predictor of board performance.  Take home – in this older study, but the most robust in our field, doing well on your OKAPs just prior to taking the boards is way more predictive of board pass rate than USMLE scores. 

All data and conclusions need careful scrutiny, but based on what I’ve seen, there is little evidence to support using USMLEs as a residency screening tool.  Having hopefully established there is not much demonstrating the scores are helpful, in an upcoming post I’ll cover why this practice is potentially harmful.  In the subsequent post (and last on this topic), I’ll discuss a proposed and seemingly likely major change to USMLE reporting coming our way, and what may (or may not) replace the void in screening. 

Thanks for reading.  Comments are most welcome!