In the past two blog posts, I discussed the initial intent, current use, validity and consequences of the USMLE as a residency screening tool. Thankfully, this wasn’t just an exercise; most of my interest and information on this topic came after learning of some very potentially disruptive changes in the near future: moving to a pass-fail scoring system.
As stated in a recent viewpoint on this topic in JAMA: “Changing the USMLE to a pass-fail format would require residency programs to find other, potentially more meaningful, ways of evaluating applicants. Although a more thorough review of applications would be resource intensive, programs might identify outstanding applicants who would have been overlooked based on a numerical cutoff.”
This sounds encouraging, but it would be naïve to think that changing to a pass/fail system would improve much if it occurs in isolation. The authors go on to state:
“Moving to a pass-fail system for USMLE could make it more difficult to counsel students because each residency program would develop independent review standards. Furthermore, the movement over the past decade to pass-fail grading in many medical schools could exacerbate this problem, making it difficult to predict success in the match. Unless the ERAS significantly improves the capacity for programs to screen applicants based on individual characteristics (key words, research area, etc.), program directors may use the variables they have access to such as placing more emphasis on medical school reputation or location. Changing such a complex system must be addressed carefully because it is a crucial factor in determining the specialty training of thousands of medical school graduates.”
Therein lies my biggest concern – if the USMLE is changed to pass/fail without other carefully considered, well implemented, uniform policies from our governing organizations, the process may become even more arbitrary and discriminatory. My hope is our leaders will be proactive in addressing these issues and start trialing and debating potential solutions in the near future.
What could these “more meaningful” evaluation tools look like? A brief literature review provides some insight, although most of these models are rather esoteric to me, and there is very little substantial evidence or use of them for the purpose of residency selection. I’ll briefly (and poorly) describe some of them below.
Holistic Review. Holistic assessment is defined by the American Association of Medical Colleges as ““flexible, individualized way of assessing an applicant’s capabilities by which balanced consideration is given to experiences, attributes, and academic metrics and, when considered in combination, how the individual might contribute value as a medical student and physician.” This involves using programmatic or institutional values to review applications, with all domains of the application measured similarly in regard to the institutional needs. Some of these domains could include “ethics, leadership, cultural competency, diversity, communication, healthcare disparities, patient and family centered care, and innovation.” Such an approach is currently being used in some medical school admissions, and “does not necessarily need to be complex or time intensive.”
Gateway Exercises. These are uniform evaluation opportunities during various checkpoints in medical school training – for example, at the end of required clerkships or beginning/end of the academic year. A common exercise currently implemented by many medical schools are Objective Structured Clinical Examinations (OSCEs).
Simulated Assessments. Simulated encounters and exercises are frequently used in medical training including the OSCE above or other standardized patient evaluations, computer-based cases, written clinical scenarios, mannequins (in bow ties) or a combination thereof.
Competency-Based Assessments. The transition to a competency-based curriculum in medical school (and residency) has been slow but persistent. The evaluation system necessitates assessment across multiple domains, thus creating a more comprehensive portfolio as students progress through training. Along those lines, longitudinal tools or dashboards that provide trajectories over time, as compared to one’s peers, may be a potential tool for comparing residency applicants .
Standardized Video Interview or Letter of Evaluation. Emergency Medicine is on the innovative and “early adapter” end of the curve as it applies to residency application. Not only have they adapted a standardized letter of evaluation at the end of required externships (which is felt to be the most important factor for determining which applicants to interview), but also piloted a standardized video assessment during the 2018 interview cycle. This assessment is an online interview involving questions based on “knowledge of professional behaviors and interpersonal and communication skills”. The interviews are scored by a third-party and both the scores and video are provided to residency programs.
To be honest, I’m not sure I really understand any of what was just stated, especially on a practical level. Some additional (major) limitations to any of these assessment models include: increased cost and complexity, lack of validation for resident selection, difficulty implementing and then comparing across schools, and program director and faculty acceptance.
Lastly, as mentioned in the quote above from the JAMA viewpoint, these measures and application review will likely be more “resource intensive”. I doubt many program directors, coordinators and review committees currently feel capable of devoting additional time and money to this process. To that end, I believe the application and interview process must also change, with several currently debated suggestions highlighted in this prior post. Again, my hope is that our leaders will be proactive rather than reactive with all this, mindful that the party with the most at stake is the applicants.
I hope these posts have been informative and provocative. This is a pivotal aspect of the application process that is likely changing in the near future, so it will be hard to ignore. As always, please let me know any thoughts or perspectives! I have one additional topic to discuss in the near future that has been quite eye opening for me – I imagine it will be much the same for many of you.
University of Kentucky Department of Ophthalmology 