“Are you OK?” a medical school classmate asked when I snuck back into the lecture hall. I had just learned that I bombed arguably the most important test of my life. I was not OK. I was employing every technique I could muster so I wouldn’t break down in public.

I had just checked my score on the United States Medical Licensing Examination (USMLE) Step 1 test. The first of three “Step” exams that physicians-in-training must pass to become licensed to practice independently, the computer-based test focuses on the basic sciences. It’s an eight-hour grind done in a single sitting.

This test, once completed in paper and pencil by students en masse and viewed mostly as a tedious hurdle towards licensure, has since become an obsessive focus among medical students due to its elevation in the residency application process, fueling anxieties, ambitions, and evaluations of self-worth.

advertisement

I will never forget the sensation of shock that followed the big reveal: I had passed, but performed worse than my lowest expectations. I was already beginning to reimagine my future, whole careers disappearing before my eyes.

Now that I’m nearing the completion of my residency in internal medicine and pediatrics at the Harvard combined program at Brigham and Women’s Hospital and Boston Children’s Hospital, it all seems so melodramatic. But it is a familiar story for the thousands of students each year who land in the grey area between passing and the cutoff scores many residency programs use to choose whom to interview, unable to retake the test (which can only be done if you fail) and uncertain about what comes next.

I have felt different ways about Step 1 since that day. The first was shame, as I tried to understand how I had let such a thing happen to me and hid my poor performance from others. Then came redemption, as I worked hard to compensate, over-performing on my clinical clerkships and subsequent Step examinations.

Self-doubt followed, as a handful of residency program directors could not look past my Step 1 score to extend me an offer to interview. Finally, conquest, after I opened my envelope on Match Day to learn I had matched at what had always been my top-choice program.

But the feeling I share today is anger: At the unnecessary anguish an exam that was designed to be pass/fail has caused so many. At the harmful impact the test has had on learning in medical school. At the rejection of common sense, evidence, and empathy that so many in leadership positions have fallen prey to by reducing residency applicants into three-digit numbers. At the laziness of this approach and disingenuous claims that the test “levels the playing field” for applicants. And at the greed of a multimillion-dollar industry that has exploited the opportunity to extract money from already overly indebted students (registering for the test costs $630, while Step 1 preparation materials and courses can run much higher).

On March 11, leaders from the parent organizations of USMLE — the Federation of State Medical Boards and the National Board of Medical Examiners — along with the American Medical Association, the Association of American Medical Colleges, and the Educational Commission for Foreign Medical Graduates will convene in Philadelphia for the Invitational Conference on USMLE Scoring, also known as InCUS. The main objective of the conference is to reach consensus on the issue of whether the USMLE should continue to report numeric scores on its licensing exams, including Step 1.

I see only two acceptable paths forward. One is to completely reform Step 1 (and undergraduate medical education alongside it, standardizing the way curricula are structured and how student performance is reported) to make the Step 1 score truly meaningful. This is unlikely because it would require massive coordinated effort. The other is to return the test to what it was originally intended to do: provide a benchmark assessment of competency for the purpose of professional licensure (like the bar examination for lawyers) by eliminating numeric scoring and making it pass/fail. That would put the onus back on residency programs and medical schools to agree on more suitable ways to evaluate students.

Academic and evidence-based arguments abound against the continued “Step 1 climate,” which is directly engendered by its numeric scoring system. In recent issues of the journal Academic Medicine and its podcast, a rich exchange has highlighted the evidence that Step 1 is a poor predictor of future clinical performance; the negative impact of the test on students’ mental health and engagement with institutional curricula; and the discriminatory effect of the test on underrepresented minorities, women, and those from lower socioeconomic backgrounds.

I am sympathetic to the challenges residency program directors face in reviewing ever-mounting numbers of applications on a tight schedule with diminishing numerical or comparative data for use in decision-making. However, the common practice of using Step 1 cut-off scores and its corresponding notion that Step 1 is some sort of equalizer across applicants — and therefore has singular value to the resident selection process — can’t reasonably be justified.

First, there are wide discrepancies across medical schools that challenge the notion that the test is “standardized.” Some schools teach to the test, while others focus on the exam’s many blind spots, knowing full well that students will study the Step 1 material on their own. The amount of time off students are given to independently prepare for the test varies from a few weeks in some medical schools to several months in others.

The test’s flexible timing is also nonstandard: Many schools require students to pass the test before starting rotations in the hospital, while others push it back until after clinical clerkships, knowing students will benefit from the additional experience. Even the version of the test a student sees is not standard, requiring convoluted statistical corrections that make direct comparisons of scores between test sittings (and applicants) even more problematic.

On an individual level, students’ personal, family, and health circumstances, as well as their financial ability to afford the services of the growing and predatory Step 1 prep industry, have an enormous impact on achievement on the exam. And the capriciousness of assessing a person’s performance on a single day is unavoidable, particularly when the Prometric centers that administer Step 1 impose steep penalties on test takers for changing test dates and applicants cannot retake the test if they do poorly.

A number of medical schools have been inadvertently complicit in the growing importance Step 1 has assumed in the resident selection process by adopting pass-fail grading for students in their coursework. Often well-intended and couched in the language of wellness, this transition to pass/fail grading has precluded schools from making meaningful claims on the relative strengths of their students to residencies, leaving Step 1 as a pillar of objectivity to fill a perceived vacuum.

Grading fairly is always a challenging task, but it must not be one that schools shy away from tackling, as they are certainly better positioned to evaluate the performance of their students over four or more years than a nameless licensing exam completed in one day.

I was extraordinarily lucky. I benefited from going to a prestigious medical school with a rigorous grading system that allowed me to distinguish myself when it came time to apply to residency programs. I had the time and the capability to prove myself on future USMLE exams. I worked with faculty members who knew me and could advocate for me. My desired specialty was not dermatology, orthopedics, or otolaryngology, competitive specialties that are notorious for their emphasis on Step 1 scores in resident selection decisions. And my top choice residency program did not rely on Step 1 cut-off scores and invests in a laborious, holistic process when deciding who to interview.

My Step 1 score did not ultimately limit my professional ambitions. But it does limit others’. Given the lack of standardization across medical school curricula and the misappropriation of test scores for screening medical students for residency programs, the USMLE should no longer report numerical scores for Step 1.

There are many possible paths beyond a numerically scored Step 1 that have been presented by others and merit further discussion.

The time has come for the medical profession to apply the same thoughtfulness and rigor that physicians aspire to in clinical practice into deciding how it evaluates trainees. We must not contract out this essential responsibility to licensing boards and rely on Step 1 numbers for simplicity’s sake.

Nicholas Cuneo, M.D., is a fourth-year resident in internal medicine and pediatrics at Brigham and Women’s Hospital, Boston Children’s Hospital, and Boston Medical Center and a member of the Doris and Howard Hiatt Residency Program in Global Health Equity.

Leave a Comment

Please enter your name.
Please enter a comment.

  • Incredibly stupid change that removes merit-based assessment … therefore opening up the residency application process to severe institutional bias, nepotism, and the mental gymnastics of “holistic” review.

    Any sensible medical student should fight such decisions immediately.

    It’s also a remarkable coincidence that the author didn’t mention that those from his alma mater, and others with similar goals, will disproportionately benefit from said changes.

    The overwhelming majority of students DO NOT WANT THIS.

  • Selection for T5-10 schools isn’t just standardized tests. In fact, many people with MCAT scores higher than those at T5-10 schools elect not to go to them because of better financial aid deals or location preferences. The average USMLE for the top 50 schools are within a 10 point range. Why should “prestige” even be an advantage? How does going to an institution with a good name demonstrate that you’re going to be a better doctor?
    Turning your argument around, why should they go to Ivy schools just because they tested better one day on the MCAT? If you look at the MCAT distribution, there are individuals in almost every school who performed better on the MCAT than those who attend T5-10 schools.
    Also, if someone from a t5 school scores a 245 and someone not from there gets a 270, I would want the 270 to be my doctor.

  • This is essentially communism. This is taking out a way to distinguish med students for residency, and giving a huge amount of match-power to students from top schools. It also makes things easier for students who have primary care aspirations and don’t care at all about the different aspirations of their peers. This “author” would immensely benefit from p/f step 1 as the best opportunities would go much more frequently to students from his alma mater, thereby increasing the value of his degree. Therefore, his credibility is entirely shot, and he may be willfully misleading other med students reading his article. If you aren’t from a T5-10 school, you best fight this tooth and nail.

    • EXACTLY what I thought. The author can’t speak for the majority of med students when he doesn’t come from anywhere near the same circumstances.

    • So if you are so good at standardized tests, why aren’t you at a T5-10 school? Your argument assumes that students from these schools do poorly on the USMLE. This author is the obvious exception, not the rule. The average USMLE scores for students at Ivy League schools is 245+. And if you think that a selective program is going to take you over a student from one of these schools, you’d better come in with at least a 10-20 point advantage to overcome their prestige advantage anyhow. Maybe you are a great test taker, and a scored test benefits you personally. If so, good for you. But look at the averages, and it’s clear that it doesn’t benefit everyone in the way you seem to suggest. In my opinion, the real question is not whether some lower tier students can beat out top students on the USMLE, the real question is whether the USMLE measures what competitive residency programs are really looking for anyway. If it doesn’t, then why are you any more deserving for a residency position just because you tested better on one day? I get that we need a way to distinguish ourselves, but another system would probably be more fair than the USMLE (since most of us are at state schools precisely because we didn’t do as well as the Ivy Leaguers on the MCAT).

  • “I was extraordinarily lucky. I benefited from going to a prestigious medical school with a rigorous grading system that allowed me to distinguish myself when it came time to apply to residency programs… I worked with faculty members who knew me and could advocate for me.”
    This is exactly why a pass/fail Step 1 is detrimental for the majority of medical students. Those who are not from prestigious medical schools like the author won’t have the opportunity to distinguish themselves in applications to competitive specialties. A Pass from a place like Johns Hopkins will be worth more to program directors than a Pass from a state school, no matter how much people pretend that other metrics might take precedence. This will just increase the chasm between tiers of medical schools and give subjective measures like connections and research opportunities (which are already more available at higher-tier schools) more import. Any student at a low- to mid-tier medical school (not to mention DO schools) will tell you that this is would be an unfair and negative development, and will make it more difficult for us to stand out and prove objectively that we have the dedication and knowledge to work side-by-side with students from schools of more prestige.

    • Nadia,

      I think you bring up a valid and fair point about the negative outcome this result could have to students at schools perceived to be not as prestigious. I strongly believe there needs to be a discussion regarding the bias and inequity created via name recognition and opportunity. There has been long driven stigma particularly against those from DO institutions that have been a detriment to fair distribution of placement.

      That being said, I still respectfully disagree with the notion that having a Step 1 score is a necessity to standing out and giving those students an opportunity. The fact is that the test in itself is not predictive of success in a future career and does not identify who will be the better physician. Understanding the serious nature of competency in medicine, I get why a test exist. But nevertheless, like the author has stated it should be placed strictly on competency (Pass/Fail). The fact is such standardized test isolate and exclude far more quality applicants than prestige ever will. Studies demonstrate tests like the MCAT and Step are correlated with household income and racial background. These tests for a long time have excluded people of color from entering more prestigious programs and areas of specialties for nearly its entire existence. Not to mention, as the author alludes to the inequity created by ability to purchase resources from this predatory industry. I myself, as a person of color and a person from a low income background with no connections to medicine have found the indulgence of my peers into external resources that have rounded up into over a thousand dollars very disheartening. In my program, a State US MD program, students have completely forgone school material and dedicated their entire schedule to the pathomas, sketchys, and B&Bs of the world. If the intention of the first 2 years of medical school was only to master the Step 1 test then I see no value in even having the school in the first place. But I know that is not the intention and why so many physicians enter residency burnt out and disconnected from their patients. I cannot argue for a high stakes, life altering test like it has become. I deeply sympathize with your concerns of inequity and know that it must be addressed in some fair way but ultimately, I do believe this is the right move and can increase diversity across medicine and a more fair matching process.

  • If one is worried about diversity, imagine the impact on under represented minorities if residency selection were made with NO objective measures at all. Medical school grades are highly subjective, often with no defined criteria. Dean’s letters would move from nearly meaningless to completely so. Everyone can get someone to write a nice letter of recommendation.

    Residency selection would be even more “this person looks like the rest of us” than it is now.

    Not clear that moving USMLE to pass/fail would solve the problem of competition for certain specialties, make evaluations more reliable or make the experience less stressful for the students. It would just shift to a different set of, more subjective, criteria.

    • On the other hand, URM cohorts score lower on standardized metrics like the Step1 and MCAT, and are thus disproportionately screened out by having minimums. Removing this screening system would greatly increase URM access. For example in surgical subspecialties where a 250+ step1 is currently expected, having only a “Pass” listed the same on everyone’s app would greatly increase the numbers of URMs that could apply and get interviewed.

  • I am afraid this is a narrow look at the problem. The first thing that popped into my mind was substituting “MCAT” for “USMLE”. There is some evidence that MCAT scores predict USMLE scores and success on certifying board examinations.
    The emphasis on MCAT scores has created an even larger industry of preparing students for this test. The scores have great influence on whether one attends a highly prestigious medical school. If there were no USMLE score, then we could end up with a de facto reliance on the MCAT to determine who gets into Harvard and Stanford for med school. With no other way of comparing medical school applicants, the Harvard and Stanford grads would get the best residencies.
    A smart student from a less prestigious school would have nothing to show to compete. Perhaps the residencies would start requiring students to submit their SAT and MCAT scores?
    If we get rid of scores on the USMLE, I can see the competitive residencies demanding that their applicants take yet ANOTHER test, whose only purpose is selection.

    • I think you are approaching the asymptote. I am inclined to think that success at any level is predicted only by the selection pool. In general, if you get into the pool your success rate is high. Even as parity advances, it is effervescent due to bias. Those with the “power” will always determine the pool size and relent only to placate. The line will never reach to fairness.

  • Like most outcome specific curriculum or programs; law, medicine, dental, Phd, vet, etc., as long as everyone knows well the areas of the testing and it is applied with consistency, it in some ways seems not a good test of a candidates capabilities but it is the only objective sieve there is. Not everyone is capable. It doesn’t discriminate about anything and is just a fact. Who is best prepared to move up? I competed against multiple people with advanced degrees that were way ahead in specialty fields selections, it was as if they had been pre-ordained from day one. In looking back I could have excelled but I am satisfied I got out in the 4 years instead of 2-4 more years. Most of those going into the specialty fields were damn smart and motivated. I just wanted to get on with it and did well.

  • We can debate whether Step 1 scores benefit individual candidates or program directors, or whether reporting scores provides a net benefit to society. But one thing that is not debatable is how incredibly beneficial our idolatry of Step 1 has been for the test’s sponsors. NBME revenues have more than tripled in the past two decades, despite no real increase in USMLE test-takers, and their top executives earn seven-figure salaries. The NBME has a powerful incentive to maintain the status quo.

    • Congratulations for such an inspiring story. This shows again everything is about networking and what school you come from. If you had had that score and been an IMG (international medical graduate), you wouldn’t have any chances, not even for family medicine in the least prestigious program of the country. On the other hand, an IMG with 270 would never be called for an IV for IM in Harvard. Scores do correlate with how well you can perform during medical residency. However, if you have the right contact you can be in the program you desire. This is a reflection on how unfair is the system.

  • I offer my congratulations on your angst. However, changing a simple test scoring method will not change privilege. I don’t know whether you are worthy of your place or not. I do know many never get the chance to compete fairly. It is the arrogance of the fraternity (club) that breeds contempt.

Sign up for our Daily Recap newsletter

A roundup of STAT’s top stories of the day in science and medicine

Privacy Policy