Sorry, you need to enable JavaScript to visit this website.



Certification Exams: Are We Testing Recall or Judgment?


I have a recurring nightmare: I’m a sophomore in college, taking an engineering exam on differential equations that I don’t understand at all. It’s an open book test, and I still have no idea what to do! Somehow, in my dream, I miraculously pass the test, but know with certainty that I should never take another engineering course, just for fun, ever again. So, I became a doctor.

Fast-forward 30 years and I am making rounds with residents and fellows. The computer on wheels (COW) accesses orders, EMR entries, and PubMed. The residents use tablets and smart phones to log in to sites that provide practice guidelines, differential diagnoses, and more. ABP exams have evolved from oral exams in front of a daunting panel of learned practitioners, to unproctored exams on floppy computer disks to today’s proctored, computer-based exams. Real time internet access for clinical care prompts the question: Should my old college open-book exam be reincarnated as an ABP open-internet exam?

Unfortunately, there is precious little research to guide answers to this question. Overall performance on open-book/internet exams does not seem to change much, with those who benefit from the approach offsetting a population of examinees who become more confused by it.  Historically, data on the effects of open-book exams reveal conflicting results on whether the open-book (and presumably the open-internet) exam encourages or hinders learning.


There are also philosophical disagreements on the role of recall in testing. A closed-book exam requires test-takers to recall facts, which are then used to address the diagnosis, treatment, and management of a given clinical scenario. Proponents of the closed-book exam argue that the practice of medicine requires a certain amount of “pocket knowledge” that every practitioner must have to deliver excellent care. Opponents of the closed-book exam chafe at this requirement for fact recall precisely because technology-aided retrieval of information has become reality – arguably a necessity – in everyday practice. It is increasingly unrealistic to expect physicians to recall information the way they did in the past as the body of knowledge in medicine expands exponentially. Regardless of one’s philosophical perspectives, the current closed-book exam design has produced ample psychometric data in support of its fairness, validity, and reliability.


If the ABP and other certifying boards were to shift to an open-internet approach, a substantial amount of preparatory work and pilot testing would be needed. Exam questions would presume access to facts and focus on assessing critical thinking, effective search strategies, decision skills, and complex data analysis. Such questions might have more than one correct answer, and the correct answers might be weighted differently such that one of the correct answers might be considered “more correct” than another. Thorough psychometric analysis would be required to document the validity and reliability of such an approach before it could become the basis for certification.

An open-internet exam would still be timed. So, just as in the clinic, there would be insufficient time to look up every imaginable fact. Some examinees might even feel greater pressure with the combination of access to resources and time limits.

Predictably, open access to the internet carries a host of security concerns, which would have to be solved before this approach becomes viable. The June 3, 2012 issue of The Chronicle of Higher Education (“Online Classes See Cheating Go High-Tech” by Jeffrey R. Young, documents the technological arms race between innovators in online education and those intent on using holes in the online systems to cheat. Robust exam security will be necessary to ensure that everyone taking the exam has an equally fair chance at passage.

Finally, economic concerns have been raised. Although many or most pediatricians have already purchased subscriptions to online databases and information resources, this is not true for all. The ABP would want to avoid any notion that purchasing such a subscription was “required” in order to pass the test.

The ABP is thinking hard about these issues and will conduct the necessary research to evaluate open-internet testing. In the meantime, I encourage you to let us know what you think. Please leave a comment below.


David G. Nichols, MD, MBA
President and CEO