Sorry, you need to enable JavaScript to visit this website.



Future of Testing: Can You Draft a Question Requiring Skills Beyond Recall?


Over the past year, I have had the opportunity to hear about advances and issues facing our profession from pediatricians from all across the country. Your thoughtful comments to my last post provide an even greater wealth of insights into Maintenance of Certification in general and testing in particular.

Many agreed that an “open-Internet” approach to testing is important to consider, and many recognized the complexity of such a change. An overarching theme in the responses was the purpose of the test, which is to assess knowledge as an integral component of medical practice. Many of the comments echoed my earlier assertion that technology-aided retrieval of information has become reality – arguably a necessity – in everyday practice. Conversely, several of you pointed out there is not enough time to look up everything in daily practice and worried whether today's residents were becoming too dependent on the smartphone.

Your comments raised other pertinent questions:

  • If we allowed access to the Internet during tests, could we agree on selected resources and websites that should be accessible (such as Red Book, ePocrates, UpToDate)?
  • Is the use of the Internet appropriate for initial certification as well as Maintenance of Certification?
  • Could we allow an open-Internet test if we could not guarantee that test-takers are prevented from copying and emailing test questions or entering chat rooms?

Right now, the leading test delivery providers are researching the best ways to allow for open-Internet testing while still ensuring the integrity of the examination process, but we’re not there yet.

Assuming that the security challenges can be resolved, an even more important issue is the kinds of questions that would be appropriate for a section of the exam that included Internet access to certain resources. We would want questions that do not simply test recall, but also focus on analytic thinking, data synthesis, evidence assessment, judgment and decision-making.

Our subboards and question writing committees are made up of board-certified pediatricians who see the same kinds of patients as any practicing pediatrician. They work extremely hard at writing fair and valid questions that require higher-level thinking and critical reasoning skills. For now, we will continue to use closed-book, multiple-choice examinations, which offer the most objective methods for scoring. If, at some point in the future, we incorporate Internet-access during examinations, we would need an expanded pool of questions that examine critical reasoning skills.

We invite you to join the effort by drafting sample questions that you think would assess the higher order thinking skills that define a good pediatrician -- even if he or she can look up information before making decisions about management or treatment. Our shared goal should be an examination that mirrors to the greatest degree possible the increasingly complex world in which we practice. And as one commenter suggested, physicians don’t practice in a “closed-book world.”

Before you get started, I strongly encourage you to look at some of the resources used by our question writing committees: one is an overview of Bloom's taxonomy; the other is an excellent tutorial developed by the National Board of Medical Examiners (NBME) on how to write good questions.

You may also want to beta test your question(s) with your colleagues before you submit them. We look forward to seeing the kinds of questions you come up with.


We will share your submissions with our subboards and question writing committees as they wrestle with the same challenges. You shouldn’t expect to recognize questions that you submit on a future examination. All questions undergo multiple rounds of thorough review and careful editing.

In fact, we have always accepted questions from diplomates, and they all go through the same process. You can find our item writing portal through your portfolio on the ABP website. For now, and for ease of use, we created a simpler method for you to share your submissions, accessed through the teal button.

Realistically, developing new questions and designing new exams takes time. We have begun a rigorous discussion about testing, and we are reviewing the best thinking from testing experts as well as medical educators and expert clinical practitioners. In the meantime, we welcome your input as we explore this important issue.


So give question-writing a try. As you do, I hope you will also continue to share your ideas about what makes a good question. I invite you to use the comment section of this blog to contribute to the discussion. As always, I look forward to reading your insights and thoughtful suggestions.


David G. Nichols, MD, MBA
President and CEO