Piloting a New Approach to Assessment
For some time now, the ABP has been working on improvements to Maintenance of Certification (MOC) that will make the activities more relevant and better reflect modern pediatric practice. My last blog described expanded pathways to receive Improvement in Practice (Part 4) credit. We also hear from diplomates that the MOC exam (Part 3), required once every 10 years, is stressful and may not make us better pediatricians.
After much research and discussion culminating in our recent Future of Testing Conference, the ABP has decided to pilot a new kind of assessment delivered via the internet.
This approach is called MOCA – Maintenance of Certification Assessment.
If the pilot proves successful, MOCA could eventually replace the half-day long exam every 10 years at a secure test center.
MOCA – Pioneering Work by the American Board of Anesthesiology
MOCA is based on pioneering work by the American Board of Anesthesiology (ABA), which coined the term “MOC Anesthesiology (MOCA) Minute” to describe their assessment tool. Last year, 1,400 ABA diplomates participated in the initial MOCA Minute pilot. They received one multiple choice question via email every week. They could choose when to open the question, but once it was opened, a limited amount of time was allowed to answer (one minute for ABA questions). Immediately after submitting the answer, a feedback page appeared indicating whether the answer was correct, as well as a brief explanation of the correct answer, the learning objective, key references, a comment box, and links to learning resources provided by the various anesthesiology specialty societies. If the diplomate answered the question incorrectly, some follow-up questions on the same general topic continued to appear in subsequent weeks or months.
The ABA’s MOCA Minute pilot was so successful that ABA has announced that it will replace its current every 10-year Part 3 MOC exam in general anesthesiology with an expanded MOCA Minute pilot as part of their redesigned MOCA program (i.e., MOCA 2.0). Their diplomates will be required to answer 30 questions per quarter (120 questions per year). MOCA 2.0 will be “summative,” meaning that a pass/fail decision is made based on the diplomate’s answer pattern over time. Physicians will know how they are performing relative to the board’s standard. Those approaching the minimum standard will receive an alert and have the opportunity to improve their performance. Compared to the every 10-year exam, this approach offers the possibility of greater validity, because the assessment sample is significantly larger as the number of questions answered one at a time over a 10 year period is more than the 200 questions in a single high stakes secure exam. It also provides an earlier alert to significant knowledge gaps, which should benefit the diplomate and offer greater assurance to the public.
The ABP’s Plans
The ABP is planning to conduct its own pilot study and will be developing a version of MOCA. I need to stress immediately that it is a pilot for MOC only. No changes are planned for the initial certifying exam. The pilot will start with the General Pediatrics MOC exam and participation in the pilot will be voluntary. We will collect feedback from multiple sources, including diplomate and public reaction to judge the success of the pilot. (Our pilot is part of an American Board of Medical Specialties study that several other boards are participating in.)
A great deal of planning will be needed to launch the pilot no later than January 2017. We will do extensive listening and field-testing before launching the pilot, but here are some provisional features (which may change):
- Diplomates will establish a practice profile when registering for MOCA, so that the content can be weighted to suit the type of practice.
- Diplomates may receive 1-3 multiple choice questions per week.
- The amount of time allowed for the answer may vary depending on the complexity of the question.
- Online resources or books may be used, but because each question is timed, the diplomate will need to judge carefully whether to invest time in searching through a resource.
- A feedback page will pop up after submitting the answer.
- A randomization protocol will minimize the likelihood that any two diplomates receive the same questions during a given time period.
- Flexibility will allow diplomates to decide when to respond based on their schedule and time availability. Test security provisions may vary depending on whether the diplomate chooses to answer questions during the week they are delivered or wait to answer a batch of questions.
- If MOCA is ultimately adopted, the ABP will make pass/fail decisions based on the response patterns. Those who successfully participate will meet standards for Part 3 of Maintenance of Certification.
The Social Contract
Every time I cared for someone’s child, I would tell the resident or student with me that the family’s assessment of us would be far more consequential than any examination either of us would ever take. Yet as part of the social contract between patients and physicians1, families presume our profession has examined our knowledge before allowing us to care for their children. Current knowledge in the field is a necessary, but hardly sufficient, requirement for caring for someone’s child.
An examination by peers (originally oral, in-person exams) has been a central component of the ABP certification process since the 1930s. With the well-known explosion of medical knowledge over the past 30 years, the ABP has been administering periodic examinations to diplomates who choose to demonstrate to themselves, their peers, and their patients that they retain a current knowledge base after completing training.
This tradition continues with the proposed ABP MOCA pilot. If MOCA works in pediatrics, the requirements of a recent expert consensus conference for a “good assessment” will have been met, namely to catalyze learning in the course of assessing knowledge in a fair, valid, and reliable manner.2 Those who participate will have demonstrated to themselves and others that they have the knowledge needed for the assessment that matters most – the encounter with a sick child.
This is what I think. If you would like to tell me what you think, please leave comments below.
David G. Nichols, MD, MBA
President and CEO
1. Cruess SR, Cruess RL. Professionalism and Medicine's Social Contract with Society. AMA Journal of Ethics. 2004;6(4):1-4. 2. Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011;33(3):206–214.