Carol Carraccio Reflects on Receiving St. Geme, Jr. Award

Carol Carraccio, MD, MA, is Vice President of Competency-Based Assessment for the American Board of Pediatrics. Dr. Carraccio has received the 2017 Joseph W. St. Geme, Jr., Leadership Award for more than three decades of work to improve medical education and find objective ways through multiple sources to provide useful assessments and feedback to medical trainees.

"Teaching others made me a better clinician." 

When I was chief resident at St. Christopher’s Hospital for Children in Philadelphia, I fell in love with medical education. While I enjoyed clinical care, it was when I had the opportunity to work with junior colleagues and medical students that I became truly excited about my work. And teaching others made me a better clinician. In those days, assessments were limited by both lack of tools and structured programs of assessment. The rule of thumb was that if you did something wrong, you heard about it, otherwise no news was good news. But during that year, I made the commitment to build my career around medical education.

I’m humbled by this award, and I know that the progress we have made in improving the feedback and evaluation of medical trainees is the result of much dedication and hard work by many medical educators. I’ve been asked to explain in this blog what the excitement is all about, so I’ll ask my learned colleagues to forgive any over simplification – they only gave me a little space to explain half a century of work by people from around the world.

So here goes:       

In recent years, medical educators have been attempting to develop reliable approaches to assessing the competence of medical trainees by using the direct observations of members of the health care team. In order to do so, medical educators have created programs of assessment using frameworks such as Entrustable Professional Activities (EPAs), Competencies and Milestones. We also have found that advances in technology – particularly the use of mobile devices -- can make doing these assessments easier, faster and perhaps, because less time elapses, more accurate. These new approaches are currently being developed and refined by various groups within the education community, including the American Board of Pediatrics, the Council on Pediatric Subspecialties (CoPS), Accreditation Council for Graduate Medical Education (ACGME), the Association of Pediatric Program Directors (APPD), the National Board of Medical Examiners (NBME) and others.

Research is underway to study the validity of these approaches. Recent results from two of these projects indicate that assessment of EPAs using supervision scales and competencies using, milestones, each contribute to informing the decision about whether a resident or fellow is ready to advance to the next step in training.

So what are these approaches?

EPAs are essential activities that physicians are entrusted to perform safely and effectively without supervision. Competencies are the skills a physician has to acquire in order to perform those EPAs. You might think of it this way: if riding a bicycle were an EPA, then pedaling, maintaining balance, and using the brakes would be competencies.

Milestones are narrative descriptions of behaviors for each of the competencies along a continuum of development ranging from novice –early medical student—to an expert clinician who is years into practice. A novice bike rider would likely find it difficult to judge the distance needed for stopping even when the stop is anticipated whereas an expert bike rider could do this even when an urgent stop is required.

We create these brief narrative descriptions of behaviors for a given competency at different performance levels, from novice to expert. They give us a shared mental image of what trainees look like delivering care at various levels of performance.

Here’s an example of a trainee demonstrating behaviors of a novice for the competency of performing an thorough and accurate physical exam. She or he quickly takes a newborn from a mother’s arms and lays the baby on a cold examining table which prompts the baby to cry and disrupts any opportunity to perform a useful physical exam. A resident who is much farther along the development continuum might come into the room, establish rapport with the mother, and observe the baby in the mother’s lap for rate and ease of breathing, color, and use of both arms and legs before ever touching the baby, maximizing the information gained.

Milestones are descriptions of the stages of development within an area of competence. As such, milestones provide a roadmap for learners. They can refer to these descriptions of behaviors at each phase of their training, using them to help reinforce and set learning goals. Integrating the behaviors of all competencies that are needed to perform an entrustable professional activity, such as “providing care to a well newborn,” is what trainees are called upon to do in delivering care to patients.

When you put EPAs, competencies and milestones together for purposes of assessment, you get information on how well a learner does an individual task such as a physical examination as well as integrating the individual tasks to provide care to a well newborn.

Studying These Approaches

EPAs, Milestones and Competencies are the subjects of much research to see how well they work and how they can be improved. Two of the latest studies are described in the ABP’s 2016 Annual Report. One is led by Richard B. Mink, MD, MACM, Chief of the Division of Pediatric Critical Care, and Director of the Pediatric Critical Care Fellowship Program at Harbor-UCLA Medical Center in Torrance, Calif., and Professor of Pediatrics at the David Geffen School of Medicine at the University of California-Los Angeles (UCLA). Another is from Patricia J. Hicks, MD, MHPE, Professor of Clinical Pediatrics at the Perelman School of Medicine at the University of Pennsylvania and a general pediatrician at the Children’s Hospital of Philadelphia. Dr. Hicks is director of the Pediatrics Milestones Assessment Collaborative (PMAC), a joint effort of the ABP, APPD and NBME. I invite you to take a closer look at those promising studies.

One of the most exciting things is the large number of training programs that have partnered with us to do these studies. Residency and fellowship program directors and associate program directors are committing significant time and energy to these studies and Alan Schwartz, PhD, Director of APPD LEARN (Longitudinal Educational Assessment Research Network), is helping with the management and data input/analyses from the hundreds of program study sites. These data all live within the LEARN database, a rich resource for our community.

Related Article: Carol Carraccio Receives 2017 Joseph W. St. Geme, Jr. Leadership Award

About the AuthorCarol Carraccio, MD, MA

Dr. Carraccio oversees the design and development of the ABP's competency-based assessment programs, including the Pediatrics Milestones Project., In addition, she works with subspecialty communities to identify “Entrustable Professional Activities” and map these to Milestones. Dr. Carraccio was a member of the ABP Board of Directors and Executive Committee before joining the ABP’s senior management team in 2011. She is board certified and maintaining certification in general pediatrics.