Dramatic changes are coming to the way resident physicians in all medical specialties are evaluated as they advance through training. Faculty with the Department of Anesthesiology at Vanderbilt University School of Medicine (VUSM) have developed an automated tool to help more accurately assess residents’ performance as they meet educational goals and ultimately become better doctors.
The changes are due to the Next Accreditation System (NAS) being implemented by the Accreditation Council for Graduate Medical Education (ACGME), a nonprofit organization that evaluates and accredits more than 9,000 medical residency programs throughout the United States.
New NAS requirements are intended to track learning ‘Milestones,’ and they take effect in July 2014 for anesthesiology residencies nationwide. To assist, Vanderbilt’s Department of Anesthesiology faculty developed an automated case evaluation tool to provide immediate feedback on residents’ performance and track their progress. The case evaluation tool pulls data related to specific clinical tasks performed during patient cases and fulfills some of the new assessment requirements in ACGME’s Milestones system.
“This near real-time, detailed feedback will be critical to help residents accurately evaluate their practice as they relate to the published guidelines and continually improve the quality of care they deliver,” said Matthew McEvoy, M.D., vice chair for educational affairs and residency program director for VUSM’s Department of Anesthesiology. “In light of the coming Milestones system, we decided to be proactive and put our extensive data collection systems to work for us. We have been able to get this tool developed, tested and into practice well before the new ACGME standards are put into effect.”
A major concern of many residency program directors is there will be a tremendous burden in documenting residents’ progress through the multi-level Milestones system. For instance, there are 60 residents in the Vanderbilt Department of Anesthesiology training program, and there are 25 defined Milestones, or competency-based performance areas, that are to be demonstrated by residents from the beginning of their education through graduation.
Each Milestone has five possible levels of performance, and the ACGME requires that each resident is thoroughly assessed every six months on all of the Milestones. This translates into 1,500 data points that must be reported to the ACGME during each evaluation cycle, in addition to a report of personal performance for each resident.
“Assessing and reporting resident performance on all Milestones and core competencies is quite challenging for faculty members,” said Warren Sandberg, M.D., Ph.D., chair of the Vanderbilt Department of Anesthesiology. “Having a tool that reduces the administrative workload while also providing precise, reliable data, puts us at a tremendous advantage.”
While developed initially to evaluate anesthesiology residents, the infrastructure exists for the automated case evaluation tool to be customized for use by other residency programs at VUSM. The tool has been presented by its developers to program directors of other specialties at Vanderbilt, as well as to the American Medical Association’s Consortium to Accelerate Change in Medical Education. It will also be presented at a panel presentation at the ACGME’s spring 2014 annual meeting.
“We encourage our training programs to share best practices, and this tool is one of those,” said Donald Brady, M.D., senior associate dean for GME and Continuing Professional Development.
“The anesthesiology residency, under the leadership of Dr. McEvoy, is at the forefront of helping residents use their own data for personal and systematic quality improvement. This project is a novel combination of educational innovation and informatics, the latter of which required partnership with the Vanderbilt Anesthesiology and Perioperative Informatics Research (VAPIR) division, led by Dr. Jesse Ehrenfeld.”
The evaluation tool is the focus of a study published by Vanderbilt anesthesiology faculty in the January issue of the medical journal Anesthesiology. The study shows that anesthesiology residents want frequent updates on their clinical performance, as well as evaluations of their performance compared to that of faculty and their fellow residents as a whole.
For the initial study, data was collected from 24,154 completed anesthetic cases at Vanderbilt University Medical Center from February 2010 to March 31, 2013. The data covered five measurable skill areas: antibiotic administration, glucose monitoring, temperature management, pain management and central line insertion.
Additional metrics have been added to the evaluation tool, including the rate of post-operative nausea and vomiting, unanticipated ICU admission, and acute kidney injury. This information is of great benefit in training residents to see the perioperative outcomes of care provided in the operating room.
“The training of physicians is undergoing enormous changes in the coming years,” said Jesse Ehrenfeld, M.D., MPH, first author on the study. “This new feedback system is one part of a larger system being designed at Vanderbilt that will ultimately track residents throughout their training and allow them to graduate when they have reached an appropriate competency level, rather than simply after having served as a resident for a conventional period of time following medical school.”
