Evalumetrics—a performance measurement tool and more

Anne Ford

August 2014—Change takes time. Ask anyone who’s ever joined a gym, coached an underperforming sports team, or felt themselves growing older—cell by graying, wrinkling cell—in the change-of-address line at the DMV.

Or just ask Donald Karcher, MD. Since 2008, when the Joint Commission began mandating that health care organizations rigorously evaluate physician performance when granting or renewing practice privileges, Dr. Karcher has watched laboratories gradually move from noncompliance to curiosity to comprehension and finally to compliance.

“Around 2010, it was very clear that pathologists—actually, physicians of many types—knew almost nothing about this,” says Dr. Karcher, who is chair of pathology, director of laboratories, and professor of pathology at George Washington University Medical Center, Washington, DC. By “this,” of course, he means Ongoing Professional Practice Evaluation and Focused Professional Practice Evaluation, or OPPE/FPPE, the screening processes the Joint Commission requires health care organizations to use when granting or renewing physician privileges.

When Dr. Karcher and Jennifer Hunt, MD, chair of pathology at the University of Arkansas College of Medicine, gave a presentation at the CAP’s 2011 annual meeting about OPPE/FPPE, “most of the audience members who attended this workshop knew very little about it,” he says. By the time he and Dr. Hunt gave the same presentation at the 2013 meeting, “the audience was way more sophisticated. We could actually witness the evolution of the pathologists’ knowledge.”

Still, knowledge doesn’t always equal power. While laboratories might have learned more about the Joint Commission’s requirements, Dr. Karcher says, by and large they did not have a simple method of recording and monitoring their OPPE/FPPE efforts, until now.

Last year, the CAP launched Evalumetrics, a cloud-based software application that combines a library of CAP-defined metrics for pathologists with a performance-tracking database and a reporting module for generating OPPE and FPPE reports. The application was designed to give subscribers flexibility in data-entry frequency and performance thresholds, as well as to design custom metrics. (See “Top five metrics in use,” page 42.)

Conceived by Dr. Karcher and Dr. Hunt and developed by the CAP’s Quality Practices Committee (of which Dr. Karcher is a member), the application comes loaded with more than 100 anatomic and clinical pathology metrics, from autopsy and blood banking to point-of-care and surgical pathology. “The metrics are completely fleshed out,” Dr. Karcher says. “Everything you would need to know about that metric is there—the description of it, the competency domain. And each metric has a benchmark which the subscriber can choose to use or not use. We recommend a threshold with many of the metrics. We give them references. So it’s really a complete package. This is a turnkey solution.”

Subscribers, whose yearly subscription payment consists of a base price plus an additional fee for each provider they enroll, enter their own providers and data and generate their own reports. “It really gives the subscriber almost total flexibility to design it the way they want,” Dr. Karcher says.

Though early Evalumetrics adopter Raynette Kaneshiro, PA(ASCP)—who maintains performance records for 13 pathologists at The Queen’s Medical Center, Honolulu—appreciates the application’s flexibility, it is its ease of use she praises most. Previously, she relied on Excel, and she had to transfer the data from Excel into another format to report it to her hospital OPPE committee.

Clunky as that method was, it represented an improvement from even earlier days, when her supervisor “used to take all the reports and working drafts and count them by hand and type everything into a Word document and then send it to the OPPE committee,” she laughs. “He used to bust out the calculator. Now it’s easy for me to compile the data and enter it into Evalumetrics, and he just reads the reports and signs off on the bottom.”

In addition, she says, Evalumetrics has helped highlight problems that may have otherwise gone unnoticed: “It brings up issues that weren’t really noticed before. For example, we might discover that for a pathologist who is away most of the time, maybe we need to change the values, because they’re not even there to sign off on cases. So it identifies not only pathologist-related issues, but processing and protocols issues as well.”

Mark vanGorder, MD, a pathologist at Franciscan St. James Health–Olympia Fields and Chicago Heights, Ill., became an Evalumetrics subscriber only recently. He is rolling it out now across a newly expanded group of pathologists and facilities across northwest Indiana and northeast Illinois. Like Kaneshiro, Dr. vanGorder and his colleagues previously relied on a homemade system to keep track of OPPE and FPPE data.

“The vast majority of it was in paper form,” he explains. “The system was started by one of the more senior people in our group, and as new medical directors came in, they basically borrowed it. I think it worked well, but it had never been computerized and organized.” With the expansion of his group, that had to change. “Our administration wants us to standardize the evaluation of the pathologists and the performance of the pathology departments,” Dr. vanGorder says, “and as part of the process, they were quite interested in the Evalumetrics system.”

One of Dr. vanGorder’s favorite aspects of Evalumetrics so far is its use of automatic monthly or quarterly reminders to users to enter data. “One of the problems with QA programs is that they take time,” he says. “It just takes time, between committee meetings and surgical sign-outs and CAP inspections and all the other things pathologists do. You’re not really crazy about someone giving you another task: ‘OK, let’s start comparing the thyroid FNAs with the thyroid resections,’ or ‘Let’s re-review your flow cytometry data,’ or ‘Let’s re-review your smear interpretations.’”

In his view, Evalumetrics’ reminders make it more difficult to put off OPPE/FPPE-related tasks. “If you’ve signed up, you know you have to enter the data into it monthly or quarterly, and it reminds you if you don’t. It’s like a little alarm clock. It kind of forces you. It’s like counting calories,” he says.

Once Evalumetrics has been fully rolled out across all of his institution’s facilities, Dr. vanGorder hopes to take advantage of its customizable nature. “One of our ideas is to begin to follow cytology and histology correlations—thyroid FNAs and thyroid sections and bladder biopsies and bladder resections—to get a sense of the atypical and concordance rates,” he says. “But since we have to do this across the board, it will be slow going. Every hospital has a slightly different mix of cases and different specialties.”

As for broader utilization trends, Dr. Karcher says that so far, “we’re being validated that we developed the right metrics” for Evalumetrics. Among the most popular: turnaround time for signing out of small biopsies, comparison of frozen-section diagnosis to final diagnosis, and the rate of amended reports.

In particular, he says, the clinical pathology metrics appear to be especially well received by users. “Metrics in anatomic pathology are much easier for heads of groups to design,” he says. “In five seconds, the average head of a group could list four or five things for anatomic pathology that they would like to measure. In clinical pathology, it’s much more difficult, and we feel one of the great advantages of this program is that we have written a whole series of metrics for clinical pathology. People are now latching on to these and recognizing these are important metrics that they may have had a difficult time designing themselves.”

Will Evalumetrics’ usefulness eventually extend beyond OPPE/FPPE? Yes, say its creators. In the future, subscribers will receive cross-institutional metric utilization reports from Evalumetrics so they can benchmark their own quality management efforts against those of other subscribers with similar demographics.

Top five metrics in use

There are 101 CAP-defined metrics covering 18 pathology practice areas, with current subscribers electing to use 89 of them. Evalumetrics subscribers are using an average of eight metrics for each pathology provider. About half of all subscribers use the top five metrics listed here: