Feature Story

Probing employee competency in POC testing

May 2004
Anne Ford

In the 16 years since CLIA ’88 began requiring laboratories to evaluate employee competence, surprisingly few studies have examined just how well labs are complying. Competence assessment "has not become an industry the way continuing education has become an industry," says Paul Valenstein, MD, president of Pathology and Laboratory Associates, Ann Arbor, Mich. "There is remarkably little interest in how competence is being assessed and whether competence assessment methods mean anything in the real world."

If he were a patient, he says, he’d care more about whether the people taking care of him were competent to perform their tasks than whether they had learned something new from a continuing education program in the past year. "If you’re about to fly on an airplane, do you care most about whether the pilots learned something new about aviation last month or whether they know how to fly the plane?"

So what are labs doing to make sure their employees don’t crash and burn? A recent Q-Probes study, "Competency of Point-of-Care Glucose Testing Personnel," asked just that. The study, which surveyed employees of 114 hospitals during the last three months of 2003, was coauthored by Dr. Valenstein and Peter J. Howanitz, MD, director of clinical laboratories at State University Hospital and vice chairman and professor of pathology at State University of New York Health Science Center, Brooklyn.

The study asked participating laboratory management to fill out a questionnaire about their competence assessment procedures, review employee records to determine whether they were complying with their own assessment requirements, and then to administer two written case studies to POC glucose testing personnel "to see what they really knew," Dr. Valenstein says. Data were taken from three nursing units in which POC glucose testing was frequently performed (most often a medical unit, ICU, and surgical unit). The records of 3,136 employees were reviewed. Just over 1,900 staff members answered questions about case study 1, and 2,014 answered questions about case study 2.

Drs. Howanitz and Valenstein found that almost all of the hospitals surveyed (98 percent) had standardized POC glucose competence assessment programs. Almost 96 percent of POC personnel had been tested for competence in the past year. And in all but one percent of hospitals surveyed, employees underwent competence assessment each year whether they had made performance errors that year or not.

Dr. Valenstein, chair of the CAP Quality Practices Committee, was gratified by most of the study’s findings. "The average facility did quite well, both in developing a program and in following through to make sure employees were being evaluated," he says. He was distressed, however, by the discovery that four percent of participating hospitals allowed employees to continue performing patient testing after those employees had failed competence assessment. "Competency assessment is not just an activity to be checked off in a quality logbook—it’s supposed to allow only the competent to practice," he says. "When an employee fails a POC competency examination, it should throw up a red flag that says, ’We need to teach you how to do your work before you go back on the floors.’" The Q-Probes data analysis and critique puts it in starker terms: "[I]f a catastrophic event occurred and incompetent laboratory testing personnel were responsible, the institution and its supervisory personnel would be in an indefensible position."

In addition to recommending that employees who fail competence assessment be suspended from patient testing until they demonstrate proficiency, the study suggests that competence assessment be part of an employee’s regular annual performance review. That recommendation stems from the discovery that the participating hospitals that combined competency assessment with annual performance review were less likely to forget to administer it. Bundling it with an employee’s regular yearly review means that "no assessments are left to the last minute shortly before an inspection," reads the recommendation.

The assessment methods themselves varied among hospitals. Direct observation was used by 67 percent of participants, with review of proficiency testing and daily quality control coming in second with 56 percent. (Many laboratories used more than one method to measure competence, so the individual percentages add up to more than 100 percent.) Written testing, review of patient results, and use of manufacturers’ materials were other common methods. Two methods, on average, were used per employee. "There are particular methods suggested to measure competence that are spelled out in CLIA regulations, but we weren’t able to demonstrate that one method is better than another," Dr. Valenstein says.

Still, he would like to see case studies used more frequently. "A correct response indicates that an employee is likely to solve problems in conformance with good medical practices and an institution’s policies," Dr. Valenstein says. Participants in the Q-Probes study were asked to write answers to two common problems: what to do when the linearity of the method is exceeded and what to do when QC values are out of range. A response was considered correct if it conformed to the institution’s written policies. For both case studies, approximately 10 percent of respondents answered incorrectly. "That doesn’t mean 10 percent of them are not competent to perform POC testing," Dr. Valenstein adds, "because in any testing situation some people are going to become confused and not apply what they know. But an incorrect response should prompt a closer look at an employee’s capabilities."

Meanwhile, who should be responsible for assessing POC competence—the laboratory or the nursing unit? Study results suggest the laboratory should at least share the duty. "When the clinical lab was involved in administering the assessment program, more employees were assessed," Dr. Valenstein says. At institutions where laboratory personnel were responsible for assessment, 100 percent of POC testing employees had been evaluated for competence in the last year. That percentage dropped to 95 at hospitals where the lab and the nursing unit shared the responsibility, and dipped to 94 if the nursing unit alone was responsible.

"The difference was statistically significant," Dr. Valenstein says. "So while nursing staff may be very capable of performing POC glucose testing, at this point they don’t seem to be as good at administering a competency assessment program."

The study suggests also that hospitals make greater use of manufacturers’ competence assessment products. These products are tailored to the idiosyncrasies of the instrument the hospital has purchased and are generally provided by manufacturers without charge. Interested readers are referred to several studies, among them "Characteristics of educational software use in 106 clinical laboratories" (Astion ML et al. Am J Clin Pathol. 2002; 18: 494-500) and "But can they do it? Clinical competency assessment" (Schwabbauer M. Clin Lab Sci. 2000; 13: 47-52).

Overall, Dr. Valenstein says, competence assessment practices have improved since 1996, when he and Dr. Howanitz, chair of the CAP Point-of-Care Testing Committee, conducted a similar study on competence assessment of lab personnel: "I believe laboratory management is more comfortable with the idea of measuring competence now. We’re learning how to do it and how to remind ourselves to do it regularly."

Anne Ford is CAP TODAY senior editor. For more information on Q-Probes and how to enroll, call CAP customer service at 800-323-4040 or 847-832-7000 option 1#.