Feature Story

Promise versus practice of expert systems

What is an expert system

November 2000
Eric Skjei

"There’s clearly some tension between the promise that expert systems technology has held out over the last 20 or 30 years and what it has actually achieved in real-world applications," says Glenn Edwards, MBBS, MD, a pathologist and CEO of Pacific Knowledge Systems, Sydney, Australia."There really haven’t been a lot of off-the-shelf or commercial products made available," adds Dr. Edwards, "and that clearly raises the question of why? What happened?"

This technology has its success stories, Dr. Edwards and others agree. Some laboratory information systems vendors have developed and integrated expert systems technology so well that it functions so much in the background that many of their customers may not know they are using it daily.

But examples such as this are the exception, not the rule. "In the labs I’ve had experience with," Dr. Edwards says, "the LIS itself will sometimes have some basic rules capacity built into it, but the application of these tools tends to be very narrow, without a great deal of functionality."

Dr. Edwards and others contend that clinical laboratory expert systems generally have not realized their full potential. They have been largely relegated to discrete, niche areas, such as determining which hematology samples require a film, and function where the logic that this technology automates is limited, well defined, and not likely to change much.

Much is lost as a result, says Dr. Edwards. When, for example, such relatively limited rules in an LIS trigger on a high cholesterol or trigylceride result and offer a comment that goes no farther than "’common causes of raised lipids are genetic, diabetes, obesity,’ and so on, that’s pretty shallow," says Dr. Edwards. "That level of functionality, that level of comment, is not very specific, and it betrays a lack of depth that is often reflected in the response of clinicians, which is by and large to ignore such comments or even to actively critique them."

Bruce Friedman, MD, too, says expert systems technology has not fulfilled its potential. "It’s the little [inference] engine that couldn’t," he says, citing his own laboratory’s experience. Dr. Friedman is professor of pathology, University of Michigan Medical School, and director of clinical support information systems, University of Michigan Health System, Ann Arbor.

"We’ve only been successful in deploying sets of rules that have internal relevance for the lab, a kind of lab-directed reflexive testing that doesn’t raise political implications or consequences," says Dr. Friedman. (By "political implications," Dr. Friedman is alluding to the sensitivity that a clinical laboratory must exercise regarding the issue of medical authority for ordering tests.)

In other words, Dr. Friedman and his colleagues have benefitted from this technology primarily by defining tests where the laboratory, having run test A, can automatically order test B, without having to check back with the initiating physician. This is a classic, if limited, application of expert systems.

Vendors generally have supplied a toolkit but have not supplied robust, preformed rules, says Dr. Friedman. This practice may have been appropriate some years ago, when the technology was relatively new, Dr. Friedman notes, and vendors were concerned about the implications of even appearing to concede some level of medical decisionmaking to a nonhuman process. Offering a toolkit rather than fully developed rules is one way to reduce potential legal liability. But the problem with that strategy, Dr. Friedman explains, is that "generally, these toolkits were complicated enough and required so much overhead that many lab personnel, particularly in smaller labs, really never deployed the rules, even when they purchased the additional rules module."

A common complaint about expert systems technology has been that its rules can be complex and difficult to maintain. Helping a physician say, in effect, to a computer system, "I’m only at the hospital Tuesday afternoon and Thursday morning, and I only want to see test results for patients in the following critical care units," is a potentially useful role for rules technology. But if that physician’s schedule changes, then the rules, too, must be changed. And in many expert systems installations, the physician must update the rules to reflect these changes. If he or she decides to see different results, or only abnormal results, or results for different or additional care units, then he or she must again change the rules to reflect these new criteria. "Forget to make these changes, or forget that these criteria are in place, and you may be deprived of critical information and not even know that you’re not getting it," says Dr. Friedman.

Also impeding the use of such rules is that many physicians still order tests using paper and pencil. If they use an automated order-entry system, it may not be rules-enabled or operate efficiently enough to quickly offer the ordering physician feedback on, for instance, a test order that may be redundant or inappropriate.

"Take a system like ours where, by and large, physicians are using paper and pencil ordering," Dr. Friedman says. In this system, a clerk takes the order form and enters the data into the laboratory information system. If a rule then fires, indicating to that clerk that white blood cell counts have been ordered at four-hour intervals for patient A, but, since they’ve all been normal, the next order for that test may not be necessary, "you’re then placing a burden on that clerical worker that he or she is not prepared or allowed to accept-namely, canceling an order," Dr. Friedman says.

Created unequal Rules have met with mixed success in the clinical laboratory world, concurs Hal Weiner, president of Weiner Consulting, Eugene, Ore., and a pioneer in the use of expert systems technology in the laboratory. Weiner says several factors are responsible.

First, he points out, what is meant by the term expert system varies greatly. "What one vendor may call an expert system may be nothing more than a bunch of tables," Weiner says, "while another vendor’s expert system may not only support traditional rules logic but also handle fuzzy logic, a relatively advanced system capability that makes it possible to understand the nuances of nebulous terms like ’tall’ or ’overweight’ or ’is not consistent with Crohn’s disease.’"

Similarly, some expert systems have a relatively simplistic mechanism for activating a rule, such as only when an order is entered or a result recorded, Weiner says. Others have so thoroughly integrated expert system logic into their basic, routine operation that the appearance of a new piece of data in the database may activate a rule.

Consider, for example, when a mistake is made in entering a patient’s age. The mistake is later discovered and then corrected in the hospital information system, and this change is communicated to the LIS in a way that says, in effect, "Patient A was born in 1973 not 1937."

This change could have significant implications for many aspects of the patient’s care and interpretation of his or her laboratory and other medical information. "Most LISs can, at best, issue a message saying, ’The patient’s age got changed, so you better go look at all the reports,’" says Weiner. "Newer, more advanced systems are smart enough to actually go back and recalculate all of the normal ranges and everything else pertaining to that change, as well as issue alerts and messages to physicians and others."

Cerner Corp., which has long been developing and deploying expert systems technology, offers useful conceptual distinctions between simple and advanced levels of this technology in the laboratory. "Internally, we talk about high-value and low-value rules," says Vanetta Wick, product specialist for Discern, a rule-based event-driven expert knowledge system. Any rule that solves a problem for a client, "regardless of how complex or simple it is in our terms," becomes a high-value rule, she says. To illustrate, Wick cites the case of Dartmouth-Hitchcock Medical Center in New Hampshire, which has written a rule that simply tracks and reports venipuncture charges.

"That client could not figure any other way, through their billing system or any other system available to them, to obtain a correct venipuncture charge, one charge, regardless of how many specimens were being drawn, so they did not charge for venipunctures," says Wick. Employing the expert systems capabilities in its Cerner LIS, staff members at the medical center wrote a rule that, running 24 hours a day, 365 days a year, is allowing them to charge appropriately and is delivering a significant financial return. "Some people might call that a low-value rule," says Wick, "but to the client who is running it, it is a high-value rule."

A second reason expert systems technology has not fulfilled its potential is because it primarily has been applied inside the laboratory.

"There are systems out there that have the capability to reach outside the laboratory," Weiner says, "and I think that’s where we’re going to see significant advances in the usefulness and value of this technology." Such advances likely would affect the clinical and the business side of laboratory operations.

"Consider the situation where you’ve received a test order for a patient who is a member of an HMO," Weiner says. "Before you do that test, you really would like to be able to review that patient’s eligibility and make sure you’re going to get paid." Now you can, depending on what technology you use. Some expert systems can log onto the Web, access that HMO’s membership database, and verify the patient’s eligibility before the order is entered.

"One of the prime uses of our product is billing," says Wick. Cerner has a tool, she says, "to enable a customer to cut through the complexities of ordering this and not ordering that, calling it this and not calling it that, assigning this code or that code, to get the billing they need for their site straightened out so that it happens correctly."

Citing another example, Wick says: "One of our clients had a billing issue in the anatomic pathology area. Charging for specimens as they came in was a chronic problem for the staff members who processed those specimens. They constantly found themselves looking through code books, flipping pages, trying to figure out what the right code and charge should be."

The client solved the problem by writing a rule that associated the correct charge with the specimen when it arrived. The laboratory then wrote another rule that interfaced the laboratory software with a device that produces labels for the specimens, applying the correct identifiers, so the staff would not have to create the labels manually. "That application was so successful that the producer of the device asked how it had been done and whether he could make use of it for his other clients," Wick says.

Reaching beyond the laboratory into other clinical areas holds great potential for expert systems technology, proponents agree. Take the patient who is placed on digitalis, says Weiner. "One of the things that an expert rule system should be able to do is see if this patient has a potassium test scheduled at some point in the future, and if not, suggest that this be done," he says. And if the result of that test is low, rather than just report the low number, the expert system also should be able to remind the physician that the patient is on digitalis and recommend that he or she consider putting the patient on a potassium supplement.

Adverse drug events are an obvious fit for this functionality, and some vendors are targeting this area. According to Discern product manager Jana Malinowski, Cerner has taken a proactive approach to implementing adverse drug event rules. She cites Our Lady of the Lake Hospital, Baton Rouge, La., as an example of a site where an ADE alert system recently was implemented.

Keeping the technology fresh
Distinguishing between effective and less-than-effective expert systems, some vendors say, has everything to do with the sophistication of the technology on which they are built. Soft Computer Corp., or SCC, has achieved success in this area, according to CEO Gilbert Hakim, by staying ahead of the technology curve. SCC has, for several years, written its software with the complexity of expert systems in mind, and, perhaps more important, its products take advantage of parallel processing.

"Most systems using purely relational or object-oriented databases slow down if you apply thousands of rules," Hakim says. "Ours doesn’t, because we use two systems in tandem all the time, two live databases running in tandem." Consequently, SCC’s product can handle sites that process as many as 10,000 requisitions a day.

Expert systems can be complex and pose maintenance challenges, says Hakim. "When you go into an environment with 2,000 tests and you have 2,000 doctors, each with at least three or four rules of his or her own basically involving combinations of diagnosis, age, sex, and multiple tests done at different times, possibly at different sites, and sometimes under different medical record numbers," he says, "it is a given that your expert systems installation will encounter and must be able to handle complexity."

What do client physicians obtain as a result? "The expert systems technology in our software helps them find patterns in the results in the patient’s history," says Hakim. "Every discipline finds something different-cardiologists have four or five patterns they look for, endocrinologists look for another set of patterns. . . . They love it."

The business side of health care also benefits, adds Hakim. Rules technology can help reduce unnecessary testing, not only by reducing test orders but also by not performing all the tests specified in a profile. "For example," he says, "instead of ordering a T3 and a T4, the system orders a TSH, and only if the result is abnormal are additional tests reflexed."

Similarly, says Hakim, autoverification of results is a process that lends itself to rules technology but that can become complex. "In a traditional system, the practice might be to say, ’If this test result is above this or below that level, let it go,’" he explains. But that kind of logic does not take into account the significant variations among patients in age, gender, disease, other test results, and so forth. "A lot of times, a normal result is abnormal for certain other groups of patients," Hakim says, "so you can’t use a standard range-and this is where a rules-based system like ours becomes very effective."

A wizard in the laboratory
"The clinician—the primary care physician who is the lab’s client—really needs expert input from specialists," says Dr. Edwards, of Pacific Knowledge Systems. Clinicians need the assistance of specialists who can examine a patient’s test results and offer insight into what those results mean in the context of everything the laboratory has determined about that patient, including demographic data, clinical data, and any other past or present test results.

For example, Dr. Edwards says, it’s important for interpretive reports to be able to say, "’Here’s a patient with hyperlipidemia, and I can see this patient has been in a similar state for the past two or three years. I can see that the patient is on a lipid drug, and, oh, by the way, it appears that this patient has never been screened for diabetes . . . that might be an omission, something we can help with.’" The optimal role of the clinical pathologist, at least in the Australian health care system, he says, has been to supply this kind of specialist feedback, observations that put the results into a context.

This kind of deeper interpretation has, however, been the exception rather than the rule, Dr. Edwards adds. Why? Because pathologists usually work in a highly automated setting and because thousands, or even tens of thousands, of orders and results pass through a typical laboratory each day. Furthermore, little opportunity exists, even for those with the necessary clinical expertise, to provide individual interpretive comment on many reports. Pathologists instead tend to focus on niche and highly complex areas, such as fertility. "The bulk of the work passing through the clinical pathology lab really goes without any kind of clinical interpretation at all," says Dr. Edwards.

Expert systems technology has great potential to help alleviate this problem, Dr. Edwards says, and he and his colleagues at Pacific Knowledge Systems have developed a tangible, commercial expression of that conviction. The product, called Lab Wizard, has been in clinical use for two years. "Lab Wizard makes management of the clinical commenting process relatively simple and straightforward," adds Dr. Edwards.

The development of Lab Wizard is a step forward, he says, in part because it is explicitly designed for those in the laboratory who understand and drive clinical commenting-pathologists and clinical laboratory scientists. "This is a substantial breakthrough," says Dr. Edwards, "first, because it means that ownership of the clinical commenting process is retained by the pathologist, and, second, because the software does not require those individuals to understand what’s going on under the hood. There’s no need for them to know how to program or maintain computer code, decision trees, or anything else."

Lab Wizard presents a relatively simple graphical user interface, he adds, one that guides the user through the process in a relatively intuitive way. The process is based on comparing and contrasting real reports the pathologist would see in his or her routine work and on eliciting from the user the key features of those reports and how they are distinguished.

Consider the patient who has abnormal lipid tests and has been screened for diabetes. In the normal course of events, pathologists, if they had time, would offer interpretations of this patient’s case by writing a note or a comment about whether those results had met target levels, whether the results indicated improvement, how those results may have changed over time, and perhaps what was done with the results of the diabetes tests. Pathologists would assemble this information to help guide the primary care physician in making future management decisions. "By helping assemble and integrate all of that data, Lab Wizard helps the pathologist reach that point where he or she can offer real support to the GP," says Dr. Edwards.

For about 18 months, a handful of pathologists and laboratory scientists have been using Lab Wizard in their daily work at a Pacific Knowledge Systems’ test site lab in Perth, Western Australia, which processes more than 3,500 samples per day. "They require no support from the IT department there," says Dr. Edwards, "other than routine system backup and maintenance, and they need no special computer skills or training, other than typical familiarity with desktop computers and standard business applications."

These users, says Dr. Edwards, are driving the development of knowledge bases, or "projects," as he refers to them, across a range of applications that span many laboratory functions. "At last count, there were 34 test panels included in some 16 projects dealing with such areas as diabetes, lipids, thyroid function, endocrinology, fertility, viral serology, iron studies and anemia," he says.

Does the product help meet the demand imposed by high test volumes? Yes. Do pathologists find they now need to review every report that leaves the laboratory? No, says Dr. Edwards. The laboratory chooses criteria for reports that it autovalidates and sends to the physician client without review. It also separates out a select subgroup of other reports, which are then reviewed and released by laboratory specialists. These reports typically are unusual, complex, or complicated, or for other reasons require manual attention and sign off.

The feedback from laboratory clients has been positive. "The primary care physicians, the GPs, really do relish this level of professional feedback from their expert colleagues because it helps them sift through what would otherwise be a lot of raw data," says Dr. Edwards. "Both partners in the process-pathologists and GPs-have found that this is a way of restoring and invigorating their clinical role." The laboratory is not put in the position of telling general practitioners what to do, and conversely, pathologists aren’t threatened by a technology that may diminish their role. "In fact, it’s empowering them to do much, much more," Dr. Edwards says. "They can see that the lab reports now have much more value than before, and they see that they are the ones driving this improvement and making it happen."

Recognizing that some laboratories need preformed comments and not just a toolkit, Pacific Knowledge Systems plans to offer not only Lab Wizard, but also seed knowledge bases in such areas as diabetes, lipids, anemia, serology, and hepatitis. "We will be working with partners-teaching hospitals, for instance-to build these seed knowledge bases, and we already have one hospital interested in working with us in that way," says Dr. Edwards. "Their pathologists with particular expertise in certain areas will help us build up knowledge bases which we can then make available to other labs that may not have that expertise."

Don’t forget the customer’s role
Many vendors perceive the limited impact of expert systems technology, says Mary Beach, director of ancillary systems for McKesson HBOC, and they share the frustration others have experienced as a result of this unrealized promise. But she adds that much, if not most, of that limitation can be attributed to the highly variable way customers relate to the technology.

"I think how an expert system is implemented and how it fulfills the inherent promise of the technology has a great deal to do with the particular customer that is utilizing it," says Beach. "If they’re designed properly, expert systems place all the control in the user’s hands and allow the system to be tailored and to interact in whatever way the lab can dream up."

Beach sees customers interacting with McKesson HBOC’s laboratory systems in a variety of ways, ranging from the creative and exploratory to the indifferent and unimaginative. It isn’t always a function of how advanced the technology is, she says. Even users of some of the vendor’s legacy products have become adventurous, perhaps because they have been using their LIS for many years and have become comfortable with it, she adds.

"They’ve had a system for a long time," says Beach, "and they’re now saying, ’Well, what else can I get it to do?’" In many of these cases, she adds, customers are pushing the technology to realize its full potential in such areas as interpretation and clinical evaluation. "They’re looking at lots of different lab results, evaluating them, and coming up with either an alert or an interpretation for a physician," she says.

Pathways Laboratory, McKesson HBOC’s new client/server LIS, combines much of the power of expert systems that was built into its predecessor, Advantage, explains Beach. Advantage itself was based in large part on the Advanced Laboratory Systems product acquired by HBOC a few years ago. Moreover, adds Beach, HBOC also inherited technology from 3M that was integrated into its PathLab III product, which antedates Advantage.

As a result of this evolution, Beach says, expert systems technology has become an integral and extensive part of Pathways Laboratory, so much so that some customers may not know if and when this technology is working for them. "It occurs so far behind the scenes they don’t even realize it, and may even say, if asked, that they aren’t doing much with expert systems," she says. "But at the same time, they are probably doing quite a lot—using calculations, alerts, reflex ordering—functions that are in fact rooted in and dependent on rules."

Other users, however, are aware of the presence and power of these tools and use them to fulfill and extend the clinical value and processes in the laboratory. Beach sees this as particularly important in the area of taking multiple laboratory results from more than one instrument or work area in the laboratory and combining them to provide more information to the clinician.

"In older systems, the expert technology was often limited to a single test or a specific instrument," she says. "Now, though, to be able to take hematology results, chemistry results, pathology results, and to then be able to say, ’Okay, doctor, this is what you should be thinking about,’ is an exciting change."

McKesson HBOC recently announced a product, Horizon Care Alert, that will allow users to build expert system functionality across an organization, making it possible to immediately alert clinicians to problems involving interactions between medications. Expert systems were, in the past, often difficult to maintain, says Beach, who outlines several steps McKesson HBOC has taken in response. "One is to permit the sharing of rules," she says. "So now one customer who writes a fantastic rule can actually save it, and it can be imported into someone else’s system."

The company also has made it possible to create a general rule and then show how to apply it in multiple instances. For example, the calculation involved in a 24-hour urine test is essentially the same calculation for many other tests. "So we create the rule once, then explain that it also applies to this test, that test, and those other tests," says Beach.

Finally, McKesson HBOC has taken what might be called a staged approach to implementing the expert system functionality in new installations. "We want to have a lab go up with what we call a base of expert rules, all the different things that are common to all laboratory operations," says Beach. That base is installed as part of the standard implementation. But only after the site has gone live and the customer is comfortable with the product does the company go back in and teach them how to use the expert systems technology. "We have found that expecting the lab to try to understand rules up front, during the installation, when they don’t even really understand how the system itself functions yet, is too much of a burden," she says.

McKesson HBOC also is concentrating on finding ways to apply expert systems technology to not only results processing but also ordering and other front-end processes. "So customers can set up rules that require, for example, that the user check for other information," Beach says. "For example, that he or she obtain screening results before doing the detail titer test, or, for a verbal order, that the system automatically fax a document to the doctor to confirm it."

Eric Skjei is a freelance writer in Stinson Beach, Calif.