Anne Paxton
August 2021—The downtime manual that the anatomic pathology laboratory at the University of Vermont Medical Center maintained in 2020 was never intended to be used for dealing with a cyberattack.
In fact, it wasn’t actually a manual. It was a laboratory-wide policy essentially consisting of one instruction to be used in the event of a power failure or short-term IT disruption or other emergency: “Bring everything to a halt.”
In anatomic pathology, “Our downtime protocol was: You stop in your tracks,” says dermatopathologist Anne M. Stowman, MD, director of surgical pathology operations at the University of Vermont Medical Center (UVMMC). “For the urgent/emergent specimens, you get out your paper logs, you do paper recording of the cases coming in, and you handwrite your cassettes, your descriptions, your slides.” That would be a bit slower and less efficient, but it would work for brief, temporary outages and disruptions.
But the cyberattack that UVMMC experienced in October 2020, cutting off the labs’ access to the medical center’s information technology systems and disabling operations for more than three weeks, was an abrupt wake-up call. “A cyberattack really hits the AP laboratory very hard. In the clinical lab, the analytics are still functioning; the machines are still functioning. But for producing results in AP? We were essentially dead in the water,” says Alexandra N. Kalof, MD, UVMMC division chief of anatomic pathology.
She sees the impact of the cyberattack on the AP laboratory as extreme and unprecedented. “It was like nothing I could have possibly imagined,” Dr. Kalof says. “I mean, the impact COVID had on us paled in comparison to this attack.”
The cyberattack led to a massive rethink of the risks that AP laboratories face and how they should respond when an IT crisis occurs. One legacy of the cyberattack is a 37-page Anatomic Pathology Downtime Manual, probably more aptly named a disaster manual. Developed by the UVMMC AP lab team and published in March, the manual is tailored to the specific needs of the AP lab. It has helped turn UVMMC’s unwelcome cyberattack experience into a useful resource for AP laboratories across the country.
“The Downtime Manual that emerged after the cyberattack was a cumulative total of everything we’ve learned and the processes that worked the best,” Dr. Kalof says. Intended as a “road map to provide guidance and a framework to anatomic pathology” in the event of a prolonged system downtime, the new Downtime Manual outlines an overall “incident command structure” to confront the range of issues likely to arise for an AP lab in the high-risk areas of accessioning, processing specimens, reporting, and billing/reconciliation.
That incident command structure designates one section medical director and one staff member to lead each of those areas plus communications, services schedule, logistics/supplies, and safety. Included are templates that provide instructions for core tasks like manual printing of cassettes and slides and forms for tasks like scheduling staff for phone results triage. Among useful cautions in the Downtime Manual: a warning to store the manual itself, and all necessary downtime forms, outside the networked system.
[dropcap]D[/dropcap]r. Stowman, one of the coauthors of the Downtime Manual, didn’t worry the first day in October when the Epic EHR stopped responding, since seeing the “spinning wheel of death” on the computer screen was something everyone had experienced. “We thought it was just an outage or a downtime or routine maintenance.” Never before, though, had there been a downtime of more than 24 hours. “It wasn’t until several hours later that we wondered, why has nobody said anything? There was no communication from the lab or hospital administration at that point.” Work on the processor was still being processed; slides could still be cut to a certain point, she says. “We just couldn’t enter any data on the computer.”
For Dr. Kalof, also a coauthor of the manual, the cyberattack came to light when the AP manager said during a conference call that some messages were coming through and the computer was glitchy. But looking back, she says, “we had a prodrome” before the attack. “In the weeks prior to the cyberattack things were operating oddly. There were glitches in communication. Outlook would freeze. Another program malfunctioned a couple of days before the attack and we all kind of thought this is just another glitch, or they’re going to update something. But it didn’t come back.” All of this may have been occurring while cyberattackers were slowly punching holes in the IT system.
After turning her computer off as a first reaction to news of the cyberattack, Dr. Kalof found she needed to get up to speed quickly on a number of things to manage the crisis. By three days in, the backlog was mounting and the faculty were anxious. “They didn’t understand why we couldn’t just put cases through. People were ready to sign cases out. But we couldn’t—because we didn’t have any way to report the results.”
In the square footage each occupies, the UVMMC AP lab and the clinical lab are about the same, Dr. Stowman says. But the two labs’ experiences of the cyberattack were significantly different. A key difference, Dr. Kalof says: “We couldn’t generate actionable reports. One AP specimen goes through so many steps in grossing, then histology, then interpretation by the pathologist. And those steps couldn’t even start until we figured out how to label and identify the specimens.” That was a complex process. “We had to map that out and make sure it was rock solid before we processed anything in the laboratory.” (When the lab was later able to transition from handwritten slides and blocks to printed ones with the name of the surgical accession number, she adds, “that was a huge relief from a patient safety standpoint.”)
For the clinical lab at UVMMC, on the other hand, the added complexity of generating the report wasn’t there. “It seemed like their biggest issue was how to get those results back to the providers. Throughout the entire downtime, their machines were still running. But we had that lag of a week and a half where we were trying to figure out what our workflows were.” As a result, the AP lab didn’t start to think about establishing a call center until about two weeks into the attack.
“We didn’t know we would have to build an anatomic pathology laboratory from the ground up in, essentially, a weekend,” Dr. Kalof says. That involved creating a new report, making sure it was in compliance, getting results to providers, dealing with the backlog, and deciding which cases to send out to reduce the volume. But the need to employ manual processing to make sure there was little risk of error was one of Dr. Kalof’s top worries. Along those lines, she made a trip to Staples to secure dozens of flash drives, five Dictaphones, and 10 “FAXED” stamps for the lab. The Dictaphones, as it happened, went unused and were returned to the store, but the FAXED stamps became an important tool for verifying that reports had been transmitted when no computer was around to do the job.
[dropcap]O[/dropcap]utages generally have an asymmetrical impact on laboratories versus other hospital departments, Dr. Stowman notes. “The operating room, for instance, wouldn’t operate in the dark, but they could keep seeing patients.” So in any outage, the clinical and AP labs would continue getting specimens. And at UVMMC, both had the same instinct: “We’ve got to stop the inflow.”
UVMMC’s clinical lab, however, would typically have 5,000 specimens per day requiring results to be sent out by fax, while in the AP lab, “we maybe had 200. But our process is much more error-prone because of the number of people touching each AP specimen compared with a CP specimen, which goes on a machine and then comes off.”
With its hands-on procedures, the AP lab could seem less vulnerable to hackers who figure out how to disrupt or bring down a hospital’s IT system, whether by installing ransomware or through other malicious code. But most manual processes can’t escape reliance upon IT.
During the cyberattack, “the problem we had was there was no connectivity in the lab, there was no way of tracking the specimens, and there was no backup for that,” Dr. Stowman says. “Our computers couldn’t communicate with each other so that we could print our slides. The grossing technologist’s description of the case could not get transferred to the pathologist’s desk for the final diagnostic portion of their report. So the reason the lab doesn’t say, ‘Oh, we’ll just go to manual processing’ is that with the manual nature of AP, there is too much room for error.”
In its near-paralyzed state, the laboratory early on decided to offload some of its dermatopathology and GI specimens to a neighboring hospital 45 minutes away. “We siphoned off almost a third of our high-volume GI and dermatopathology to our network partner and we decided to keep the complex cancer care cases,” Dr. Kalof says.
“We packed them up, cataloged them, emergency-credentialed our hospital’s pathologists, and got a cryostat and two pathology assistants and a histology tech to drive them to our partner hospital each day,” Dr. Stowman says. “That offloaded a significant amount of stress.”
Contributing to that decision was the laboratory information system that the lab had. Because the UVMMC AP laboratory had recently transitioned to Epic’s Beaker for its LIS, “we had no independent LIS,” Dr. Kalof says, so the LIS went down along with the EHR. “But in the very first days of the cyberattack when we chose to funnel our significant backlog to our network partner, in part it was because they were undergoing a phased integration and still had Meditech; they had not yet transitioned to Beaker. We were lucky because they were able to take our samples and generate reports in their own system.”
As other departments in the hospital started relying on WhatsApp Messenger, the popular chat and phone app, to stand in for the disabled phone system, the AP lab followed suit starting on day two of the downtime. “We adopted it immediately,” Dr. Stowman says, because none of the staff emails were functional and the communication gap was crippling some people who were working remotely. “We needed a bidirectional information system for the faculty to communicate and the same for staff,” Dr. Kalof explains. “WhatsApp is not HIPAA-compliant, so we didn’t use it to transmit patient information, but at least it left us able to transmit updates and let people know what was going on.”
Remobilization of staff was required right off the bat. “The grossing room was a significant bottleneck,” she recalls. “A person who was all gloved up and touching patient specimens could not type, so we developed a scribe system. Now every grossing bench has a grosser and a scribe. The grosser would talk aloud and the scribe would take down their words on a piece of paper that was then sent to the AP support room.” There, the report waited in a queue to be typed and printed by the one laboratory computer with an attached printer, merged with the final diagnosis, then for a pathologist to proofread it.
Normally, answering the phones and coordinating cases and send-outs, not transcription, makes up the majority of the work of AP support staff. But during the cyberattack, “they went from zero to 100 percent of their job consisting of reporting and transcribing these reports,” Dr. Stowman says. Since the AP support staff time was dedicated to this process, it was crucial to adopt a triage system to handle the phones. “We developed a schedule from 7 AM to 5 PM to have different people displaced within the lab to answer the phone and take down all the information needed to call back or to discover a particular result in whatever way they could. This was a lifesaver in alleviating the burden on the AP support staff.”
Similarly, all of the lab’s cytology service was redeployed—some of them became scribes in partnership with grossers while others were assigned to answering phones—since all Pap tests had to be redirected. “We processed no Pap smears during the downtime. We got in touch with two Mayo Clinic labs and a sister hospital in New York and diverted that workload entirely.”
With the urgent and ad hoc nature of some of the staff redeployments, it soon became clear that a special management structure was needed. “We were having miscommunication about staffing and not understanding that AP support was burdened to a significant degree, while histology was surviving, and cytology was sitting idle.” By forming an incident command team, the lab was able to assign one staff member to histology, one to accessioning, and so on. “And we said we are going to meet twice a day, at 9:00 AM and 3:00 PM, to report and work through problems.” Those daily huddles “were a huge game changer for us as well,” Dr. Stowman adds.
Fortunately, the many restrictions and new procedures stemming from the pandemic did not provide added aggravation to the AP lab. “Clinical pathology had COVID samples coming in that needed to be handled differently, and in AP we didn’t,” Dr. Stowman notes. “So I think the pandemic probably had less impact on AP. Since we had already become accustomed to Zoom and Microsoft Teams meetings, perhaps the pandemic was a blessing because we knew how to meet remotely and efficiently.”
Initially, the concerns of most clinicians had centered on their lack of access to reports needed for clinical care, Dr. Stowman says. “They would say, ‘I took this patient’s history two weeks ago and I haven’t looked at the result yet. I’ve got the patient coming in this afternoon and I have no idea what their diagnosis is. Can you help me find the report?’” But the AP lab had no way to find that report. “There were also a number of biopsies that were signed out basically at the start of downtime and never reached the in-basket of the provider, or perhaps they reached the in-basket of the provider but the provider never got to it in time. So we had no way of tracking those results.”
“But once we moved to the processing and started reporting and filing, we had no complaints from our providers because we faxed diligently all the reports as they came”—using the FAXED stamp as evidence. “There was a lag, but no one was complaining about a lag during the cyberattack.”
By day 22 of the cyberattack, when the team had gotten into a comfort zone with its new procedures, word was out that they would need to prepare for uptime; the system was prepared to come back in three days. “It threw us back into a whole new round of stress because we thought: What are we going to do about billing? About getting all these patient reports that are sitting in a file cabinet into the EHR? There were all these unknowns,” Dr. Stowman says.
[dropcap]F[/dropcap]or this cyberattack, the UVMMC AP lab had to learn downtime processes on the fly—but it doesn’t plan to have that happen again. “As far as preparedness goes, I do think we have the necessary stockpile of items, and we know what to stockpile for another prolonged downtime,” Dr. Stowman says. She advises labs to be especially sure of one thing—that they have a current printout of their contact lists. “Having a list of every clinician is paramount, so every lab needs to make sure their Rolodex is updated and printed once a week, because if there is a cyberattack all of the shared drives that hold contact information will go offline.”
WhatsApp, daily huddles, phone triage, the incident command structure—they were all part of what Dr. Kalof and the laboratory team called “coordination in crisis.” With preparedness, the laboratory can use coordination in crisis and not only mitigate the effects of an attack but also, in her view, be ready for any acute or crisis-like situation.
It’s critical that AP labs understand the trends that make labs increasingly vulnerable to cyberattacks, Dr. Kalof says. “We’re so reliant on the EHR to transmit results. It was bizarre to be so advanced and automated and integrated in the EHR and we took for granted the ease of it. Without it, we reverted to practices from the 1990s.” When a chair at an academic institution in the Midwest mentioned to her that their AP lab had been down for two weeks from a cyberattack, she began to realize how many laboratories had experienced such an attack already but weren’t sharing their experience and their information. “I think their institutions discourage the conversation because of the sense that it would increase their vulnerability even more.” If the stigma of being a victim of a cyberattack is dwindling, that is all to the good, she believes, because willingness to share experiences will help other labs be ready to respond.
The Downtime Manual that the UVMMC AP lab created has already come to the rescue of the AP lab at Scripps Health in San Diego, after a ransomware attack on May 1 took down Scripps’ EHR, website, and patient portal. “They read about us in CAP TODAY and called and said, ‘What do we do?’ It was awful what they were going through,” Dr. Stowman says. With the Downtime Manual, “I do think we significantly helped them through their situation.”
The cyberattack highlighted for Dr. Stowman that routine operations can mask the need for process improvement, especially in academic medicine, she notes, where “changes can occur at a snail’s pace.” During the attack, there was fast emergency action to reset processes, switch to paper, bring in substitute devices, and redeploy staff. It all showed how much can be accomplished when there is a crisis. Post-cyberattack, the virtual daily huddles became part of the new routine. “We started those to be on top of things and maintained them twice a day for a couple of months, then once a day. Now we’re down to once every other week, but we still do them with the same group of people and find them incredibly useful.” Ordinarily, “people tend to under-realize the capacity for change,” Dr. Stowman adds. Through its response to the cyberattack, “our lab saw the capacity for change and flexibility in workflow.”
Dr. Kalof, too, sees the positive effects of the cyberattack on the AP lab over the long term. “We struggled with communications structures and coordination of efforts before, and the incident command structure that ultimately ended up being developed has resulted in a whole different way that we’re looking at leadership.”
UVMMC cytopathologist Nora K. Frisch, MD, medical director of surgical pathology quality assurance, will co-present with Andrew J. Goodwin, MD, professor of pathology and laboratory medicine at the University of Vermont Larner College of Medicine and UVMMC vice chair for quality and clinical affairs, in a Sept. 26 CAP21 session on “The Pathology of a Cyberattack: How Can Your Laboratory Be Prepared?”
Anne Paxton is a writer and attorney in Seattle.