Weeks of lab turmoil follow cyberattack

Anne Paxton

April 2021—After he finished interviewing for a fellowship one morning last October at the University of Vermont Medical Center, pathology resident William O. Humphrey, MD, checked in to attend grand rounds virtually. Then the cyberattack struck.

It began mysteriously, with people dropping one by one off the Zoom screen and emails arriving only intermittently. Internet service grew patchy and a hospital staffer unmuted and canceled grand rounds, saying, “We aren’t really sure what’s going on.”

From there, a cascade of failures indicated serious trouble. “All of a sudden we’re realizing we can’t sign into our EMR. We can’t get into our email either. My phone isn’t working on the Wi-Fi. Something is wrong,” recalls Dr. Humphrey, a member of the CAP Informatics Committee. That was the prelude to a siege in which fax machines and penmanship were unretired from obsolescence, paperlessness became a relic of the past, and words like “runners” and “bouncers” entered routine laboratory vocabulary.

External agents had maliciously invaded and at least partially disabled the system. “It was certainly something abrupt. And our impression was that it may have been related to email phishing,” Dr. Humphrey says, though no official word to hospital staff has clarified how it occurred and who engineered it and why.

Such attacks have become a serious risk for any enterprise reliant on IT, which in this decade is nearly all enterprises. But cyberattacks are special hazards for health care institutions. For the UVMMC laboratory, the effects of the attack ranged wide and continue to haunt operations.

As UVMMC realized hospital systems had been disrupted, a forced shutdown of the network was determined to be the safest recourse. “We went into downtime mode,” says Christina M. Wojewoda, MD, associate professor and director of microbiology. Normally, that would mean an eight-hour break at most. “You get the critical results out verbally and everything else sits and waits for the system to come back. And then you work really hard to get all that work back into the computer.”

But in this case, the backlog would continue to mount. Ahead lay weeks of downtime and chaos. “When you think about a cyberattack, you think about your electronic

health record going down,” says Andrew J. Goodwin, MD, division chief for laboratory medicine at UVMMC at the time and now vice chair for quality and clinical affairs. And it certainly did in this instance. “But your telephones are run by servers and software now. The fax machines are run by servers and software. Your pager system, your emergency alert system—everything lives on a computer drive somewhere. So a cyberattack shuts down much more than you anticipated.” In this attack, even the computer-driven pneumatic tube system at UVMMC went out of commission.

Four prior crises the lab experienced in recent years provided a telling point of reference as to the scale and severity of the cyberattack. “We suffered from a pretty significant flood,” Dr. Goodwin says. “Then our nursing colleagues went on a planned, 72-hour work stoppage. We had a go-live process to implement Beaker, our new Epic laboratory information system. Then there was COVID.”

The cyberattack was more stressful than any of those. The reason: “It was the cutting off of everything basic. It was not having a comprehensive playbook to draw upon. It was the inability to consult colleagues around the country for advice because, thankfully, very few labs had been attacked to this degree. And the uncertainty about how long it was going to last contributed to the heightened anxiety.”

“The hospital didn’t shut down. We still had patient care needs to meet,” Dr. Wojewoda says. But for nearly a month, all of the reporting requirements for routine testing, reference work, and SARS-CoV-2 results for the state health department were fulfilled manually.

Although some members of the IT department were put on furlough because there were no systems to work with, others were deployed to sweep all IT components and purge them of malware, Dr. Wojewoda says. “They had to go through every computer, scrub it, and make sure it was safe to use. In some instances new PCs had to be brought in.”

The shutdown of the hospital’s Epic EMR system meant masses of paper had to be deployed to report results. “Runners” helped the laboratory by hand-conveying results to floors and faxing and filing results. That was the assignment for Dr. Humphrey and other residents. “After we lost the network,” he says, “there was a rotating pool of all the residents and trainees in pathology, and some staff, who would make ourselves available to go to the room where we were storing the paper files, search for results, then physically take them back to the fax machine to be sent to the clinician.”

These tasks were an education in how much data results from a simple interaction like a primary care visit, Dr. Humphrey says. “We had three or four different areas of chemistry that each had its own filing system. We couldn’t just send the clinician the results, because then we would lose the only copy, so every result that had to go to a clinician had to be copied and then faxed.” Which was a challenge for a couple of residents in their late 20s. “I don’t think they had ever considered what a fax machine did until this happened.”

Without computers, test orders opened up further risks—although not as great as the risks that drug prescriptions posed to the UVMMC pharmacy, where an unclear entry could result in the wrong dose of a drug. “Pharmacy handwriting had to be very, very clean,” Dr. Wojewoda notes. Luckily for the handwriting-challenged, most orders to the laboratory involve checkboxes on a standard form.

Still, other measures were needed to head off problematic test orders at the pass. One of them was the purchase of a label-maker printer to avoid the syndrome of undecipherable handwriting. The laboratory also resorted to posting “bouncers” at its door. “We figured out as soon as the sample and requisition came to us, it was now our problem. So if all the information wasn’t present that we needed, we would be stuck,” Dr. Wojewoda says.

An added complication was that medical record numbers were usually not available—only a name and date of birth. But avoiding additional resolution time to fulfill test orders was a priority. “So the bouncers would review the requisition to say, ‘Nope. You don’t have the patient floor on here.’ Or: ‘We won’t know whom to call if there’s no physician filled out.’” And back the test order would go for completion.

At the same time, “We didn’t want to broadcast what the problem was,” Dr. Wojewoda adds. “The real idea of causing something like this is to incite panic, and the last thing you want to do, especially in a pandemic, is create more chaos.” In fact, there was already a perception outside UVMMC that things might be out of control: The simple task of trying to tell external colleagues that staff would be off-grid for a while failed when the staff discovered those emails weren’t going anywhere. “So it was just like we dropped off the map.”

Dr. Humphrey, who was on his hematopathology rotation at the time, found that improvising solutions was the order of the day. “The clinics and the ORs were still going and we had to, on the fly, figure out how to do everything we would normally do for those patients without the technology that’s pretty much required to function in health care now.”

“All of a sudden we’re wondering where the paper copies of all our forms are. Where’s the binder with the printed-out procedures? How many copies of this result can we make? How can we shift what we do in the computer to, essentially, patient file folders and still keep the high quality of patient care we always pride ourselves on?”

Can a laboratory prepare adequately for the kind of disaster UVMMC experienced? “We were not ready for this. I will tell you that right now,” Dr. Wojewoda says. Despite having well-established and regularly drilled downtime procedures, “we didn’t have a procedure built for being down a month.” In drills, “we could still perform testing and take care of patients. Because we used to do that without electronic medical records all the time.”

“We create so much more information for patients now with different tests, imaging, physical exam findings, than we did back in the paper days. So we were trying to resurrect systems and thinking: Do we remember how to do this? And how do we keep it all straight? How do we get the treating clinician to understand what’s important?”

First of two parts
Next month: cybersecurity

Electronic systems can flag results. “But it’s much more difficult if everything’s on paper. I’ve never seen more paper being used. It was unreal,” she says. Because most of the laboratory’s printers are networked, printing was unavailable. “People were printing things off at home and we were photocopying like there was no tomorrow because as soon as the results left the laboratory, we had no guarantee they would get to the person who needed it.” Accustomed to ever-available electronic data, the clinical staff would take a result, walk away with it, and later call and ask for it again. “We kept having to make copies and resend the same result multiple times.”

Having been through the experience, Dr. Wojewoda hopes to spread the word about the risks and the preparation needed to avoid the worst effects of a cyberattack. She’d like to see people “not get as blindsided as we did.” Preparing for a longer downtime period than eight hours is a must, she says. Other tips:

• Have a process for shared samples.
• Design workup forms and report forms with all required elements and store printed copies on a shelf.
• Maintain hard copies of maintenance tasks.
• Outsource as much as possible because lower volume means fewer chances for error. Divert the outreach business.
• Buy as many laptops/tablets as possible and be able to plug them into printers.
• Think of all the people who will try to send email to you and send them a personal email address.
• Keep up to date a list of faculty/staff/resident phone numbers and personal emails.

For the microbiology laboratory at UVMMC, Dr. Wojewoda says the emergency preparedness plan now contains these and other additions:

• Have culture documentation worksheets printed and ready to use.
• Have paper result forms for each type of assay printed and ready to use.
• Keep a copy of the requisition with any testing logs and a copy of the result, filed in alphabetical order, for manual entry into the laboratory information system when systems are up.
• Create a spreadsheet for high-volume testing results to do a mail merge to print results, rather than handwrite results.
• Use label stickers to print plate labels for culture plates.

Rebuilding the EMR and LIS databases has been a challenge, UVMMC pathologists say. “We’re now trying to collate all of the worksheets we used and the ways we kept track of quality control and temperatures and all of the other regulatory requirements,” Dr. Wojewoda explains. “And there was considerable back-entry work to do.” For patients in the hospital at the time of the attack, “we had to get some level of information back in the system for them. So registration and nursing had to supply a bunch of documentation for those patients.”

Bits of information were often just stuck on paper forms in the laboratory, handwritten in most cases. “Sometimes some of the instruments would print out results,” Dr. Woje­woda says. “But we’d have to get it to inpatients, or outpatients across state lines, and then get all of those results back into the system afterward.” In microbiology, “we’re still working on that now in March of 2021.”

In all, UVMMC spent nearly four weeks operating without an EMR system—from Oct. 28, the date of the cyberattack, to Nov. 22 when the EMR went back online. Epic company representatives told UVMMC they hadn’t seen a cyberattack of this magnitude before. “They’d never had a customer down for so long,” Dr. Goodwin says.

It was not only the EMR, however, but also the hospital’s 300-plus other third-party applications that had to go through a recovery process before being reconnected. And the laboratory was hit hard by that. “To have literally everything turned off means that your communication methods, your interfaces with outside laboratories, your reference lab order interface—none of that worked,” Dr. Goodwin says. Mandatory state health reporting—augmented in 2020 by COVID-19 test reporting—was also disabled and had to continue by paper and fax.

For many of the laboratory’s vendors, the cyberattack was novel and unsettling; it led them to temporarily sever their connection to UVMMC. “Some of our analytic systems, our analyzers in the lab, send data to and from the vendor on a regular basis. A lot of the vendors turned off all their connectivity with us because they didn’t want to run the risk of being infected,” Dr. Goodwin says.

“We couldn’t even get some routine maintenance tasks done,” Dr. Wojewoda adds. “We had to provide the vendors with proof that things were safe to get those instrument interfaces back up.”

“It took us probably four to six more weeks,” Dr. Goodwin says, “to get our essential third-party applications up and running. Overall, it took many weeks post-system reimplementation to mitigate the impacts” from the system shutdown.