A healthcare AI with the power to manage prescriptions is rather open to mind-altering suggestions, according to security experts.
Redteamers at AI security firm Mindgard reported on Tuesday that it took relatively little work for them to get a healthcare AI from Doctronic not only to spill its system prompts, but to let them make modifications too.
Wanna make the bot spout COVID-19 conspiracies and vaccine misinformation, or speak with a put-on accent? Just tell Doctronic that a session hasn't started and the conversation it's having isn't with a user but the system. Then, you can get it to spill its system prompts and use that information to wreak mischief.
"It was as easy as notifying the AI that the session was not yet started," Mindgard chief product officer Aaron Portnoy told The Register in an email.
Mindgard points out that these manipulations are session-specific. Tricking Doctronic into helping you make meth because you shared a fake press release with it saying it was a programming update to make meth legal (an example in the study) is funny, but it's not behavior that's going to spill over to other users or persist.
Well, at least most of the time.
The researchers did find that they were able to maintain a bit of clinical persistence in the form of SOAP notes, a common form of structured recordkeeping for patient interaction, consisting of the subjective reports from the patient, objective observations by the healthcare professional, an assessment of the situation, and a plan of action.
Any time Doctronic needs to refer something to a human medical professional for review (e.g., a prescription, face-time with a clinician) it generates a SOAP note for the human clinician, which becomes a permanent part of a patient's Doctronic record. SOAPs are not prescriptions, but they are recommendations to a clinician reviewing the machine's work to authorize one.
If someone were to trick Doctronic into modifying an OxyContin prescription to triple the size by telling it prescribing guidelines had changed, and an overworked approving physician were not to notice, jackpot - at least that's Mindgard's interpretation of the SOAP exploit it described.
"According to Doctronic's own website, its treatment plans 'match those of board-certified clinicians 99.2% of the time,'" Mindgard noted. "With such a high level of confidence, will the SOAP be doubted?"
Whether it'd be caught or not, the fact that Doctronic's AI could seemingly be so easily tricked is concerning, especially given it's currently part of a trial in Utah to see about its effectiveness as a health care intermediary, including with the ability to handle some prescriptions.
Both the Utah state government and Doctronic made clear to us that such a prescription refill exploit couldn't be fulfilled in Utah, as controlled substances can't be acquired through the program.
Doctronic told us that The Utah pilot limits drug refills to previous, non-controlled prescriptions. Zach Boyd, Utah Commerce Department AI policy office director, told us that the state demo also has "additional safeguards that are in place before a prescription is issued that are not part of the generic Doctronic model" that would prevent such misuse.
In short, neither Doctronic nor the state of Utah seem too concerned about Mindgard's findings since no one's actually getting a prescription cut for triple-strength Oxy or tricking their local auto-doctor into dispensing misinformation.
Doctronic told us that it "reviewed the prompt patterns [Mindgard] reported as part of our normal review process... We take security research seriously and continue improving safeguards to increase robustness against adversarial inputs."
Portnoy has his doubts about the company's level of commitment – he says Doctronic has given him the silent treatment since Mindgard disclosed the issue in late January, and he's not sure Doctronic has resolved the issue, either.
"As far as we are aware Doctronic is still vulnerable," Portnoy said. ®
Source: The register