The Digital Veil: How AI’s Hidden Threat to Patient Privacy is…

In an era where data is king and cyber threats loom large, medicine remains a sanctuary of confidentiality, fostering patient trust and enabling physicians to handle sensitive information with care. The Hippocratic Oath, one of the oldest and most revered medical ethics texts, has stood the test of time, emphasizing the sacred trust between physician and patient. It reads: “Whatever I see or hear in the lives of my patients, whether in connection with my professional practice or not, which ought not to be spoken of outside, I will keep secret, as considering all such things to be private.” This trust is the bedrock of the healthcare system, ensuring that patients can confide in their doctors without fear of their information being exposed.

However, a recent revelation from MIT researchers has cast a shadow over this trust. A study presented at the 2025 Conference on Neural Information Processing Systems (NeurIPS) investigates the potential for artificial intelligence models trained on de-identified electronic health records (EHRs) to memorize patient-specific information. This groundbreaking research underscores the need for rigorous testing to ensure that targeted prompts cannot reveal sensitive data, emphasizing the importance of evaluating leaks in a healthcare context to determine their meaningful impact on patient privacy.

Foundation models, designed to generalize knowledge and make better predictions by drawing upon many patient records, are normally expected to operate in this manner. However, the risk of “memorization” looms large, where the model draws upon a singular patient record to deliver its output, potentially violating patient privacy. This risk is not hypothetical, as foundation models are already known to be prone to data leakage.

Sana Tonekaboni, a postdoc at the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard and first author of the paper, highlights the dual-edged sword of these high-capacity models. While they can be a resource for many communities, adversarial attackers can prompt a model to extract information on training data. Given this risk, Tonekaboni notes, “this work is a step towards ensuring there are practical evaluation steps our community can take before releasing models.”

To delve deeper into this potential risk, Tonekaboni approached MIT Associate Professor Marzyeh Ghassemi, a principal investigator at the Abdul Latif Jameel Clinic for Machine Learning in Health (Jameel Clinic) and a member of the Computer Science and Artificial Intelligence Lab. Ghassemi, a faculty member in the MIT Department of Electrical Engineering and Computer Science and Institute for Medical Engineering and Science, runs the Healthy ML group, which focuses on robust machine learning in health.

The research team developed a series of tests to assess the potential risk EHR foundation models could pose in medicine. These tests are designed to measure various types of uncertainty and assess their practical risk to patients by measuring various tiers of attack possibility.

“We really tried to emphasize practicality here; if an attacker has to know the date and value of a dozen laboratory tests from your record in order to extract information, there is very little risk of harm. If I already have access to that level of protected source data, why would I need to attack a large foundation model for more?” says Ghassemi.

In the past 24 months, the U.S. Department of Health and Human Services has recorded 747 data breaches of health information affecting more than 500 individuals, with the majority categorized as hacking/IT incidents. This trend underscores the need for robust privacy measures in the healthcare sector.

Patients with unique conditions are especially vulnerable, given how easy it is to pick them out. “Even with de-identified data, it depends on what sort of information you leak about the individual,” Tonekaboni says. “Once you identify them, you know a lot more.”

In their structured tests, the researchers found that the more information the attacker has about a particular patient, the more likely the model is to leak information. They demonstrated how to distinguish model generalization cases from patient-level memorization, to properly assess privacy risk.

The paper also emphasized that some leaks are more harmful than others. For instance, a model revealing a patient’s age or demographics could be characterized as a more benign leak, whereas a model revealing a patient’s diagnosis or treatment plan could be characterized as a more severe leak.

The implications of this research are profound. As we move further into the digital age, the potential for AI to memorize and leak sensitive patient information is a real and present danger. The MIT researchers’ work is a crucial step towards understanding and mitigating this risk.

The AI Privacy Paradox: Balancing Innovation and Protection

The advent of AI in healthcare has brought about a paradigm shift in how medical data is managed and utilized. On one hand, AI has the potential to revolutionize healthcare by enabling more accurate diagnoses, personalized treatments, and efficient management of patient records. On the other hand, the same technology poses significant risks to patient privacy, as demonstrated by the MIT study.

The challenge lies in finding a balance between the benefits of AI and the need to protect patient privacy. This is not an easy task, as AI models are becoming increasingly sophisticated, making it harder to ensure that they do not leak sensitive information. Moreover, the healthcare sector is under constant pressure to innovate and improve patient outcomes, which can sometimes lead to a compromise on privacy measures.

One of the key issues in this debate is the concept of de-identification. While de-identification is a common practice in healthcare to protect patient privacy, the MIT study shows that it is not foolproof. Even with de-identified data, AI models can still memorize and leak sensitive information, especially if the attacker has some knowledge about the patient.

This raises the question of whether de-identification is still an effective measure in the age of AI. The answer is not straightforward, as it depends on the specific context and the level of risk involved. For instance, de-identification may be sufficient for certain types of data, such as demographic information, but it may not be enough for more sensitive data, such as diagnosis or treatment plans.

Another issue is the ethical implications of using AI in healthcare. While AI has the potential to improve patient outcomes, it also raises important ethical questions about the use of patient data and the responsibility of healthcare providers. For example, should healthcare providers be held accountable if an AI model leaks sensitive patient information? And what are the ethical implications of using AI to make medical decisions that could have a significant impact on a patient’s life?

The Future of AI in Healthcare: A Privacy-First Approach

As AI continues to play a larger role in healthcare, it is crucial to address the privacy concerns raised by the MIT study. One way to do this is to adopt a privacy-first approach to AI development and deployment. This approach emphasizes the importance of privacy and data protection from the outset of the AI development process, rather than as an afterthought.

A privacy-first approach involves several key steps. First, it is essential to ensure that AI models are trained on de-identified data, and that the de-identification process is robust and effective. Second, it is important to implement rigorous testing to assess the potential risk of data leakage, as demonstrated by the MIT study. Third, it is crucial to establish clear guidelines and best practices for the use of patient data in AI models, and to ensure that healthcare providers are aware of these guidelines and are held accountable for their compliance.

Another important aspect of a privacy-first approach is the development of ethical guidelines for the use of AI in healthcare. These guidelines should address the ethical implications of using AI to make medical decisions, and should provide clear guidance on the responsibilities of healthcare providers and AI developers.

In conclusion, the MIT study highlights the potential risks of AI to patient privacy, but it also provides valuable insights into how to mitigate these risks. By adopting a privacy-first approach to AI development and deployment, healthcare providers can ensure that they are using AI in a way that balances innovation and protection, and that they are meeting their ethical and legal obligations to protect patient privacy.

FAQ

Q: What is the Hippocratic Oath, and why is it important in healthcare?

A: The Hippocratic Oath is one of the oldest and most revered medical ethics texts, emphasizing the sacred trust between physician and patient. It is important in healthcare because it sets the standard for patient confidentiality and trust.

Q: What is the potential risk of AI memorization in healthcare?

A: The potential risk of AI memorization in healthcare is that AI models trained on de-identified electronic health records (EHRs) could memorize patient-specific information and leak sensitive data, violating patient privacy.

Q: How can healthcare providers balance the benefits of AI with the need to protect patient privacy?

A: Healthcare providers can balance the benefits of AI with the need to protect patient privacy by adopting a privacy-first approach to AI development and deployment. This involves ensuring that AI models are trained on de-identified data, implementing rigorous testing to assess the potential risk of data leakage, and establishing clear guidelines and best practices for the use of patient data in AI models.

Q: What are the ethical implications of using AI in healthcare?

A: The ethical implications of using AI in healthcare include questions about the use of patient data, the responsibility of healthcare providers, and the potential impact of AI on patient outcomes. It is important to establish clear ethical guidelines for the use of AI in healthcare to address these issues.

Q: What is the future of AI in healthcare, and how can healthcare providers ensure that AI is used responsibly?

A: The future of AI in healthcare is promising, but it also poses significant challenges, particularly in terms of patient privacy. Healthcare providers can ensure that AI is used responsibly by adopting a privacy-first approach to AI development and deployment, establishing clear ethical guidelines, and ensuring that healthcare providers are aware of these guidelines and are held accountable for their compliance.