0
An image of the flag of Europe, which consists of twelve golden stars forming a circle on a blue field.

EDPB Opinion on AI Models and GDPR Principles: Key Takeaways

AI & Algorithms, EEA & UK, EU, GDPR , , , , ,
In December 2024, the European Data Protection Board (EDPB) issued an Opinion in response to a request from the Irish supervisory authority, focusing on the application of GDPR principles in the context of AI models. The Irish supervisory authority posed three specific questions:
  1. When and how can an AI model be considered “anonymous”?
  2. What is the appropriateness of legitimate interest as a legal basis for AI deployment and development?
  3. What are the consequences of unlawful processing of personal data on subsequent operations of the AI model?
Through its answers, the EDPB provided key guidance on how AI models interact with fundamental rights to privacy and data protection established in the GDPR.

Anonymous AI Models

According the EDPB, “[f]or a model to be anonymous, it should be very unlikely 1) to directly or indirectly identify individuals whose data was used to create the model, and 2) to extract such personal information from the model through queries.” While anonymous data can help mitigate privacy concerns, it does not automatically make the AI model completely exempt from GDPR compliance. When a model is claimed to be anonymous, supervisory authorities will evaluate the claims of anonymity on a case-by-case basis, considering “all the means likely to be used” by the controller or a user. The Opinion states that supervisory authorities should review the documentation provided by the controller when assessing if the model is truly anonymous. The EDPB outlines methods that the controller may use to demonstrate anonymity, which may include: 1) reducing the amount of personal data used during training, 2) taking steps to ensure this data cannot be identified, and 3) utilizing technical safeguards to prevent data extraction from the AI model using prompts or queries. Key Takeaway: If a business claims an AI model relies on anonymous data, the claims of anonymity should be substantiated on a case-by-case basis with sufficient evidence and documentation. To do this, businesses with allegedly anonymous AI models may need to implement technical measures to limit the collection of data, reduce the likelihood of data being identifiable, protect against that data being extracted by users during deployment, and create documentation capable of demonstrating these efforts.

Legitimate Interest as a Legal Basis

Under the GDPR, a legitimate interest may constitute a legal basis for companies to process personal data when they have a justifiable reason to do so (beyond obtaining consent). However, the legitimate interest should be balanced against the data subject’s rights and interests, which requires careful consideration and justification when processing information from data subjects. The Opinion provides a framework to assess if a legitimate interest can be a valid legal basis for processing personal data in AI development and deployment. The framework is comprised of a three-step test:
  1. Identify the legitimate interest pursued by the controller;
  2. Assess the necessity of the processing for purposes of the legitimate interest; and,
  3. Balance the legitimate interests against the rights and freedoms of the data subjects.
When conducting this test, the controller should be careful to identify an interest that is lawful, clearly articulated, and non-speculative. For example, a legitimate interest may be to develop an AI model’s conversational agent or to improve threat detection in an information system. The controller should also adhere to GDPR data minimization principles, which state that the processing activities must be proportionate and in line with only what is necessary to achieve the legitimate interest. Finally, controllers should conduct a nuanced balancing test. This test considers the unique circumstances of each case, which may include the data subject’s interest in retaining control over their data, personal benefits, or socioeconomic interest. The Opinion notes, the more precisely an interest is defined in relation to the purpose of the processing, the more precise the estimation of benefits and risks will be. By employing this framework, developers and deployers should be able to decrease the likelihood that their AI models are disproportionately infringing on individual privacy rights and better align their AI practices with GDPR requirements. Key Takeaway: The three-step analysis, according to the Opinion, is crucial to improving compliance for organizations relying on legitimate interest as a legal basis for processing in AI development or deployment. Organizations relying on legitimate interest in this AI context should review their processing activities to determine whether they are proportionate, transparent, and aligned with GDPR principles—like data minimization—to justify the reliance on legitimate interest as a legal basis for processing.

Consequences of Unlawful Processing

The Opinion notes that supervisory authorities enjoy discretionary powers to investigate and assess violations, and they can choose appropriate remedial measures based on the context of the case. However, the EDPB also provides guidance for the supervisory authorities, based on three scenarios.
  1. In the first scenario, personal data is retained in the AI model. The Opinion states that supervisory authorities will need to consider the surrounding circumstances of the AI model to determine if the development and deployment phases of the model involve different legitimate purposes for processing. If so, each should be examined separately.
  2. In the second scenario, personal data is retained in the model and is processed by another controller during deployment. In this instance, the supervisory authorities should determine if the deploying controller conducted an appropriate assessment to demonstrate accountability with Articles 5(1)(a) and 6 of the GDPR. This assessment should show that the AI model was not developed by unlawfully processing personal data.
  3. In the final scenario, a controller unlawfully processes personal data to develop the AI model, and then anonymizes the data before processing it in the context of deployment. The Opinion states that, if it can be demonstrated to the supervisory authorities that the deployment of the AI model does not entail the processing of personal data, then the GDPR does not apply. Therefore, the unlawfulness of the initial processing in development should not impact the deployment operation of the model.
While supervisory authorities do have substantial discretion in oversight of processing activities, the scenarios highlighted by the EDPB show that the development and deployment phases, while connected, may need to be evaluated independently. Key Takeaway: Organizations should proactively ensure compliance at both the development and deployment stages of an AI model. Supervisory authorities will likely use the above examples as guidance, emphasizing the important of demonstrating lawful practices through each stage of the model. The EDPB’s Opinion is an important guide for organizations navigating the intersection of AI and data privacy law. By addressing issues around anonymous AI models, legitimate interest, and lawful processing in development and deployment stages, the Opinion emphasizes responsible AI development. As AI technologies continue to advance, businesses should be aware of the ways supervisory authorities are overseeing their AI models. The insights provided by the EDPB provide a foundation to help businesses to advance and develop new AI models, while also helping to safeguard and protect the rights of individuals.
0

Hoyoverse, developer of Genshin Impact, to pay $20 million to settle FTC complaint

Children, Gaming , , ,
On January 17, the Federal Trade Commission (FTC) announced a proposed settlement with Cognosphere Pte. Ltd and its subsidiary Cognosphere, LLC, doing business as Hoyoverse, developer of gacha video games such as Genshin Impact and Zenless Zone Zero, over allegations that Hoyoverse’s loot boxes and children’s data collection practices violated various federal laws. What is a gacha video game? Generally, a “gacha” video game is one that can be downloaded and played for free but is monetized by selling in-game currency that can be spent on chance-based rewards, which the FTC refers to as “loot boxes.” The loot box rewards range from playable characters to cosmetics to equipment for specific characters, but the reward a player receives is based on chance (e.g., one percent chance to receive X reward) and which reward a player received is revealed only after the player has paid to open the loot box. In games such as Genshin Impact, certain rewards are often featured and available for limited periods of time. For example, if a new character is introduced into the game, the character is typically only available as a rare loot box reward for, say, three weeks. The character is not available for direct purchase, and if the player misses the character as a reward, a rerun of the character as a loot box reward may not happen for months or even years. According to the FTC, this causes players to “purchase dozens of loot boxes, at the cost of hundreds of dollars,” to obtain the featured characters within the limited availability time frame. What did Hoyoverse allegedly do? According to the FTC’s complaint, Hoyoverse violated the FTC Act by misrepresenting the odds and cost of their loot boxes and violated the Children’s Online Privacy Protection Act (COPPA) by failing to provide notice to and collect sufficient consent for children younger than 13 years old.
  • The FTC Act
The FTC claims that Hoyoverse violated the FTC act by making false or misleading representations in advertisements, marketing, and promotions about the odds of obtaining a particular reward in a loot box. For example, Hoyoverse’s social media ads claimed that certain rewards would have a “huge drop-rate boost,” when in reality, the purported “boost” in odds was referring to a featured prize being available to obtain “at all” during the limited availability period, “while the underlying odds of obtaining the featured prize remain[ed] the same.” So, the odds for the reward essentially went from zero percent to the standard percent for the given reward tier (e.g., 5-star rewards may be one percent). The FTC further claims that Hoyoverse’s loot box system constitutes an unfair act or practice, because purchasing a loot box requires the player to navigate a “complex and confusing multi-tier virtual currency exchange system” to purchase a loot box. This system typically requires the player – including children and teenagers, according to the FTC – to purchase in-game currency with actual money and transform that in-game currency into other in-game currency, sometimes multiple times, before being able to purchase a loot box. This multi-tier virtual currency system poses an increased risk for children and teenagers, “whose executive function skills are not yet fully developed” and therefore are “particularly susceptible” to the system’s monetization and pressure to spend money on virtual currency. As such, because children and teenagers can purchase virtual currency in the multi-tier system without first obtaining parental consent to such purchases, the FTC alleged that the system, in the context of children and teenagers, violated the FTC Act as an unfair act or practice.
  • COPPA
According to the FTC, Hoyoverse’s Genshin Impact is covered by COPPA because it is an online service directed to children, and Hoyoverse had actual knowledge that it collected personal information from children under the age of 13. The FTC alleged that Genshin Impact is directed to children under 13 because, in part, the game features matter, visual content, animated characters, and activities that are directed to children. For example, the gameplay and subject matter revolve around exploring, role-playing and collecting a team of heroes, and “engaging in fantasy combat with no blood or gore,” which the FTC claimed are all mechanics like those in other games “popular with children.” And Genshin Impact’s use of anime-style cartoon graphics and colorful animation, according to the FTC, further emphasizes the game’s appeal to children. In particular, Hoyoverse’s use of child-like characters such as Paimon and Klee in promotional materials (e.g., the game’s icon in app stores) serves as evidence of the game’s appeal to children. Despite the applicability of COPPA to Genshin Impact, Hoyoverse failed to satisfy COPPA’s requirements. Specifically, the FTC alleged that Hoyoverse violated COPPA by:
  1. Failing to provide notice on their website or in Genshin Impact of the information collected from children, how they used that information, and to whom they disclosed the information;
  2. Failing to provide the above information directly to parents; and,
  3. Failing to obtain consent from parents before collecting personal information from children.
In addition, because violations of COPPA can constitute an unfair or deceptive act or practice under the FTC Act, the FTC also included such a violation amongst their COPPA-related allegations. What does the $20 million settlement obligate Hoyoverse to do? To settle the FTC’s claims against Hoyoverse, Hoyoverse entered into a proposed settlement order, which will require Hoyoverse to pay a $20 million fine and make changes to address the allegations in the complaint. Hoyoverse will be:
  • Prohibited from allowing children under 16 to purchase loot boxes in Genshin Impact or other Hoyoverse video games without a parent’s affirmative express consent;
  • Prohibited from selling loot boxes using virtual currency without providing an option for consumers to purchase loot boxes directly with real money;
  • Prohibited from misrepresenting loot box odds, prices, and features;
  • Required to disclose loot box odds and exchange rates for multi-tiered virtual currency;
  • Required to delete any personal data previously collected from children under 13 unless they obtain parental consent to retain such data; and,
  • Required to comply with COPPA, including its notice and consent requirements.
Key takeaways? While much of the FTC’s complaint and proposed settlement order references loot boxes in the alleged violations, the FTC’s allegations are more focused on how Hoyoverse promoted and operated its loot box system and to whom they were selling loot boxes. First, if a video game company seeks to monetize their game using loot boxes, the company should consider whether their advertising and promotional material obscures or otherwise inaccurately details the odds of winning a particular reward. For example, there is greater risk in saying a particular reward is “boosted” or its odds are “increased,” when the odds are going from zero percent to the usual percentage rate for a given reward rarity. Second, if a video game uses anime-style graphics, child-like characters, and no blood or gore, the video game should consider satisfying COPPA’s obligations, which may include informing children and parents about the game’s information practices and collecting consent from parents to collect such information. Lastly, if the game sells loot boxes to children under 13 and teenagers under 16, the game may need to satisfy a parental consent requirement before allowing either the children or teenagers to purchase virtual currency or loot boxes.
0
Image depicting the flag of Texas, which is blue, white, and red, with a lone white star.

Texas sues Allstate, continuing Lone Star’s focus on vehicle data regulation

Data / Web Scraping, Data Brokers, United States, Vehicle Data , ,
Update: On Jan. 29, 2025, it was reported that on Jan. 12, 2025, Texas sent Kia America, Inc., a notice of their alleged violations of the Texas Data Privacy and Security Act. Kia has 30 days to cure the alleged violations. Everything is bigger in Texas, including data privacy enforcement. On January 13, 2025, Texas continued its recent regulatory focus on vehicular and geolocation data by initiating a lawsuit against Allstate and its subsidiary, Arity, alleging the companies violated numerous consumer protection and data privacy laws by unlawfully collecting, using, and selling personal, vehicular, and location data without consumers’ knowledge or consent. What led to this lawsuit? For more than half a year, Texas has been leading the regulatory enforcement of vehicular and geolocation data practices. On June 6, 2024, Texas Attorney General Ken Paxton announced that his office had opened an investigation into various car manufacturers “after widespread reporting” that those manufacturers had been secretly collecting mass amounts of data about drivers and selling that data to third parties. The “widespread reporting” cited by Paxton as the seed for the investigation was most likely a nod toward the Mozilla Foundation’s “Privacy Not Included” report published in September of 2023. The report expressly declared that modern vehicles are a “privacy nightmare” and that all 25 car brands researched for the report were labeled as having the worst privacy ever reviewed by the Foundation. The investigation initiated by Paxton in June of 2024 eventually led to a lawsuit against General Motors, filed on August 13, 2024, alleging the company engaged in false, deceptive, and misleading business practices related to its unlawful collection and sale of driving data to insurance companies without the consumers’ knowledge or consent. Following this suit, Paxton’s office sent a notice in November of 2024 to Arity, LLC, a data analytics company founded in 2016 by Allstate, alleging that Arity was in violation of Texas’s recently enacted state privacy law, the Texas Data Privacy and Security Act (the “TDPSA”). The notice identified specific provisions of the TDPSA that Arity was allegedly violating and requested that Arity cure the violations within 30 days, in accordance with the TDPSA’s cure period. But according to Texas’s petition against Allstate and Arity filed on January 13, 2025, Arity failed to cure the alleged TDPSA violations within the 30-day cure period, thereby allowing Texas to include these alleged violations in the lawsuit. What did Allstate and Arity allegedly do? According to Texas’s petition, defendants Allstate and Arity developed and integrated software into third-party apps so that when consumers downloaded the third-party app, they also “unwittingly” downloaded the defendants’ software. The defendants presented the software as “providing a necessary function,” but Texas claims the software does little more than scrape data from the third-party app. Once downloaded, the defendants’ software, through the third-party apps, monitored the consumer’s location and movement “in real-time” and collected trillions of miles of consumer driving data, including geolocation data, accelerometer data, gyroscopic data, and more. The defendants then sold that data to third parties or used it for Allstate’s insurance underwriting. To encourage third parties to integrate the defendants’ software, the defendants paid app developers and offered an incentive program that provided “generous bonus incentives” if developers increased the size of their dataset. All the while, according to Texas, consumers did not consent to, nor were they made aware of, the full extent of defendants’ collection and sale of data. Instead, defendants entered into agreements with the third-party app developers to mandate, to some degree, that certain privacy disclosures and consent language were presented by the third-party apps to consumers, but those third-party disclosures and consent, according to Texas, never mentioned the existence of the defendants, “let alone any of Defendants’ data collection or sales.” Nor did the defendants provide consumers with any of their own notices regarding their data collection practices, and even if consumers did happen to take the extra step to investigate defendants’ policies, those policies contained “untrue and contradictory statements that do not reflect Defendants’ practices.” For example, the policies expressly stated that the defendants do not sell personal data for monetary value, which Texas alleges is untrue, and the policies do not provide consumers with the ability to request that defendants stop selling their data. Taken together, Texas claims these alleged facts establish the basis for numerous legal violations, including violations of the TDPSA, the Texas Data Broker Law, and the Texas Insurance Code. Key takeaways?
  1. Texas is – and will likely remain – focused on regulating vehicular data practices. As the saying goes, once is a coincidence, twice is a pattern, and thrice is a regulatory enforcement focus. Within the short span of half a year, Texas opened an investigation, submitted a notice to cure under the TDPSA, and initiated two lawsuits, all targeting vehicular data practices. Given the rapidity in which Texas is bringing these actions, Texas will likely continue making this an enforcement priority for the near future.
  2. Relying on third-parties to provide notices and collect consent on your behalf may not be enough. The facts allege that the defendants had entered into agreements that, to some degree, obligated the third-party apps to provide notices and collect consumer consent for the collection and sharing of data with the defendants. Yet, according to Texas’s petition, these third-party disclosures and consent collection mechanisms failed to sufficiently inform consumers about the defendants’ data practices.
  3. SDKs remain an area of risk. In recent years, there has been a string of federal and state enforcement action over the use of software development kits (SDKs) to collect and share data. The Federal Trade Commission entered a settlement agreement with InMarket and others; California entered a stipulated judgment of $500,000 with Tilting Point Media; and now a core fact of Texas’s petition is that the defendants developed and integrated SDKs into third-party apps to scrape data.
  4. Carefully consider whether data is being “sold.” Under the TDPSA, a “sale” occurs when personal data is shared, disclosed, or transferred for monetary or other valuable consideration, and Texas alleges that Allstate and Arity “sold” personal data when they sold “data-based products and services for monetary value that linked a specific [consumer] to their alleged driving behavior.” Often, the language of “selling” something conjures to mind ideas of direct financial transactions – exchanging personal data expressly for money or other benefits – but regulators, including those in Texas and California, interpret “selling” personal data more broadly. Thus, companies should carefully review whether their data disclosure and access practices may constitute “selling” personal data, and if so, whether they satisfy the relevant obligations when data is being “sold.”
0

Metaverse Law in Orange County Lawyer Magazine

AI & Algorithms, CCPA, Cybersecurity, GDPR, Health Data, State Privacy Laws , , , , , , ,
The January 2025 edition of Orange County Lawyer magazine features an article written by Metaverse Law’s Lily Li. Read “AI and Machine Learning in Drug Development and Clinical Trials” below or in Orange County Lawyer magazine.
[Originally published as a Feature Article: AI and Machine Learning in Drug Development and Clinical Trials, by Lily Li, in Orange County Lawyer Magazine, January 2025, Vol. 67 No.1, page 28.]   AI and Machine Learning in Drug Development and Clinical Trials by Lily Li   In 2013, sleep medication zolpidem (Ambien, Ambien CR, and Edluar) swept headlines. Marie Claire reported on an alarming and suspicious rise in users experiencing irrational eating, gambling, and even “sleep-driving” while in a hypnotic trance—waking with no memories of their actions.[1] In several cases, women arrested and convicted for driving under the influence contested their convictions, arguing that they were not liable for these undisclosed drug-related side effects. At the same time, several clinical studies suggested that women metabolized zolpidem differently from men. By reviewing existing literature, Japanese researchers out of Shimane University identified 40% higher concentrations of zolpidem in women than men following use, and higher rates of visual hallucinations and sensory distortions.[2] The FDA released a safety advisory, warning users of the risks of “next-morning impairment” for the use of Ambien and related drugs.[3] In addition, the FDA took the unusual step of recommending a 50% cut in the dosage for women. When asked about the change, an FDA director told ABCNews.com: “The changes are different in women and men . . .We don’t understand why yet, but women are more susceptible to next-morning impairment.”[4] Yet, a decade later, the evidence supporting different zolpidem dosages for women and men is unclear.[5] In part, this is due to the lack of research surrounding sex differences in drug impact and drug treatment, as well as substantial gaps in the inclusion of women in clinical studies. From 1977 to 1993, FDA policy recommended excluding women of childbearing potential from Phase 1 and early Phase II drug trials.[6] Even after this policy was removed in 1993, industry fears remained with respect to drug interactions with pregnancy. This episode with zolpidem raised several concerns in the drug development and clinical trial process:
  • How do we recruit representative candidates for drug trials?
  • How do we ensure the quality and availability of datasets for clinical research?
  • How do we measure potential impacts of drug dosing on different populations?
  • What are the legal implications for failing to address appropriate drug doses?
  AI and ML to the Rescue? Now that artificial intelligence is being used in research and development, one wonders: Can artificial intelligence (AI) and machine learning (ML) reduce bias and risks during drug development? Or will it create new legal risks due to bias, privacy intrusions, and lack of transparency? The FDA released a discussion paper on AI, Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products, to discuss potential regulatory frameworks to address the use of AI and ML.[7] In this discussion paper, the FDA released a set of fascinating case studies into existing research and uses of AI in the clinical trial process. Several of these case studies are discussed below, as well as an analysis of their potential impact on the zolpidem example.
  1. Recruitment. According to the FDA, “AI/ML is being used to mine vast amounts of data, such as data from clinical trial databases, trial announcements, social media, medical literature, registries, and structured and unstructured data in EHRs [electronic health records], which can be used to match individuals to trials (Harrer, 219 Shah, Antony, & Hu, 2019).” In this manner, researchers can combine huge quantities of publicly available data and individual health data from prior research to identify participants with certain medical conditions (or lack of adverse conditions) for investigational treatments. For zolpidem, the use of AI/ML may have been able to identify a much broader list of participants for initial clinical testing, making it easier to assess and identify adverse reactions.
  2. Selection and Stratification of Trial Participants. In addition to initial recruitment, AI/ ML has the capability improve intake, selection, and classification of clinical trial participants. Based on baseline characteristics selected by the researchers, such as prior clinical data, and vitals/labs taken during intake, predictive algorithms can help identify high-risk participants.[8] These groups can then be randomized and then subject to more strict monitoring protocols. In the case of zolpidem, alcohol use is associated with sometimes severe adverse effects from the drug, and so it would be beneficial to screen out candidates with a history of alcoholism or, on the flip side, assess drug interactions for this high-risk group with additional support, monitoring, or counseling.
  3. Dose/Dosing Regimen Optimization. AI/ML can be used to predict drug exposure for different populations based on factors such as weight, height, sex, and other characteristics that might impact drug metabolism. Based on prior drug exposure and response profiles for similar drugs and similar populations, AI/ML can help to narrow the dose/dosing regimen selected for a study. As noted by the FDA’s discussion paper, this can help optimize drug dosing “in special populations where there may be limited data (e.g., rare disease studies, pediatric and pregnant populations).” Based on this research, we can imagine future scenarios where AI/ML could have avoided zolpidem dosing concerns, where graduated and limited dosing was tested and applied to different sex, age, and metabolism categories to determine ideal dosing.
  4. Data Analysis. On a more intriguing level, the FDA AI discussion paper discussed the concept of creating “digital twins” of patients for clinical trials. Essentially, an AI version of the clinical participant is created, using the existing candidate’s electronic health records, vital signs, labs and other records. Researchers can assess how the digital twin would react under normal conditions using AI/ML modeling based on data gathered from similar individuals. This digital twin would then act as a substitute for a placebo candidate in a clinical trial, and act as a benchmark against the actual patient undergoing investigational treatment. For zolpidem, this could be used to assess candidates that already have underlying medical conditions such as anxiety, depression, or other confounding factors, to see whether an adverse effect from a trial is due to the investigational treatment or something that is likely to occur to the same individual from anxiety alone.
  5. Postmarketing Safety Surveillance. Finally, AI/ML can help detect and assess adverse events once the drug enters the market. This is not just limited to individual case safety reports (ICSR), required by regulators, but can include adverse events reported publicly on social media and the wider internet. This type of postmarketing safety surveillance could assist researchers and drug companies in identifying potential drug risks, prior to landing on primetime news.
  Quality and Reliability Risks While AI/ML can help to address the costs and efficiency of clinical trials, this relies substantially on the underlying data used to train AI. The quality and reliability of any AI/ML model requires similar quality controls for underlying training data. Given the safety risks of inappropriate drug dosing, or recruiting candidates with severe medical conditions, AI developers cannot rely solely on self-reported healthcare data with no external medical testing or validation. Developers should be equally wary of training on third-party data sets that do not provide documentation on the collection of data and data validation. Within an existing healthcare organization, if the organization is big enough, aggregate and de-identified data may be obtained from existing electronic health care records and prior clinical trials. Yet, even within these large datasets, errors may surface during training. Medical providers may code the same procedure, and similar symptoms, a dozen different ways. Even drug names can be misspelled and coded incorrectly within existing records. While many of these errors may end up being statistically insignificant with enough data, there is the risk of missing one or two major adverse events, or “black swan” events, that would otherwise change the entire risk profile of a drug. In addition to quality and reliability, the underlying dataset needs to be representative of the population that will be studied for the clinical trial. If the underlying dataset is only trained on a handful of individuals with a certain medical predisposition, age, sex, weight, etc., it will be difficult for the AI model to make predictions for that group. As an example, if the training data only contains the medical information for two individuals over the age of sixty, and shows no adverse effects from a particular drug dose, this information is not enough to generalize that the drug at that dosage is appropriate for all individuals over the age of sixty. For all we know, these two candidates could be a former Olympic diver and a nutrition coach, two outliers that completely skew the data. Consequently, the underlying training data for any AI model should also be assessed for bias and representativeness as it applies to the proposed clinical trial.   Data Privacy, Cybersecurity, and AI Risks The data privacy and cybersecurity risks associated with the foregoing uses of AI/ML cannot be underestimated. The quality and representativeness of any AI system in this field will rely heavily on large swathes of healthcare data, fine-tuned and, at times, personalized in the case of digital twins. This is sensitive or special category data at its finest, triggering heightened scrutiny under the EU’s data privacy law, the GDPR, and U.S. data privacy and data breach laws. To date, most healthcare organizations have sidestepped data privacy concerns by relying on HIPAA’s de-identification standard to remove personal information and other identifiers from healthcare data, making it difficult to associate with an individual. While the FDA requires Institutional Review Board (IRB) review of most biomedical research involving human subjects, this generally does not apply to de-identified personal information that cannot be linked to an individual. Simply de-identifying data and then running with it is not enough, however. Under the California Consumer Privacy Act and similar state laws, for example, recipients of de-identified data need to affirm that they will not attempt to reidentify the data (except to test their de-identification methods). The GDPR has a much higher “anonymization” standard, which looks at the re-identifiability of personal information, given all the different datasets that an organization may have access to. AI/ML itself is making the de-identification process harder. As it is capable of slicing and dicing data by age, race, sex, and medical condition, and combining multiple large datasets, it is easy to run the risk of re-identifying data. While several thousand people might have the same configuration of eye color, age, gender, and weight, only one or two may have participated in a clinical trial at a particular location, or have specific allergies or side effects to certain types of medication. As a result, in circumstances where healthcare data is not de-identified, or the risk of reidentification is heightened, then it behooves clinical organizations and their AI developers to implement written information security programs and associated privacy and security controls.   Legal Liability and Drug Dosing In several notable cases, defendants on zolpidem were able to contest or overturn DWI or even vehicular manslaughter cases. Essentially, these defendants argued that they were not aware of the potential dangers of zolpidem, and so could not be liable for their actions while “sleep driving.” This raises the question: If AI gets good enough, and can tell you exactly the right dose to take of a drug, will you (or your doctor) be liable if you deviate from the AI’s recommendations? Will the AI’s recommendations be discoverable in court (and surfaced via AI-enhanced search)? Only time will tell what this brave new world will bring.   ENDNOTES [1] Kai Falkenberg, While You Were Sleeping (September 27, 2012), Marie Claire, https://www.marieclaire.com/culture/news/a7302/while-you-were-sleeping/.   [2] Takuji Inagaki, Tsuyoshi Miyaoka, Seiichi Tsuji, Yasushi Inami, Akira Nishida, and Jun Horiguchi, Adverse Reactions to Zolpidem: Case Reports and a Review of the Literature, 12 Prim Care Companion J Clin Psychiatry 6 (2010), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3067983/.   [3] U.S. FDA, Drug Safety Communication: FDA approves new label changes and dosing for zolpidem products and a recommendation to avoid driving the day after using Ambien CR (May 14, 2013), https://www.fda.gov/drugs/drug-safety-and-availability/fda-drug-safety-communication-fda-approves-new-label-changes-and-dosing-zolpidem-products-and.   [4] FDA: Cut Ambien Dosage for Women, ABC News (January 10, 2013, 6:03AM), https://abcnews.go.com/Health/fda-recommends-slashing-sleeping-pill-dosage-half-women/story?id=18182165.   [5] David J Greenblatt, Jerold S Harmatz, & Thomas Roth, Zolpidem and Gender: Are Women Really At Risk?, 39(3) J. Clinical Psychopharmacol. 189 (May/Jun 2019), https://pubmed.ncbi.nlm.nih.gov/30939589/.   [6] NIH Inclusion Outreach Toolkit: How to Engage, Recruit, and Retain Women in Clinical Research, last accessed September 16, 2024: https://orwh.od.nih.gov/toolkit/recruitment/history.   [7] FDA, Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products (May 10, 2023), https://www.fda.gov/media/167973/download; see also Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products; Availability, 88 FR 30313 (May 11, 2023), https://www.federalregister.gov/documents/2023/05/11/2023-09985/using-artificial-intelligence-and-machine-learning-in-the-development-of-drug-and-biological.   [8] Thi Tuyet Van Tran, Hilal Tayara, and Kil To Chong, Artificial Intelligence in Drug Metabolism and Excretion Prediction: Recent Advances, Challenges, and Future Perspectives, 15 Pharmaceutics. 1260 (Apr 17, 2023), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10143484/.   Lily Li is an AI, data privacy, and cybersecurity lawyer and founder of Metaverse Law. She is a certified information privacy professional for the United States and Europe and is a GIAC Certified Forensic Analyst for advanced incident response and computer forensics. She can be reached at info@metaverselaw.com.
0

HHS releases proposed rule to modify HIPAA Security Rule requirements

Health Data, United States , ,
On December 27, 2024, the U.S. Department of Health and Human Services (HHS), through its Office for Civil Rights (OCR), announced a proposed rule that would modify the security requirements imposed by the Health Insurance Portability and Accountability Act of 1996 (HIPAA) Security Rule. The proposed rule, if adopted, would modify the HIPAA Security Rule to require covered entities and their business associates to implement more stringent cybersecurity safeguards and measures to protect electronic protected health information (ePHI). These new requirements would include, among other things:
  • Requiring written documentation of all HIPAA Security Rule policies, procedures, plans, and analyses.
  • Adding specific compliance periods for existing HIPAA Security Rule requirements.
  • Requiring the creation of a technology asset inventory and a network map that illustrates the movement of ePHI throughout the regulated entity’s electronic systems and, at least every 12 months, reviewing the asset inventory and network map.
  • Requiring notification of certain regulated entities within 24 hours when a workforce member’s access to ePHI or certain systems is changed or terminated.
  • Requiring regulated entities to conduct a compliance audit at least once every 12 months.
  • Requiring business associates verify at least once every 12 months for covered entities that they have deployed technical safeguards required by the Security Rule to protect ePHI.
  • Requiring covered entities to test the effectiveness of their security measures at least once every 12 months.
  • Requiring network segmentation.
  • Requiring vulnerability scanning at least every six months and penetration testing at least once every 12 months.
  • Requiring greater specificity for conducting a risk analysis.
These changes come in response to what the OCR sees as a “substantial increase in reports of large breach reports over the last five years.” According to the OCR, between 2018 and 2023, reports of large breaches increased by 102 percent, and the number of individuals affected by such breaches increased by ten times that, at 1002 percent. The proposed rule changes seek to improve the cybersecurity of critical health infrastructure by updating the Security Rule’s standards to better address the increase in cybersecurity threats in the health care sector. The proposed rule can be viewed in the Federal Register, where it is scheduled for publication on January 6, 2025. Stakeholders within the health care sector, including patients and covered entities, are welcome to submit comments on the proposed rule through regulations.gov for 60 days after its publication. While the proposed rule goes through the rulemaking process, the current Security Rule remains in effect. We will continue monitoring for developments.
1 … 5 6 7 8 9 … 29