0
Image of a cellular phone with the ChatGPT app open.

Overview: The EU General-Purpose AI Code of Practice

AI & Algorithms, Cybersecurity, EU , , , ,

Why Do We Need a Code of Practice?

On August 2, 2025, the general-purpose AI (GPAI) provisions of the EU AI Act went into effect. GPAI models (including models that support most generative AI, like ChatGPT), now face certain obligations in the EU, including requirements around transparency, copyright and systemic risk. However, the EU AI Act is a framework: it defines obligations but leaves technical details to harmonized standards and codes of practice. While this approach sets certain expectations and allows the EU AI Act to remain technology-neutral, it also leaves questions about how businesses substantially comply with the EU AI Act. To bridge this gap, a multi-stakeholder group drafted the General-Purpose AI Code of Practice (GPAI Code). On August 1, 2025, the European Commission issued a formal opinion confirming the GPAI Code is an “adequate tool” to help demonstrate compliance with the EU AI Act. Why is the Code significant? This opinion signals that organizations who adopt the GPAI Code may be able to demonstrate good-faith efforts to comply with the relevant provisions of the EU AI Act –  according to the Commission’s website: “The Code of Practice helps industry comply with the AI Act legal obligations…of general-purpose AI models.” In its opinion, the Commission notes that the Code provides actionable commitments and reporting mechanisms, especially for high-risk models. Additionally, the Commission emphasized that the Code provides a practical framework to demonstrate regulatory compliance. Following this endorsement, providers of GPAI models can voluntarily sign the Code, which “will reduce their administrative burden and give them more legal certainty than if they proved compliance through other methods.” Still, signatories should be aware that the Code explicitly states that adherence to the Code does not necessarily constitute evidence of compliance with the EU AI Act.

What is a General-Purpose AI Model?

A GPAI model is a component of an AI system with a wide range of possible uses, whether intentional or unintentional. It is important to note that these models are not systems in themselves but are part of AI systems. Additional elements, like user interfaces, are necessary to make these models fully operational systems. Under Article 3(63) of the EU AI Act, a GPAI model includes those trained on a “large amount of data using self-supervision at scale.”  They can be applied across sectors or tasks, usually without substantial modification, meaning GPAI models “can be integrated into a variety of downstream systems or applications.” Recital 98 of the EU AI Act states that the generality of the model can also be determined by the number of parameters, and “models with at least a billion parameters…should be considered to display significant generality and to competently perform a wide range of distinctive tasks.” GPAI models are sometimes called “foundation” or “frontier” models, and while they may include large language models (LLMs), they can also process audio, physical, textual or visual data, powering systems like DALL-E, GPT-4, Gemini, LaMDA, SEER, ALIGN, and more.

How are general-purpose AI models regulated?

Under the EU AI Act, the chapter on GPAI both addresses generative AI and outlines some of the most stringent requirements under the Act. However, all requirements for GPAI under the EU AI Act are directed to providers as opposed to deployers. Providers of GPAI models have a range of obligations under the EU AI act, both directly to supervising authorities and onward to AI providers who integrate the GPAI models into their systems. Obligations of Providers of GPAI Models If a provider places a GPAI model on the EU market, or integrates such a model into its own AI system on the EU market, it must:
  • Prepare and maintain technical documentation for regulators. This should include at least a general description of the GPAI model, including the tasks it’s designed to perform and the types of systems in which it can be integrated; acceptable use policies; and information on training process.
  • Prepare and maintain documentation for downstream providers. This should include information that allows the downstream AI system providers to comply with their own obligations under Article 53(1)(b). Similar to the technical documentation, this includes but is not limited to a general description of the model, and a description of its elements and development process.
  • Prepare an EU copyright policy. This policy should establish a means to comply with EU regulations on copyright and related rights.
  • Prepare and publish a summary of training content. Using the template provided by the AI Office, providers of GPAI must share a comprehensive summary of AI training information. This should allow stakeholders to exercise their rights by informing them of the information used to train the GPAI model.
  • Cooperate with relevant authorities and appoint an authorized representative. Providers must also cooperate with relevant authorities, and if they are established outside the EU, appoint an authorized representative located in the EU.
It is notable that under Recital 85, the EU AI Act states that GPAI systems “may be used as high-risk systems by themselves or be components of other high-risk systems.” Therefore, the providers of GPAI systems must work closely with providers of high-risk AI systems to ensure compliance with any requirements of high-risk systems under the Act. Obligations of Providers of GPAI Models with Systemic Risk What does “systemic risk” mean? GPAI models with systemic risk include models that reasonably pose foreseeable negative effects relating to major accidents, disruption of critical sectors, serious consequences to public health and safety, public and economic security, democratic processes, and the dissemination of false or discriminatory content, or other similar effect. Under Article 51(1) of the EU AI Act, a GPAI model will be classified as having systemic risk if:
  • It has high impact capabilities, or
  • It is designated by the Commission to have high impact capabilities based on the criteria in Annex XIII (i.e., the number of parameters in the model, the size of the data set, the amount of computation used to train the model, etc.).
What are the additional obligations for these models? In addition to the requirements for all GPAI models, those with systemic risk have additional obligations related to:
  • Model evaluation, assessment, and mitigation of systemic risks;
  • Incident management and reporting; and
  • Cybersecurity protections and technical documentation.
Because there are differences in the obligations between GPAI systems generally and GPAI systems with systemic risk, this classification procedure should be noted by providers of GPAI systems; it is essential to understand where each GPAI model falls, and what requirements the model has under the EU AI Act. According to Article 52(6), a list of GPAI models with systemic risk will be published and updated by the European Commission, but it has not been published at the time of writing.

What is the General-Purpose AI Code of Practice?

While not legally binding, providers of GPAI models can use the Code of Practice to demonstrate compliance with their obligations under the EU AI Act. The Code consists of three chapters on 1) transparency, 2) copyright, and 3) safety and security. The first two chapters apply to all providers of general-purpose AI models, providing a way to demonstrate compliance with obligations under Article 53 of the AI Act. The final chapter applies only to general-purpose AI models with systemic risk under Article 55 of the AI Act. Chapter 1: Transparency Among other things, this chapter requires signatories to create and maintain documentation for all GPAI models distributed within the EU for up to ten years. There are exceptions for models that are free, open-source, and do not pose systemic risk. When completing this documentation, signatories must use a standard Model Documentation Form, which includes information on licensing, technical specifications, training data, and other parameters of the GPAI model. The Code encourages publication of this information to promote transparency. Chapter 2: Copyright This chapter requires signatories to create and maintain a copyright policy that complies with the EU’s legal standards. This includes, but is not limited to, ensuring that data collected by web crawling is lawfully accessible, and certain websites flagged for copyright infringement are avoided. Importantly, signatories must designate a contact for copyright holders to submit complaints, along with a process for handling those complaints. Chapter 3: Safety & Security (GPAI with systemic risk only) One of the main elements of this chapter is the requirement for signatories to develop a state-of-the-art Safety and Security Framework before releasing any GPAI model categorized as posing a systemic risk. Additionally, systemic risks should be identified and inventoried, and before progressing with development or deployment, the signatories should weigh the relative risks and determine if they are acceptable, among other requirements.

What’s next?

The Code will be monitored and reviewed at regular intervals by the AI Office, and may be updated in response to emerging risks, technological developments, or incidents involving general-purpose AI models.
0
Image of a computer circuit board with "AI" written on one of the chips.

AI Updates: An Overview of the Legal Landscape

AI & Algorithms, California, CCPA, EU , , , , , ,
As AI continues to advance, so do regulatory efforts. During the 2024 legislative session, 45 states along with Puerto Rico, the Virgin Islands, and Washington D.C. all introduced AI bills. With the legislative session for 2025 wrapping up, we are seeing similar tends this year. As new legal requirements emerge, organizations across the U.S. and EU may face overlapping – yet not identical – regulations that touch on issues of bias, safety, privacy, and transparency. Additionally, these laws may categorize the same AI system differently in different jurisdictions, requiring a nuanced approach to navigating these laws. Keeping this in mind, this article provides a brief overview of a handful of these laws. The practical takeaway? Businesses operating in the U.S. or EU should be aware of their legal requirements. Additionally, these organizations may want to consider a programmatic, auditable, and documented approach to AI governance, which may allow the business to map their AI controls to multiple legal frameworks.

Converging Themes

While details of AI laws differ across jurisdictions, trends seem to be converging on risk-based classification, transparency requirements, and enforcement efforts. Regulators are moving toward risk-based classification. This means AI uses are categorized according to their use case (and the risk associated with that use case). As seen in the EU AI Act, the Colorado AI Act, and TRAIGA, systems may be prohibited or classified by risk. High-risk systems tend to have stricter governance, testing and documentation requirements. Another shared theme is transparency. Laws including the EU AI Act, Colorado AI Act, Utah AI Policy Act, may require covered entities to tell people when AI is in use, while other laws may require the developer or deployer to explain the logic behind certain outputs, and provide consumers with a methods of contesting certain decisions, or opt out of certain types of decisionmaking entirely. The California AI Transparency Act and the EU AI Act may also require labeling of certain AI-generated content. Finally, enforcement is sharpening. The EU AI Act comes with regulatory teeth, with fines of the higher of €35,000,000 or 7% global annual turnover for violation of prohibited practices. In the U.S., state attorneys general and regulators have been active in monitoring AI missteps, including consumer protection and privacy violations. For example, attorneys general in Massachusetts and Oregon have issued advisories on how consumer protection laws apply to AI, while Texas Attorney General Ken Paxton reached the first-of-its-kind settlement in a healthcare generative AI investigation.

The European Union Artificial Intelligence Act (EU AI Act)  

Overview: The EU AI Act is the world’s first comprehensive AI regulation and sets a high-water mark for governance expectations. The Act is technology neutral and uses risk-based classification to sort AI systems into risk-tiers, each with escalating obligations. Key Provisions:
  • Prohibited systems include cognitive behavioral manipulation, most real-time biometric identification, and systems used for social scoring. These systems are considered to pose an unacceptable risk to safety or fundamental rights.
  • High-risk systems include hiring tools, biometric identification, and critical safety technology. They must undergo conformity assessments, maintain technical documentation, and ensure human oversight.
  • Limited-risk systemsinclude chatbots, deepfake generators, and public facing generative AI. These systems have transparency obligations to ensure users understand they are interacting with AI.
  • Minimal-risk systems include AI-enabled spam filers, grammar checkers, and basic AI in video games. These systems have no specific obligations under the Act, but best practices are encouraged.
Key Dates & Enforcement:
  • February 2, 2025: Prohibitions on certain AI systems and requirements on AI literacy start to apply.
  • August 2, 2025: Rules on general practice AI models, governance, confidentiality, and penalties start to apply.
  • August 2, 2026: The remainder of the AI Act (except for Article 6(1)) applies.
The Act will be enforced by European AI Office and national market surveillance authorities. Non-compliance with the prohibition of AI practices is subject to an administrative fine of up to €35,000,000 or up to 7% worldwide annual turnover, whichever is higher. Non-compliance with other provisions shall be subject to administrative fines of up to €15,000,000 of up to 3% of its total worldwide annual turnover, whichever is higher.

Colorado: Consumer Protections for Artificial Intelligence Act (CO AI Act)

Overview: Enacted in May 2024, the CO AI Act was the first far-reaching AI law in the United States. This Act primarily focuses on high-risk AI systems, including but not limited to those which influence “consequential decisions” – those impacting areas such as employment, education, housing, healthcare, finance, insurance, legal services, and essential government services. Key Provisions: Developer and deployers must both exercise “reasonable care” to protect consumers from known or reasonably foreseeable risks of algorithmic discrimination. For both, this may include providing notice to the Colorado Attorney General within 90 days of becoming aware of new discrimination risks.
  • Developers. There is a rebuttable presumption that the developer used reasonable care if they disclose, among other things:
    • reasonably foreseeable uses and known inappropriate or harmful uses of the AI system (including of algorithmic discrimination) and the measures taken to mitigate them;
    • the intended purpose, benefits, uses and outputs of the AI system; and
    • high-level summaries of the data types used to train the AI system, including data governance measures.
  • Deployers must also exercise reasonable care to protect consumers from any known or reasonably foreseeable risks of algorithmic discrimination. Similarly, there is a rebuttable presumption that the deployer used reasonable care if they complete the following, among other things:
    • a risk-management program that considers the NIST AI Risk Management Framework (AI RMF) or another similarly recognized risk management framework with substantially similar requirements (for more information about conducting an AI Risk Assessment, you can check out our post here);
    • an impact assessment, that includes the purpose, use cases, deployment context, and an analysis of whether it poses any foreseeable risks of discrimination, along with steps taken to mitigate those risks;
    • notice to consumers when certain systems are being used that include the system purpose, contact information, and options to opt-out of AI processing for that purpose, correct personal information used in the decisionmaking process, and appeal the decisionmaking process.
  • Disclosure should be clear. Regardless of risk level, any AI system that is directly interacting with Colorado consumers must disclose that it is an AI system, unless that would be obvious to a reasonable person.
Key Dates & Enforcement: While this law was originally set to take effect in 2026, Colorado Governor Polis called a special legislative session to address budget issues, taking place on August 21. The impact of SB24-05 (Consumer Protections for AI) is on the agenda, which may result in a delayed enforcement deadline and substantive changes to the law’s provisions. Violations are treated as deceptive trade practices under Colorado’s Consumer Protection Act, subject to enforcement by the Colorado Attorney General and penalties of up to $20,000 per violation.

Texas Responsible AI Governance Act (TRAIGA)

Overview: While TRAIGA originally provided a comprehensive AI framework, the final version has been significantly pared down. With narrow substantive provisions, TRAIGA focuses on harms caused by AI, and the Act regulates – or completely bans – certain uses of these systems. TRAIGA applies broadly to private sector companies if they provide AI-generated content or services to Texas residents, even if they are located outside the state of Texas. Additionally, government agencies interacting with the public fall squarely within the scope of the Act. You can read more about TRAIGA at our blog post covering the Act here. Key Provisions:
  • Prohibited AI For Public and Private Sectors include but are not limited to intentionally inciting self-harm, violence or crime; infringing on an individual’s rights; or unlawfully discriminating (with purposeful intent). The Act also prohibits deploying AI systems that intentionally generate illegal content, as well as child sexual abuse material or sexually explicit chat systems that impersonate children.
  • Prohibited AI uses for the Public Sector include but are not limited to social scoring and uniquely identifying individuals with biometric data (with limited exceptions).
  • Transparency Requirements for Public Sector may require governmental agencies to, among other things, provide conspicuous notice to consumers that they are acting with an AI system.
Key Dates & Enforcement:   TRAIGA was signed into law in June 2025 and takes effect on January 1, 2026. With no private right of action, the Act can only be enforced by the Texas Attorney General. The Act requires the Attorney General to create an “online mechanism” on their website where consumers can submit complaints of potential violations. If the Attorney General determines a violation has occurred, there is a 60-day cure period. If the violation continues after this period, the Attorney General may bring a claim for, among other things:
  • an injunction;
  • a civil penalty for curable breaches between $10,000 and $12,000;
  • a civil penalty for uncurable breaches between $80,000 and $200,000; and
  • a civil penalty for each day of continued violation between $2,000 and $40,000.
 

California CCPA Draft Regulations

Overview: On July 24, 2025, the California Privacy Protection Agency (CCPA) board voted 5-0 to finalize Draft Regulations to the California Consumer Privacy Act (CCPA). The CPPA sent the rulemaking package to the Office of Administrative Law, which has 30 days to approve the regulations. For a deeper dive on the CCPA Draft Regulations, please see our post here. Key Provisions:
  • Automated-decisionmaking (ADMT): Businesses must inform consumers with a pre-use notice and provide opt-out rights when AI or automated tools influence “significant decisions,” including those about employment, education, housing, healthcare, financial or lending services, and similar areas.
  • Risk Assessments: Organizations engaging in high-risk data processing (such as the decisions covered in ADMT, above) must conduct risk assessments before beginning processing, and must update them regularly, including within 45 days of any material change of the system. For more information about conducting an AI Risk Assessment, you can check out our post here.
  • Cybersecurity Audits: Businesses meeting certain thresholds must undergo annual, evidence-based audits carried out by a “qualified, objective, independent professional.” The audits must rely on specific evidence (as opposed to assertions by the business management), and all information related to the audit should be kept for a minimum of five years after completion.
Key Dates & Enforcement: Compliance with these Draft Regulations will be required once they are approved by the Office of Administrative Law. The deadlines include:
  • ADMT Regulations: January 1, 2027
  • Privacy Risk Assessments: December 31, 2027
  • Cybersecurity Audits:
    • For businesses with $100+ million in annual gross revenue: April 1, 2028.
    • For businesses between $50 million and $100 million in annual gross revenue: April 1, 2029.
    • For businesses with less than $50 million in annual gross revenue: April 1, 2030.

Other Laws to Consider

Along with the more far-reaching laws provided above, there are additional laws that businesses may want to consider when building, implementing, or otherwise engaging with AI tools or systems.
  • Utah’s Artificial Intelligence Policy Act
    • Effective as of May 2024, this Act mandates certain disclosures when businesses use generative AI to interact with consumers. This applies specifically to “regulated professions,” where the provider shall make the disclosure prominently, regardless of whether it is obvious the person is interacting with an AI system or not.
  • New York City’s Local Law 144 (and other AI employment regulations)
    • Signed in 2021, this law applies to employers and employment agencies in New York City that use “automated employment decision tools” to screen candidates or employees. It requires that an independent bias audit be conducted within one year of using the AI tools. For more information on AI in employment, see our article on AI In the Workplace: Legal Considerations for Leadership Teams.
  • California’s AI Transparency Law (SB 942)
    • Effective January 1, 2026, this law applies to “covered providers” – those offering generative AI systems with over 1 million monthly users in California. These providers must provide: 1) a free, public AI detection tool; and 2) certain disclosures as a label or embedded within their content.
  • California’s Data Transparency Law (AB 2013)
    • Effective January 1, 2026, developers of generative AI systems must post a disclosure on their website including documentation used to train the AI system. This documentation includes high-level summary of datasets used in the development of the AI system – the sources or owners of the datasets, how they further the purpose of the AI system, the number of datapoints in the datasets, and more.

Key Takeaway

As lawmakers race to keep up with the breakneck speed of AI implementation, guidance is quickly becoming enforcement. While specific requirements between these laws vary, the common thread is clear: covered entities are expected to understand, document, and justify their AI systems’ design, data, and impact. Additionally, organizations utilizing AI should consider building responsible AI governance into their operations. By incorporating these governance processes into everyday systems and – similar to those for privacy and cybersecurity – organizations may proactively protect against legal, ethical and operational risk when implementing AI.
0
Image depicting the flag of Texas, which is blue, white, and red, with a lone white star.

Overview of the Texas Responsible AI Governance Act

AI & Algorithms, United States , ,
In June 2024, Texas Governor Abbott signed HB 149 for the Texas Responsible Artificial Intelligence Governance Act (TRAIGA, or the Act) into law, which will go into effect on January 1, 2026. With this law, Texas joins California, Colorado and Utah in implementing AI-specific state laws. While TRAIGA was originally provided a comprehensive AI framework, the final version has been significantly pared down. With narrow substantive provisions, TRAIGA focuses on harms caused by AI, and the Act regulates – or completely bans – certain uses of these systems. Businesses that develop or deploy AI systems in Texas should consider measures for compliance with the Act. This may include reviewing system uses and updating the system’s documentation with details the Texas Attorney General (AG) may need to investigate a complaint. Scope & Applicability TRAIGA applies broadly to private sector companies if they provide AI-generated content or services to Texas residents, even if they are located outside the state of Texas. Additionally, government agencies interacting with the public fall squarely within the scope of the Act. Substantive Provisions
  • Prohibited Uses for all AI. TRAIGA prohibits certain uses of AI systems for both the public and private sector, including intentionally inciting self-harm, violence or crime; infringing on an individual’s rights; or unlawfully discriminating. The Act also prohibits deploying AI systems that intentionally generate illegal content, as well as child sexual abuse material or sexually explicit chat systems that impersonate children.

Notably, accidental or disparate impact alone is most likely not enough to violate TRAIGA, as there most likely must be a purposeful intent to discriminate using the AI system.

  • Public Sector: While the scope of TRAIGA includes both public and private entities, there are additional requirements for Texas governmental agencies.
    • Prohibited Uses: Prohibited uses include, among other things, social scoring and uniquely identifying an individual using biometric data, with limited exceptions.
    • Transparency Requirements: Governmental agencies must, among other things, provide conspicuous notice to consumers that are interacting with the AI system – even if this would be obvious to the user; and are prohibited from gathering biometric data without the individual’s informed consent if doing so would infringe that individual’s rights.
Enforcement With no private right of action, TRAIGA can only be enforced by the Texas AG. The Act requires the attorney general to create an “online mechanism” on the AG’s website where consumers can submit complaints of potential violations. If the AG investigates a complaint, developers and deployers of the AI systems may be required to provide information including, but not limited to, a description of the:
  • purpose, intended use, deployment context, and associated benefits of the AI system;
  • categories of data used as inputs and of the outputs produced by the system;
  • types of data used to train the system;
  • evaluation criteria of the performance of the system;
  • known limitations of the system; and
  • post-deployment and user safeguards (e.g., oversight, use and learning processes established to address issues).
If the AG determines a violation has occurred, there is a 60-day cure period. If the violation continues after this period, the AG may bring a claim for, among other things:
  • an injunction;
  • a civil penalty for curable breaches between $10,000 and $12,000;
  • a civil penalty for uncurable breaches between $80,000 and $200,000; and
  • a civil penalty for each day of continued violation between $2,000 and $40,000.
Safe Harbor Notably, TRAIGA states that a defendant may not be found liable for violations if the defendant substantially complies with the most recent version of the Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (AI RMF Framework) published by the National Institute of Standards and Technology (NIST) or another recognized risk management framework for AI systems. Sandbox Program Under TRAIGA, the Texas Department of Information Resources is required to establish a sandbox program. This program would enable developers and deployers of AI systems to obtain legal protection and limited market access. The AG may not file or pursue charges against a program participant for violations of TRAIGA that occur during the testing period. Effective Date The Act goes into effect on January 1, 2026.
0
An image of the flag of Europe, which consists of twelve golden stars forming a circle on a blue field.

EDPB Opinion on AI Models and GDPR Principles: Key Takeaways

AI & Algorithms, EEA & UK, EU, GDPR , , , , ,
In December 2024, the European Data Protection Board (EDPB) issued an Opinion in response to a request from the Irish supervisory authority, focusing on the application of GDPR principles in the context of AI models. The Irish supervisory authority posed three specific questions:
  1. When and how can an AI model be considered “anonymous”?
  2. What is the appropriateness of legitimate interest as a legal basis for AI deployment and development?
  3. What are the consequences of unlawful processing of personal data on subsequent operations of the AI model?
Through its answers, the EDPB provided key guidance on how AI models interact with fundamental rights to privacy and data protection established in the GDPR.

Anonymous AI Models

According the EDPB, “[f]or a model to be anonymous, it should be very unlikely 1) to directly or indirectly identify individuals whose data was used to create the model, and 2) to extract such personal information from the model through queries.” While anonymous data can help mitigate privacy concerns, it does not automatically make the AI model completely exempt from GDPR compliance. When a model is claimed to be anonymous, supervisory authorities will evaluate the claims of anonymity on a case-by-case basis, considering “all the means likely to be used” by the controller or a user. The Opinion states that supervisory authorities should review the documentation provided by the controller when assessing if the model is truly anonymous. The EDPB outlines methods that the controller may use to demonstrate anonymity, which may include: 1) reducing the amount of personal data used during training, 2) taking steps to ensure this data cannot be identified, and 3) utilizing technical safeguards to prevent data extraction from the AI model using prompts or queries. Key Takeaway: If a business claims an AI model relies on anonymous data, the claims of anonymity should be substantiated on a case-by-case basis with sufficient evidence and documentation. To do this, businesses with allegedly anonymous AI models may need to implement technical measures to limit the collection of data, reduce the likelihood of data being identifiable, protect against that data being extracted by users during deployment, and create documentation capable of demonstrating these efforts.

Legitimate Interest as a Legal Basis

Under the GDPR, a legitimate interest may constitute a legal basis for companies to process personal data when they have a justifiable reason to do so (beyond obtaining consent). However, the legitimate interest should be balanced against the data subject’s rights and interests, which requires careful consideration and justification when processing information from data subjects. The Opinion provides a framework to assess if a legitimate interest can be a valid legal basis for processing personal data in AI development and deployment. The framework is comprised of a three-step test:
  1. Identify the legitimate interest pursued by the controller;
  2. Assess the necessity of the processing for purposes of the legitimate interest; and,
  3. Balance the legitimate interests against the rights and freedoms of the data subjects.
When conducting this test, the controller should be careful to identify an interest that is lawful, clearly articulated, and non-speculative. For example, a legitimate interest may be to develop an AI model’s conversational agent or to improve threat detection in an information system. The controller should also adhere to GDPR data minimization principles, which state that the processing activities must be proportionate and in line with only what is necessary to achieve the legitimate interest. Finally, controllers should conduct a nuanced balancing test. This test considers the unique circumstances of each case, which may include the data subject’s interest in retaining control over their data, personal benefits, or socioeconomic interest. The Opinion notes, the more precisely an interest is defined in relation to the purpose of the processing, the more precise the estimation of benefits and risks will be. By employing this framework, developers and deployers should be able to decrease the likelihood that their AI models are disproportionately infringing on individual privacy rights and better align their AI practices with GDPR requirements. Key Takeaway: The three-step analysis, according to the Opinion, is crucial to improving compliance for organizations relying on legitimate interest as a legal basis for processing in AI development or deployment. Organizations relying on legitimate interest in this AI context should review their processing activities to determine whether they are proportionate, transparent, and aligned with GDPR principles—like data minimization—to justify the reliance on legitimate interest as a legal basis for processing.

Consequences of Unlawful Processing

The Opinion notes that supervisory authorities enjoy discretionary powers to investigate and assess violations, and they can choose appropriate remedial measures based on the context of the case. However, the EDPB also provides guidance for the supervisory authorities, based on three scenarios.
  1. In the first scenario, personal data is retained in the AI model. The Opinion states that supervisory authorities will need to consider the surrounding circumstances of the AI model to determine if the development and deployment phases of the model involve different legitimate purposes for processing. If so, each should be examined separately.
  2. In the second scenario, personal data is retained in the model and is processed by another controller during deployment. In this instance, the supervisory authorities should determine if the deploying controller conducted an appropriate assessment to demonstrate accountability with Articles 5(1)(a) and 6 of the GDPR. This assessment should show that the AI model was not developed by unlawfully processing personal data.
  3. In the final scenario, a controller unlawfully processes personal data to develop the AI model, and then anonymizes the data before processing it in the context of deployment. The Opinion states that, if it can be demonstrated to the supervisory authorities that the deployment of the AI model does not entail the processing of personal data, then the GDPR does not apply. Therefore, the unlawfulness of the initial processing in development should not impact the deployment operation of the model.
While supervisory authorities do have substantial discretion in oversight of processing activities, the scenarios highlighted by the EDPB show that the development and deployment phases, while connected, may need to be evaluated independently. Key Takeaway: Organizations should proactively ensure compliance at both the development and deployment stages of an AI model. Supervisory authorities will likely use the above examples as guidance, emphasizing the important of demonstrating lawful practices through each stage of the model. The EDPB’s Opinion is an important guide for organizations navigating the intersection of AI and data privacy law. By addressing issues around anonymous AI models, legitimate interest, and lawful processing in development and deployment stages, the Opinion emphasizes responsible AI development. As AI technologies continue to advance, businesses should be aware of the ways supervisory authorities are overseeing their AI models. The insights provided by the EDPB provide a foundation to help businesses to advance and develop new AI models, while also helping to safeguard and protect the rights of individuals.
0

Metaverse Law in Orange County Lawyer Magazine

AI & Algorithms, CCPA, Cybersecurity, GDPR, Health Data, State Privacy Laws , , , , , , ,
The January 2025 edition of Orange County Lawyer magazine features an article written by Metaverse Law’s Lily Li. Read “AI and Machine Learning in Drug Development and Clinical Trials” below or in Orange County Lawyer magazine.
[Originally published as a Feature Article: AI and Machine Learning in Drug Development and Clinical Trials, by Lily Li, in Orange County Lawyer Magazine, January 2025, Vol. 67 No.1, page 28.]   AI and Machine Learning in Drug Development and Clinical Trials by Lily Li   In 2013, sleep medication zolpidem (Ambien, Ambien CR, and Edluar) swept headlines. Marie Claire reported on an alarming and suspicious rise in users experiencing irrational eating, gambling, and even “sleep-driving” while in a hypnotic trance—waking with no memories of their actions.[1] In several cases, women arrested and convicted for driving under the influence contested their convictions, arguing that they were not liable for these undisclosed drug-related side effects. At the same time, several clinical studies suggested that women metabolized zolpidem differently from men. By reviewing existing literature, Japanese researchers out of Shimane University identified 40% higher concentrations of zolpidem in women than men following use, and higher rates of visual hallucinations and sensory distortions.[2] The FDA released a safety advisory, warning users of the risks of “next-morning impairment” for the use of Ambien and related drugs.[3] In addition, the FDA took the unusual step of recommending a 50% cut in the dosage for women. When asked about the change, an FDA director told ABCNews.com: “The changes are different in women and men . . .We don’t understand why yet, but women are more susceptible to next-morning impairment.”[4] Yet, a decade later, the evidence supporting different zolpidem dosages for women and men is unclear.[5] In part, this is due to the lack of research surrounding sex differences in drug impact and drug treatment, as well as substantial gaps in the inclusion of women in clinical studies. From 1977 to 1993, FDA policy recommended excluding women of childbearing potential from Phase 1 and early Phase II drug trials.[6] Even after this policy was removed in 1993, industry fears remained with respect to drug interactions with pregnancy. This episode with zolpidem raised several concerns in the drug development and clinical trial process:
  • How do we recruit representative candidates for drug trials?
  • How do we ensure the quality and availability of datasets for clinical research?
  • How do we measure potential impacts of drug dosing on different populations?
  • What are the legal implications for failing to address appropriate drug doses?
  AI and ML to the Rescue? Now that artificial intelligence is being used in research and development, one wonders: Can artificial intelligence (AI) and machine learning (ML) reduce bias and risks during drug development? Or will it create new legal risks due to bias, privacy intrusions, and lack of transparency? The FDA released a discussion paper on AI, Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products, to discuss potential regulatory frameworks to address the use of AI and ML.[7] In this discussion paper, the FDA released a set of fascinating case studies into existing research and uses of AI in the clinical trial process. Several of these case studies are discussed below, as well as an analysis of their potential impact on the zolpidem example.
  1. Recruitment. According to the FDA, “AI/ML is being used to mine vast amounts of data, such as data from clinical trial databases, trial announcements, social media, medical literature, registries, and structured and unstructured data in EHRs [electronic health records], which can be used to match individuals to trials (Harrer, 219 Shah, Antony, & Hu, 2019).” In this manner, researchers can combine huge quantities of publicly available data and individual health data from prior research to identify participants with certain medical conditions (or lack of adverse conditions) for investigational treatments. For zolpidem, the use of AI/ML may have been able to identify a much broader list of participants for initial clinical testing, making it easier to assess and identify adverse reactions.
  2. Selection and Stratification of Trial Participants. In addition to initial recruitment, AI/ ML has the capability improve intake, selection, and classification of clinical trial participants. Based on baseline characteristics selected by the researchers, such as prior clinical data, and vitals/labs taken during intake, predictive algorithms can help identify high-risk participants.[8] These groups can then be randomized and then subject to more strict monitoring protocols. In the case of zolpidem, alcohol use is associated with sometimes severe adverse effects from the drug, and so it would be beneficial to screen out candidates with a history of alcoholism or, on the flip side, assess drug interactions for this high-risk group with additional support, monitoring, or counseling.
  3. Dose/Dosing Regimen Optimization. AI/ML can be used to predict drug exposure for different populations based on factors such as weight, height, sex, and other characteristics that might impact drug metabolism. Based on prior drug exposure and response profiles for similar drugs and similar populations, AI/ML can help to narrow the dose/dosing regimen selected for a study. As noted by the FDA’s discussion paper, this can help optimize drug dosing “in special populations where there may be limited data (e.g., rare disease studies, pediatric and pregnant populations).” Based on this research, we can imagine future scenarios where AI/ML could have avoided zolpidem dosing concerns, where graduated and limited dosing was tested and applied to different sex, age, and metabolism categories to determine ideal dosing.
  4. Data Analysis. On a more intriguing level, the FDA AI discussion paper discussed the concept of creating “digital twins” of patients for clinical trials. Essentially, an AI version of the clinical participant is created, using the existing candidate’s electronic health records, vital signs, labs and other records. Researchers can assess how the digital twin would react under normal conditions using AI/ML modeling based on data gathered from similar individuals. This digital twin would then act as a substitute for a placebo candidate in a clinical trial, and act as a benchmark against the actual patient undergoing investigational treatment. For zolpidem, this could be used to assess candidates that already have underlying medical conditions such as anxiety, depression, or other confounding factors, to see whether an adverse effect from a trial is due to the investigational treatment or something that is likely to occur to the same individual from anxiety alone.
  5. Postmarketing Safety Surveillance. Finally, AI/ML can help detect and assess adverse events once the drug enters the market. This is not just limited to individual case safety reports (ICSR), required by regulators, but can include adverse events reported publicly on social media and the wider internet. This type of postmarketing safety surveillance could assist researchers and drug companies in identifying potential drug risks, prior to landing on primetime news.
  Quality and Reliability Risks While AI/ML can help to address the costs and efficiency of clinical trials, this relies substantially on the underlying data used to train AI. The quality and reliability of any AI/ML model requires similar quality controls for underlying training data. Given the safety risks of inappropriate drug dosing, or recruiting candidates with severe medical conditions, AI developers cannot rely solely on self-reported healthcare data with no external medical testing or validation. Developers should be equally wary of training on third-party data sets that do not provide documentation on the collection of data and data validation. Within an existing healthcare organization, if the organization is big enough, aggregate and de-identified data may be obtained from existing electronic health care records and prior clinical trials. Yet, even within these large datasets, errors may surface during training. Medical providers may code the same procedure, and similar symptoms, a dozen different ways. Even drug names can be misspelled and coded incorrectly within existing records. While many of these errors may end up being statistically insignificant with enough data, there is the risk of missing one or two major adverse events, or “black swan” events, that would otherwise change the entire risk profile of a drug. In addition to quality and reliability, the underlying dataset needs to be representative of the population that will be studied for the clinical trial. If the underlying dataset is only trained on a handful of individuals with a certain medical predisposition, age, sex, weight, etc., it will be difficult for the AI model to make predictions for that group. As an example, if the training data only contains the medical information for two individuals over the age of sixty, and shows no adverse effects from a particular drug dose, this information is not enough to generalize that the drug at that dosage is appropriate for all individuals over the age of sixty. For all we know, these two candidates could be a former Olympic diver and a nutrition coach, two outliers that completely skew the data. Consequently, the underlying training data for any AI model should also be assessed for bias and representativeness as it applies to the proposed clinical trial.   Data Privacy, Cybersecurity, and AI Risks The data privacy and cybersecurity risks associated with the foregoing uses of AI/ML cannot be underestimated. The quality and representativeness of any AI system in this field will rely heavily on large swathes of healthcare data, fine-tuned and, at times, personalized in the case of digital twins. This is sensitive or special category data at its finest, triggering heightened scrutiny under the EU’s data privacy law, the GDPR, and U.S. data privacy and data breach laws. To date, most healthcare organizations have sidestepped data privacy concerns by relying on HIPAA’s de-identification standard to remove personal information and other identifiers from healthcare data, making it difficult to associate with an individual. While the FDA requires Institutional Review Board (IRB) review of most biomedical research involving human subjects, this generally does not apply to de-identified personal information that cannot be linked to an individual. Simply de-identifying data and then running with it is not enough, however. Under the California Consumer Privacy Act and similar state laws, for example, recipients of de-identified data need to affirm that they will not attempt to reidentify the data (except to test their de-identification methods). The GDPR has a much higher “anonymization” standard, which looks at the re-identifiability of personal information, given all the different datasets that an organization may have access to. AI/ML itself is making the de-identification process harder. As it is capable of slicing and dicing data by age, race, sex, and medical condition, and combining multiple large datasets, it is easy to run the risk of re-identifying data. While several thousand people might have the same configuration of eye color, age, gender, and weight, only one or two may have participated in a clinical trial at a particular location, or have specific allergies or side effects to certain types of medication. As a result, in circumstances where healthcare data is not de-identified, or the risk of reidentification is heightened, then it behooves clinical organizations and their AI developers to implement written information security programs and associated privacy and security controls.   Legal Liability and Drug Dosing In several notable cases, defendants on zolpidem were able to contest or overturn DWI or even vehicular manslaughter cases. Essentially, these defendants argued that they were not aware of the potential dangers of zolpidem, and so could not be liable for their actions while “sleep driving.” This raises the question: If AI gets good enough, and can tell you exactly the right dose to take of a drug, will you (or your doctor) be liable if you deviate from the AI’s recommendations? Will the AI’s recommendations be discoverable in court (and surfaced via AI-enhanced search)? Only time will tell what this brave new world will bring.   ENDNOTES [1] Kai Falkenberg, While You Were Sleeping (September 27, 2012), Marie Claire, https://www.marieclaire.com/culture/news/a7302/while-you-were-sleeping/.   [2] Takuji Inagaki, Tsuyoshi Miyaoka, Seiichi Tsuji, Yasushi Inami, Akira Nishida, and Jun Horiguchi, Adverse Reactions to Zolpidem: Case Reports and a Review of the Literature, 12 Prim Care Companion J Clin Psychiatry 6 (2010), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3067983/.   [3] U.S. FDA, Drug Safety Communication: FDA approves new label changes and dosing for zolpidem products and a recommendation to avoid driving the day after using Ambien CR (May 14, 2013), https://www.fda.gov/drugs/drug-safety-and-availability/fda-drug-safety-communication-fda-approves-new-label-changes-and-dosing-zolpidem-products-and.   [4] FDA: Cut Ambien Dosage for Women, ABC News (January 10, 2013, 6:03AM), https://abcnews.go.com/Health/fda-recommends-slashing-sleeping-pill-dosage-half-women/story?id=18182165.   [5] David J Greenblatt, Jerold S Harmatz, & Thomas Roth, Zolpidem and Gender: Are Women Really At Risk?, 39(3) J. Clinical Psychopharmacol. 189 (May/Jun 2019), https://pubmed.ncbi.nlm.nih.gov/30939589/.   [6] NIH Inclusion Outreach Toolkit: How to Engage, Recruit, and Retain Women in Clinical Research, last accessed September 16, 2024: https://orwh.od.nih.gov/toolkit/recruitment/history.   [7] FDA, Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products (May 10, 2023), https://www.fda.gov/media/167973/download; see also Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products; Availability, 88 FR 30313 (May 11, 2023), https://www.federalregister.gov/documents/2023/05/11/2023-09985/using-artificial-intelligence-and-machine-learning-in-the-development-of-drug-and-biological.   [8] Thi Tuyet Van Tran, Hilal Tayara, and Kil To Chong, Artificial Intelligence in Drug Metabolism and Excretion Prediction: Recent Advances, Challenges, and Future Perspectives, 15 Pharmaceutics. 1260 (Apr 17, 2023), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10143484/.   Lily Li is an AI, data privacy, and cybersecurity lawyer and founder of Metaverse Law. She is a certified information privacy professional for the United States and Europe and is a GIAC Certified Forensic Analyst for advanced incident response and computer forensics. She can be reached at info@metaverselaw.com.
1 2 3 4 5