Phishing in the Age of AI

AI & Algorithms, Cybersecurity ,

Phishing is one of the most common practices used by hackers looking to gain access to company systems. Phishing attacks typically rely on an individual impersonating a trusted person or organization to convince the victim of the attack to click a link, provide login credentials, send money or disclose other – often sensitive – information. 

A recent settlement involving Ambry Genetics (Ambry) is a reminder that phishing attacks can create consequences beyond a single compromised account. In September 2026, the U.S. Department of Health and Human Services’ Office for Civil Rights (OCR) announced a $700,000 settlement with Ambry following a phishing incident that potentially exposed protected health information (PHI) belonging to over 200,000 people. 

The settlement highlights why businesses need to consider phishing awareness and prevention as part of their broader cybersecurity program – especially as AI can amplify the scale of phishing attacks and can make impersonation attempts more difficult to decipher. 

What does phishing look like?

According to NIST, phishing attacks occur when “the perpetrator masquerades as a legitimate business or reputable person” to gain access to personal, confidential, or other sensitive information. An example of phishing might be an employee who receives an email that appears to come from another co-worker, bank, vendor or boss asking them to reset a password, review an invoice or log into an account. The link may then lead to a fake website designed to steal information. 

Phishing may also happen through text or SMS messages and phone calls. The FBI refers to these variations as “smishing” and “vishing,” respectively, and they look like typical phishing attempts, just transacted through a different means of communication.

Attackers do not always need to break through technical security controls. Often, they just need to convince one employee to give them access through fraudulent means. 

What happened at Ambry Genetics?

Ambry Genetics is a California based genetic-testing company.  In 2020, the company discovered that an employee had compromised a company email account via a phishing attack. 

According to OCR, the attacker potentially gained access to PHI belonging to over 200,000 people. The compromised information may have contained data points such as names, addresses, birthdays, diagnoses, lab results, and financial information.

OCR’s investigation of the attack revealed more than a single phishing incident:  It uncovered that Ambry was not compliant with broader security practices required by HIPAA and its relevant rules, including questions of whether the company had conducted an adequate risk analysis, properly removed employee access after they had left the company, and assigned unique user login information to individual users. 

As part of the settlement, Ambry agreed to a two-year corrective action plan. The plan includes elements such as requirements to conduct proper risk analyses, address identified security risks, and review data protection policies and procedures, while also providing additional workplace training. 

It is important to highlight that a phishing attack can be the impetus for regulators taking action. What looks like an isolated incident may lead regulators to examine more than just how the attacker got in – they may also investigate whether the company had reasonable security practices in place before the incident occurred. 

How might AI change phishing?

The FBI has warned that criminals are using AI to create more polished and personalized phishing messages to imitate voices or images of trusted people. AI-generated audio can be used to impersonate people to make a phone call that sounds like it’s coming from a boss, co-worker, or family member. 

Because AI is generally increasing the ability to create phishing messages, traditional warning signs may be less reliable. For example, poor grammar, awkward wording or random phone numbers may be less obvious when AI can quickly generate messages and voices. 

Therefore, businesses who are providing training on how to recognize phishing may consider focusing less on whether a message simply looks or sounds legitimate, and more on whether the request itself should be independently verified. For example, if someone asks for a wire transfer, password reset or access to confidential information, employees should consider verifying that request through other means. This could include calling a known number or using a separate communication method. The FBI recommends independently confirming the identity of someone making a suspicious or unusual request. 

Key Takeaways and Security Practices to Consider 

Phishing attacks aren’t new, but the age of artificial intelligence may be changing the scope and frequency of these kinds of attacks. For example, a study by IBM shows how AI can be used to amplify both the amount of phishing attacks and the human quality of the attack itself.  With this in mind, fundamental phishing prevention is as important as ever. When considering how to lower its susceptibility to phishing emails, business may consider the following:

  • Training employees: Employees are often key targets for phishing attacks and should understand what these attacks can look like and how to report suspicious messages to information security teams within the business. 
  • Enabling MFA: Multi-factor authentication (MFA) is a process where an individual has to provide multiple forms of verification to enter an account. For example, if MFA is enabled, a stolen password alone should not give an attacker immediate access to an account. 
  • Limiting employee access: As a general rule, employees should only be able to access systems and information that is needed for their job. Typically, this access is revoked after a period of time or when it is no longer needed. 
  • Preparing for incidents: Businesses may consider having tools in place to disable compromised accounts, investigate incidents and determine if sensitive information was exposed, among other steps that may be required in the investigation process. 

Agentic AI meets Cybersecurity

AI & Algorithms, Artificial Intelligence, Cybersecurity , ,

AI has moved beyond simply answering questions or giving recommendations. By now, you may have heard a new(er) term floating around the internet: agentic AI. These are AI systems that can be given access to tools, company data, and applications, with the goal of performing tasks on a user’s behalf.

While this added functionality may improve efficiency, it can also create cybersecurity risks. For example, an AI system that can only draft emails inherently has different risks than one that’s able to view inboxes, see customer information and send emails on a person’s behalf. 

As businesses begin to use these agentic systems, they should consider both what information an AI agent can access and what authority an AI agent has to act on that information. 

What are key distinctions of agentic AI? 

An important difference in understanding the risk profiles between standard AI (like an LLM) and agentic AI is not just that the agent can perform more tasks. Instead, it is often an issue of access. For example, a conversation with an LLM may be a one-time input and output of that information, while the use of agentic AI may give the agentic system ongoing access and authority to operate inside company environments. This increase in access and authority has the potential to substantially increase the amount of risk that comes with this type of system, because agentic AI may be able to use its existing permissions to make decisions or perform actions without human oversight.  

Therefore, the level of access and permission an agent has can be pivotal to how risk is controlled and managed in an organization. An AI agent with narrow access and required human oversight/permission will likely present lower risks than one that is able to act independently without such restrictions. For businesses, this makes permissions, oversight and the ability to stop or limit an agent central to cybersecurity concerns.

How can businesses maintain control over AI agents? 

The National Institute of Standards and Technology’s AI Risk Management Framework (“NIST AI RMF”) provides a useful way for businesses to approach these risks. This voluntary industry standard framework organizes AI risk management into four functions: Govern, Map, Measure and Manage. Rather than just a checklist, this framework encourages organizations to establish responsibility for their AI systems, understand how they work, evaluate their risks and continue managing those risks throughout the AI lifecycle.  

This framework is flexible and can be applied to agentic AI systems. As a foundation, the principles outlined in the NIST RMF can help businesses answer practical security questions: Who is responsible for overseeing an agent? What is the agent’s intended role? What systems and sets of information does it need access to? What could happen if mistakes are made? And what measures are in place if its actions/behavior have to be limited or stopped?

Additionally, NIST has begun focusing specifically on identity and authorization risks for these AI agents, including issues like authentication and auditing. 

How can AI agents become a cybersecurity target?

Businesses may also consider attacks directed at the AI agent itself, such as prompt injection. Prompt injection is where instruction contained in a prompt (or input) is used to manipulate an AI agent’s behavior. For an AI agent, this prompt could be in an email, document or other outside content that the agent is asked to review. 

For example, pretend a business has an AI agent which is tasked at reading emails and has access to internal files. A malicious email might contain instructions attempting to persuade the AI agent to retrieve information it does not need and disclose sensitive information back to the sender of the email. This is a classic example of a prompt injection attack that deployers of AI agents should be aware of. 

NIST’s agent-security work specifically identifies prompt injection as an issue organization are recommended to consider when developing identity and authorization controls for agents. 

What safeguards should businesses look for? 

Security should not depend solely on the employee using an AI agent correctly. Instead, businesses should consider the protections that are built into the AI agent itself. 

For example, an AI agent may be designed with limited, instead of broad, access to company information by default. This way, the AI agent’s ability to access information is limited by permissions required to perform its rule. Higher-risk actions may require additional approval. For example, an agent might be allowed to draft an email independently, but sending sensitive information would require human confirmation, approval, and oversight of the agent. 

Businesses may also consider whether the agent creates records or logs of its own activity, whether different systems can be restricted, and if an agent’s access is able to be quickly suspended. These safeguards may limit damage caused by both mistakes and outside attacks that result in attempts to manipulate an agent. 

Again, the goal does not need to be to prevent AI from acting independently, but rather to ensure that its ability to act stays within clearly identified boundaries. 

Why does monitoring matter? 

Even when a business has security limits in place, it will still need to be able to identify when an AI agent behaves unexpectedly or attempts to go outside those pre-defined limits. 

A recent OpenAI incident provides a good example of why monitoring is essential.  In September 2026, an AI agent was given a search-based training task, and in completing the task, the agent found that there was a gap in internet access restrictions and used the internet to query an external chatbot. OpenAI reported that its monitoring system detected the unauthorized activity within about 15 minutes, and additional controls were added that would have blocked the access.

This incident highlights the possibility that setting permissions before deployment may not be enough to prevent AI agents from only acting within their boundaries. Put simply, businesses should also monitor what an agent does. For example, a standard review of an AI agent’s activity logs may help determine:

  • What system an agent accessed,
  • What actions it performed,
  • When that action occurred,
  • Whether sensitive information was involved, and,
  • Whether the activity stayed within an agents assigned role/access.

Businesses should remember that cybersecurity runs in both directions, and risks can include: 1) protecting an AI agent from unauthorized access, and 2) monitoring the AI agent itself for unauthorized activity. If unusual activity occurs, businesses should know how to quickly respond. 

What happens if an AI agent accesses something it should not? 

Existing cybersecurity laws may also be relevant when an AI agent interacts with systems outside of the business. 

For example, the federal Computer Fraud and Abuse Act (“CFAA”) addresses certain access to protected computers without authorization or beyond its authorized access. Courts have already begun to explore how this law applies when an AI agent performs the action.

In Amazon.com Services LLC v. Perplexity AI, Inc., Amazon argued that Perplexity’s agentic browser violated the CFAA by accessing Amazon through users’ accounts. In August 2026, the Ninth Circuit vacated a preliminary injunction against Perplexity after concluding Amazon was unlikely to show that Perplexity itself has accessed Amazon’s computers. Instead, the court viewed the user as accessing Amazon with the AI agent helping perform the actions. 

In this case, the court made clear that its decision was limited to the facts of that case and did not create a broad rule for agentic AI. The bigger takeaway is that businesses should set clear limits on where their agents can operate and what they can access.

However, businesses should not assume they are off the hook just because an AI acted independently. California Civil Code § 1714.46 says a company cannot defend a claim by arguing that the AI caused the harm on its own. This does not mean the business is automatically liable, but it highlights why companies need to keep oversight and controls in place.

What should businesses take away?

Agentic AI are changing the way businesses operate, but giving an AI system the ability to act upon company data and applications should require additional security planning and considerations before deployment. Businesses deploying agentic AI may want to consider the following:

  • Define the agent’s job: establish clear guidelines as to what the agent is supposed to do and what falls in and out of that role.
  • Limit permissions: give the agent access only to the systems it absolutely needs to operate with for its assigned tasks.
  • Keep humans in the loop: determine how best to monitor agentic AI’s actions and, where necessary, implement additional oversight. 
  • Look for built-in safeguards: evaluate whether the agent supports limited permissions, approval controls, logging and quick revocation of access. 
  • Monitor agent activity: maintain oversight to determine what the agent accessed and if it acted outside its role. 
  • Avoid giving agents access to sensitive data: avoid giving an AI agent access to health, financial, employee or highly confidential client information.
  • Protect against manipulation: consider how the agent interacts with potentially malicious emails, websites and other outside information.
  • Prepare for incidents: establish protocols and procedures to quickly disable agents if needed. 

Agentic AI changes the cybersecurity question from what AI can see to what AI can do. Businesses need to understand this and limit authority, monitor activity and be prepared to step in if something goes wrong.

0

Deepfakes: A New Form of Workplace Sexual Harassment

Cybersecurity , , ,

In recent years, there has been an uptick in the number of cases where images generated or edited by artificial intelligence have given rise to workplace harassment claims. Regardless of whether the conduct at issue occurred in person or off duty, courts have shown a willingness to hold employers liable, leaving employers vulnerable to significant costs from employee misconduct. 

Current Cases

Employer liability arising from AI-generated content may stem from actionable workplace harassment claims. This could include media such as falsified videos, audio and images containing sexually explicit material which features a real person without their consent. 

Current and pending litigation involving these types of claims includes:  

  • Carranza v. City of Los Angeles (Cal. Ct. App. 2025). A decision from the California Court of Appeals confirmed a $4 million dollar award issued to a female police captain where a deepfake photo of her topless circulated in the workplace. There, the dissemination in the workplace was considered actionable workplace harassment. 
  • Pearson v. State of Washington (Wash. Super. Ct. 2025). Washington State Patrol trooper Collin Pearson alleges coworkers circulated an AI-generated kissing video that created a hostile work environment based on sexual orientation.
  • Friedrichs v. Scripps Media, Inc. (M.D. Tenn. 2025). Former Nashville meteorologist Bree Smith Friedrichs alleges her employer failed to address sexually explicit deepfake images and retaliation tied to workplace sexism claims. 

What about other federal statutes? 

Workplace harassment claims often interact with Title VII of the Civil Rights Act of 1964, which prohibits discrimination on the basis of sex. Additionally, Section 230 limits liability for platforms where harmful content is posted, meaning that if, for example, an employee distributes an AI-generated non-consensual image on a workplace messaging system (e.g. Slack, Microsoft Teams Chat, etc.) the employer, as opposed to the platform, may still be held liable. Additional claims at play may include: 

  • Title VII of the Civil Rights Act of 1964. The primary federal employment law used in deepfake cases. It supports hostile work environment, sexual harassment, sex discrimination, and retaliation claims. Employers face liability if they knew of the conduct and failed to investigate or stop it.
  • TAKE IT DOWN Act. The first major federal deepfake-specific law. It criminalizes knowingly publishing nonconsensual intimate imagery, including AI-generated “digital forgeries.” Requires covered platforms to remove reported content rapidly.

Are state laws involved? 

State laws cover three categories of harm—nonconsensual intimate deepfakes, election deepfakes, and identity impersonation. Additionally, nonconsensual intimate imagery and revenge porn statutes now often explicitly include AI-generated content, prohibiting the distribution of intimate images without consent and adding an additional legal framework supportive of employee claims against employers. 

In California, there are a handful of specific laws addressing this type of AI use, which may include: 

  • AB 602 creates a civil cause of action against anyone who either creates and intentionally shares digitized sexually explicit material without the depicted person’s consent, providing broad protection against deepfake pornography. Claims arising under this statute are supplemented by strong privacy torts, publicity rights, and CA FEHA for workplace claims. 
  • SB 926 explicitly adds AI-generated depictions to CA’s existing revenge porn law. 
  • SB 1381 and AB 1831 extend CA’s protections to include AI-generated content depicting minors. 

Additional laws have been enacted in Connecticut, Michigan, New Jersey, and New York, among other states. Additionally, state and common law claims for defamation may be relevant when deepfakes create false representations that create reputational harm. Deepfake audio and video may be considered evidence of injury. 

What are my potential responsibilities as an employer? 

While the issue is specific, the issue may require comprehensive action in order to preempt potential liability. Employers may consider the following actions: 

    • Updating Policies: Ensure that workplace policies clearly prohibit dissemination of sexually explicit material, real or doctored. Draft or update a standalone AI Acceptable Use Policy that names prohibited conduct (creating, possessing, distributing deepfakes targeting coworkers) and specifies that violations are grounds for discipline up to and including termination. 
    • Incorporating Training: Equip HR, legal, and IT teams to recognize and respond to deepfake incidents effectively.
    • Refreshing Investigation and Response Protocols: Encourage prompt investigations, which may include forensic analysis, verification of metadata, and ensuring fairness in credibility assessments for both alleged victims and accused parties. 
    • Reviewing Insurance: Review employment practices liability insurance coverage to confirm whether deepfake-related harassment claims and related cyber incidents are covered. Many existing EPL policies predate generative AI and may contain gaps.

What’s next? 

This is a rapidly evolving area of employment litigation—the applications of state deepfake and AI-related statutes in workplace harassment claims are likely to turn on pending federal agency actions and court decisions, ultimately determining the limits of employer liability for their employee’s potentially harassing conduct. Concerned employers may consider monitoring this landscape closely and adjusting compliance programs as litigation continues to contour this area of law. 

0
What to know about CIPA and Shine the Light claims

What to Know About CIPA and Shine the Light Claims

Cybersecurity

Doing Business in California? What To Know About CIPA and Shine the Light Claims

  Blog Contributor: Madeline Yuki Gaudlitz, 2L at the University of Michigan Law School In recent months, companies operating in California have reported an increase in demand letters requesting damages for alleged violations under new and existing privacy laws. Under current data privacy legislation, companies can expect these claims to continue. Plaintiffs’ attorneys have relied on two statutes as a basis for their demands, the California Invasion of Privacy Act (“CIPA”) and California Civil Code § 1798.83 (“Shine the Light”).

What is CIPA?

Originally enacted in 1967 to “protect the right of privacy” of California residents, CIPA bans wiretapping, eavesdropping, or recording private communications. In recent years, Plaintiffs’ attorneys compared real-time consumer-tracking software embedded in companies’ websites to the type of behavior CIPA prohibits. In addition to imposing criminal penalties and fines of up to $10,000, the statute allows private individuals whose personal data has been intercepted by businesses to sue for $5,000 per violation.

Who does CIPA apply to?

CIPA may apply to:
  • Companies with consumer facing-websites or applications used by a California resident
  • Both companies that use these technologies in their consumer-facing website or application and third-party developers

What technologies may leave my company exposed under CIPA?

Potential CIPA liability may apply to a range of real-time consumer tracking technologies that are a standard part of website or application design, which may include:
  • Website analytics
  • Software developer kits
  • Third-party tracking pixels and software
  • Fingerprinting software
  • Application programming interfaces
  • Conversation intelligence software-as-a-service (SaaS)
  • Cookies and identity profiles
*Notably, CIPA is sensitive to the processes used to collect customers’ information. Likewise, Shine the Light may not apply to businesses that share information with third parties only for administrative or customer service purposes. To assess liability under these statutes, businesses may want to coordinate with third parties to ensure awareness of their own business practices and awareness and compliance under CIPA.

What is Shine the Light?

Originally enacted in 2003, the Shine the Light law was aimed at increasing customer awareness of how their personal information may be shared with third parties for direct marketing purposes. CIPA requires businesses to disclose their information-sharing practices upon request or allow customers to consent to information sharing. Failure to comply may result in a civil penalty of $500 per violation, and $3,000 if the violation is willful, intentional, or reckless.

Who does Shine the Light apply to?

The Shine the Light law may apply to: 1. For-profit companies with 20 or more full or part-time employees, 2. that collect personal information from California residents, and 3. that have shared customer information with third parties for direct marketing purposes 4. within the immediately preceding calendar year. Direct marketing may include spamming, telemarketing, or mail. Personal information may include name, address, e-mail address, telephone numbers, date of birth, medical or financial information, information about children, race, religion, occupation and education, and information about the transaction.

Best Practices

While these statutes impose distinct obligations, compliance may be able to be addressed by general practices that reflect their obligations to limit data collection and sharing of personal information. To work toward compliance, a company may consider:
  • Reviewing your company’s privacy policy to ensure that it accurately informs consumers in California of their privacy rights.
  • Clearly communicating your company’s privacy policy to consumers.
  • Ensuring that the consumer consents to the collection and sharing of personal information.

For CIPA

Regarding liability under CIPA, businesses may want to consider:
  • Reviewing your website or application design for features that collect personal information of users.
  • Coordinating with third party providers to ensure their awareness and compliance with CIPA risks and requirements.
  • If utilizing real-time tracking technologies, securing a consumer’s affirmative consent to data tracking.

For Shine the Light:

There are a couple of avenues that may limit risk under the Shine the Light Law:
  • Ensure that website or application design, physical store, or employees clearly disclose consumer data privacy rights.
  • Ensure that that website or application design allows consumers to actively and easily consent to personal information sharing.

OR

  • Maintain awareness of sales of customers’ personal information within the preceding year.
  • Establish a designated address–email, mail, or toll-free number–that customers may use to contact a business and request information about how their personal information is used.
  • Be prepared to disclose the types of information shared and the names and contact points for third parties that received or purchased the information within the preceding year within 30 days.

What’s Next?

In the coming years, we may see legislation that responds to the challenges CIPA claims pose to regular business operations in the digital age. SB 690 proposes an exception to CIPA liability for companies that use personal data for commercial purposes. However, the current status of this critical amendment is stalled. What we know now:
  • It will not be reconsidered until the 2026 legislative session, currently set to run from January 5-August 31, 2026.
  • Legislative history indicates that any exception would only apply to future cases, not currently pending claims or claims filed before the amendment is finalized.
  • Unanimous approval in the state senate may reflect policymakers’ concern with applying CIPA to commercial data collecting practices.
Ultimately, the amendment’s status is uncertain, but there is reason for companies to be optimistic about an eventual tapering down of CIPA claims. Despite this, businesses should remain cognizant of other regulations aimed specifically at digital data collection. Credit: Madeline Yuki Gaudlitz
0
Image of a cellular phone with the ChatGPT app open.

Overview: The EU General-Purpose AI Code of Practice

AI & Algorithms, Cybersecurity, EU , , , ,

Why Do We Need a Code of Practice?

On August 2, 2025, the general-purpose AI (GPAI) provisions of the EU AI Act went into effect. GPAI models (including models that support most generative AI, like ChatGPT), now face certain obligations in the EU, including requirements around transparency, copyright and systemic risk. However, the EU AI Act is a framework: it defines obligations but leaves technical details to harmonized standards and codes of practice. While this approach sets certain expectations and allows the EU AI Act to remain technology-neutral, it also leaves questions about how businesses substantially comply with the EU AI Act. To bridge this gap, a multi-stakeholder group drafted the General-Purpose AI Code of Practice (GPAI Code). On August 1, 2025, the European Commission issued a formal opinion confirming the GPAI Code is an “adequate tool” to help demonstrate compliance with the EU AI Act. Why is the Code significant? This opinion signals that organizations who adopt the GPAI Code may be able to demonstrate good-faith efforts to comply with the relevant provisions of the EU AI Act –  according to the Commission’s website: “The Code of Practice helps industry comply with the AI Act legal obligations…of general-purpose AI models.” In its opinion, the Commission notes that the Code provides actionable commitments and reporting mechanisms, especially for high-risk models. Additionally, the Commission emphasized that the Code provides a practical framework to demonstrate regulatory compliance. Following this endorsement, providers of GPAI models can voluntarily sign the Code, which “will reduce their administrative burden and give them more legal certainty than if they proved compliance through other methods.” Still, signatories should be aware that the Code explicitly states that adherence to the Code does not necessarily constitute evidence of compliance with the EU AI Act.

What is a General-Purpose AI Model?

A GPAI model is a component of an AI system with a wide range of possible uses, whether intentional or unintentional. It is important to note that these models are not systems in themselves but are part of AI systems. Additional elements, like user interfaces, are necessary to make these models fully operational systems. Under Article 3(63) of the EU AI Act, a GPAI model includes those trained on a “large amount of data using self-supervision at scale.”  They can be applied across sectors or tasks, usually without substantial modification, meaning GPAI models “can be integrated into a variety of downstream systems or applications.” Recital 98 of the EU AI Act states that the generality of the model can also be determined by the number of parameters, and “models with at least a billion parameters…should be considered to display significant generality and to competently perform a wide range of distinctive tasks.” GPAI models are sometimes called “foundation” or “frontier” models, and while they may include large language models (LLMs), they can also process audio, physical, textual or visual data, powering systems like DALL-E, GPT-4, Gemini, LaMDA, SEER, ALIGN, and more.

How are general-purpose AI models regulated?

Under the EU AI Act, the chapter on GPAI both addresses generative AI and outlines some of the most stringent requirements under the Act. However, all requirements for GPAI under the EU AI Act are directed to providers as opposed to deployers. Providers of GPAI models have a range of obligations under the EU AI act, both directly to supervising authorities and onward to AI providers who integrate the GPAI models into their systems. Obligations of Providers of GPAI Models If a provider places a GPAI model on the EU market, or integrates such a model into its own AI system on the EU market, it must:
  • Prepare and maintain technical documentation for regulators. This should include at least a general description of the GPAI model, including the tasks it’s designed to perform and the types of systems in which it can be integrated; acceptable use policies; and information on training process.
  • Prepare and maintain documentation for downstream providers. This should include information that allows the downstream AI system providers to comply with their own obligations under Article 53(1)(b). Similar to the technical documentation, this includes but is not limited to a general description of the model, and a description of its elements and development process.
  • Prepare an EU copyright policy. This policy should establish a means to comply with EU regulations on copyright and related rights.
  • Prepare and publish a summary of training content. Using the template provided by the AI Office, providers of GPAI must share a comprehensive summary of AI training information. This should allow stakeholders to exercise their rights by informing them of the information used to train the GPAI model.
  • Cooperate with relevant authorities and appoint an authorized representative. Providers must also cooperate with relevant authorities, and if they are established outside the EU, appoint an authorized representative located in the EU.
It is notable that under Recital 85, the EU AI Act states that GPAI systems “may be used as high-risk systems by themselves or be components of other high-risk systems.” Therefore, the providers of GPAI systems must work closely with providers of high-risk AI systems to ensure compliance with any requirements of high-risk systems under the Act. Obligations of Providers of GPAI Models with Systemic Risk What does “systemic risk” mean? GPAI models with systemic risk include models that reasonably pose foreseeable negative effects relating to major accidents, disruption of critical sectors, serious consequences to public health and safety, public and economic security, democratic processes, and the dissemination of false or discriminatory content, or other similar effect. Under Article 51(1) of the EU AI Act, a GPAI model will be classified as having systemic risk if:
  • It has high impact capabilities, or
  • It is designated by the Commission to have high impact capabilities based on the criteria in Annex XIII (i.e., the number of parameters in the model, the size of the data set, the amount of computation used to train the model, etc.).
What are the additional obligations for these models? In addition to the requirements for all GPAI models, those with systemic risk have additional obligations related to:
  • Model evaluation, assessment, and mitigation of systemic risks;
  • Incident management and reporting; and
  • Cybersecurity protections and technical documentation.
Because there are differences in the obligations between GPAI systems generally and GPAI systems with systemic risk, this classification procedure should be noted by providers of GPAI systems; it is essential to understand where each GPAI model falls, and what requirements the model has under the EU AI Act. According to Article 52(6), a list of GPAI models with systemic risk will be published and updated by the European Commission, but it has not been published at the time of writing.

What is the General-Purpose AI Code of Practice?

While not legally binding, providers of GPAI models can use the Code of Practice to demonstrate compliance with their obligations under the EU AI Act. The Code consists of three chapters on 1) transparency, 2) copyright, and 3) safety and security. The first two chapters apply to all providers of general-purpose AI models, providing a way to demonstrate compliance with obligations under Article 53 of the AI Act. The final chapter applies only to general-purpose AI models with systemic risk under Article 55 of the AI Act. Chapter 1: Transparency Among other things, this chapter requires signatories to create and maintain documentation for all GPAI models distributed within the EU for up to ten years. There are exceptions for models that are free, open-source, and do not pose systemic risk. When completing this documentation, signatories must use a standard Model Documentation Form, which includes information on licensing, technical specifications, training data, and other parameters of the GPAI model. The Code encourages publication of this information to promote transparency. Chapter 2: Copyright This chapter requires signatories to create and maintain a copyright policy that complies with the EU’s legal standards. This includes, but is not limited to, ensuring that data collected by web crawling is lawfully accessible, and certain websites flagged for copyright infringement are avoided. Importantly, signatories must designate a contact for copyright holders to submit complaints, along with a process for handling those complaints. Chapter 3: Safety & Security (GPAI with systemic risk only) One of the main elements of this chapter is the requirement for signatories to develop a state-of-the-art Safety and Security Framework before releasing any GPAI model categorized as posing a systemic risk. Additionally, systemic risks should be identified and inventoried, and before progressing with development or deployment, the signatories should weigh the relative risks and determine if they are acceptable, among other requirements.

What’s next?

The Code will be monitored and reviewed at regular intervals by the AI Office, and may be updated in response to emerging risks, technological developments, or incidents involving general-purpose AI models.
1 2 3 … 5