AI Price Fixing and Discrimination with Recommendations for Retailers

AI & Algorithms, Artificial Intelligence , , , , , , , , ,

Next year will likely bring more lawmaking surrounding artificial intelligence algorithms used to set consumer pricing. Two themes are emerging: price fixing and price discrimination.

AI price fixing uses algorithms to coordinate pricing, which is raising increased scrutiny in the U.S. and Europe. Companies provide competitively sensitive data to third-party providers that analyze it to recommend prices for participating companies, leading to artificially inflated prices and reducing competition.

After over a year of litigation, the Department of Justice (DOJ) recently filed a proposed settlement with a revenue management software company, related to its algorithmic rental pricing software. State AGs continue to litigate against participating companies, and lawmakers have introduced bills to ban algorithmic price fixing. California has passed laws to restrict the use of common pricing algorithms, and regulators in the EU and UK are also focusing on algorithmic collusion. 

Regulation challenges include the argument that antitrust laws may have loopholes that make AI-driven pricing hard to prosecute; distinguishing collusion from companies’ legitimate responses to market conditions; and AI complexities creating difficulty proving violations.

AI price discrimination uses algorithms to set personalized prices for products or services based on the individual consumer’s behavior, demographics or perceived willingness to pay. The process is often hidden, making it hard to know who pays what. That raises fairness concerns (algorithms might exploit vulnerable groups and charge them more), prompting legal scrutiny and proposed regulations. 

The FTC is investigating AI’s role in price discrimination, while recently passed regulations and laws in California aim to ban using personal data for discriminatory pricing or make it easier to sue over algorithmic pricing. New York has enacted several price discrimination laws requiring algorithmic pricing disclosure and prohibiting gender-based pricing. 

Recommendations for retailers

While it is perfectly legal to test prices and adjust them to market conditions, retailers should be careful when reviewing and adopting AI pricing tools. The recent proposed settlement from the DOJ highlights some best practices for using these tools: 

  • Confirm that the AI tools train on publicly available pricing data or aggregated data, not the non-public and sensitive pricing strategies of competitors. 
  • Ensure that the AI tools allow prices to be lowered as well as increased within user-defined price ranges. 
  • Keep humans in the loop and check that users can reject or override price suggestions. 

In addition, businesses may consider the following best practices to lessen the risks of allegations of unlawful AI price discrimination:

  • The AI tools should not adjust pricing dynamically based on consumer profiles or other sensitive personal information (e.g., veteran status, religious beliefs, health condition). 
  • If the AI tool adjusts pricing based on location, retailers should confirm that this reflects legitimate market conditions and business concerns, rather than historic bias. 
  • Confirm that the AI tool uses de-identified and/or aggregated data, both to train its models and during deployment, so information is not linkable to an individual. 

Shadow AI: How Can Companies Protect Against Unknown Uses?

AI & Algorithms, Artificial Intelligence , , , , , , , , , , ,

Employees don’t always wait for their employers to approve new technology. This could be using a personal chatbot account to summarize a document, installing an AI browser extension, uploading information into an AI analysis tool or using an AI feature built into software without their employer knowing about it. This practice is often referred to as “shadow AI.”

Shadow AI is typically not malicious. Often, it’s the result of employees wanting to work more efficiently, or as a result of unclear AI use policies. However, shadow AI can create security and confidentiality issues. If employees send personal or company information into an AI system that has not been reviewed, the business may not have an accurate picture of where its information is going, how it’s being used, or for how long it is being stored. 

What does shadow AI look like?

Shadow AI could include any number of unapproved AI tools used for work purposes. For example, an employee may paste customer information into an LLM to create a summary, upload internal presentation information for editing or use an AI tool to analyze a spreadsheet. 

The problem is that using shadow AI can result in company information being sent to a third-party provider that the original company may not know about. By hiding in the “shadows,” employees who use AI tools in this way can create governance gaps. Without proper review of the AI tools being used by its employees, a company may not know what information is being provided, how long it will be retained for, who can access it or whether the vendor can use it for other purposes, like training its own systems. 

How can companies address employee AI usage?

Company AI policies can bring clarity to issues surrounding shadow AI. 

By addressing what kinds of technologies may be used, these policies can explain which AI tools are approved, what information employees may enter into those tools, and when a new AI tool or use requires review. In general, companies drafting these policies may want to specifically and clearly state what information may and may not be used. For example, a general policy telling employees to use AI responsibly may not provide enough guidance when someone is deciding whether to upload a confidential document or customer data into a new tool. 

These policies should also be communicated clearly. Without understanding the policies that a company has in place, employees may inadvertently engage in shadow AI use. With clarity on approved tools, uses, and inputs, an effective employee AI use policy could lower these risks. Within this policy, a company may may consider creating a process to request new AI tools. This way, relevant business teams can review any AI tools and uses before company information is provided.  

What should businesses take away?

Businesses do not necessarily need to prohibit AI use. However, companies may consider reviewing which tools employees are using and what information is being provided to them. Some steps companies may consider in this review process include, but are not limited to: 

  • Identifying AI tools employees are actually using, including personal accounts, browser extensions and AI features within existing software.
  • Explaining which tools are approved, restricted or prohibited and what information employees may enter.
  • Considering privacy, confidentiality, retention, deletion and AI-training terms with vendors.
  • Determining whether AI uses involve processing that requires a CCPA risk assessment or updates to privacy documentation.
  • Establishing rules preventing employees from providing sensitive company information to unapproved AI tools.
  • Giving employees a clear way to request new AI tools before using them for company work.

While AI in the workplace may boost efficiency, it may also create risk if the company is not aware of it. By providing employees with guidance on AI, businesses may be able to reduce the risks of shadow AI. 

AI Notetakers: Key Takeaways for Recording Calls

AI & Algorithms , , , , , , , , , , ,

Use of AI notetakers is quickly becoming routine. These tools can automatically join video calls, listen to conversations, generate transcripts, create summaries, and identify key points in a meeting. 

While these tools have an argument for efficiency, they also create legal, privacy, and confidentiality risks. 

This issue is a lot more complicated than simply asking participants of a meeting for their consent to be “recorded.”  This is because recording, transcription, and AI processing are separate activities – and each may have its own notice and consent requirements.

As a result, businesses using these tools need to understand both what is happening during the meeting and what happens to the information after the meeting has ended.

What are AI notetakers?

An AI notetaker is a tool that captures meeting content and uses artificial intelligence to create transcriptions, summaries, action items, and log other important meeting records. Most operate as a bot that joins a video conference meeting as an additional participant. 

Many AI notetakers process information from meetings through cloud-based services rather than keeping the conversation only on an employee’s local computer. Depending on the provider, the service may receive audio transcripts and other meeting information and then use an AI system to analyze and summarize the dialogue. This means meeting content may be transferred outside of the company’s own system and processed or stored by a third-party provider, which may have privacy and data-sharing implications. 

Why do recording and consent laws matter?

Recording laws differ across the United States. Some states generally follow a one-party consent approach, meaning the consent of one participant is sufficient to record a conversation.  However, other states follow an all-party consent approach, sometimes referred to as “two-party consent” which generally requires the consent of everyone involved in a conversation for recording. The specific details and exceptions vary by jurisdiction. 

What actually counts as consent?

Sometimes consent can be more complicated than just simply displaying a recording symbol. Video-conference platforms may use different methods to alert participants when recording a meeting begins: a pop-up requiring participants to acknowledge the recording, an audible announcement, a banner or an icon that’s displayed during the meeting, some hosts may also require verbal consent.

But notice and consent are not always the same thing and what constitutes legally sufficient consent can depend on the applicable law and the circumstances at hand.

What happens to meeting information after the call?

Once an AI notetaker has captured a meeting, businesses should understand what rights the provider has over that information. Vendor terms can address data ownership, licensing, data retention, and whether information may be used to develop or improve the provider’s service. 

This becomes especially important as ordinary workplace conversations frequently include information that employees may not intentionally send to a third party: customer information, personnel issues, financial projections, internal strategy, product development and many other confidential materials may all be discussed while the AI notetaker captures the conversation. 

The transcription or summary can also create additional security concerns once the meeting ends. For example, automatically generated notes may be distributed via email, downloaded, forwarded and stored in employee accounts long after an original conversation has occurred. Meeting summaries create additional opportunities for sensitive information to spread or remain stored indefinitely. 

Businesses should therefore review not only what the tool captures but also who receives the resulting transcript, where it is stored, how long it is retained and who has the ability to delete it.

What about attorney-client privilege?

Adding an AI notetaker to a legal discussion can create questions about whether confidentiality has been maintained, and depending on the circumstances, whether privilege could be challenged or waived. You can read more updates from Federal District Courts on the issue here. 

This does not mean that all use of technology during a legal meeting automatically destroys attorney-client privilege. Whether privilege is affected may depend on the circumstances such as how the AI provider handles information and why the tool is being used. Therefore, businesses should be more cautious about allowing AI notetakers into meetings that involve legal advice or other professionally protected information. 

The same concern applies to trade secrets and other confidential business information. Meetings involving sensitive product plans or internal strategies and other proprietary information may not be appropriate for AI transcription unless the tool and its data practices have been reviewed carefully. 

What should businesses take away?

AI notetakers can be useful workplace tools, but businesses should have clear policies in place before employees begin using them regularly. Below are some high-level tips that businesses may want to consider when onboarding a new AI notetaker: 

  • Approve specific tools: Employees should know which AI notetakers are permitted and how those tools record, transcribe, store and process meeting information.
  • Create clear notice and consent procedures: Businesses may consider the laws that may apply to meeting participants and make sure notices accurately reflect how AI is being used.
  • Limit use in sensitive meetings: Legal, HR, disciplinary investigation and other confidential discussions may require additional approval or no AI notetaker at all.
  • Review vendor data practices: Companies should understand how meeting information is used, stored, retained and deleted, including whether it may be used to train or improve AI systems.
  • Set rules for transcripts and summaries: Businesses may want to determine who can access or share AI-generated notes, how long they are kept and whether they are treated as official company records.

Healthcare AI data breaches: Why companies need to protect more than patient records

AI & Algorithms, Artificial Intelligence , , , , , , , , , , ,

Healthcare and life science companies are not only protecting traditional patient records – things like names, diagnoses, treatment notes, lab results, and insurance information. They may also be protecting clinical trial data, research data, vendor systems, proprietary models, and other information used to support patients and drug development. When these companies and systems are involved in data breaches, the legal risks can become broader than a standard data breach. 

A useful example is the recent Novo Nordisk incident. Novo Nordisk is the pharmaceutical company that makes drugs such as Ozempic and Wegovy and recently disclosed a cyber incident involving patient data from clinical trials. It has also been reported that a hacking group claimed to have stolen over a terabyte of data and attempted to extort the company, though Novo Nordisk has not confirmed the entire scope of those claims. 

The Novo Nordisk incident shows that healthcare breaches can involve more than names and medical records. Companies also need to consider whether a cyber incident involves clinical trial information, healthcare provider data, research files, AI model information, or other confidential business materials. When this type of data is involved, companies may need to take precautions to protect the data, document their response, and comply with applicable privacy and cybersecurity obligations.

Recent health-related breaches also highlight how these incidents create major legal and business consequences. The Change Healthcare cyberattack that occurred in 2024 shows the scale of healthcare cybersecurity risks with reports that 197.2 million individuals were impacted by this breach. For scale, 197.2 million people would be more than half of Americans. The 23andMe breach also highlights the litigation risks tied to sensitive genetic information, with a bankruptcy judge approving a $46.75 million settlement for victims in the breach. While the facts of these examples differ, they all highlight why companies handling health-related data need to treat cybersecurity, AI governance and breach response as connected compliance and a priority in business practices. 

Why are AI-related healthcare breaches unique?

A standard healthcare breach analysis often focuses on whether names, medical records, insurance information, or other identifiable patient information was exposed. These concerns are highly important and sensitive. But as artificial intelligence becomes more integrated into healthcare, it creates additional risks because AI tools rely on large volumes of sensitive data that may also be tied to valuable research and business information.

For example, an AI model used in drug development may involve clinical trial data, molecular research, imaging data, prompts, outputs and other model training information. If these systems are accessed by an unauthorized actor, the incident may raise privacy concerns and may also create IP and competitive risks. 

This means healthcare AI security should not only focus on protecting patient records. Companies should also consider whether their AI models, training datasets, research systems, and vendor environments are adequately secured.

Why does de-identification matter?

De-identification can be key when healthcare data is used with AI. In simple terms, de-identification means removing information that could identify a person up to a certain legal standard. This matters because companies may want to use patient data or clinical data to train, test or improve their models. 

If health data is properly de-identified before being used with AI, the company may reduce privacy risks. However, if the data is not properly de-identified, several questions remain. The company may need to ask whether its privacy notices clearly explained that health data could be used for AI training or analysis. State privacy laws may also require clear disclosures about how personal information is collected and used. For example, California’s CCPA requires covered businesses to provide notice about the categories of personal information collected and the purposes for which that information is collected or used. This is especially important when health-related data is later used in a way that patients or consumers may not have expected. 

The company may also need to consider whether protected health information was shared with an outside AI vendor, whether a business associate agreement was required, and whether patient authorization was needed.

What happens if PHI may have been compromised? 

Breach response is also an important issue. If a healthcare company experiences a ransomware attack or an unauthorized access incident, they may need to assess whether protected health information (PHI) was compromised, This may include reviewing what information was involved, whether it could identify an individual, who accessed it, whether it was viewed or acquired and if the risk was mitigated or not.

The Novo Nordisk incident highlights why this analysis matters. Novo Nordisk stated that the clinical trial information involved was not directly linked to patient names or other direct identifiers. However, the incident still raised questions about patient-related data, de-identification, and whether any protected health information may have been compromised. This is why companies should be able to document how health-related data was stored, protected, and separated from information that could identify individuals.

What should companies take away? 

For healthcare and life science-related companies, the key takeaway is that AI governance should include privacy and cybersecurity from the start. Before using health-related data with AI, companies should ask:

  • What data goes into the AI system?
  • Is the data identifiable or properly de-identified?
  • Who can access the data, prompts, outputs, or model?
  • Are outside vendors involved?
  • Can the vendor use the data to train or improve its own models?
  • What security controls apply if the system is breached?

Companies should also treat AI models, training datasets, prompts, outputs, and research tools as sensitive assets as well since they can carry PHI and may lead to inadvertent disclosures. Vendor contracts should address confidentiality, data retention, security controls and model training. 

AI may help healthcare and life science companies innovate and operate faster, but innovation cannot replace privacy and regulatory accountability. Companies using AI with health-related data need to confirm de-identification practices, strengthen vendor contracts, and properly prepare for responses before incidents occur.

0
A minimalistic picture of a human brain being digitized into technological lines that also look like a human brain.

What is an AI risk assessment? And how is one conducted?

AI & Algorithms, EU, United States , , , , ,
AI is ubiquitous, and organizations are adopting AI solutions at a rapid pace. Findings from the first nationally representative survey in the US on generative AI use suggest that “U.S. adoption of generative AI has been faster than adoption of the personal computer and the internet.” With this proliferation comes legal risk, and AI risk assessments are essential tools for organizations to understand the risks and the legal requirements that come with AI adoption. Like privacy risk assessments, AI risk assessments aim to identify, evaluate and mitigate potential risks associated with systems or processes. Because AI can introduce unique challenges—including algorithmic bias, transparency issues, and accountability concerns – the assessment should be tailored to the unique elements of the AI system being implemented. Below is a general overview on how to conduct an AI risk assessment. While the scope and specific frameworks of each risk assessment will vary, it is essential to maintain a structured, systematic approach to ensure the system is being evaluated thoroughly.

Determine Which Laws Apply

To begin any risk assessment, the first step is to determine which laws, regulations, and standards apply. For AI systems, these laws may include, but are not limited to, AI-specific laws, sector-specific laws, and state privacy laws. How to identify applicable laws Begin by identifying the jurisdictions where the AI system will be deployed, accessed, or will otherwise impact individuals. Then, assess which sectors the AI system will be operating in (e.g., finance, employment, healthcare) and whether any AI-specific or general laws apply to the system or its use. Applicable AI-specific laws may include, but are not limited to:
  • California Training Data Transparency Act. In effect on January 1, 2026, this law requires documentation about any generative AI system available to consumers in California. This documentation must be posted on the developer’s website and includes, among other things, a summary of the datasets used in the development of the system, the source of the datasets, how these datasets further the AI system’s intended purpose, and a description of the types of data points within the data sets.
  • California AI Transparency Act. In effect on January 1, 2026, this law covers providers with generative AI systems that are accessible in California and have over one million monthly users. Under this law, covered entities are required to make an AI-detection tool at no cost to users of the AI system. The law also requires the covered entity must provide an optional and mandatory embedded disclosure for all outputs, among other things.
  • Colorado Artificial Intelligence Act. Enacted in 2024, this Act includes parameters around “high-risk” AI systems—those which make, or are a substantial factor in making, consequential decisions. This Act is designed to protect against algorithmic discrimination and imposes obligations relating to transparency and disclosures, risk analysis and mitigation, and impact assessments for both developers and deployers.
  • Utah Artificial Intelligence Policy Act. Enacted in early 2024, this Act requires providers of generative AI systems to ensure that the system discloses whether the user is talking with a generative AI system. In some instances, this disclosure must be made at the beginning of the interaction with the user.
  • Illinois Human Rights Act. In effect on January 1, 2026, amendments to the Illinois Human Rights Act will address the use of AI systems, specifically in employment contexts. The Act currently prohibits discrimination for protected classes in Illinois, and the amendments to the Act will expand its scope to include employment discrimination resulting from the use of AI. For more about this Act, visit our previous article here.
  • EU AI Act. The EU AI Act entered into force on August 1, 2024, but its provisions are phased into effect over time. Under this Act, AI systems are categorized into one of three risk levels: unacceptable, high and low. While AI systems with unacceptable risk are prohibited under this Act, those models classified as high or low risk are subject to additional transparency, risk, and safety obligations.
Additional privacy laws & standards Data protection and privacy laws and regulations, like the California Consumer Privacy Act (CCPA) or General Data Protection Regulation (GDPR), should be taken into consideration, because AI systems frequently process personal or sensitive data. For an overview of the current US state comprehensive privacy laws, visit our previous article here. In addition to identifying applicable laws, it is also helpful to understand emerging standards and ethical guidelines for responsible AI, such as those from ISO, IEEE, or NIST. Although not legally binding, these frameworks can provide best practices to align the AI system or processes with industry standards.

Choose Your Framework

After understanding the legal requirements that apply to your AI system, your organization should select a risk assessment framework that aligns with the type of AI system being implemented and your organization’s goals. Because AI is still relatively new, frameworks are still in development. However, there are a handful of frameworks currently available, which include, but are not limited to:
  1. NIST AI Risk Management Framework. This framework – and its accompanying playbook – was developed by the National Institute of Standards and Technology (NIST) and is designed to “increase the trustworthiness of AI systems, and to help foster the responsible design, development, deployment, and use of AI systems over time.” Because the NIST framework addresses risks to organizations, people, and society in general, it offers a flexible approach that can be used across various industries.
  2. ISO/IEC 42001:2023. This framework focuses on AI management system standards across all types of AI applications and contexts, and offers organizations guidance on creating, deploying, and monitoring AI systems. This standard is particularly useful for organizations seeking international recognition for their AI governance practices, and covers areas including responsible AI, reputation management and user trust, managing AI-specific risks, and innovating within the ISO/IEC framework.
  3. CNIL Self-Assessment Guide for Artificial Intelligence (AI) Systems. This framework offers organizations an analysis grid to assess the maturity of their AI systems in light of the GDPR. Published by the CNIL, the French data protection authority, this framework outlines general aspects of data protection law as well as specific elements that should be more thoroughly reviewed in the context of AI. Because this assessment focuses on the GDPR, it is best for organizations seeking compliance with European data protection and AI laws.
Regardless of the framework, any organization implementing an AI system or process should conduct an assessment using a structured approach. Not only will this approach help provide a more comprehensive assessment, but it will enable greater consistency with each iteration of the assessment, allowing the organization to more effectively compare risks and manage accountability.

Identify AI Stakeholders

Identifying relevant stakeholders in the organization’s AI system or process ensures that all relevant perspectives and concerns are considered. In turn, this helps provide a more thorough, well-rounded assessment. Who are stakeholders? A stakeholder is anyone who is affected by, has an interest in, or has control over an AI system. Key groups often include developers, engineers, product owners or managers, compliance teams, organizational leadership teams, and users. How to identify stakeholders To identify relevant stakeholders for an AI system or process, start by analyzing the AI system’s lifecycle. Consider who is involved in each phase, from design and development to deployment. For example, developers and engineers play vital roles in understanding technical implications throughout the lifecycle, while leadership teams can help guide the intended purpose and evolution of the system. Users should also be considered, as they can provide use-case examples after deployment and feedback on their interactions with the system. Additionally, it is essential to include a diverse range of stakeholders. Balancing differing priorities, such as ensuring fairness, reducing bias, and operational efficiency, will help address potential risks more comprehensively. A range of perspectives can help uncover blind spots, build trust, and ensure that the AI system aligns with legal standards and user expectations.

Map Your System

Mapping your AI system will help provide a clear understanding of how the AI system operates, interacts with, and impacts its environment. By accounting for system components, data flows, and dependencies, an organization can better pinpoint potential risks of bias, inaccuracies, or other issues at each stage of the AI system’s lifecycle. Outline the system  Start by outlining the AI system’s purpose and scope. Define each input, output, and process, and include algorithms, data sources, and models that the AI system relies on. Integrations with other platforms should also be considered and documented. During this process, the organization should refer back to the roles of all stakeholders to ensure each is accounted for. Define the data journey After the system’s structure is defined, trace the data journey from collection, to decision-making, to output. During this process, it is important to highlight any personal data and sensitive data. Processing of this information can lead to issues where errors, biases, or other vulnerabilities may emerge, and may implicate specific AI or other data privacy laws. Identify monitoring methods Finally, map feedback loops and other mechanisms for monitoring the system after deployment. AI systems evolve through updates and learning processes, and it is essential to understand how these changes can expose additional risks. By creating a detailed data map, the organization can establish a comprehensive foundation to carry out the remainder of the risk assessment in a thorough manner.

Set Quality and Accuracy Metrics

For any assessment, metrics must be compared to ensure the system operates as intended, delivers meaningful results, and meets stakeholder expectations. To determine these metrics, the organization should first define the goals of the AI system. Key questions to ask may include:
  • What specific problem is the AI system designed to solve?
  • What value does the AI system contribute?
  • What decisions or actions will the AI influence or automate?
  • What are the users’ needs and expectations from the system?
  • Are there specific fairness, inclusivity, or accessibility goals?
  • How should the system evolve with time or use?
The organization’s metrics should be tailored to address the answers to these and related questions. Next, consider the datasets used to train and evaluate the system. Ensuring that data is complete, consistent, and representative will help ensure the AI system reflects real-world usage. Therefore, datasets should also have metrics to ensure reliable data is being used to assess the system. The reliability of the system should also be defined. Consider metrics like error rates, false positives, and false negatives to gain insight on how AI handles instances like edge cases or unexpected inputs. Finally, user metrics can also be insightful into how well the AI system is performing. These could include satisfaction scores, task success rates, or other metrics to determine how well the AI meets user expectations. After each metric is defined, establish a threshold or benchmark for each. Continuous monitoring and regular evaluation against these standards will help ensure the AI system maintains reliability over time. For dynamic AI systems – which continuously evolve with new data or updates – assessing quality and accuracy is an ongoing process.

Assess Privacy and Cybersecurity

Privacy and cybersecurity are both deeply interconnected components of AI risk assessments. Taking steps to assess these elements helps ensure user safety – particularly when the system collects or otherwise processes personal or sensitive information. Increased Risk of Vulnerability in AI Systems AI systems can handle large amounts of data, making them targets for malicious actors and raising significant threats for privacy concerns. In an evaluation of the cyber security risks to AI by the UK’s Department for Science, Innovation and Technology, vulnerabilities from malicious actors were identified at each stage of an AI system’s lifecycle. Without robust security measures, these vulnerabilities can be more easily exploited.  However, by mitigating these vulnerabilities, organizations can enhance their security measures to better protect against a range of cyber threats. Data Protection Impact Assessments (DPIAs) Most U.S. states with comprehensive data privacy laws require organizations to conduct a data protection impact assessment or data privacy impact assessment (DPIA) for high-risk data processing activities. DPIAs are systematic evaluations that require organizations to adopt privacy-forward practices and require close interaction between privacy and cybersecurity functions. DPIAs help organizations evaluate how personal data is collected, stored, processed and shared. In the context of AI, DPIAs are essential for identifying privacy risks in the training, deployment, and maintenance phases of the AI system. In many instances, DPIAs are required in Europe and the U.S. in the case of:
  • Deployment of high-risk AI systems, as defined under the EU AI Act;
  • Evaluation of personal aspects relating to individuals based on automated processing. This includes profiling, or decisions made on an evaluation that produces legal effects, or similar impacts on a natural person;
  • Systematic monitoring of a publicly accessible area on a large scale;
  • Processing personal data that constitutes sensitive personal data;
  • Processing personal data where it could present a heightened risk of consumer harm, such as unfair or deceptive treatment; financial, physical or reputational injury to consumers; or physical or other intrusion on solitude or private affairs;
  • Processing personal data for purposes of targeted advertising; or
  • Sales of personal data.
Like frameworks for the overarching AI assessment, there are also frameworks to help conduct a DPIA, including the:
  1. NIST Risk Management Framework (RMF). This framework is designed to provide a structured yet flexible approach for managing security and privacy risks, including conducting a DPIA. Through this framework, an organization can link risk management processes at the system level and organizational level. The NIST Cybersecurity Framework can be aligned with the NIST RMF and can be implemented through NIST risk management processes.
  2. ISO/IEC 29135:2023. This document provides guidelines for the process of a privacy impact assessment, and the structure and content of a DPIA report. It is applicable to all types of organizations, regardless of size, including public and private companies, government entities, and not-for-profit organizations.
  3. ICO Sample DPIA Template. This template from the UK’s Information Commissioner’s Office provides an example of how an organization can record the DPIA process and outcome. This template should be read alongside the guidance for an acceptable DPIA set out in the European Guidelines for DPIAs.
The frameworks to conduct a DPIA are similar those used to conduct an overarching AI risk assessment. While both identify and mitigate potential risks, a DPIA will focus on personal data privacy concerns arising from or within the AI system. While NIST points out that “there is no foolproof way” to protect AI from attacks, using a DPIA to understand privacy and cybersecurity risks can help reduce damage to or by an AI system.

Review Bias

After the groundwork of the assessment has been completed, it is essential to understand the results of the assessment – specifically when it comes to bias and discrimination. Bias in an AI system occurs when a model produces unfair or skewed outcomes due to issues in the data, algorithms, or deployment of the system. These skewed outcomes pose significant ethical, legal, and regulatory risks, making a comprehensive review of bias an essential part of an AI risk assessment. Bias from Training Data & Algorithms To review bias, the organization should start by examining the data used to train the AI system. The training data helps AI systems learn to make decisions and should be carefully reviewed. This data should be representative of the context in which the AI system will operate, and issues with this dataset – such as under or overrepresentation of certain groups – can lead to discriminatory outcomes. In addition to issues with training data, the algorithms used can also introduce or amplify bias. According to a report on managing bias in AI, NIST points out that these situations “often arise when algorithms are trained on one type of data and cannot extrapolate beyond those data.” This could be due to an issue with the data itself or because of the mathematical representations of the data in the algorithms. Bias from Deployment Context After reviewing the technical elements of the AI system, bias review should also include deployment contexts. This is because even seemingly neutral or well-trained models can produce biased results if deployed in contexts the AI system was not trained for. Differences in user behavior may create unintended outcomes. To mitigate these risks, organizations should ensure datasets are diverse, representative, and regularly audited for imbalances or stereotypes. Additionally, organizations should conduct context-specific testing before deployment and implement feedback mechanisms to monitor and address bias over time.

Manage Risks

Effective risk management is the final step of conducting an AI risk assessment. Per NIST, “[a]ddressing, documenting, and managing AI risks and potential negative impacts effectively can lead to more trustworthy AI systems.” This process should be done through a proactive, iterative, and comprehensive approach to identify and assess risks – especially for systems that evolve over time. Using the steps above, organizations can conduct regular performance reviews and implement feedback loops to better pinpoint potential risks as well as their severity and likelihood of harm. After identifying risks, organizations should clearly document and communicate risk management processes to stakeholders, ensuring that system limitations and safeguards are understood. Additionally, businesses should take a collaborative approach with stakeholders to mitigate risks and help align practices with best-in-class recommendations. Key practices for managing risk include adopting policies for system oversight and adopting regular assessments to ensure ongoing compliance with laws and regulations. AI systems will never be risk-free. However, businesses can effectively use AI risk assessments to safeguard against potential harms. Through a systematic evaluation of the AI system, organizations can create more trustworthy and reliable AI systems, while ensuring compliance and protecting user privacy.
1 2 3