Discover the privacy, confidentiality, and compliance challenges of AI in clinical trials, from HIPAA and FDA rules to AI vendor risks.

AI Use in Clinical Trials: Privacy, Confidentiality, and Compliance Risks

AI & Algorithms , , ,

Artificial Intelligence (AI) is becoming increasingly common in health care clinical trial operations. Sponsors, contract research organizations, research sites and vendors may use AI for many tasks such as patient recruitment, eligibility screening, and informed consent support among other things. While this can improve efficiency, these tools may also create privacy, confidentiality and compliance risks in an already highly regulated environment. 

Clinical trials often involve sensitive patient information, confidential study data, and strict documentation obligations. This means that AI use in clinical research cannot be treated like traditional workplace software. Before deploying an AI tool, organizations should understand what data the tool will access, where that data will go, whether the vendor can use it to train models, and whether the output can be explained, reviewed, and preserved.

How is AI being used in clinical trials? 

AI tools may be used throughout the clinical trial process. For example, AI can be used to help identify potential participants, screen patient eligibility, summarize medical records, draft study documents, monitor data and flag adverse events. Generative AI may also be used for other tasks in the clinical trial process such as summarizing meeting notes, clinical records or other trial related documents.

These uses can be helpful, but they also raise basic compliance questions. Before using an AI tool, organizations need to ask whether the tool fits within the trial’s legal and contractual framework. Clinical trial agreements and site agreements often control who may access trial data and how that data can be used, but those contracts may have been drafted without considering the use of AI in the trial. If an AI vendor is introduced without updating those agreements, contracts that are not AI-specific may create gaps in data access, confidentiality and regulatory accountability.

What legal obligations matter?

AI use in clinical trials must also fit within existing privacy, research and documentation obligations. HIPAA and its related rules may apply when AI vendors receive, store and analyze PHI. This means, organizations may need privacy and security safeguards before sharing their trial data with the AI vendor.

FDA electronic records requirements should also be considered as clinical trial records have to be reliable, auditable and traceable. If an AI tool creates a summary or other output without prompt logs or documented human review, it may create problems for recordkeeping and data integrity. 

Consent for the use of AI in the clinical trial process is also a key factor, especially when AI is used in recruitment processes or eligibility screenings. In these situations, participants should receive clear information about how AI is involved and how their information may be used.

Finally, confidentiality should remain at the forefront. Clinical trials do not only involve patient data, but may also involve sensitive commercial information, such as investigational product data, interim results, safety signals, and proprietary research methods. If an AI tool takes in this information for transcription, summarization, or analysis without clear confidentiality restrictions, it may create risks for sponsor confidentiality, trade secret protection, or attorney-client privilege.

Why are AI vendors a risk?

AI vendors can be one source of risk in clinical trials because they may receive and process very sensitive trial data. Standard AI vendor terms may not be enough when clinical trials involve protected health information or FDA-regulated records. Before using a specific AI vendor, organizations should review whether the vendor can retain prompts or outputs, use trial data to train its models, allow employee access to uploaded data, or store information outside approved systems.

Vendor contracts should clearly address data retention, model training, confidentiality, audit rights, security controls and human oversight. Additionally, companies should not rely only on what the vendor promises in the contract. They should verify how the vendor actually handles trial data before using the AI tool.

What should companies take away? 

For sponsors, meaning the organizations responsible for clinical trials, the main takeaway is that AI should be reviewed before it is used with clinical trial data. The organization should review what the AI tool will be used for, whether it will access PHI or confidential study information, whether the vendor can use trial data to train its models and if AI-generated outputs can become part of the trial record.

There are AI vendors that offer privacy-protective features like PHI redaction and de-identification. However, companies should not assume these features automatically make the tools compliant. Companies still need to verify how the vendor stores, processes, deletes and protects clinical trial data before deployment. 

AI may help clinical trials become more efficient, but efficiency does not replace privacy, confidentiality, consent, or regulatory accountability. Companies should update agreements, limit data exposure, verify vendor practices, document human review, and clearly disclose AI involvement where appropriate.

AI Companion Chatbot Regulations: New State Laws Target Child Safety, Disclosures & Crisis Response

Eyes on AI Chatbots: New State Laws Target Child Safety, Disclosures and Crisis Response 

AI & Algorithms , , ,

AI companion chatbots have recently become a focus of state AI regulation. Unlike task-oriented chatbots, companion chatbots may be designed to simulate conversation, friendship, emotional support or other ongoing personal relationships. These features may create heightened legal and safety concerns, especially when users are minors, emotionally vulnerable or may mistake automated responses for human support. 

State legislatures are beginning to address some of these risks.  While some enacted laws focus more on transparency and AI disclosures, proposed bills go further by addressing youth safety, emotional dependence, crisis response, age verification, data protection and human oversight. The Future of Privacy Forum is currently tracking 98 chatbot-specific bills across 34 states and three federal proposals, showing how quickly and unevenly chatbot regulation is developing.

Why are companion chatbots receiving regulatory attention?

Companion chatbots raise unique risks compared to traditional automated tools as they are often designed to boost user interaction, and keep users engaged over longer periods of time. In some cases, they may remember previous conversations, provide emotionally validating responses and create the impression of a meaningful relationship. These features may make the product engaging but also raise concerns about manipulation, emotional dependency and the collection of sensitive and personal data.

These risks are especially significant when the user is a minor. Because minors may have a harder time recognizing the limits of AI systems or identifying persuasive design techniques, they may be more vulnerable to mistaking automated responses for genuine human support. Regulators are also paying closer attention to situations in which a user discloses mental health concerns or expresses self-harm thoughts during a chatbot interaction. 

Across these proposals, there are several recurring regulatory themes: transparency, age verification, content safety, harm prevention, data protection, liability and enforcement practices. These themes suggest that lawmakers are not only concerned with whether users know they are interacting with AI but also with how chatbot systems are designed, how they collect data, and how they respond if a user may be at risk. 

What laws have already been passed?

Connecticut recently enacted one of the broader state laws addressing youth online safety and AI-related protections. On June 2, 2026, Governor Lamont signed Public Act 26-15, describing it as a bipartisan law intended to protect children and adults from digital-age harms, including youth social media addiction and concerns over the growing use of AI. The law also includes chatbot-related protections, such as requiring chatbot operators to make reasonable efforts to detect suicidal ideation or indicators of self-harm expressed by users and to maintain a protocol for responding with appropriate resources. Connecticut’s law goes into effect October 1, 2026.

California has also already taken steps to regulate companion chatbots through SB 243, which established baseline disclosure and safety requirements for companion chatbot operators. This law is in effect, with additional requirements for operators beginning July 1, 2027. 

Together, these enacted laws show that states are beginning to regulate AI systems directly, even without a comprehensive federal AI law. For businesses, this means AI compliance may increasingly depend on tracking different state requirements rather than relying on one national standard.

What pending proposals should companies watch?

Pending California bills highlight how companion chatbot regulations may become more specific. For example, SB 1119 focuses on chatbot interactions with child users, defined as consumers under 18 years of age. If enacted, it would require operators to take a more proactive approach to safety by conducting annual child safety risk assessments, creating public child safety policies, setting privacy and safety defaults for minors, providing parental controls, and conducting audits. It would also restrict certain chatbot behaviors like responses that encourage self-harm, substance use, disordered eating, or harm to others. This bill is currently active in the Assembly committee process.

California’s AB 1988, also known as the PAUSE Act, focuses more directly on crisis response. Unlike SB 1119, this bill is not limited to minors. If enacted, AB 1988 would require companion chatbot operators to identify and respond to credible crisis expressions, provide 988 Suicide and Crisis Lifeline information, pause chatbot responses after repeated crisis expressions, and require human moderator review before ending that pause. This bill is currently active in the Senate committee process.

These proposals show that chatbot regulation is moving beyond basic transparency requirements. Lawmakers are increasingly focused on how AI systems are designed, how they interact with vulnerable users, and whether companies have real safety procedures in place when a chatbot conversation becomes harmful or high risk.

What should companies take away?

For companies that are developing or deploying companion chatbots, one key takeaway is that basic AI disclosure may not be sufficient. Emerging state laws are narrowing in on how chatbots interact with minors, respond to self-harm or crisis-related statements, collect sensitive data, and whether meaningful human oversight is available. Businesses operating across multiple states should track both enacted laws and pending proposals as chatbot obligations may differ by state and are continuing to rapidly change. 

Why the EU AI Act belongs on every general counsel's radar – now Governance Intelligence

Why the EU AI Act belongs on every general counsel’s radar – now Governance Intelligence

AI & Algorithms, EU

Why should general counsel in the US care about a European law? Because the EU AI Act’s governance demands are set to reach far beyond the bloc.

The European Union’s AI Act has captured the attention of legal departments worldwide, but many US-listed companies may be making a critical mistake: treating its deferred implementation dates as a reason to postpone governance planning.

General counsel and chief legal officers should not focus on when the EU AI Act’s requirements become enforceable, but how long it will take their organizations to build the governance infrastructure needed to comply.

The answer is likely much longer than most companies expect.

Understanding the EU AI Act

The EU AI Act is the world’s first comprehensive artificial intelligence law. Like the General Data Protection Regulation (GDPR), its reach extends beyond Europe and can apply to companies outside the European Union that place AI systems on the EU market or make them available to EU users.

The law takes a risk-based approach, categorizing AI systems according to their potential impact on individuals and society. Certain AI applications are prohibited outright, while ‘high-risk’ systems used in areas such as employment, education, healthcare, financial services and critical infrastructure face extensive governance, documentation, testing and oversight requirements. The Act also imposes transparency obligations on many general-purpose AI systems, requiring organizations to disclose when users are interacting with AI-generated content or AI systems.

In-house counsel’s role in EI Act compliance

For legal departments, the significance of the Act extends beyond compliance. It establishes a governance model that increasingly treats AI systems like regulated products, requiring organizations to demonstrate that risks have been assessed, controls implemented and ongoing oversight mechanisms are in place.

Compliance with the EU AI Act cannot be achieved by the legal department alone. For example, legal departments can help oversee risk assessments and draft required disclosures. However, product and technical teams will be responsible for documenting AI training data, testing system accuracy, monitoring outputs and identifying potential performance failures. In-house counsel cannot independently validate those requirements without close collaboration across the organization.

This is particularly important for companies developing or deploying AI systems that may be classified as high risk under the EU AI Act. Regulators are expected to examine not only how a system functions, but also how it is marketed, documented and contractually restricted. As a result, legal teams will increasingly be responsible for reviewing customer agreements, product documentation and marketing materials to ensure they align with the system’s intended use.

The overlap between EU and US AI governance

The governance challenge becomes more urgent when viewed through a US lens.

Many organizations are waiting for greater clarity from European regulators before investing in governance programs. However, some US requirements are already moving forward. California’s automated decision-making technology regulations create risk assessment obligations for certain AI systems used to make significant decisions affecting consumers and workers. Organizations using AI in areas such as employment, housing, healthcare, education or financial services may already need to begin building the documentation and assessment processes that will eventually support both California and EU compliance efforts.

From a governance perspective, the overlap is significant. Rather than creating separate frameworks for each jurisdiction, legal departments should be identifying opportunities to build a common foundation that can support multiple regulatory regimes.

Transparency requirements offer another example. The EU AI Act requires disclosures for certain general-purpose AI systems, including situations where users interact with AI-generated content. Similar concepts are emerging across US states, particularly for consumer-facing AI tools and companion chatbots.

This creates a layered governance challenge. Organizations need a broad framework that addresses enterprise-wide AI risk while also accounting for state-specific requirements that may influence product design and deployment decisions.

The companies best positioned for compliance will not be those waiting for regulatory certainty. They will be the organizations already bringing legal, product, security and compliance teams together to build repeatable governance processes now.

The EU AI Act may be European legislation, but for US-listed companies, its governance implications have already arrived.