The January 2025 edition of Orange County Lawyer magazine features an article written by Metaverse Law’s Lily Li. Read “AI and Machine Learning in Drug Development and Clinical Trials” below or in Orange County Lawyer magazine.
[Originally published as a Feature Article: AI and Machine Learning in Drug Development and Clinical Trials, by Lily Li, in Orange County Lawyer Magazine, January 2025, Vol. 67 No.1, page 28.]AI and Machine Learning in Drug Development and Clinical Trialsby Lily Li
In 2013, sleep medication zolpidem (Ambien, Ambien CR, and Edluar) swept headlines. Marie Claire reported on an alarming and suspicious rise in users experiencing irrational eating, gambling, and even “sleep-driving” while in a hypnotic trance—waking with no memories of their actions.[1] In several cases, women arrested and convicted for driving under the influence contested their convictions, arguing that they were not liable for these undisclosed drug-related side effects.
At the same time, several clinical studies suggested that women metabolized zolpidem differently from men. By reviewing existing literature, Japanese researchers out of Shimane University identified 40% higher concentrations of zolpidem in women than men following use, and higher rates of visual hallucinations and sensory distortions.[2]
The FDA released a safety advisory, warning users of the risks of “next-morning impairment” for the use of Ambien and related drugs.[3] In addition, the FDA took the unusual step of recommending a 50% cut in the dosage for women. When asked about the change, an FDA director told ABCNews.com: “The changes are different in women and men . . .We don’t understand why yet, but women are more susceptible to next-morning impairment.”[4]
Yet, a decade later, the evidence supporting different zolpidem dosages for women and men is unclear.[5] In part, this is due to the lack of research surrounding sex differences in drug impact and drug treatment, as well as substantial gaps in the inclusion of women in clinical studies. From 1977 to 1993, FDA policy recommended excluding women of childbearing potential from Phase 1 and early Phase II drug trials.[6] Even after this policy was removed in 1993, industry fears remained with respect to drug interactions with pregnancy. This episode with zolpidem raised several concerns in the drug development and clinical trial process:
How do we recruit representative candidates for drug trials?
How do we ensure the quality and availability of datasets for clinical research?
How do we measure potential impacts of drug dosing on different populations?
What are the legal implications for failing to address appropriate drug doses?
AI and ML to the Rescue?Now that artificial intelligence is being used in research and development, one wonders: Can artificial intelligence (AI) and machine learning (ML) reduce bias and risks during drug development? Or will it create new legal risks due to bias, privacy intrusions, and lack of transparency?
The FDA released a discussion paper on AI, Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products, to discuss potential regulatory frameworks to address the use of AI and ML.[7] In this discussion paper, the FDA released a set of fascinating case studies into existing research and uses of AI in the clinical trial process. Several of these case studies are discussed below, as well as an analysis of their potential impact on the zolpidem example.
Recruitment. According to the FDA, “AI/ML is being used to mine vast amounts of data, such as data from clinical trial databases, trial announcements, social media, medical literature, registries, and structured and unstructured data in EHRs [electronic health records], which can be used to match individuals to trials (Harrer, 219 Shah, Antony, & Hu, 2019).” In this manner, researchers can combine huge quantities of publicly available data and individual health data from prior research to identify participants with certain medical conditions (or lack of adverse conditions) for investigational treatments. For zolpidem, the use of AI/ML may have been able to identify a much broader list of participants for initial clinical testing, making it easier to assess and identify adverse reactions.
Selection and Stratification of Trial Participants. In addition to initial recruitment, AI/ ML has the capability improve intake, selection, and classification of clinical trial participants. Based on baseline characteristics selected by the researchers, such as prior clinical data, and vitals/labs taken during intake, predictive algorithms can help identify high-risk participants.[8] These groups can then be randomized and then subject to more strict monitoring protocols. In the case of zolpidem, alcohol use is associated with sometimes severe adverse effects from the drug, and so it would be beneficial to screen out candidates with a history of alcoholism or, on the flip side, assess drug interactions for this high-risk group with additional support, monitoring, or counseling.
Dose/Dosing Regimen Optimization. AI/ML can be used to predict drug exposure for different populations based on factors such as weight, height, sex, and other characteristics that might impact drug metabolism. Based on prior drug exposure and response profiles for similar drugs and similar populations, AI/ML can help to narrow the dose/dosing regimen selected for a study. As noted by the FDA’s discussion paper, this can help optimize drug dosing “in special populations where there may be limited data (e.g., rare disease studies, pediatric and pregnant populations).” Based on this research, we can imagine future scenarios where AI/ML could have avoided zolpidem dosing concerns, where graduated and limited dosing was tested and applied to different sex, age, and metabolism categories to determine ideal dosing.
Data Analysis. On a more intriguing level, the FDA AI discussion paper discussed the concept of creating “digital twins” of patients for clinical trials. Essentially, an AI version of the clinical participant is created, using the existing candidate’s electronic health records, vital signs, labs and other records. Researchers can assess how the digital twin would react under normal conditions using AI/ML modeling based on data gathered from similar individuals. This digital twin would then act as a substitute for a placebo candidate in a clinical trial, and act as a benchmark against the actual patient undergoing investigational treatment. For zolpidem, this could be used to assess candidates that already have underlying medical conditions such as anxiety, depression, or other confounding factors, to see whether an adverse effect from a trial is due to the investigational treatment or something that is likely to occur to the same individual from anxiety alone.
Postmarketing Safety Surveillance. Finally, AI/ML can help detect and assess adverse events once the drug enters the market. This is not just limited to individual case safety reports (ICSR), required by regulators, but can include adverse events reported publicly on social media and the wider internet. This type of postmarketing safety surveillance could assist researchers and drug companies in identifying potential drug risks, prior to landing on primetime news.
Quality and Reliability Risks
While AI/ML can help to address the costs and efficiency of clinical trials, this relies substantially on the underlying data used to train AI. The quality and reliability of any AI/ML model requires similar quality controls for underlying training data. Given the safety risks of inappropriate drug dosing, or recruiting candidates with severe medical conditions, AI developers cannot rely solely on self-reported healthcare data with no external medical testing or validation. Developers should be equally wary of training on third-party data sets that do not provide documentation on the collection of data and data validation.
Within an existing healthcare organization, if the organization is big enough, aggregate and de-identified data may be obtained from existing electronic health care records and prior clinical trials. Yet, even within these large datasets, errors may surface during training. Medical providers may code the same procedure, and similar symptoms, a dozen different ways. Even drug names can be misspelled and coded incorrectly within existing records. While many of these errors may end up being statistically insignificant with enough data, there is the risk of missing one or two major adverse events, or “black swan” events, that would otherwise change the entire risk profile of a drug.
In addition to quality and reliability, the underlying dataset needs to be representative of the population that will be studied for the clinical trial. If the underlying dataset is only trained on a handful of individuals with a certain medical predisposition, age, sex, weight, etc., it will be difficult for the AI model to make predictions for that group. As an example, if the training data only contains the medical information for two individuals over the age of sixty, and shows no adverse effects from a particular drug dose, this information is not enough to generalize that the drug at that dosage is appropriate for all individuals over the age of sixty. For all we know, these two candidates could be a former Olympic diver and a nutrition coach, two outliers that completely skew the data. Consequently, the underlying training data for any AI model should also be assessed for bias and representativeness as it applies to the proposed clinical trial.
Data Privacy, Cybersecurity, and AI Risks
The data privacy and cybersecurity risks associated with the foregoing uses of AI/ML cannot be underestimated. The quality and representativeness of any AI system in this field will rely heavily on large swathes of healthcare data, fine-tuned and, at times, personalized in the case of digital twins. This is sensitive or special category data at its finest, triggering heightened scrutiny under the EU’s data privacy law, the GDPR, and U.S. data privacy and data breach laws.
To date, most healthcare organizations have sidestepped data privacy concerns by relying on HIPAA’s de-identification standard to remove personal information and other identifiers from healthcare data, making it difficult to associate with an individual. While the FDA requires Institutional Review Board (IRB) review of most biomedical research involving human subjects, this generally does not apply to de-identified personal information that cannot be linked to an individual.
Simply de-identifying data and then running with it is not enough, however. Under the California Consumer Privacy Act and similar state laws, for example, recipients of de-identified data need to affirm that they will not attempt to reidentify the data (except to test their de-identification methods). The GDPR has a much higher “anonymization” standard, which looks at the re-identifiability of personal information, given all the different datasets that an organization may have access to.
AI/ML itself is making the de-identification process harder. As it is capable of slicing and dicing data by age, race, sex, and medical condition, and combining multiple large datasets, it is easy to run the risk of re-identifying data. While several thousand people might have the same configuration of eye color, age, gender, and weight, only one or two may have participated in a clinical trial at a particular location, or have specific allergies or side effects to certain types of medication. As a result, in circumstances where healthcare data is not de-identified, or the risk of reidentification is heightened, then it behooves clinical organizations and their AI developers to implement written information security programs and associated privacy and security controls.
Legal Liability and Drug Dosing
In several notable cases, defendants on zolpidem were able to contest or overturn DWI or even vehicular manslaughter cases. Essentially, these defendants argued that they were not aware of the potential dangers of zolpidem, and so could not be liable for their actions while “sleep driving.”
This raises the question: If AI gets good enough, and can tell you exactly the right dose to take of a drug, will you (or your doctor) be liable if you deviate from the AI’s recommendations? Will the AI’s recommendations be discoverable in court (and surfaced via AI-enhanced search)? Only time will tell what this brave new world will bring.
ENDNOTES
[1] Kai Falkenberg, While You Were Sleeping (September 27, 2012), Marie Claire, https://www.marieclaire.com/culture/news/a7302/while-you-were-sleeping/.
[2] Takuji Inagaki, Tsuyoshi Miyaoka, Seiichi Tsuji, Yasushi Inami, Akira Nishida, and Jun Horiguchi, Adverse Reactions to Zolpidem: Case Reports and a Review of the Literature, 12 Prim Care Companion J Clin Psychiatry 6 (2010), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3067983/.
[3] U.S. FDA, Drug Safety Communication: FDA approves new label changes and dosing for zolpidem products and a recommendation to avoid driving the day after using Ambien CR (May 14, 2013), https://www.fda.gov/drugs/drug-safety-and-availability/fda-drug-safety-communication-fda-approves-new-label-changes-and-dosing-zolpidem-products-and.
[4] FDA: Cut Ambien Dosage for Women, ABC News (January 10, 2013, 6:03AM), https://abcnews.go.com/Health/fda-recommends-slashing-sleeping-pill-dosage-half-women/story?id=18182165.
[5] David J Greenblatt, Jerold S Harmatz, & Thomas Roth, Zolpidem and Gender: Are Women Really At Risk?, 39(3) J. Clinical Psychopharmacol. 189 (May/Jun 2019), https://pubmed.ncbi.nlm.nih.gov/30939589/.
[6] NIH Inclusion Outreach Toolkit: How to Engage, Recruit, and Retain Women in Clinical Research, last accessed September 16, 2024: https://orwh.od.nih.gov/toolkit/recruitment/history.
[7] FDA, Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products (May 10, 2023), https://www.fda.gov/media/167973/download; see also Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products; Availability, 88 FR 30313 (May 11, 2023), https://www.federalregister.gov/documents/2023/05/11/2023-09985/using-artificial-intelligence-and-machine-learning-in-the-development-of-drug-and-biological.
[8] Thi Tuyet Van Tran, Hilal Tayara, and Kil To Chong, Artificial Intelligence in Drug Metabolism and Excretion Prediction: Recent Advances, Challenges, and Future Perspectives, 15 Pharmaceutics. 1260 (Apr 17, 2023), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10143484/.
Lily Li is an AI, data privacy, and cybersecurity lawyer and founder of Metaverse Law. She is a certified information privacy professional for the United States and Europe and is a GIAC Certified Forensic Analyst for advanced incident response and computer forensics. She can be reached at info@metaverselaw.com.
This article was initially published in Reuters and Thomson Reuters Westlaw Today.
Lily Li of Metaverse Law discusses the landscape for AI legislation, with the passage of the European Union’s AI Act while states pass AI bills with differing thresholds, coverage and subject matter.
The landscape for EU and US AI legislation feels like a rinse and repeat of data privacy legislation in 2020. Back then, the General Data Protection Regulation (GDPR) was in full force and effect, while California and other states were developing privacy laws at breakneck speed. Many companies were caught unaware by GDPR, only to face a new onslaught of US state-by-state privacy laws.
Now, companies face the same problem. The EU has just passed a comprehensive AI law, the EU AI Act, which imposes significant compliance obligations and antitrust-style mega fines.
In the United States, state legislatures are passing AI bills at a breakneck speed, with differing thresholds, coverage and subject matter. Do global companies bite the bullet and comply with the EU AI Act globally, or should there be a more nuanced jurisdiction-by-jurisdiction approach?
Comprehensive and imposing
The EU AI act is a comprehensive law that has been in development for years by EU regulators. One of its unique features, not seen in US legislation, is a complete ban on certain “prohibited AI practices” (Article 5, https://bit.ly/4gQHfe8). Some of these prohibited practices include assessing whether an individual is likely to commit a crime and real-time biometric identification by law enforcement (think Minority Report), as well as social scoring of individuals.
In addition to setting forth prohibited practices, the EU AI Act designates a list of high-risk AI practices. This includes, but is not limited to, use of AI in employment decisions, credit scores, insurance and access to services. For these high-risk AI practices, AI providers need to implement a full risk management program that considers the following factors:
Data governance
Technical documentation
Recordkeeping
Human oversight
Accuracy, robustness, and cybersecurity management
Quality management
Like the GDPR, the EU AI Act imposes significant fines. This can be up to $35,000,000 or 7% of total worldwide revenue, whichever is higher, for engaging in prohibited AI practices (Article 99, https://bit.ly/3XRewgl), and up to $15,000,000 Euros or 3% of the total worldwide annual turnover, whichever is higher for other violations (Article 99, https://bit.ly/3XRewgl). The law requires each EU country to designate at least one independent and impartial body to monitor and enforce the EU AI Act’s requirements.
In contrast, the US is following a patchwork approach. Instead of comprehensive federal legislation, we are seeing a state by state and agency approach. To date, these laws generally fall into four main categories: (i) consumer protection; (ii) employment rights; (iii) image and likeness rights; and (iv) transparency/ risk assessment requirements for high-risk AI processing.
Consumer protection
For state consumer protection laws governing AI, Utah is one of the first movers. In May of 2024, it added requirements governing AI to its consumer protection statutes. Utah’s AI Policy Act requires businesses in Utah to disclose the use of generative AI tools, and also makes businesses liable for any consumer protection violations by these generative AI tools.
At the federal level, the FTC has used its consumer protection authority under Section 5 of the FTC Act, in order to regulate against unfair and deceptive practices in commerce concerning AI. In 2022, Weight Watchers agreed to pay a $1.5 million civil penalty in a settlement with the FTC, in part over allegations that the company improperly collected children’s data to train its models and algorithms. This settlement included “algorithmic disgorgement” — i.e., Weight Watchers was required to delete any models trained on such data.
More recently, on Sept. 25, 2024, the Federal Trade Commission (FTC) has cracked down on companies that make misleading or fraudulent claims about their use of AI tools. This included taking action against DoNotPay (https://bit.ly/3BtSWXW), a company that claimed to offer an AI service that was “the world’s first robot lawyer.”
DoNotPay agreed to a $193,000 settlement with the FTC, pursuant to a consent order. The consent order (https://bit.ly/4dNyjmN) also requires DoNotPay to refrain from “representing that its Service or any other internet-enabled product or service that it offers operates like a human lawyer or any other type of professional, unless that representation is not misleading and DoNotPay possesses competent and reliable evidence to substantiate the representation.” In addition, DoNotPay is required to notify consumers of the order and to submit compliance reports to the FTC.
AI in employment decisionmaking
At the employment level, Illinois recently enacted a law that prohibits the use of AI systems from discriminating against employees or job applicants based on any protected classes.
In addition, this amendment explicitly bans the use of race or zip code when used as a proxy for race in AI systems making employment decisions. Illinois’ requirements join New York City Local Law 144 (https://on.nyc.gov/3zHlSva) in regulating automated employment decision-making tools. While Local Law 144 does not include an explicit ban on the use of race or zip code in AI systems, it has very stringent notice and audit rights.
Where employers use AI systems “to substantially assist or replace discretionary decision making,” Local Law 144 requires publicly available third-party bias audits of automated employment decision-making tools.
Image and likeness rights
Generative AI is also regulated by state laws and cases governing image and likeness rights. Following the actors and writers strike in Hollywood, and high-profile litigation by Sarah Silverman and others, California has acted. In the last week, Governor Gavin Newsom signed two AI bills designed to protect entertainers.
AB 2602 requires contracts with actors and other performers to specify whether generative AI will be used to create a replica of the performer’s voice or likeness. AB 2836 bans the use of digital replicas for deceased performers, without the consent of the performer’s estate.
Transparency and risk assessment
The majority of US state comprehensive data privacy laws require transparency concerning the use of AI to process personal data and make decisions that impact important rights, such as employment, housing, and access to services. In addition, these laws generally give consumers the right to opt out of such processing.
Colorado’s AI Act, slated to go in effect in 2026, goes even further. It imposes risk assessment and bias assessment requirements for any “high-risk artificial intelligence system” that makes or is a substantial factor in making a consequential decision.
For purposes of the law, “consequential decision” means a decision that has a material or similarly significant effect on the provision or denial to any consumer of, or the cost or terms of:
Education
Employment
Financial or lending services
Essential government services
Health-care services
Housing
Insurance
Legal service
The Colorado AI Act has even more substantial transparency and notification obligations. As just one example, developers and deployers of “high-risk” AI systems are required to publicly post on their websites a description of the high-risk systems, as well as describe how the AI system manages the risks of bias. This includes further reporting to the Attorney General of “any known or reasonably foreseeable risks of AI discrimination arising from the intended use of the system.” Section §6-1-1702(5).
Where to go from here?
The trend lines are clear, and AI legislation is here to stay. While the US has not enacted federal AI legislation of the same scope as the EU AI Act, we already see significant risk assessment and transparency requirements. As a result, AI companies need to go global with their AI risk management strategies and not get left behind.
Lily Li is the founder and president of Metaverse Law. She advises global clients on their AI risk assessments and data protection impacts assessments, and supports her clients’ overall governance, risk, and compliance (GRC) programs. In addition, she holds the GIAC Certified Forensic Analyst (GCFA) certification for advanced incident response and digital forensics and certifications in information privacy such as the FIP, CIPP/US/E/M. She is based in Newport Beach, California, and can be reached at info@metaverselaw.com.
The original article can also be found on the OneTrust DataGuidance website by clicking on this link.
On September 19, 2024, the California AI Transparency Act (the Act) was signed into law by the California Governor. The Act follows in the steps of other US states that have developed laws requiring transparency in the use of artificial intelligence (AI). The Act, however, is unique in that it has specific watermarking requirements. In this Insight article, OneTrust DataGuidance breaks down the key provisions of the Act and who it applies to, with comments provided by Jacob Canter, Counsel at Crowell & Moring LLP, and Lily Li, Founder of Metaverse Law Corporation.
Definitions
The Act provides definitions for key terms such as ‘personal information,’ ‘personal provenance data,’ and ‘metadata.’ Among the notable, ‘artificial intelligence’ is defined as ‘an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.’
Under the Act, ‘generative artificial intelligence system’ is defined as ‘an artificial intelligence that can generate derived synthetic content, including text, images, video, and audio, that emulates the structure and characteristics of the system’s training data.’
Scope
The Act applies to covered providers, who must comply with the Act from January 1, 2026, when it becomes operative.
The Act defines ‘covered provider’ as ‘a person that creates, codes, or otherwise produces a generative artificial intelligence system that has over 1,000,000 monthly visitors or users and is publicly accessible within the geographic boundaries of the state.’
Regarding the 1 million monthly visitors or users, Jacob notes “This is a bit ambiguous because it does not explain how to calculate ‘over 1,000,000 visitors or users.’ Is this based on an average number of visitors or users from the prior year? Does your obligation to comply change every month depending on how many users you had in the prior month? Until that ambiguity is clarified, the safer approach may be to prepare for compliance even if your company does not consistently have over 1 million visitors.”
Lily adds that “according to Governor Newsom, California is ‘home to 32 of the world’s 50 leading AI companies,’ many of which will be required to comply with this Act due to the nature of their AI systems and number of monthly users.”
Jacob furthers that “Most of the generative AI laws in the U.S. have been subject-matter specific. Some states have either enacted or passed laws related to transparency and fairness in elections (for example, PA, MA, NC, WA, and CA). Many states have passed laws that seek to limit the dissemination of deepfakes (for example, TX, FL, IL, NY, and CA). And Colorado and New York City have passed laws that seek to limit discriminatory uses of generative AI (for example, CO and NY). In contrast, the AI Transparency Act is general. It covers all generative AI content that a covered company’s product generates. On these terms, the Act is actually quite broad.”
Obligations
Regarding the implications of the Act on businesses, Jacob explains that “California’s AI Transparency Act will have a direct impact on businesses that develop generative AI systems and have over 1 million monthly visitors or users. These businesses must comply with the law’s requirements: to create an ‘AI detection tool,’ to embed ‘latent-disclosure’ data into their AI-generated content, and to make ‘manifest disclosures’ available for the content as well.”
The Act requires covered providers to provide an AI detection tool to users at no extra cost, that:
allows users to assess whether an image, video, or audio content has been created or changed by the covered provider’s generative AI tool;
outputs any system provenance data detected in the content;
does not output any personal provenance data detected in the content;
subject to certain exceptions, is publicly accessible;
allows users to upload content or provide a URL for online content; and
supports an application programming interface that allows users to use the tool without visiting the covered provider’s website.
Under the Act, covered providers should also collect user feedback on the AI detection tool and incorporate this feedback to improve the tool’s efficacy.
In addition, covered providers should not:
collect or retain personal information from users of the AI detection tool, except where exceptions apply;
retain content provided to the AI detection tool for longer than necessary to comply with the Act; and/or
retain personal provenance data from content submitted to the AI detection tool.
Lily adds that “While other AI laws in the US are focused on risk assessment, notice, and disclosure obligations, this is the first major AI law that imposes product requirements on AI developers. Now, AI developers need to code in a digital watermark on generative AI content and provide the tools to detect this watermark. This is different from written disclosures on a browser or app, which can easily get lost or obscured when generative AI content is copied or embedded downstream.”
Covered providers should offer users to option to include a manifest disclosure in image, video, or audio content that has been created or altered by the covered provider’s generative AI system that:
identifies the content as being generated by AI;
is clear, conspicuous, and appropriate for the content, as well as understandable to a reasonable person; and
is permanent or difficult to remove.
Covered providers should also include a latent disclosure in AI-generated image, video, or audio content generated by AI system that:
communicates the name of the covered provider, the name and version number of the generative AI system used, the time and date the content was created or altered, and a unique identifier – to the extent technically feasible and reasonable, the disclosure should be direct or through a link to a permanent internet website;
is detectable by the covered provider’s AI detection tool;
is consistent with industry standards; and
is permanent or extraordinarily difficult to remove.
Lily explains that “Additionally, the Act includes a requirement that these covered providers enter contracts with their licensees that contain specific provisions. (22757.3(c).) This means that businesses that incorporate AI or are considering implementing AI systems from covered providers may want to ensure the appropriate contracts are in place.”
If covered providers license their generative AI systems to third parties, they must ensure that licensees maintain these disclosure requirements. If covered providers know that a third-party licensee is no longer capable of including such disclosures, they will be required to revoke their license within 96 hours of discovering this fact. Following the revocation of the license, the third party must cease using a licensed generative AI system.
Enforcement
The Act will be enforced by the Attorney General, a city attorney, or a county counsel and provides that violations of the Act are liable for civil penalties.
Lily notes that “Under this Act, fines can add up quickly: A covered provider found in violation of this Act will be liable for $5,000 per violation – and each day the provider is in violation of the Act counts as a new violation. (22757.4(a-b).) For those who contract with covered providers, a violation may result in an injunction along with reasonable attorney’s fees and costs (22757.4(c).).”
Next steps
Jacob states that “Indirectly, the Act may create opportunities. Technical know-how is required to develop the AI detection tools, and both the latent and manifest disclosures. As often happens, companies can use this change in policy as an opportunity to build a product that facilitates compliance.”
Lily adds that “This Act goes into effect on January 1, 2026, but covered providers should act now given the significant technology requirements of the Act. Covered providers need to:
make an AI detection tool;
include both an optional and a latent disclosure in all AI generated content; and
enter contracts with licensees to ensure such latent disclosures.”
Do you use AI transcription tools for meetings? While helpful, these tools present legal, privacy, and security concerns. For example, do you have to obtain consent from attendees to use the AI tools? Do you own the audio, video, and transcription data?
Metaverse Law’s Lily Li joined David Lee of Do What Works and Jason Makevich of Greenlight Information Services to discuss these AI tools. A video and transcription of the conversation can be found below.
Intro
Dave: As a marketer, one of the AI tools that absolutely saves me time and increases my productivity are these AI transcription tools.
You’ve seen them everywhere between Zoom, Otter AI, Riverside, Descript, the list goes on and on. And a lot of them offer their transcription services for free.
Who doesn’t really like free stuff, but there are some things that we should really know about from both a privacy and a legal point of view, when you do use recording and AI transcription services, when you’re on calls with you and your participants.
So today I’m joined by Jason Makovich, a security and privacy expert and Lily Lee, an attorney specializing in privacy and security.
Welcome both.
Lily: Thanks for having us
Dave: So why don’t we kind of just start out with an introduction, Lily, why don’t you tell us a little bit about yourself
Lily: Sure.
My name is Lily. I’m the founder of Metaverse law, and it’s a boutique firm that focuses exclusively on cybersecurity, AI and data privacy law, and we work with funded startups all the way to public companies.
Dave: And Jason, why don’t you tell us a little bit about yourself.
Jason: Yeah, sure.
Jason Mankiewicz here with GreenLight Cyber.
We’re a managed cybersecurity company in Southern California, clients all across the country. So I’m a CISSP, which is a certification in our industry for cybersecurity. And, provide a lot of advice to small businesses, mid sized businesses as well around cyber risk. My team provides all the managed cybersecurity solutions for them.
How do AI note taking services work?
Dave: Very cool.
So Jason, why don’t you kind of start us out with from like a technology standpoint, kind of explain to us lay people, how does this how do these like AI transcription software, you know, services, how do they work?
Jason: mean, they’re pretty great. You mentioned in the beginning. I mean, we see ’em all the time now.
I love ’em for a lot of reasons. You know, it makes our lives a lot easier.
We’re in a lot of meetings and it’s great to have summary of those and next steps, right? And so it does a good job of capturing those.
So it works by joining the meeting as a bot and it listens to everything going on transcribes use large language models to take that information and make sense of it. And then it can summarize it. It can also interpret things like let’s say we were in a zoom call right now. It might say,
Hey, David mentioned he was going to do these three things.
And Lily mentioned that she would do these and she also told Jason to do these.
And so then it can actually give you kind of those summarize next steps, which awesome. It can give you know screenshot recording summaries and everything else so really, really powerful tools.
Many of them out there that are great but I know we’re here today to talk about some of the things to think about when when using those.
Dave: So does the technology, I mean, does it run locally on my computer or does this stuff all somehow get shoved into this mystical cloud?
And we don’t have, possession for lack of a better word of that conversation and data.
Jason: I mean, it’s, it’s more in the cloud, right?
So it’s a bot that is cloud driven, joins a meeting from the cloud.
There’s nothing really local about it per se and I guess it could
be mystical to some but basically, it’s all cloud driven,
By whatever that provider would be.
So, typically they’re going to be operating in things like. Amazon web services,
Microsoft Azure, Google cloud platform. One of those generally.
AI Notetakers – Who owns the data?
Dave: then Lily, from, like a data and ownership perspective, it’s not like it’s on our local computers anymore. It’s somewhere out there in the cloud.
What do we need to think about from that perspective?
Lily: Yeah, for sure.
So when I take a look at AI note takers or any type of AI providers, I’m really careful to check their terms. There’s data ownership and also data licensing.
And so the idea is that you’re in a contract with the AI note taker.
And you’re granting them either ownership of all the data that you’re providing in the meeting, or you’re granting them a license to use the data.
The terms will decide whether or not the license is limited so that they’re only using your data to give you the services and nothing else.
Or if it’s a really broad license, so your data is actually being fed into their systems to improve their services or for them to bundle up and sell or share elsewhere.
How are AI transcription services using our data?
Dave: Wow. So that was at, when you first described that to me a while back, I was kind of like, Oh, what do I need to look at? So let me, I guess my question is, what do we need to look at to understand how these AI transcription services are using our data?
Lily: I’ll definitely check out the terms and definitely check out the privacy policy.
You’ll probably notice as you’re signing up for these services that there are free or freemium options.
And then there are business or enterprise options. Generally, there are 2 different sets of terms. And if the product is free, then more likely than not, your data is going to be used and input into the services themselves.
Also, check the privacy policies. If they are legitimate, first of all, they’ll have an updated privacy policy, and then they’ll go into how they use the data and how they share their data.
One thing to note is that regardless of which service you use, whether or not it’s free enterprise there will always be caveats for law enforcement and other types of requests for your data.
And so, that’s why, when, when we think about use cases for these AI note takers, there are certain highly sensitive cases that I can touch on further that shouldn’t be provided to AI note takers at all.
Because in any circumstance, the AI company can provide this information to legitimate law enforcement requests.
When should you turn off AI transcriptions tools?
Dave: Well, not that we’re proposing to do anything illegal here, but what, what are, I mean, what are some of the specific use cases, as an, can I actually even say this as an attorney, would you recommend not using transcription services if you’re on a zoom call or whatnot?
Lily: Yes, and so again, this is just general information, this is not legal advice. No one listening to this is getting a bill.
But there definitely are certain use cases where I’d avoid a note takers.
One of them is you know, conversations with your attorney. Again, generally, you’re seeking legal advice. You’re seeking information about what’s okay. What’s not and you’d prefer not that information not to be shared with the 3rd party or with other companies.
Again, depending on your use case, if you have really sensitive trade secret information this might be the time to set down the AI notetaker, because again, let’s pretend you’re in a big tech company and it’s another big tech company’s AI notetaker.
Do you necessarily want to disclose to them how you’re developing your product.
Dave: That’s a great point.
So anytime in conversations with an attorney. Think twice, as well as, in internal IP, be careful on when you’re using that.
So, I mean, that, that starts addressing things from like a cloud services side.
Data & Security concerns with summarized notes
But Jason, we also talked a little bit about, it’s not just that another company has that data. Typically we talked about, when a meeting ends, you get these nice summaries and these nice emails.
Maybe can touch a little bit upon that from a privacy and data security standpoint.
Jason: Yeah.
I mean, that’s one of the things that worries me actually is I remember the first time I, learned about these, bots maybe a year or two ago. I didn’t even realize there was one in a meeting.
I mean, it’s hard to know sometimes, right? And, and I was in a meeting with I think a vendor of ours.
And after the meeting we got, I got this email and it was summarizing the meeting and it was pretty cool, but It was an email and I could forward that to anybody.
And then email is inherently insecure platform. It was not developed in really the way that we, the ways that we use email. And so we’re constantly trying to chase security for email, which just inherently is insecure, right?
So now you’ve got potentially sensitive information on a platter for, you know, who’s ever, who’s ever eyes are on that email. And if that email gets forwarded, or the email mailbox gets compromised, or goes into some shared mailbox because you’ve signed up with zoom on some other account or whatever.
Who knows where that’s going to end up, right? And then that gets stored in email.
Well do you ever delete those? Do you ever go back and purge your emails from it? So how long is that going to be there? And who else might end up getting access to that?
And so there’s just so much to think about.
It’s like you’re in a meeting, you think it’s just confined to this space and these people, and then all of a sudden now you have these captures of the meeting that go into this you know, email, a bunch of email boxes, no less, right? Everyone that was in the meeting generally.
So that’s kind of a big fear or risk that I think about.
You know, I think the other challenge is the overall, is it, you know, how do you ask, I always wonder, how do you even ask, hey, can we turn off the note taker, like, it’s not mine, it’s someone else’s, right?
So, I think we all need to kind of step up and say, hey guys, you know, we’re starting to, to Lily’s point, we’re starting to talk about trade secret or something important.
Why don’t we go ahead and stop recording this and take this, you know, offline effectively or just live. So just things to think about, but the email part, it really leads to that potential of data leakage that I worry about.
Dave: Yeah.
Something definitely to think about.
Beware of transcript bots joining automatically
And then, everyone has an AI bot, right?
So we’ve all joined zoom meetings and we see all these different bots pop up and which bot wins. One of the questions I has, how do you know it’s an actual bot? We talked a little bit about that.
Cause I can just rename myself from Dave Lee at the, at the zoom and I can just say, Oh, I’m Dave’s note taker.
Jason: Absolutely.
I mean, that’s a good point. You have to be really pay close attention to who’s in your meetings. Right? A lot of times I’ll be in a meeting. I don’t even know everyone in there. Of course. Right? So. It’s one of the, it’s like wedding crashers.
It’s like, everyone just kind of assumes well, they’re with the bride or they’re with the groom, right? And that’s kind of how, how these meetings are probably half the time.
You’re not going to know everyone in there and everyone’s going to assume, well, part of the other party or someone else.
But now with the note takers, yeah, I mean, we were talking about this the other day. What would stop someone from sneaking into a meeting, but renaming their zoom account to, you know, one of these products and keeping their camera off and staying on mute, they’d probably look identical to one of those.
Jason: So, yeah, I mean, it’s just things to think about and anytime you’re, dealing with sensitive information in a meeting and you think it’s private.
Hey, very close attention. Who’s in that meeting. both human and bot.
Jason: Yeah.
Dave: I’m certainly guilty of this.
Sometimes it’s just more from a, let’s see, how should I say this? A lazy slash convenience standpoint. Where you initiate a zoom meeting and you’re like, ah, whatever, just let anyone in.
But what you’re saying is really, you know, pay close attention to who you’re actually letting in. So putting people into a waiting room and making sure that they are who they say they are is probably a good best practice around there.
So, in terms of Lily for. From, I don’t think this is asking legal advice, is it?
But from, from a legal standpoint, are there things that we have to be concerned about when, we’re just firing up, zoom or an AI bot in terms of, I remember it’s kind of like, way before the pandemic. Right. I always used to say like, hey, is it, is it okay if I record this meeting?
Mostly from a permission standpoint, because I want to make sure people understand where we’re recording.
Some states require consent before recording
But, are there other considerations beyond that in today’s, you know, post COVID everyone’s using zoom and do we still need to ask that, have that permission structure or is it automatically assumed, do we need to be concerned about that state by state?
I don’t, I don’t know anything about that.
Lily: Sure, yes,
So there are quite a few states, including California, where it is an all party consent state,
which means that all parties to a phone communication, a video communication needs to consent to the recording.
Zoom is very interesting because they’ve actually been through the ringer
a few times with different types of privacy class actions.
For instance, they actually paid out an $85 million dollar privacy settlement as a result of a class action, claiming that they inadvertently allowed third parties to come in and intercept calls or record calls without the parties consent.
So this is a real issue.
You’ll notice that a lot of platforms have now a recording notice or some other recording alert for kind of the default platform recording.
But that doesn’t mean that it covers your own use of an AI bot or an AI recording system if it’s not integrated with the platform and so it doesn’t trigger that notice. So, again, best practice, still a good idea to alert people what you’re doing.
Another thing to be aware of is that there are AI laws that govern notice and consent requirements if you’re using AI for certain activities.
For instance, if you are engaging in interviews and you’re using AI in order to assess an interviewee. Then you may have certain notice requirements in certain states.
And now with the new EU AI Act, there’s going to be requirements going forward to notify individuals if you’re using AI, again, for interview purposes, performance evaluations, testing and other things that might impact an individual’s rights and abilities and access to services.
Dave: Oh, wow. I didn’t even think about the EU and this is probably a topic for another day, but the whole GDPR and the privacy insecurity along with the California CCP, I’m sure there’s, there’s stuff that governs around that.
So that’s really good information to know.
From your experience, has there been any, issues in people using AI. And this is more like within the public domain, right?
But are there things that we can take a look at and that makes us, that we can get smarter on in terms of how people or how zoom was in a class action suit or any advice in that perspective, what we need to start thinking about.
Lily: I mean, definitely from a legal and privacy perspective.
At least in the United States, there’s a lot more permissiveness around using AI and around using people’s data as long as there’s notice regarding it.
And you’re giving individuals the ability to opt out of such recording and giving them certain rights to their data.
Again, if you’re going abroad, and you’re looking at the EU, then that’s a whole separate conversation, and it becomes more about opting into a process.
Dave: Got it.
Jason, in terms of selecting a particular AI transcription tool, is there anything that we should probably think about besides looking at the T’s and C’s and privacy from a technical standpoint, when we decide to use one platform or another? Cause I mean, there are so many of them out there.
Yeah, I would say
Risks of using free transcription tools
Jason: Lily touched on the whole freemium thing. If you’re not paying for a product, then chances are you’re the product, right?
That’s the case with so many things out there and in our digital world. Stop using free stuff.
I mean, look at the terms and conditions, look at the privacy policy, just look at, compare what you get from the paid versions of things and just understand what you’re dealing with.
Sure. There are certain use cases where the free version of something makes sense, including I mean, if all you’re doing is talking about, you know, you’re talking to friends or whatever.
And yeah, fine. But I mean, gosh, in my business, there’s no way we’d be using something that might compromise you know, the privacy.
I will add further outside of the meeting assistant, but in the, under the guise of the AI bot within the meeting, there are tools out there that are marketing as noise cancelling tools for the products we use like Zoom and Teams and all that, noise cancelling software.
Go buy noise cancelling hardware, because when you start, and I’m not naming any products, but when you start looking at the privacy policies of these noise cancelling products out there, really anything that joins a meeting or intercepts the audio on the meeting, there are free versions out there that are they’re allowed to in the privacy policy, some of them, they’re they own that data now.
They they’re allowed to do anything they want with it.
Potentially. I don’t know all the legalese, but so pay very close attention to the software that you and everyone in your company uses.
Corporate governance and use of AI tools
I think the real answer here is around governance and having governance around the use of AI and software in general technology in general in your organization.
There needs to be governance policies established, talked about, and they’re not policies to put in a drawer.
They’re policies that , should be really worked on and really, really thought through and then enforced throughout the entire organization.
And there are ways to enforce. to the point where it would be very difficult, if not impossible for users to use something that they shouldn’t use.
But it starts with a conversation with the leadership of the organization and usually some experts outside of the organization like Lily on the legal side, myself on the technical side, or folks like us that can help guide, help ask the right questions.
And ultimately come up with you know, what’s going to make sense for you and your business.
Dave: That’s a really good point on governance.
Back in my old it days and you know, Jason, what that was way back when, I mean, you can enforce policies so that you can have your, company computers locked down.
So they, you know, the end user can’t install things.
But with remote work and people using their own personal computers it really becomes less of something of that you can actually Enforced from a policy perspective, but it’s also a, I don’t know what you, what you guys call it today, but like a socialization standpoint, because if I’m at home and I’m on my own personal PC and I decided to install some software, but I’m using it for business purposes, it’s yeah, there should be a governance process that when you use any asset for a company purposes, it probably should go through the, the it department and have been vetted and approved.
Not just for us. It doesn’t crash your computer, but
Jason: And not just going through the IT department, but have the business leadership working with IT to define what should be and should not be allowed.
And having generally outside expertise guiding around that because most IT departments may not have that experience or perspective.
And, you know, if they’re working in that IT department every day, they’re not going to necessarily know what else is going on and all these other environments are working with, you know, providers or, you know, attorneys or anyone outside that does that for a living, that really understands governance and risk.
And there’s a lot of different folks out there, that, that can help with that. But I think that’s key too, is, is just business leaders really getting involved in defining, understanding and defining what should be allowed in the organization,
Final Thoughts
Dave: Cool. Are there, are there any last words of advice that we would like to give to you know, managers, business owners in terms of using these recording software and AI transcription tools?
Lily: I mean, like Jason mentioned, you know, have a policy in place even from a cost savings point of view, if you’re going to pay for an enterprise version of some software, make sure all your employees know about it and are using that rather than using company funds to develop or work with competing software.
And then also, you know, there are a lot of different settings in many of the default platforms that allow you to give notice that allow you to use waiting rooms and allow you to develop that functionality that limits data leakage.
So, again, just really get to know what you’re using in the company.
Dave: Yeah. Not just from a usability standpoint. Cause you know, that’s what I’m always looking at. Like, Oh my gosh, how easy is it to use this thing? But there’s things that need to be looked at behind the scenes.
Jason: But on the flip side, I would also add, don’t be too scared of technology not to adopt it, right?
I mean, it’s happening, whether we like it or not, like we’re entering this new AI world and don’t be so scared of it that you’re not going to.
You leverage the technology that’s out there because your competitors will, and you’ll be left behind.
In fact, you need to be very forward thinking. You need to be paying attention to what’s out there. What, you know, if chat GPT caught you off guard, and you were the last person in your family to learn about it, chances are, you’re not doing enough to stay in, in front of technology, right?
And that, that it’s just one of those things where, you know, you want it, you want to know what’s going on, you not want to know what’s out there and really pay attention because, you know, things are happening very quickly, but always have that mindset of security, privacy, governance, and be working with the right people, to make sure that, that you’re making good decisions.
Dave: Yeah.
I don’t think any, any of us here are saying that we should not use these AI transcription services.
It’s more along the lines of we have to be aware of how it works and what are the security and legal ramifications we need to think through before we hit that record button.
Very cool.
Outro
So thank you, Jason and Lily.
For those who would like to get in touch with them, I’m going to leave their contact information and the website below in the transcript.
Thanks everyone for listening and spending time with us.
There’s more great educational content and interviews for you at do what. works slash podcasts.
Jason Makevich, CEO, Greenlight Information Services
Metaverse Law’s Lily Li recently spoke at Arthur’s inaugural AI Fest. Lily was on a panel that spoke about legal considerations for enterprise use of AI.
The panel explored the complex legal landscape surrounding AI adoption in business. The expert panelists discussed regulatory compliance, data privacy, intellectual property, and ethical concerns, providing actionable insights for companies integrating AI into their operations. Attendees gained a deeper understanding of the potential legal risks and how to navigate them effectively to ensure responsible and compliant AI use in the enterprise.
