The January 2025 edition of Orange County Lawyer magazine features an article written by Metaverse Law’s Lily Li. Read “AI and Machine Learning in Drug Development and Clinical Trials” below or in Orange County Lawyer magazine.
[Originally published as a Feature Article: AI and Machine Learning in Drug Development and Clinical Trials, by Lily Li, in Orange County Lawyer Magazine, January 2025, Vol. 67 No.1, page 28.]AI and Machine Learning in Drug Development and Clinical Trialsby Lily Li
In 2013, sleep medication zolpidem (Ambien, Ambien CR, and Edluar) swept headlines. Marie Claire reported on an alarming and suspicious rise in users experiencing irrational eating, gambling, and even “sleep-driving” while in a hypnotic trance—waking with no memories of their actions.[1] In several cases, women arrested and convicted for driving under the influence contested their convictions, arguing that they were not liable for these undisclosed drug-related side effects.
At the same time, several clinical studies suggested that women metabolized zolpidem differently from men. By reviewing existing literature, Japanese researchers out of Shimane University identified 40% higher concentrations of zolpidem in women than men following use, and higher rates of visual hallucinations and sensory distortions.[2]
The FDA released a safety advisory, warning users of the risks of “next-morning impairment” for the use of Ambien and related drugs.[3] In addition, the FDA took the unusual step of recommending a 50% cut in the dosage for women. When asked about the change, an FDA director told ABCNews.com: “The changes are different in women and men . . .We don’t understand why yet, but women are more susceptible to next-morning impairment.”[4]
Yet, a decade later, the evidence supporting different zolpidem dosages for women and men is unclear.[5] In part, this is due to the lack of research surrounding sex differences in drug impact and drug treatment, as well as substantial gaps in the inclusion of women in clinical studies. From 1977 to 1993, FDA policy recommended excluding women of childbearing potential from Phase 1 and early Phase II drug trials.[6] Even after this policy was removed in 1993, industry fears remained with respect to drug interactions with pregnancy. This episode with zolpidem raised several concerns in the drug development and clinical trial process:
How do we recruit representative candidates for drug trials?
How do we ensure the quality and availability of datasets for clinical research?
How do we measure potential impacts of drug dosing on different populations?
What are the legal implications for failing to address appropriate drug doses?
AI and ML to the Rescue?Now that artificial intelligence is being used in research and development, one wonders: Can artificial intelligence (AI) and machine learning (ML) reduce bias and risks during drug development? Or will it create new legal risks due to bias, privacy intrusions, and lack of transparency?
The FDA released a discussion paper on AI, Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products, to discuss potential regulatory frameworks to address the use of AI and ML.[7] In this discussion paper, the FDA released a set of fascinating case studies into existing research and uses of AI in the clinical trial process. Several of these case studies are discussed below, as well as an analysis of their potential impact on the zolpidem example.
Recruitment. According to the FDA, “AI/ML is being used to mine vast amounts of data, such as data from clinical trial databases, trial announcements, social media, medical literature, registries, and structured and unstructured data in EHRs [electronic health records], which can be used to match individuals to trials (Harrer, 219 Shah, Antony, & Hu, 2019).” In this manner, researchers can combine huge quantities of publicly available data and individual health data from prior research to identify participants with certain medical conditions (or lack of adverse conditions) for investigational treatments. For zolpidem, the use of AI/ML may have been able to identify a much broader list of participants for initial clinical testing, making it easier to assess and identify adverse reactions.
Selection and Stratification of Trial Participants. In addition to initial recruitment, AI/ ML has the capability improve intake, selection, and classification of clinical trial participants. Based on baseline characteristics selected by the researchers, such as prior clinical data, and vitals/labs taken during intake, predictive algorithms can help identify high-risk participants.[8] These groups can then be randomized and then subject to more strict monitoring protocols. In the case of zolpidem, alcohol use is associated with sometimes severe adverse effects from the drug, and so it would be beneficial to screen out candidates with a history of alcoholism or, on the flip side, assess drug interactions for this high-risk group with additional support, monitoring, or counseling.
Dose/Dosing Regimen Optimization. AI/ML can be used to predict drug exposure for different populations based on factors such as weight, height, sex, and other characteristics that might impact drug metabolism. Based on prior drug exposure and response profiles for similar drugs and similar populations, AI/ML can help to narrow the dose/dosing regimen selected for a study. As noted by the FDA’s discussion paper, this can help optimize drug dosing “in special populations where there may be limited data (e.g., rare disease studies, pediatric and pregnant populations).” Based on this research, we can imagine future scenarios where AI/ML could have avoided zolpidem dosing concerns, where graduated and limited dosing was tested and applied to different sex, age, and metabolism categories to determine ideal dosing.
Data Analysis. On a more intriguing level, the FDA AI discussion paper discussed the concept of creating “digital twins” of patients for clinical trials. Essentially, an AI version of the clinical participant is created, using the existing candidate’s electronic health records, vital signs, labs and other records. Researchers can assess how the digital twin would react under normal conditions using AI/ML modeling based on data gathered from similar individuals. This digital twin would then act as a substitute for a placebo candidate in a clinical trial, and act as a benchmark against the actual patient undergoing investigational treatment. For zolpidem, this could be used to assess candidates that already have underlying medical conditions such as anxiety, depression, or other confounding factors, to see whether an adverse effect from a trial is due to the investigational treatment or something that is likely to occur to the same individual from anxiety alone.
Postmarketing Safety Surveillance. Finally, AI/ML can help detect and assess adverse events once the drug enters the market. This is not just limited to individual case safety reports (ICSR), required by regulators, but can include adverse events reported publicly on social media and the wider internet. This type of postmarketing safety surveillance could assist researchers and drug companies in identifying potential drug risks, prior to landing on primetime news.
Quality and Reliability Risks
While AI/ML can help to address the costs and efficiency of clinical trials, this relies substantially on the underlying data used to train AI. The quality and reliability of any AI/ML model requires similar quality controls for underlying training data. Given the safety risks of inappropriate drug dosing, or recruiting candidates with severe medical conditions, AI developers cannot rely solely on self-reported healthcare data with no external medical testing or validation. Developers should be equally wary of training on third-party data sets that do not provide documentation on the collection of data and data validation.
Within an existing healthcare organization, if the organization is big enough, aggregate and de-identified data may be obtained from existing electronic health care records and prior clinical trials. Yet, even within these large datasets, errors may surface during training. Medical providers may code the same procedure, and similar symptoms, a dozen different ways. Even drug names can be misspelled and coded incorrectly within existing records. While many of these errors may end up being statistically insignificant with enough data, there is the risk of missing one or two major adverse events, or “black swan” events, that would otherwise change the entire risk profile of a drug.
In addition to quality and reliability, the underlying dataset needs to be representative of the population that will be studied for the clinical trial. If the underlying dataset is only trained on a handful of individuals with a certain medical predisposition, age, sex, weight, etc., it will be difficult for the AI model to make predictions for that group. As an example, if the training data only contains the medical information for two individuals over the age of sixty, and shows no adverse effects from a particular drug dose, this information is not enough to generalize that the drug at that dosage is appropriate for all individuals over the age of sixty. For all we know, these two candidates could be a former Olympic diver and a nutrition coach, two outliers that completely skew the data. Consequently, the underlying training data for any AI model should also be assessed for bias and representativeness as it applies to the proposed clinical trial.
Data Privacy, Cybersecurity, and AI Risks
The data privacy and cybersecurity risks associated with the foregoing uses of AI/ML cannot be underestimated. The quality and representativeness of any AI system in this field will rely heavily on large swathes of healthcare data, fine-tuned and, at times, personalized in the case of digital twins. This is sensitive or special category data at its finest, triggering heightened scrutiny under the EU’s data privacy law, the GDPR, and U.S. data privacy and data breach laws.
To date, most healthcare organizations have sidestepped data privacy concerns by relying on HIPAA’s de-identification standard to remove personal information and other identifiers from healthcare data, making it difficult to associate with an individual. While the FDA requires Institutional Review Board (IRB) review of most biomedical research involving human subjects, this generally does not apply to de-identified personal information that cannot be linked to an individual.
Simply de-identifying data and then running with it is not enough, however. Under the California Consumer Privacy Act and similar state laws, for example, recipients of de-identified data need to affirm that they will not attempt to reidentify the data (except to test their de-identification methods). The GDPR has a much higher “anonymization” standard, which looks at the re-identifiability of personal information, given all the different datasets that an organization may have access to.
AI/ML itself is making the de-identification process harder. As it is capable of slicing and dicing data by age, race, sex, and medical condition, and combining multiple large datasets, it is easy to run the risk of re-identifying data. While several thousand people might have the same configuration of eye color, age, gender, and weight, only one or two may have participated in a clinical trial at a particular location, or have specific allergies or side effects to certain types of medication. As a result, in circumstances where healthcare data is not de-identified, or the risk of reidentification is heightened, then it behooves clinical organizations and their AI developers to implement written information security programs and associated privacy and security controls.
Legal Liability and Drug Dosing
In several notable cases, defendants on zolpidem were able to contest or overturn DWI or even vehicular manslaughter cases. Essentially, these defendants argued that they were not aware of the potential dangers of zolpidem, and so could not be liable for their actions while “sleep driving.”
This raises the question: If AI gets good enough, and can tell you exactly the right dose to take of a drug, will you (or your doctor) be liable if you deviate from the AI’s recommendations? Will the AI’s recommendations be discoverable in court (and surfaced via AI-enhanced search)? Only time will tell what this brave new world will bring.
ENDNOTES
[1] Kai Falkenberg, While You Were Sleeping (September 27, 2012), Marie Claire, https://www.marieclaire.com/culture/news/a7302/while-you-were-sleeping/.
[2] Takuji Inagaki, Tsuyoshi Miyaoka, Seiichi Tsuji, Yasushi Inami, Akira Nishida, and Jun Horiguchi, Adverse Reactions to Zolpidem: Case Reports and a Review of the Literature, 12 Prim Care Companion J Clin Psychiatry 6 (2010), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3067983/.
[3] U.S. FDA, Drug Safety Communication: FDA approves new label changes and dosing for zolpidem products and a recommendation to avoid driving the day after using Ambien CR (May 14, 2013), https://www.fda.gov/drugs/drug-safety-and-availability/fda-drug-safety-communication-fda-approves-new-label-changes-and-dosing-zolpidem-products-and.
[4] FDA: Cut Ambien Dosage for Women, ABC News (January 10, 2013, 6:03AM), https://abcnews.go.com/Health/fda-recommends-slashing-sleeping-pill-dosage-half-women/story?id=18182165.
[5] David J Greenblatt, Jerold S Harmatz, & Thomas Roth, Zolpidem and Gender: Are Women Really At Risk?, 39(3) J. Clinical Psychopharmacol. 189 (May/Jun 2019), https://pubmed.ncbi.nlm.nih.gov/30939589/.
[6] NIH Inclusion Outreach Toolkit: How to Engage, Recruit, and Retain Women in Clinical Research, last accessed September 16, 2024: https://orwh.od.nih.gov/toolkit/recruitment/history.
[7] FDA, Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products (May 10, 2023), https://www.fda.gov/media/167973/download; see also Using Artificial Intelligence and Machine Learning in the Development of Drug and Biological Products; Availability, 88 FR 30313 (May 11, 2023), https://www.federalregister.gov/documents/2023/05/11/2023-09985/using-artificial-intelligence-and-machine-learning-in-the-development-of-drug-and-biological.
[8] Thi Tuyet Van Tran, Hilal Tayara, and Kil To Chong, Artificial Intelligence in Drug Metabolism and Excretion Prediction: Recent Advances, Challenges, and Future Perspectives, 15 Pharmaceutics. 1260 (Apr 17, 2023), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10143484/.
Lily Li is an AI, data privacy, and cybersecurity lawyer and founder of Metaverse Law. She is a certified information privacy professional for the United States and Europe and is a GIAC Certified Forensic Analyst for advanced incident response and computer forensics. She can be reached at info@metaverselaw.com.
In a vote of 4-1, the California Privacy Protection Agency (CPPA) has decided to move forward with rulemaking of its draft regulations concerning AI, cyber audits, profiling and risk assessments, despite complaints of regulatory overreach.
On Friday, November 8, the CCPA held a public meeting to discuss proposed updates to the California Consumer Privacy Act (CCPA) regulations. The hybrid meeting included public comments from a broad range of stakeholders – nearly 45 public comments were heard from business representatives, privacy advocates, and industry experts. While the passing vote would have typically triggered a 45-day public comment period on the draft regulations, Chairperson Urban requested flexibility, considering the upcoming holidays.
Legal Challenges
During the meeting, the CPPA stated that it was sued for failing to promulgate regulations, specifically on opt-out rights of information processed by automated decisionmaking tools (ADMTs). At the same time, commentators argued that the breadth of the proposed rules overstepped the intent of the CCPA.
Board Member Alastair Mactaggart–who helped draft the CCPA–voiced concerns about the regulations, arguing that the current proposed regulation is excessively broad to the point of being unworkable. He pointed out that these regulations, as written, apply to nearly all businesses that use any kind of software to generate any type of output–whether it’s AI-powered or not. For example, a simple tool like a spreadsheet or a school admission application could fall under these rules, forcing a large swath of low-risk businesses to conduct risk assessments. Mactaggart referred to this as statutory overreach and claimed that regulations should be focused on issues that genuinely impact privacy or security.
Economic Forecasts
The CPPA also issued a Standardized Regulatory Impact Assessment (SRIA) which was discussed during the meeting. In this assessment, the CPPA estimates the total cost of this regulatory initiative to be around $3.5 billion for the first year of implementation, with an average of $1 billion each subsequent year for the first ten years. The CPPA justifies this cost, asserting that the direct benefits to California businesses will be $1.5 billion in 2027, and $66.3 billion in 2036.
However, the California Chamber of Commerce states that “[b]usinesses, consumers and governments in California will suffer net losses from the proposed rules pending before the [CPPA] this week.” This statement stems from a report prepared for the Chamber of Commerce by Capitol Matrix Consulting, which concludes that the regulations are likely to “result in a substantial net losses to businesses, consumers, and governments in this state, both in the near and long term.”
Industry groups including TechNet, the Civil Justice Association of California, and the Interactive Advertising Bureau voiced concern about the heavy compliance burden that regulations place on businesses–especially small businesses that may not have the recourses to implement the required risk assessments or redesign their services to accommodate opt-out provisions.
Behavioral Advertising & Opt-Out Provisions
Another key point of contention during the meeting was the opt-out provision for consumers related to decisions made by AI systems.
The draft regulations govern a large range of AI. Under the draft, AI is defined as a “machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments.” Additionally, the draft defines ADMTs as “any technology that processes personal information and uses computation to execute a decision, replace human decisionmaking, or substantially facilitate human decisionmaking.”
Together, these definitions are more expansive than the definition of the high-risk automated processing addressed in Article 22 of the EU’s GDPR, the source of the original opt-out language. Under Article 22, a consumer has the right to opt out of decisions made by solely automated systems. The intent of this provision is to give consumers the ability to opt out of decisions that may be made on solely automated processes, such as targeted advertising.
However, critics argue that including the opt-out language in the draft in combination with an expansive definition of AI and ADMTs could have unintended consequences, especially for small businesses. Mactaggart, for instance, is concerned that applying this opt-out rule too broadly could lead to a breakdown of essential services. For example, online booking services for airlines and automated reservation software for hotels may rely on software that would be categorized as “AI” under this definition. Allowing users to opt out of using AI when asking for these services may be untenable, which could cause friction in these industries and ultimately could cause harm to consumers by limiting access to these services or increasing costs.
Risk Assessments
A central component of the draft regulation is for businesses who use AI, as defined above, to conduct risk assessments. While the goal of this requirement is to ensure that businesses are aware of and mitigate any potential privacy risks that arise from these technologies, critics believe the regulations go too far by applying the requirement to low risk, everyday activities.
For example, a representative from the California Grocery Association expressed concerns about how the opt-out provision would impact a chain of small rural grocery stores with whom she conducts business. While these AI tools could be used to help consumers save money, the cost of compliance to integrate these tools might not be within reach, especially given the thin profit margins within the grocery industry.
Again, Mactaggart questioned the scope of the draft. He and other advocates called for a narrower focus for risk assessments that centers on significant decisions–such as those that deny individuals access to essential goods and services. This could include the denial of a loan application, exclusion from an online platform, or an adverse employment decision. One commenter stated that there have been no public comments against regulating high-risk systems, and by focusing on these issues, the CPPA could better mitigate potential harms. At the same time, this would free low-risk systems from potential overregulation.
Additionally, a commentor suggested that risk assessments should be streamlined and aligned with other state standards to reduce compliance costs. Mactaggart notes that accepting risk standards from other US jurisdictions could help businesses avoid duplicative efforts, cut compliance costs, and reduce the overall regulatory burden.
AI Training
The ability to opt out of training for AI datasets was of lesser concern but was still addressed by a number of commentors. For example, a representative from the Software and Data Industry Association argued that requiring an opt-out from consumers from AI dataset training could create a substantial burden on small businesses who already have trouble accumulating representative training data. Other commentors shares concerns that these opt-outs could compromise the quality and effectiveness for AI systems.
Ultimately, California faces a delicate balance in regulating AI and ADMT. On one hand, the state must work toward protecting consumers from privacy risks, potential discrimination, and other adverse impacts of AI. At the same time, the CPPA must ensure that rulemaking does not stifle innovation, create excessive compliance costs, or diminish competition between businesses that rely on AI.
As formal rulemaking moves forward, it will be crucial for the CPPA to consider feedback from the public comment period and to refine the regulations to ensure that they strike a balance between privacy concerns and costs to consumers and businesses alike.
Do you use AI transcription tools for meetings? While helpful, these tools present legal, privacy, and security concerns. For example, do you have to obtain consent from attendees to use the AI tools? Do you own the audio, video, and transcription data?
Metaverse Law’s Lily Li joined David Lee of Do What Works and Jason Makevich of Greenlight Information Services to discuss these AI tools. A video and transcription of the conversation can be found below.
Intro
Dave: As a marketer, one of the AI tools that absolutely saves me time and increases my productivity are these AI transcription tools.
You’ve seen them everywhere between Zoom, Otter AI, Riverside, Descript, the list goes on and on. And a lot of them offer their transcription services for free.
Who doesn’t really like free stuff, but there are some things that we should really know about from both a privacy and a legal point of view, when you do use recording and AI transcription services, when you’re on calls with you and your participants.
So today I’m joined by Jason Makovich, a security and privacy expert and Lily Lee, an attorney specializing in privacy and security.
Welcome both.
Lily: Thanks for having us
Dave: So why don’t we kind of just start out with an introduction, Lily, why don’t you tell us a little bit about yourself
Lily: Sure.
My name is Lily. I’m the founder of Metaverse law, and it’s a boutique firm that focuses exclusively on cybersecurity, AI and data privacy law, and we work with funded startups all the way to public companies.
Dave: And Jason, why don’t you tell us a little bit about yourself.
Jason: Yeah, sure.
Jason Mankiewicz here with GreenLight Cyber.
We’re a managed cybersecurity company in Southern California, clients all across the country. So I’m a CISSP, which is a certification in our industry for cybersecurity. And, provide a lot of advice to small businesses, mid sized businesses as well around cyber risk. My team provides all the managed cybersecurity solutions for them.
How do AI note taking services work?
Dave: Very cool.
So Jason, why don’t you kind of start us out with from like a technology standpoint, kind of explain to us lay people, how does this how do these like AI transcription software, you know, services, how do they work?
Jason: mean, they’re pretty great. You mentioned in the beginning. I mean, we see ’em all the time now.
I love ’em for a lot of reasons. You know, it makes our lives a lot easier.
We’re in a lot of meetings and it’s great to have summary of those and next steps, right? And so it does a good job of capturing those.
So it works by joining the meeting as a bot and it listens to everything going on transcribes use large language models to take that information and make sense of it. And then it can summarize it. It can also interpret things like let’s say we were in a zoom call right now. It might say,
Hey, David mentioned he was going to do these three things.
And Lily mentioned that she would do these and she also told Jason to do these.
And so then it can actually give you kind of those summarize next steps, which awesome. It can give you know screenshot recording summaries and everything else so really, really powerful tools.
Many of them out there that are great but I know we’re here today to talk about some of the things to think about when when using those.
Dave: So does the technology, I mean, does it run locally on my computer or does this stuff all somehow get shoved into this mystical cloud?
And we don’t have, possession for lack of a better word of that conversation and data.
Jason: I mean, it’s, it’s more in the cloud, right?
So it’s a bot that is cloud driven, joins a meeting from the cloud.
There’s nothing really local about it per se and I guess it could
be mystical to some but basically, it’s all cloud driven,
By whatever that provider would be.
So, typically they’re going to be operating in things like. Amazon web services,
Microsoft Azure, Google cloud platform. One of those generally.
AI Notetakers – Who owns the data?
Dave: then Lily, from, like a data and ownership perspective, it’s not like it’s on our local computers anymore. It’s somewhere out there in the cloud.
What do we need to think about from that perspective?
Lily: Yeah, for sure.
So when I take a look at AI note takers or any type of AI providers, I’m really careful to check their terms. There’s data ownership and also data licensing.
And so the idea is that you’re in a contract with the AI note taker.
And you’re granting them either ownership of all the data that you’re providing in the meeting, or you’re granting them a license to use the data.
The terms will decide whether or not the license is limited so that they’re only using your data to give you the services and nothing else.
Or if it’s a really broad license, so your data is actually being fed into their systems to improve their services or for them to bundle up and sell or share elsewhere.
How are AI transcription services using our data?
Dave: Wow. So that was at, when you first described that to me a while back, I was kind of like, Oh, what do I need to look at? So let me, I guess my question is, what do we need to look at to understand how these AI transcription services are using our data?
Lily: I’ll definitely check out the terms and definitely check out the privacy policy.
You’ll probably notice as you’re signing up for these services that there are free or freemium options.
And then there are business or enterprise options. Generally, there are 2 different sets of terms. And if the product is free, then more likely than not, your data is going to be used and input into the services themselves.
Also, check the privacy policies. If they are legitimate, first of all, they’ll have an updated privacy policy, and then they’ll go into how they use the data and how they share their data.
One thing to note is that regardless of which service you use, whether or not it’s free enterprise there will always be caveats for law enforcement and other types of requests for your data.
And so, that’s why, when, when we think about use cases for these AI note takers, there are certain highly sensitive cases that I can touch on further that shouldn’t be provided to AI note takers at all.
Because in any circumstance, the AI company can provide this information to legitimate law enforcement requests.
When should you turn off AI transcriptions tools?
Dave: Well, not that we’re proposing to do anything illegal here, but what, what are, I mean, what are some of the specific use cases, as an, can I actually even say this as an attorney, would you recommend not using transcription services if you’re on a zoom call or whatnot?
Lily: Yes, and so again, this is just general information, this is not legal advice. No one listening to this is getting a bill.
But there definitely are certain use cases where I’d avoid a note takers.
One of them is you know, conversations with your attorney. Again, generally, you’re seeking legal advice. You’re seeking information about what’s okay. What’s not and you’d prefer not that information not to be shared with the 3rd party or with other companies.
Again, depending on your use case, if you have really sensitive trade secret information this might be the time to set down the AI notetaker, because again, let’s pretend you’re in a big tech company and it’s another big tech company’s AI notetaker.
Do you necessarily want to disclose to them how you’re developing your product.
Dave: That’s a great point.
So anytime in conversations with an attorney. Think twice, as well as, in internal IP, be careful on when you’re using that.
So, I mean, that, that starts addressing things from like a cloud services side.
Data & Security concerns with summarized notes
But Jason, we also talked a little bit about, it’s not just that another company has that data. Typically we talked about, when a meeting ends, you get these nice summaries and these nice emails.
Maybe can touch a little bit upon that from a privacy and data security standpoint.
Jason: Yeah.
I mean, that’s one of the things that worries me actually is I remember the first time I, learned about these, bots maybe a year or two ago. I didn’t even realize there was one in a meeting.
I mean, it’s hard to know sometimes, right? And, and I was in a meeting with I think a vendor of ours.
And after the meeting we got, I got this email and it was summarizing the meeting and it was pretty cool, but It was an email and I could forward that to anybody.
And then email is inherently insecure platform. It was not developed in really the way that we, the ways that we use email. And so we’re constantly trying to chase security for email, which just inherently is insecure, right?
So now you’ve got potentially sensitive information on a platter for, you know, who’s ever, who’s ever eyes are on that email. And if that email gets forwarded, or the email mailbox gets compromised, or goes into some shared mailbox because you’ve signed up with zoom on some other account or whatever.
Who knows where that’s going to end up, right? And then that gets stored in email.
Well do you ever delete those? Do you ever go back and purge your emails from it? So how long is that going to be there? And who else might end up getting access to that?
And so there’s just so much to think about.
It’s like you’re in a meeting, you think it’s just confined to this space and these people, and then all of a sudden now you have these captures of the meeting that go into this you know, email, a bunch of email boxes, no less, right? Everyone that was in the meeting generally.
So that’s kind of a big fear or risk that I think about.
You know, I think the other challenge is the overall, is it, you know, how do you ask, I always wonder, how do you even ask, hey, can we turn off the note taker, like, it’s not mine, it’s someone else’s, right?
So, I think we all need to kind of step up and say, hey guys, you know, we’re starting to, to Lily’s point, we’re starting to talk about trade secret or something important.
Why don’t we go ahead and stop recording this and take this, you know, offline effectively or just live. So just things to think about, but the email part, it really leads to that potential of data leakage that I worry about.
Dave: Yeah.
Something definitely to think about.
Beware of transcript bots joining automatically
And then, everyone has an AI bot, right?
So we’ve all joined zoom meetings and we see all these different bots pop up and which bot wins. One of the questions I has, how do you know it’s an actual bot? We talked a little bit about that.
Cause I can just rename myself from Dave Lee at the, at the zoom and I can just say, Oh, I’m Dave’s note taker.
Jason: Absolutely.
I mean, that’s a good point. You have to be really pay close attention to who’s in your meetings. Right? A lot of times I’ll be in a meeting. I don’t even know everyone in there. Of course. Right? So. It’s one of the, it’s like wedding crashers.
It’s like, everyone just kind of assumes well, they’re with the bride or they’re with the groom, right? And that’s kind of how, how these meetings are probably half the time.
You’re not going to know everyone in there and everyone’s going to assume, well, part of the other party or someone else.
But now with the note takers, yeah, I mean, we were talking about this the other day. What would stop someone from sneaking into a meeting, but renaming their zoom account to, you know, one of these products and keeping their camera off and staying on mute, they’d probably look identical to one of those.
Jason: So, yeah, I mean, it’s just things to think about and anytime you’re, dealing with sensitive information in a meeting and you think it’s private.
Hey, very close attention. Who’s in that meeting. both human and bot.
Jason: Yeah.
Dave: I’m certainly guilty of this.
Sometimes it’s just more from a, let’s see, how should I say this? A lazy slash convenience standpoint. Where you initiate a zoom meeting and you’re like, ah, whatever, just let anyone in.
But what you’re saying is really, you know, pay close attention to who you’re actually letting in. So putting people into a waiting room and making sure that they are who they say they are is probably a good best practice around there.
So, in terms of Lily for. From, I don’t think this is asking legal advice, is it?
But from, from a legal standpoint, are there things that we have to be concerned about when, we’re just firing up, zoom or an AI bot in terms of, I remember it’s kind of like, way before the pandemic. Right. I always used to say like, hey, is it, is it okay if I record this meeting?
Mostly from a permission standpoint, because I want to make sure people understand where we’re recording.
Some states require consent before recording
But, are there other considerations beyond that in today’s, you know, post COVID everyone’s using zoom and do we still need to ask that, have that permission structure or is it automatically assumed, do we need to be concerned about that state by state?
I don’t, I don’t know anything about that.
Lily: Sure, yes,
So there are quite a few states, including California, where it is an all party consent state,
which means that all parties to a phone communication, a video communication needs to consent to the recording.
Zoom is very interesting because they’ve actually been through the ringer
a few times with different types of privacy class actions.
For instance, they actually paid out an $85 million dollar privacy settlement as a result of a class action, claiming that they inadvertently allowed third parties to come in and intercept calls or record calls without the parties consent.
So this is a real issue.
You’ll notice that a lot of platforms have now a recording notice or some other recording alert for kind of the default platform recording.
But that doesn’t mean that it covers your own use of an AI bot or an AI recording system if it’s not integrated with the platform and so it doesn’t trigger that notice. So, again, best practice, still a good idea to alert people what you’re doing.
Another thing to be aware of is that there are AI laws that govern notice and consent requirements if you’re using AI for certain activities.
For instance, if you are engaging in interviews and you’re using AI in order to assess an interviewee. Then you may have certain notice requirements in certain states.
And now with the new EU AI Act, there’s going to be requirements going forward to notify individuals if you’re using AI, again, for interview purposes, performance evaluations, testing and other things that might impact an individual’s rights and abilities and access to services.
Dave: Oh, wow. I didn’t even think about the EU and this is probably a topic for another day, but the whole GDPR and the privacy insecurity along with the California CCP, I’m sure there’s, there’s stuff that governs around that.
So that’s really good information to know.
From your experience, has there been any, issues in people using AI. And this is more like within the public domain, right?
But are there things that we can take a look at and that makes us, that we can get smarter on in terms of how people or how zoom was in a class action suit or any advice in that perspective, what we need to start thinking about.
Lily: I mean, definitely from a legal and privacy perspective.
At least in the United States, there’s a lot more permissiveness around using AI and around using people’s data as long as there’s notice regarding it.
And you’re giving individuals the ability to opt out of such recording and giving them certain rights to their data.
Again, if you’re going abroad, and you’re looking at the EU, then that’s a whole separate conversation, and it becomes more about opting into a process.
Dave: Got it.
Jason, in terms of selecting a particular AI transcription tool, is there anything that we should probably think about besides looking at the T’s and C’s and privacy from a technical standpoint, when we decide to use one platform or another? Cause I mean, there are so many of them out there.
Yeah, I would say
Risks of using free transcription tools
Jason: Lily touched on the whole freemium thing. If you’re not paying for a product, then chances are you’re the product, right?
That’s the case with so many things out there and in our digital world. Stop using free stuff.
I mean, look at the terms and conditions, look at the privacy policy, just look at, compare what you get from the paid versions of things and just understand what you’re dealing with.
Sure. There are certain use cases where the free version of something makes sense, including I mean, if all you’re doing is talking about, you know, you’re talking to friends or whatever.
And yeah, fine. But I mean, gosh, in my business, there’s no way we’d be using something that might compromise you know, the privacy.
I will add further outside of the meeting assistant, but in the, under the guise of the AI bot within the meeting, there are tools out there that are marketing as noise cancelling tools for the products we use like Zoom and Teams and all that, noise cancelling software.
Go buy noise cancelling hardware, because when you start, and I’m not naming any products, but when you start looking at the privacy policies of these noise cancelling products out there, really anything that joins a meeting or intercepts the audio on the meeting, there are free versions out there that are they’re allowed to in the privacy policy, some of them, they’re they own that data now.
They they’re allowed to do anything they want with it.
Potentially. I don’t know all the legalese, but so pay very close attention to the software that you and everyone in your company uses.
Corporate governance and use of AI tools
I think the real answer here is around governance and having governance around the use of AI and software in general technology in general in your organization.
There needs to be governance policies established, talked about, and they’re not policies to put in a drawer.
They’re policies that , should be really worked on and really, really thought through and then enforced throughout the entire organization.
And there are ways to enforce. to the point where it would be very difficult, if not impossible for users to use something that they shouldn’t use.
But it starts with a conversation with the leadership of the organization and usually some experts outside of the organization like Lily on the legal side, myself on the technical side, or folks like us that can help guide, help ask the right questions.
And ultimately come up with you know, what’s going to make sense for you and your business.
Dave: That’s a really good point on governance.
Back in my old it days and you know, Jason, what that was way back when, I mean, you can enforce policies so that you can have your, company computers locked down.
So they, you know, the end user can’t install things.
But with remote work and people using their own personal computers it really becomes less of something of that you can actually Enforced from a policy perspective, but it’s also a, I don’t know what you, what you guys call it today, but like a socialization standpoint, because if I’m at home and I’m on my own personal PC and I decided to install some software, but I’m using it for business purposes, it’s yeah, there should be a governance process that when you use any asset for a company purposes, it probably should go through the, the it department and have been vetted and approved.
Not just for us. It doesn’t crash your computer, but
Jason: And not just going through the IT department, but have the business leadership working with IT to define what should be and should not be allowed.
And having generally outside expertise guiding around that because most IT departments may not have that experience or perspective.
And, you know, if they’re working in that IT department every day, they’re not going to necessarily know what else is going on and all these other environments are working with, you know, providers or, you know, attorneys or anyone outside that does that for a living, that really understands governance and risk.
And there’s a lot of different folks out there, that, that can help with that. But I think that’s key too, is, is just business leaders really getting involved in defining, understanding and defining what should be allowed in the organization,
Final Thoughts
Dave: Cool. Are there, are there any last words of advice that we would like to give to you know, managers, business owners in terms of using these recording software and AI transcription tools?
Lily: I mean, like Jason mentioned, you know, have a policy in place even from a cost savings point of view, if you’re going to pay for an enterprise version of some software, make sure all your employees know about it and are using that rather than using company funds to develop or work with competing software.
And then also, you know, there are a lot of different settings in many of the default platforms that allow you to give notice that allow you to use waiting rooms and allow you to develop that functionality that limits data leakage.
So, again, just really get to know what you’re using in the company.
Dave: Yeah. Not just from a usability standpoint. Cause you know, that’s what I’m always looking at. Like, Oh my gosh, how easy is it to use this thing? But there’s things that need to be looked at behind the scenes.
Jason: But on the flip side, I would also add, don’t be too scared of technology not to adopt it, right?
I mean, it’s happening, whether we like it or not, like we’re entering this new AI world and don’t be so scared of it that you’re not going to.
You leverage the technology that’s out there because your competitors will, and you’ll be left behind.
In fact, you need to be very forward thinking. You need to be paying attention to what’s out there. What, you know, if chat GPT caught you off guard, and you were the last person in your family to learn about it, chances are, you’re not doing enough to stay in, in front of technology, right?
And that, that it’s just one of those things where, you know, you want it, you want to know what’s going on, you not want to know what’s out there and really pay attention because, you know, things are happening very quickly, but always have that mindset of security, privacy, governance, and be working with the right people, to make sure that, that you’re making good decisions.
Dave: Yeah.
I don’t think any, any of us here are saying that we should not use these AI transcription services.
It’s more along the lines of we have to be aware of how it works and what are the security and legal ramifications we need to think through before we hit that record button.
Very cool.
Outro
So thank you, Jason and Lily.
For those who would like to get in touch with them, I’m going to leave their contact information and the website below in the transcript.
Thanks everyone for listening and spending time with us.
There’s more great educational content and interviews for you at do what. works slash podcasts.
Jason Makevich, CEO, Greenlight Information Services
Metaverse Law’s Lily Li recently spoke at Arthur’s inaugural AI Fest. Lily was on a panel that spoke about legal considerations for enterprise use of AI.
The panel explored the complex legal landscape surrounding AI adoption in business. The expert panelists discussed regulatory compliance, data privacy, intellectual property, and ethical concerns, providing actionable insights for companies integrating AI into their operations. Attendees gained a deeper understanding of the potential legal risks and how to navigate them effectively to ensure responsible and compliant AI use in the enterprise.
Metaverse Law’s Lily Li recently spoke at the #Risk Digital UK/EU global livestream last week for two sessions: “A New Era of AI Governance” and “The Role of Technology in Modern Governance, Risk and Compliance.”
Lily’s presentations included an overview about artificial intelligence risks and discussed the latest developments in European Union and United States AI legislation and regulations. Major developments in the US include: — NY and Illinois AI bias legislation in employment — Colorado comprehensive AI legislation — California AI bills that have recently passed the legislature.
Lily also touched on how the results of the U.S. presidential election could impact the AI landscape, either through changes in FTC priority and state legislation, and the need for AI risk management and governance programs.
The law in this area is developing very quickly and will affect businesses in almost every industry. Keeping up with these changes is critical as businesses deploy and integrate AI into everyday operations.
#Risk Digital is one of a number of conferences organized by GRC World Forums, a producer of in-person and livestream educational events for governance, risk and compliance professionals.
