Meta's Llama model has memorized Harry Potter and ebonythe Sorcerer's Stoneso well that it can reproduce verbatim excerpts from 42 percent of the book, according to a new study.
Researchers from Stanford, Cornell, and West Virginia University analyzed dozens of books from the now-infamous Books3 dataset, a collection of pirated books used to train Meta's Llama models. Books3 is also at the center of a copyright infringement lawsuit against Meta,Kadrey v. Meta Platforms, Inc.The study's authors say their findings could have major implications for AI companies facing similar lawsuits.
According to the research paper, the Llama 3.1 model "memorizes some books, like Harry Potterand 1984, almost entirely." Specifically, the study found that Llama 3.1 has memorized 42 percent of the first Harry Potter book so well that it can reproduce verbatim excerpts at least 50 percent of the time. Overall, Llama 3.1 could reproduce excerpts from 91 percent of the book, though not as consistently.
"The extent of verbatim memorization of books from the Books3 dataset is more significant than previously described," said the paper. But the researchers also discovered that "memorization varies widely from model to model and from book to book within each model, as well as varying in different parts of individual books." For example, the study estimated that Llama 3.1 only memorized 0.13 percent of Sandman Slimby Richard Kadrey, one of the lead plaintiffs in the class action copyright suit against Meta.
So, while some of the paper's findings seem damning, don't call it a smoking gun for plaintiffs in AI copyright infringement cases.
"These results give everyone in the AI copyright debate something to latch on to," wrote journalist Timothy B. Lee in his Understanding AI newsletter. "Divergent results like these could cast doubt on whether it makes sense to lump J.K. Rowling, Richard Kadrey, and thousands of other authors together in a single mass lawsuit. And that could work in Meta’s favor, since most authors lack the resources to file individual lawsuits."
Why is Llama able to reproduce some books more than others? "I suspect that the difference is because Harry Potter is a much more famous book. It's widely quoted and I'm sure that substantial excerpts from it on third-party websites found their way into the training data on the web," said James Grimmelmann, a professor of digital and information law at Cornell University, who was cited in the paper.
What this also shows, Grimmelmann said, is that "AI companies can make choices that increase or reduce memorization. It's not an inevitable feature of AI; they have control over it."
Meta and other AI companies have argued that using copyrighted works to train their models is protected under fair use, a complex legal doctrine. However, the extent of memorization could complicate those arguments.
“Yes, I do think that the likelihood that LLMs are memorizing more than previously thought changes the copyright analysis,” Robert Brauneis, a professor with the George Washington University Law School, said in an email to Mashable. He concluded that the study’s findings could ultimately weaken Meta’s fair use argument.
We asked Meta for comment on the study's findings, and we'll update this article if we receive a response.
Disclosure: Ziff Davis, Mashable’s parent company, in April filed a lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.
Topics Artificial Intelligence Meta
7 wild Sora videos blowing up social media after its launch
How to Lie: Donald Trump, Jr. fails in the family trade
What Philly’s DA Win Looked Like from the DJ Booth
Trump’s Family Leave Shell Game
New 'browser syncjacking' cyberattack lets hackers take over your computer via Chrome
If They Are Not Coming For You Today
Putting Millions Back to Work
A Rich Fable
Many Indigenous communities still lack broadband internet. Here's why.
Putting People First in Pennsylvania
Elon Musk's DOGE.gov website can apparently be edited by anyone
Holy Machiavels
Poodles, doodles, and memory-challenged gnomes
Juneteenth, From Atlanta to Oakland
Thirteen Steam Features You Probably Don't Know About
Putting Millions Back to Work
Get Thee to a City of Ladies
Julian Assange and the Banality of Access
Wordle today: The answer and hints for January 23, 2025
Who Framed Augmented Reality?
Elon Musk's Starlink satellite internet shows big speed jump in first official yearYes, there are 100 million rogue black holes wandering our galaxyA list of companies reimbursing employees for abortionSoftware engineer creates AI that identifies anonymous faces in WWII photosHow to follow the 2018 midterms without losing your sanityThis stray cat who crashed a fashion show is my favorite supermodelSoftware engineer creates AI that identifies anonymous faces in WWII photosApple Store is down in the U.S.The 10 types of trolls you'll spot in the wildSpotify iceberg: How to get the latest viral music chart Icebergify'Wordle' today: Get the answer, hints for July 1RoastMe is a troll's paradise, where people ask to be insultedSoftware engineer creates AI that identifies anonymous faces in WWII photosHow period apps are affected by the overturning of Roe v. WadeWatch this donkey sing a beautiful aria'Battlefield 2042' Specialists, explainedInstagram has been blocking posts that mention abortionRoss from 'Friends' responds to his lookalike thiefThe 'falling stars' challenge is the latest way rich kids are flaunting their wealthWatch this joyous pup jump into a gigantic leaf pile 'On Becoming a God' shoves an American dream where the sun won't shine J.K. Rowling shuts down Trump's inauguration with Horcrux Twitter's Trust and Safety Council feels ignored if you can believe it Facebook emails shed light on early days of Cambridge Analytica scandal Disturbing footage shows car circling street moments before it drives into crowd This is (probably) what the OnePlus 7T will look like This is what Porsche's all Backstreet Boy says he 'has faith' in Trump's balls This man wants to make dating great again for Trump supporters Uber will pay $20 million to settle charges that it misled drivers about pay Hardcore retiree defies knitting group rules by trolling Donald Trump Here's some of the major newspaper front pages from Inauguration Day Rivian electric truck is the ultimate camping accessory in new video In 'Sound Stories,' Christian Marclay makes music out of Snapchats Chrissy Teigen once again uses stretch marks to remind us that she's normal 'World of Warcraft Classic' developers reminisce about old Azeroth Tom Holland assures fans he isn't going to stop being Spider Obamas reveal their carefully curated 44 Pixar's 'Soul': Story, art, and cast revealed at D23 How capitalism made niche nerd cultures cool