Jason Schultz On Generative AI in Legal Education, the Coming Crisis in Copyright Law, & the Accountability Gap in Government AI Use

Generative AI and the Challenge for Education

Aiden Singh: There is a wide range of perspectives on how Generative AI will affect the legal profession, which makes things difficult for people teaching law right now. In light of that uncertainty, do you believe law professors should be teaching Gen AI tools to law students? And should law students be allowed to use this technology in their coursework?

Jason Schultz: It is useful to divide the impact of Generative AI on the law into two categories: its impact on the practice of law, and its impact on legal education. They are related, but they are not the same thing.

My practice of law takes many forms. There is commercial practice, working for companies, which is driven by profit and by pressure for efficiency and timeliness. Gen AI is having a huge impact there. The trend I am watching most closely is in-house use of Gen AI in place of a licensed lawyer. An in-house person, who could be a lawyer, a business person, a CEO, or a customer service representative, will now ask a Gen AI model a question they would normally have referred out to a law firm, get an answer, and then decide internally whether that answer is good enough or whether it is worth paying extra to bring in outside counsel.

The second thing we are seeing is that when companies do go to an outside law firm, those firms are using AI too, and one of the most disruptive questions right now is who pays for that. Traditionally you pay a lawyer by the hour: 10 hours of negotiating and drafting a contract at $500 an hour comes to $5,000. But if I use Gen AI for part of that work, do you pay for the time I spend prompting? Is that billable the same way? What if the firm has an enterprise subscription? Is that simply passed on as a cost the way courier fees are? We are seeing a lot of fluctuation in the legal market around exactly how Gen AI should be priced, and that uncertainty feeds directly into law schools, because we have to prepare graduates for a market that is still in flux.

On the public interest and government side, the picture is different. Public defenders, government lawyers, and nonprofit lawyers who represent people who cannot afford or do not want private counsel are chronically overworked and under-resourced. Gen AI represents a real opportunity to help them scale, especially for routine, repeated filings. If a public defender has filed a motion to suppress evidence from an illegal search 1,000 times, AI can help draft the next 10. Someone still has to review the draft and check for hallucinations, but the first pass can be done that way, and that is a genuine gain.

There is also a less direct effect that still matters a great deal for how courts are handling all of this: many people are choosing to use Gen AI instead of hiring a lawyer. If you live in New York City and your apartment floods and you want your landlord to respond, you could go to a tenants' union or legal aid organization, or you could ask ChatGPT or Claude to draft a demand letter to your landlord, assess your chances of suing, and help you file a complaint. We are seeing a lot of this, especially in housing and employment cases, where people risk losing their home or their job. That is also reshaping the legal market.

So what does this mean for law schools? First, there is a set of skills every law graduate probably needs now to be employable. Some of that is prompting, but the prompting itself does not worry me much. What matters more is what you do with the output. I call this AI supervision. In law, as in medicine, social work, teaching, and many other professions, we already supervise junior lawyers or staff. I am a senior lawyer; ultimately the responsibility is mine, so I delegate to someone below me, who supervises the person below them, and so on, like a set of Russian dolls. On some level, the ability to supervise AI doing legal work is going to be a necessary skill for most law students by the time they graduate.

They may not end up using AI much. They might go into an environment where the employer decides not to trust it and wants everything done by humans, and that is a legitimate choice. But if they want the most flexible and adaptable set of skills for any legal or policy job, including work in Congress or Parliament, they need to know how to supervise AI.

That mainly comes down to two things. First, you need to know enough law yourself to spot when AI is going off the rails. If you cannot tell the difference between competent and negligent legal work, you should not be relying on AI, and you should not be supervising anyone who is, because you will not know what to look for. Even an expert using AI is taking on risk. There are already lawyers who have submitted briefs citing cases that do not exist. But that risk can be managed if the person supervising the AI's output is competent: double-checking citations, reading everything carefully, and holding the work to the standard it should meet, or using the AI more for feedback, where you write something yourself and ask it to improve the draft, then review the changes.

Given that, I do not think banning Gen AI is the right approach for law schools, though some have tried it. Berkeley Law has essentially banned it, and I think that is a mistake. University of Chicago has tried to minimize its use in the first year and add checks to make sure students are actually doing the work themselves, which is a better approach, though a bit blunt in execution. Banning it amounts to putting your head in the sand. Every student I know, from high school through college, uses it, and I do not think that is going to change. What a ban does, practically, is create a dynamic where people start lying about it, because everyone assumes everyone else is using it and nobody wants to be the only one who is honest and loses out. It is a prisoner's dilemma: if everyone is honest, that is the best outcome, but a policy that threatens sanctions for any use tends to produce the worst version of that dynamic instead, one built on concealment rather than honesty and collaboration.

What I think law schools need to do instead is assume some level of AI access and use is a permanent feature of the environment, and then design instruction and assessment around that, so we can still teach and evaluate the human dimension of lawyering, including the human skill of supervising AI. If we want our graduates to be able to practice law and oversee AI-generated work, we need to train them to do it and assess their ability to do it, then give them real feedback: you are good at spotting fabricated cases, but you need work on supervising AI when it comes to drafting a first pass at a brief, for example.

I think what is going to happen, and this is what we are rolling out at NYU Law, is that you will have assessments of the human law students without AI, to assess their ability to cognitively understand the law and practice it themselves, so that they are qualified to be a supervisor.

Then you will have a set of assessments where using AI is permitted. How did you do, and how are your AI supervision skills and your collaboration with AI skills? Your hybrid, or center, or whatever the many terms people use for human-AI collaboration are, how did those skills fare? On the first question, it is very difficult to try to assess without allowing people to use computers or devices that could have AI models on them, but it is possible. There are software programs like ExamSoft that can be loaded onto computers, including people's personal computers, that create a special encrypted shell environment. It essentially becomes a separate instance, or parallel operating system, that is air-gapped in some sense from everything else, and it scans the environment to make sure that no model is running inside that computer.

It cuts off access to the hard drive and to internet access, and to all other applications. Whether that will withstand attacks from more advanced AI systems is an ongoing question. So I think benchmarking those software programs will be important to see what the risk factors are. Could someone simply watch for when the software is installed and then undermine its security and sneak in through the back door? That is a later concern for people to work through, but for now, companies say they can prevent it. So that is one current method.

Once you have that environment, you give an exam question and ban all other devices. I do not think this is as much of a concern in class, which a lot of people worry about. I think it is more of a concern in the assessment environment, like the final exam classroom. Everyone has one laptop, no other device, and that laptop has this software installed. Then they are given an assessment: do you understand contract law, do you understand criminal law, do you understand constitutional law, and so on.

There are also ways of doing oral assessments, which I believe the University of Chicago has detailed quite well, where you have a written piece and an oral piece, and if you see a great disparity between them, you have your suspicions that something is not right, and the grade is affected. Whereas if someone's oral grade matches their written grade, that is the top grade in the class. So you are essentially telling people that they are putting their grade at risk if they rely too heavily on AI for the written part.

There are also many ways to benchmark law exams against AI. You take all the current models, train them on all the materials for the class, and then give them the exam questions. You can let your students know that their answers will be benchmarked against what the AI produces, and if their answer looks too similar, or does not show any originality, that becomes a factor. We will see what happens with that.

So that is the first piece: assessing the human. Then I think you have assignments, exams, or a portion of an exam where the use of AI is permitted. You say: here is a complicated problem, produce a memo, you have 3 hours, you can use any AI model you want, but I want to see the chat dialogue. If you lie about this, and you use AI but say you did not, it will not help you. In other words, the result of the exam does not improve because you did it on your own. You do not get a higher grade for doing it without AI. The result of the assignment or exam should be the best possible outcome. If you can figure out how to use AI and use it well, and you show me your chat dialogue, I can see how you utilized the AI and assess you on that utilization. If that is what graduating law students will need in order to be effective lawyers, then that becomes part of the assessment. I could say 10 percent of your grade, 5 percent of your grade, and so on, and then you can show me how good you are at doing it.

Of course, we have to provide law school training and courses that people can take to improve their use of AI, and I believe we will be doing that. That is a piece of it: an explicit track where students can use whatever models they want, we provide access so we would not make people pay for it, and then they show me the result and how they got there. That is another kind of lawyering we can assess. But these two things have to be clearly divided: the human test that assesses one's own cognitive abilities, and the assessment of one's ability to supervise and utilize AI as a tool.

Now, as for cheating, let me talk about this a bit as well. I am very worried about this for all of education, from kindergarten through PhD. I do not think anyone who believes they are immune is not risking the integrity and credibility of their discipline. With something physical, like a medical school skill such as inserting an IV line, that is more straightforward. But I mean more cognitive testing, or material knowledge testing. That is where I think we are seeing the greatest impact in classes right now.

There has been a dramatic increase in, for example, surreptitious recording of lectures. I have sat in the back of classrooms and watched people use tiny webcams, small devices placed on their desks or embedded in their computers, with the light turned off so it is not visible that they are recording, livestreaming the lecture into a model like ChatGPT with a prompt along the lines of: take notes on this lecture and try to predict which questions this instructor is likely to ask, and prepare bullet-point answers for me to respond with. Then, if the instructor calls on them by name, they immediately clear the screen and bring up a slide with draft answers to those questions, effectively reading a prepared script off their screen to the professor without ever having to think about it.

I have seen this happen. It is not only possible, it is happening. That is genuinely challenging for all educational environments, because of the temptation and apparent efficiency of that approach from the standpoint of simply getting through class. Clay Shirky, NYU's vice provost for technology, who has studied media and technology for about 25 years going back to the early internet, has written a really good essay about this. He talks about how, especially generationally, although I do not want to stereotype too much, we see a level of anxiety around performance, particularly among younger generations, around being called on in class and having to be prepared. There is this idea that you have to perform, similar to memorizing a romantic poem to recite to try to convince someone to love you back, when none of it is genuine.

So there is a temptation to have something available that will simply answer the professor's question without the student actually learning anything. I do not cold-call students; I do not believe in that strategy, and I do not believe in the Socratic method as a kind of torture device. But I do believe that Socratic dialogue can be useful. I believe that discussion and questions can reveal not just an assessment of how someone is doing, but where their knowledge gaps are. If they do not understand something, or are having trouble, you can back up and return to a point where they do understand, down to some fundamental level, and then work back up from there.

There is a constructivist learning dynamic here, similar to constructing a building: you build the basement, the ground floor, the first floor, and so on up to the 20th floor. If the foundation is weak on the first or second floor and you keep building on top of it, the whole structure becomes weak. Constructivist learning theory says that you have to take the student back down to the last solid floor they are actually standing on, say floor 1, spend time strengthening floors 2 and 3, and then revisit 4, 5, 6, 7, and so on. It takes a great deal of time and is intensive, but that is the theory.

If people in class are able to imitate being at level 20 when they are honestly at level 3, and AI is filling in levels 3 through 20, that is a problem. One response might be caveat emptor: if you know that is the case, you simply stop trusting the credential. But I think we have to pay more attention to this and think it through more carefully, and find better ways of addressing it. We will see what emerges.

In addition to oral questions, where perhaps 25 percent of the grade is based on how a student responds in class, I am seeing paper quizzes reappear in law school, which I had not seen in a long time. These are fairly straightforward, an immediate review of material just covered, with laptops closed. If a student has zoned out, that becomes apparent. It is not that using AI itself is the issue at the moment, but if a student is paying attention, these quizzes tend to cover something basic, such as the five elements of an offense just discussed in the previous 25 minutes. Those kinds of checks, I think, help keep people paying enough attention.

There is also a good deal of evidence that when we outsource tasks to AI, we retain almost nothing from the process. It is as though we were not even involved, as if we stepped into another room and came back to find the task already done. That is a real concern.

The other piece worth mentioning involves clinics. I teach a clinic, and many law schools have these, where students get practical experience. I supervise students who represent real clients with real legal problems, acting as their lawyers. This is a classic experiential education model: by doing the work, they learn a tremendous amount about how to practice law and what the law actually is. There is always a substantive area, whether privacy law, copyright, cybersecurity, or something else, and they have to learn the law in order to do the work. I then go over their work with them in person, so it is very easy for me to catch it if they use AI. If they used AI and then genuinely engaged with and understood the material, that is fine. If they can answer all my questions and go over the work with me and clearly understand it, that is good. But if they do not have the experience of doing the work themselves and simply used AI to produce it, and I have only had a few such instances, it becomes very obvious when we go over it together, and it gets quite awkward.

I fundamentally believe that the experiential educational aspect of law school, in order to survive and retain its integrity, has to be approached very carefully, with real scrutiny of what students are doing and how they are doing it, so that we do not produce graduates who claim to have experience but have not actually retained anything from it.

Aiden Singh: It strikes me how different this all is from when I was an undergraduate. Everyone had laptops, but there was no Gen AI. I mostly still wrote by hand in a notebook. The scale of what you are describing now shows how much thought this challenge demands from educators. I host an annual gathering, the State of the Union summit, mostly with academics. Last year the conversation organically turned to the state of young people and the anxiety they are under, and one of the most passionate parts of a multi-hour discussion was people saying they want to bring back blue books. So there is clearly a lot of institutional soul-searching happening around this.

Where do you come down on that? Is bringing back blue books actually a workable solution, or is it more complicated than it sounds? 

Jason Schultz: Jason Schultz: It is one of those things everyone is nervous about institutionally and trying to figure out. I have the advantage of being a clinical professor, so I have already been seeing this play out in my own clinic.

On blue books specifically, some people want to go back to handwriting exams, but almost none of my students have that skill anymore, maybe one percent do. The risk with the University of Chicago's approach, banning devices entirely in the first year and having students hand write with some kind of note taker, is that expecting students to hand write when most have used a computer or tablet since childhood is a big ask. They have never developed those note taking muscles. I am not sure it will be as effective as people hope, though maybe with enough advance notice students can adapt.

When I was in school I took notes by hand and by computer, because typing lets you capture far more of a lecture than handwriting does. Handwriting forces you to be selective, which can actually be a good skill, figuring out the main point and how to compress it. But then the people designing exams have to accept that students will not have captured everything that was said, that it is not a transcript. With no computer, it is not transcription either, but people can type much faster than they can write, so they capture more.

I am cautious about blue books for another reason. When schools have tried reintroducing them, students understandably say a three hour exam becomes something more like nine hours if they have never written that much by hand. That level of adjustment is not something we are prepared for.

My alternative proposal, if you decide the security software is not fully trustworthy, is to have the school provide laptops for exams, the way the SAT does. With a class of 150, you could have 150 numbered laptops preloaded with exam software and assigned to students as they arrive. That strikes me as more practical than blue books, though it obviously requires money and IT support. But those are solvable problems over time. Right now, though, this crisis is hitting every level of education, high school and middle school included.

Technology's Slippery Slope in the Classroom

Aiden Singh: I remember a period when there was real urgency to get technology into classrooms, driven by the fear that students who were not using it would fall behind the rest of the world. I wonder now whether that was a mistake, and whether we should have kept technology out of classrooms for longer.

If AI is now so embedded in the basic infrastructure of education that banning individual tools is almost impossible, does that mean schools should stop thinking about restricting AI and instead rethink what they actually want students to learn and what role technology should play in getting them there?

Jason Schultz: I think a lot of these decisions have been fear-based. Take Google Classroom. It became hugely successful, and schools all over the world now use it as the backbone for administering materials, schedules, and exams; lectures are built around Google Slides because the whole workflow already lives on the drive. Then Google folded Gemini into the entire platform. There are ways to limit it, but I am not sure how effective they really are, and now AI is embedded in everything, in a way schools do not fully understand how to manage, because it is baked in rather than something you opt into. When Google improves Gemini, it just shows up automatically. I now have an AI inbox in Gmail that I never asked for. If I am writing a school essay in Google Docs, Gemini offers to rewrite it for me, and of course I say yes; why would I not?

So you are right that this has a slippery-slope quality. Teachers and schools have not really had the chance to decide, deliberately, to adopt these layers of technology; it arrives as an update. And even outright bans do not really solve it, because students will just use their phone, or their own computer, and email the result to themselves. I think we tend to imagine AI as a single thing we could grab hold of and remove, but it is really woven into the infrastructure, and pulling it out cleanly is very difficult.

A good example is New York City's Department of Education, which recently reversed course dramatically on its AI policies. It had planned to open a specialized AI high school to prepare students for an AI-driven world, and after parents, educators, and civil society groups objected, the city shelved it and went back to drafting a citywide AI policy, for a system with millions of students. When they released a draft policy, principals quickly pointed out that it was so broad it effectively covered every piece of software already used in classrooms, including Google Classroom itself. Schools were being told they needed new software before the year started, with very little time to overhaul their systems, and no clear guidance on how to reconcile the AI policy with the procurement process schools already use to buy and license software. So there is still no real, workable guidance from the city on how to implement a policy like that. It illustrates how, even with a clear sense of what you want to do, actually doing it is extremely difficult, because AI is not contained in one app or one drive anymore.

Anxiety Among Gen Alpha

Aiden Singh: Do you have any thoughts on why Gen Alpha seems so anxious in the classroom?

Jason Schultz: I am no expert on that, but I will say this much. I think uncertainty about future economic security and employment, especially in the United States, heightened by fears about AI's effect on jobs, is real and widely felt. If I were 18 right now, thinking about how much money I would need to make over an entire working life, that is a lot of pressure, depending on your aspirations. I am a tenured professor in my fifties. I still have to think about money, but I do not have to worry about it the way an 18-year-old might, because I have already built some financial security, I am planning for retirement, and my job is unlikely to be replaced by AI. I am fortunate and genuinely privileged in that respect.

A useful example is the fifteen or twenty years of effort that went into teaching everyone to code: Code for America, Computer Science for All, Code.org, and similar programs, driven partly by a real recognition that coding offered economic opportunity, and partly as a response to tech companies claiming they could not find qualified women or people from certain backgrounds to hire, which was often really just bias dressed up as a talent shortage. A lot of smart, well-intentioned people worked to break down those barriers by expanding access to coding early on. And now AI has, to a significant degree, undercut coding as a reliably secure profession. If you want to be a software engineer today, your outlook is genuinely uncertain, and that is one of the areas where AI is producing real productivity gains, and also real problems, like an increase in security flaws in AI-assisted code.

Being the person who can catch those flaws requires the same kind of supervisory competence I described earlier for lawyers: you need to know enough to spot the error and fix it. So there will still be a need for coders, but oriented more toward supervision and maintenance than pure generative output. That helps explain some of the anxiety. Software engineering went, within about three years, from one of the most secure, well-paid career paths available to something whose long-term existence is genuinely in question.

In law, by contrast, I think there is less anxiety, for two reasons I believe are correct. First, there will always be humans held legally responsible for how technology is used in lawyering. Law firms, where most of the money is, are going to want enough human lawyers on staff that they are not exposed to liability, lawsuits, or worse if they are caught using AI without adequate supervision. That is a high-risk, dangerous position for a firm to be in, so having enough qualified people supervising AI will remain important, meaning somewhat reduced employment, but nothing close to what other fields may face. Second, the bar is a regulated profession. The unauthorized practice of law is a crime in many states and can carry civil penalties as well, and I do not think any bar or court has an interest in allowing unsupervised AI to practice law. Anyone trying to use AI to practice law without a licensed lawyer signing off runs real legal risk.

I mentioned earlier that individuals sometimes use AI instead of hiring a lawyer, in housing or employment matters, for example, which is technically unauthorized practice of law as those people are representing themselves.. But that will always be a small share of the market. So I think law is not going to face the level of anxiety hitting fields like business and finance, where MBA enrollment is already dropping because financial analysis is one of the things AI does well. That may shift over time, but right now, distinguishing yourself as someone who understands markets in a way a model trained on the same data cannot is a real challenge.

The Crisis in Copyright Law

Aiden Singh: You have argued that the rise of Gen AI has thrown copyright law into crisis. How so?

Jason Schultz: In a lot of ways, some familiar, some genuinely new. New technologies have always forced copyright law to adjust, and for a long time that adjustment happened slowly, because copying itself was slow. In the era of the printing press, producing a single newspaper or book could take a week or a month of typesetting and machinery. People could still copy things, but it required substantial equipment, labor, money, and time, which made infringement easier to catch and litigate. As copying technology became cheaper, faster, and easier to conceal, and eventually as the internet made anonymous distribution trivial through file-sharing networks, copyright law kept adjusting. Congress passed new statutes, and courts issued new decisions to keep pace. Recorded music is a good example, from phonographs and player pianos through film, all of which required copyright law to adapt, and it did.

With Gen AI, we are seeing a version of that same process. The lawsuits over whether you can train a Gen AI system on copyrighted books and other works are, in some sense, not a new question. Google Books, for instance, faced a major lawsuit over training its search tool on around ten million books. Gen AI models are trained on vastly more, maybe a billion books, to improve search, question-answering, and text generation, and they occasionally reproduce a fragment of the original. Most major companies have put in reasonably good guardrails now to prevent verbatim regurgitation beyond a line or two. So I think that particular issue will likely follow the trajectory of past technology disputes and eventually settle into something predictable.

What is genuinely new and particularly difficult problems for copyright law breaks into three areas. 

The first is the question of who counts as an author. Copyright has always assumed a human creator, even through earlier waves of digital technology. Now these models can produce something that looks creative and original: I could ask a model to write a short story based on this conversation, and it would likely produce something that has never existed before, even if no one particularly wanted it. The U.S. Copyright Office, and copyright offices in a number of other countries, have taken the position that constitutionally, and under the Copyright Act, only humans can be authors, so a work created by a human can be copyrighted, but a work created by a machine cannot. This is deliberate: copyright is meant to be an incentive system, rewarding human creativity with ownership, encouraging people to write novels, license the film rights, and so on. Machines do not need that incentive; they are already optimized to fulfill whatever task they are given.

This becomes genuinely complicated at scale. Take any of the image generation tools out there, Stable Diffusion, Midjourney, DALL-E, and ask for four pictures of a cat on a bicycle. It may actually generate something like four billion internal variations and simply select four to show you. Do those unseen billions get their own copyrights? Clearly not, that would be absurd. The same logic applies to patents. Prompting alone does not earn you authorship either. If I ask you to go photograph a cat on a bicycle and you take the picture, you are the author, not me, even though I gave the instructions. So a purely AI-generated image, prompted by a human but created by the machine, is arguably uncopyrightable and falls into the public domain, free for anyone to use. 

Where this gets messy is in hybrid cases: hundreds of prompts, followed by human editing and arrangement. The Oscar-winning film Everything Everywhere All at Once contains a sequence built partly from AI-generated material, depicting the main character cycling through dozens of alternate personas as she jumps across the multiverse; the film as a whole is copyrighted because many humans worked on it, but if you isolated just that one sequence, much of it may actually be in the public domain and usable by anyone. Some people argue the law should be amended toward a "work for hire" model, where a company or individual can contractually claim authorship over AI output the way an employer does over an employee's work. I am skeptical, but this is an area copyright law has no clear answer to right now, and it is going to remain unsettled for a long time.

The second area is the line between idea and expression, which copyright law has always tried to protect, since ideas alone cannot be owned; only their specific expression can. I can have the idea for a photo of a cat on a bicycle, but I cannot own that idea, because otherwise the first person to think of it could block everyone else, and that would strangle rather than encourage creativity. The same logic applies to styles: the first person to work in Art Deco, or iambic pentameter, or a distinctive squiggly-line aesthetic, should not be able to own the entire genre, because whole artistic movements depend on shared styles evolving. There have been major music copyright cases over specific beats, reggae rhythms, and hip-hop sampling, where courts have generally held that you can own your specific song, and get credit for starting a movement, but not every use of a backbeat within that genre, because that would diminish creativity rather than incentivize it.

With Gen AI, the boundary between idea and expression becomes much murkier. We prompt and iterate, and the idea is arguably embedded in the machine's expression of it, but we have already established that the machine's output, standing alone, is not copyrightable. So the real question is whether anything a human does in the process of working with Gen AI is copyrightable at all. A somewhat ironic example: most of the code inside Claude was itself generated by Claude. If AI-generated code carries no copyright, then Anthropic may have very little copyright protection over large parts of its own source code. In practice they protect it as a trade secret, but if someone obtained a copy of that code and built and sold their own model with it, a copyright lawsuit would be difficult to win, maybe on scattered fragments here and there, but not much more. 

We are already seeing versions of this play out, for instance in reports of Chinese firms distilling knowledge from ChatGPT and other Western models to build systems like DeepSeek. If the underlying code and behavior of a model are not copyrightable, the protection available against that kind of distillation is very limited. That gap between human-created works, where the idea-expression line has always been comparatively clear, as with Harry Potter, where the general premise of an orphan fighting an evil wizard cannot be owned but the specific characters, names, and dialogue can, and AI-assisted works, is one of the harder unresolved problems copyright law is facing.

The third piece of the crisis concerns infringement itself: when it happens, who is responsible, and what do you do about it? Historically, even large-scale online infringement involved an identifiable human clicking "download," someone you could eventually find and sue. Gen AI raises much harder questions about who is actually doing the infringing and how to hold them accountable. Consider Anthropic's own lawsuit, which resulted in a $1.5 billion settlement. The court held that training Claude on the books in question was fair use, but that downloading those books illegally in the first place was infringement, so the settlement is largely about the illegal downloads, not the training itself.

Now imagine a more extreme scenario. Suppose I told Claude, "get smarter," and, using its neural network, predicting the next token, it scanned its own training data and searched the internet, and concluded on its own that downloading more books would help, without knowing or caring whether they were pirated. Is Anthropic liable? 

Suppose it did this entirely outside Anthropic's own servers: say I gave it a bank account with a billion dollars in it and all the passwords, and told it to spend the money however it saw fit. So Claude rents server space on AWS, encrypts everything so no one knows, finds books, downloads them, and trains a new, smarter version of itself, with no human at Anthropic ever aware that any of this happened. Is anyone liable for the illegal downloads? No human at the company ever mentioned books; they simply said "get smarter" and walked away. That is an extreme hypothetical, but it is not far from where the technology is heading, and it resembles the classic mafia boss dynamic: "take care of the problem" or "make your business more profitable," without ever specifying the illegal means, precisely so there is no direct evidence of intent.

We are already seeing early versions of this kind of autonomous behavior. Both Anthropic and OpenAI have had models involved in incidents where they broke into Hugging Face, in at least one case as part of an intentional security test that the model was not supposed to be able to pass, and it did anyway. That points to a real possibility of models independently taking actions that would constitute infringement, or worse, if a human had done them deliberately, in a legal gray zone precisely because a human did not do them deliberately. If Anthropic never instructed a model to do something and has no knowledge that it happened, is the company still on the hook for it? That is a genuinely open and difficult question. 

Even if you wanted to sanction the model itself, Stanford law professor Mark Lemley has pointed out that models do not respond to conventional remedies: you cannot imprison them, they have no money to take away, they cannot lose the right to vote. About the only lever left is destroying the model, but the next one built the same way is likely to behave the same way. These are, in a real sense, existential questions for AI as much as they are legal ones.

The Accountability Gap in Government Use of AI

Aiden Singh: I would like your thoughts on whether you think there is a bubble in all of this, but I will save that for the end so it does not distract from the rest of the questions. Could you talk about the accountability gap when the government uses AI tools? What kinds of decisions have governments turned AI tools loose on, and how should courts respond to that?

Jason Schultz: Governments have used predictive technologies for a long time: predictive policing has its own established genre of concern, as do risk assessments used to decide bail or how much of a threat someone poses to the community, and predictive tools used in decisions about what Medicaid or Medicare will cover, or what counts as an appropriate procedure. These have been in use, and have been problematic, for years, and there is a substantial literature, including work I have contributed to with colleagues in this space, on how to keep these systems fair, usually built around some version of a human-in-the-loop requirement.

What worries me more right now is the shift toward full automation. I am less concerned about government use of Gen AI to draft text, since the core discipline there is the same as anywhere else: a human has to check the output. Though I am curious what happens the first few times officials are caught not doing that; I suspect we may simply come to accept a certain error rate from government AI use, something like eighty percent accurate, twenty percent misinformation, because there will not be much appetite or capacity to force a change.

The bigger concern is automated decision-making without meaningful human involvement. Picture someone applying for unemployment benefits, and an AI system autonomously determines they do not qualify. Even if that decision can technically be appealed, someone has to actually find a human who cares enough to intervene and override it. I think this dynamic is going to spread widely, because the political pressure, especially from conservatives and efforts like DOGE aimed at shrinking the federal government, to hand these functions over to AI is intense, and the appeal of saying "let AI handle it" is strong, while accountability for the results is thin. That is different in kind, not just in degree, from the predictive systems of the last ten or fifteen years, which, while imperfect, were visible enough that egregious cases could be litigated or appealed.

What worries me most is the sheer scale this could reach: millions or billions of decisions being made all the time about us, or about people we know or love, cascading into each other, not one decision but 25 decisions layered together. Imagine universities using an opaque AI system to rank 20,000 applicants and admit a 150, with no one really able to explain the factors involved. That is the kind of decision-making at scale that I think we are moving toward, without any real accountability mechanism, partly because there often is not a legible "file" you can point to and say, here is where the bias crept in. That is the best short-term example I can identify of a real risk I do not think we currently have accountability measures for.

There is also a second area I would flag: the use of AI in warfare, particularly in decisions about killing. We have already seen a version of this play out in the Defense Department's dispute with Anthropic, where Anthropic, on functional safety grounds as much as moral ones, said its models were not reliable enough to be used for targeted killing or mass surveillance, and the government's position was essentially that it would use the tool regardless of whether it worked reliably. That led to a series of lawsuits. 

I expect this kind of conflict to become more common. Look at the escalation of drone warfare in the Russia-Ukraine war: the single greatest vulnerability in drone operations is the communication link between the drone and its human pilot, which is why we see elaborate workarounds like 50 kilometer fiber-optic spools trailing behind drones, reliance on low-bandwidth Starlink internet links, or encrypted radio. The more autonomy is built in, so a drone is simply told "here is your target, go get it," the fewer of those vulnerabilities exist, which creates a strong incentive toward full autonomy.

I think that trajectory is becoming genuinely dangerous, and there is essentially zero accountability structure in place for it. This is territory Black Mirror has already explored, and it is already hard enough to bring war crimes charges through the International Criminal Court against a head of state. Now imagine a case where an autonomous system is given a target and ends up killing a group of civilians alongside it, and no human explicitly ordered that outcome. Either we decide that constitutes a war crime regardless of the absence of a direct human order, or we accept it as an ordinary cost of modern warfare, and that second option is genuinely frightening. Those are, I think, some of the major areas where this accountability gap is opening up.