California Admits AI Was Used to Write Bar Exam Plagued by Problems

Now here's a legal-drama-worthy twist in the recent spate of dumb lawyers getting caught using AI. It turns out that the very bar exam administered to aspiring attorneys in California was itself created with the help of a large language model, The Los Angeles Times reports. The admission was made by the State Bar of California on Monday, following widespread complaints about the quality of the test's questions, and numerous glitches experienced by test-takers when they took it in February. In a news release, the organization said that 23 of the exam's total of 171 scored multiple choice questions were […]

Apr 25, 2025 - 23:00

California Admits AI Was Used to Write Bar Exam Plagued by Problems

The State Bar of California admitted that AI was used to draft dozens of questions in the bar exam administered to aspiring lawyers.

Now here's a legal-drama-worthy twist in the recent spate of dumb lawyers getting caught using AI: it turns out that the very bar exam administered to aspiring attorneys in California was itself created with the help of a large language model, The Los Angeles Times reports.

The admission was made by the State Bar of California on Monday, following complaints about the quality of the test's questions, and numerous glitches experienced by test-takers when they took it in February.

In a news release, the organization said that 23 of the exam's total of 171 scored multiple-choice questions were drafted by the firm ACS Ventures, which developed the questions "with the assistance of AI." Another 48 questions were lifted from an older version of an exam for first-year law students.

"The debacle that was the February 2025 bar exam is worse than we imagined," Mary Basick, assistant dean of academic skills at UC Irvine Law School, told the LA Times. "I'm almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable."

Katie Moran, an associate professor at the University of San Francisco School of Law, called it a "staggering admission." The same company that used AI to draft the questions was then paid "to assess and ultimately approve of the questions on the exam, including the questions the company authored," she noted to the newspaper.

For weeks, test takers had complained that they were randomly kicked off the online platform that the bar was administered on, while screens lagged and showed error messages, per the reporting. The test itself was riddled with typos, and some questions were total nonsense.

Despite these complaints — and despite pleading guilty to AI usage — a spokesperson for the State Bar insisted that the test questions were reviewed by content validation panels and subject matter experts.

In any case, the whole situation sounds like a mortifying catastrophe. For one, the Supreme Court of California, of which the State Bar is an administrative arm, maintains it had no idea about the use of AI to create the test questions until this week — even though it had instructed the State Bar to explore the use of AI to "improve upon the reliability and cost-effectiveness of such testing" last fall, according to Alex Chan, chair of the State Bar's Committee of Bar Examiners.

Casting additional scrutiny, Basick and Moran argued that the exam questions, which should take years to develop, were drafted far too quickly, while 50 practice questions re-released just weeks before the actual exam contained numerous errors, they wrote early this month, per the LA Times.

What spurred the dubious measures sounds like a familiar tale of disastrous cost-cutting. Faced with a $22 million deficit last year, the State Bar ditched the commonly used National Conference of Bar Examiners' Multistate Bar Examination, and decided to transition to a hybrid model of in-person and remote testing. To create the new test, it inked a $8.25 million deal with Kaplan Exam Services, and contracted Meazure Learning to administer it.

In a fittingly legal result, Meazure Learning is now being sued by some of the students who took the glitchy exams. The State Bar said it will ask the California Supreme Court to adjust test scores for those who took the test in February. Chan said that the Committee of Bar Examiners will meet on May 5 to discuss other remedies, but doubted that the State Bar would release the exam questions to the public or go back to the NCBE.

More on AI: Judge Goes Ballistic When Man Shows AI-Generated Video in Court

The post California Admits AI Was Used to Write Bar Exam Plagued by Problems appeared first on Futurism.