A law professor, a law school administrator, and a recent examinee share their views of the present-day test. Is this still how we should be licensing lawyers?
This is part 2 of a two-part series on the bar exam and the future of lawyer licensing in Minnesota. Read Part 1
‘Poorly designed to serve [its] purpose’
By Carol Chomsky
We rely on the bar exam to determine whether an applicant has the minimum competence to practice law, but the exam is poorly designed to serve that purpose. In its current incarnation, the exam essentially tests the ability of applicants to take an exam, not their ability to practice law. The exam focuses heavily on memorization of legal principles and issue-spotting, rather than testing the broad range of skills that lawyers need.
Every study of lawyers, including a recent one by IAALS (the Institute for the Advancement of the American Legal System), documents that new attorneys need a range of lawyering skills more than they need recall of doctrinal specifics. Many new lawyers engage directly with clients and take primary responsibility for client matters, so they need to know how to work with clients successfully. They need to know how to manage caseloads, negotiate with opposing counsel, and develop strategies for solving client problems. None of that is tested on the bar exam.
Not only does the bar exam focus too much on knowledge of doctrine, it tests that knowledge in a way that is divorced from the way that lawyers actually use legal rules. The exam requires memorization of hundreds, maybe thousands, of detailed legal rules. Test-takers cram rules, exceptions, and exceptions-to-the-exceptions—all details that are not remembered past the exam itself. Lawyers don’t operate based on memory that way. They remember fundamental principles of law and then research or confirm the details—and check whether the law has changed. Experienced lawyers may remember more details in their areas of expertise, but even they typically refer to cases, statutes, regulations, and secondary sources, rather than relying solely on memory. The IAALS study confirms this as well. New lawyers and their supervisors noted over and over that relying on memory was the surest way to make mistakes, especially in the early years of practice. Successful lawyers know where to find the law they need and how to learn new doctrine. They do not rely on remembering legal rules.
The exam also asks test-takers to answer questions based on two-dimensional hypotheticals. But lawyers don’t react to carefully crafted fact patterns. Lawyers work with clients, witnesses, and the written record to figure out the facts—to piece together the story that they will ultimately tell as they represent their clients’ interests. Collecting and working with facts is an essential skill for lawyers, but it’s not on the bar exam. The bar exam remains stuck in an artificially constructed universe, with static, unambiguous, and predigested facts—and a mythical set of standard legal rules that exist nowhere.
Finally, the exam is what psychometricians call “speeded.” It tests the ability to answer quickly, not thoughtfully. Applicants must answer 200 multiple choice questions with just 1.8 minutes to read, understand, and analyze each question. They must write a series of essays with only 30 minutes to read the facts, analyze the situation, identify the law involved (and remember the needed details), consider how to apply that law to the facts, and write their analysis. And then in only 90 minutes, applicants must read a fact pattern and file of cases, statutes, and other materials; extract the legal principles from that file; analyze how to apply the law; and write an analysis. The exam creates pressure to answer, answer, answer, rather than to think and answer. It asks test-takers to make snap judgments when such hurried judgments harm clients. Lawyers work under time pressure, but good legal work requires preparation and care—not shooting from the hip.
A recent study by Professor Steven Foster engaged 16 practicing lawyers in taking a simulated version of the multiple-choice portion of the bar exam. Not one of the lawyers who took the exam passed it. The best of the group answered just 52 percent correctly, and most scored far below that. An exam that practicing lawyers regularly fail is not a reasonable test of minimum competence.
Because the test is so artificial, preparing for it requires purchase of expensive bar review materials and courses. NCBE, which authors the exam, gives applicants only a general outline of the subjects that will be tested. Test-takers must purchase expensive commercial outlines of the rules that the bar review companies believe the exam will cover. And it isn’t enough to purchase those outlines—preparation requires months of focused study, memory drills, and exam-taking practice. That is usually unpaid time for applicants who have just spent most or all their resources—and then some—paying for legal education. As a recent study by the AccessLex Institute confirmed, the bar exam is a test of resources, not ability.
If all of that is not enough, the exam undermines the well-being of law students, who face the prospect of an unreasonable and ineffective exam knowing that it bears no relationship to the work they will do in practice. And this ill-suited bar exam has constrained the diversity of our profession. It creates a barrier for those with disabilities and produces disparate outcomes by race. White test-takers pass in significantly higher percentages than every other racial category, an issue that has been known but ignored for decades. Whatever the explanation for the exam’s disparate racial impact (differential access to resources and the presence of stereotype threat seem likely contributors), retaining an exam that disproportionately excludes candidates of color, while failing to test the knowledge and skills actually needed by attorneys, is totally unacceptable.
Although NCBE has promised to address some of these concerns in a modified exam, a written exam will always fall short in testing many critical skills of lawyering. It is time to explore additional pathways that allow applicants to prove their competence by actually doing legal work, with access to the sources and materials used in practice.
CAROL CHOMSKY is a professor at the University of Minnesota Law School, a member of the Collaboratory on Legal Education and Licensing for Practice, and was co-chair of one of the working groups participating in the MBLE comprehensive two-year study of the bar examination.
‘A reliable, valid, and objective measure’
By Dena Sonbol
It is wise and necessary to test an applicant’s legal skills and knowledge to ensure minimum competence before granting licensure into the legal profession. As it stands today, the Uniform Bar Exam (UBE), adopted in Minnesota and 39 other jurisdictions, is effective at doing just that. Until we have a more reliable, fair, and objective alternative that will ensure minimum competence, we should maintain the bar exam. While there may be some undesirable aspects of the bar exam, there are many desirable qualities and outcomes that make it a reliable, valid, and objective measure of competence.
The UBE’s most desirable quality is that it tests core legal skills necessary for law practice: reading comprehension; legal analysis, which requires logic and critical thinking skills; and legal writing. As a profession, we should not admit people to practice who lack these core skills. While the bar exam does not test all the skills needed to practice law, it is effective at measuring the core skills and should not be dismissed simply because it does not cover everything.
The second most desirable quality of the UBE is that it is a mostly objective and fair assessment of minimum competence. The MBE (Multistate Bar Examination) counts for 50 percent of the total points and consists of multiple-choice questions covering seven subject categories. The questions are carefully and thoughtfully written and properly vetted to have little subjectivity. An examinee either knows the law and can perform legal analysis or they cannot.
One downside of the MBE is that the scope of the material tested is notably deep. In practice, lawyers must know the rules and the exceptions to the rules, and they must be able to recognize when they need to conduct more research. The MBE demands much deeper knowledge of the subjects covered than is required of many lawyers in practice. The NextGen Bar Exam is being designed to decrease the depth of tested material.
The MPT (Multistate Performance Test) portion of the UBE counts for 20 percent of the total points and equalizes all examinees by giving them the law in cases, statutes, and other guidance, and the facts in a client file containing depositions, police reports, or other documents. All examinees are measured on their ability to synthesize the law and cull through a client file to demonstrate reading comprehension, legal analysis, and legal writing. The MPT is a fair assessment of minimum competence, and the NextGen Bar Exam will incorporate more assessments that equalize examinee resources and focus on testing skills rather than memorization.
The final part of the UBE, the MEE (Multistate Essay Exam), is the most problematic in terms of fairness and objectivity. The MEE counts for 30 percent of the total points and consists of six essays on 12 subject categories. This is too much and not reflective of the requirements of law practice. A new lawyer would require on-the-job training to be considered competent to practice law effectively and ethically in all these areas. Also, there is an inherent subjectivity in grading the MEE that is more pronounced than in MPT grading. The MPT asks examinee to address specific issues and limits them to the law provided, but the MEE prompts can be open-ended, and the examinee is not limited by any law. A competent and prepared examinee may read an MEE and not identify all the issues (or identify too many). Yet the point sheet for scoring the MEE is rigid and only awards points for specific issues. Fortunately, the NextGen Bar Exam will address most of these concerns, as the entire NextGen Bar Exam will cover eight foundational subject categories instead of 12 and the pure essay format of the MEE will be eliminated.
The UBE is not perfect, but it is still a reliable, valid, and objective measure of minimum competence. Those who oppose the UBE argue that it is resource- and time-prohibitive for an applicant to successfully prepare for the exam, resulting in disparate outcomes for certain examinees. This, too, is an insufficient reason to eliminate the bar exam.
Despite the resources and time required to prepare for and pass the bar exam, thousands of examinees pass every year. Certain factors make it more likely for an applicant to pass the bar exam. The most crucial factors in determining bar passage are, not surprisingly, knowledge and skills. The extent of knowledge and skills an examinee can attain during the bar study period is correlated with their level of knowledge and skills when they begin studying, and the resources and time they have available to build their knowledge and skills. Further, the level of knowledge and skills an examinee has when they begin studying is correlated with how well they did in their bar-tested law school courses.
Whether an examinee does well in bar-tested law school courses is correlated with an examinee’s LSAT score, another exam some rally against. Yet the LSAT tests the very same skills that are core to law school success and bar passage, such as reading comprehension, logic, and critical thinking. To the extent that educational, social, and economic experiences yield lower LSAT scores and are clustered more frequently within various groups of people, the bar exam, which tests the same skills, will reflect the same disparity of experiences among people. This need not be the case, though.
There is not clear evidence of the cause of this disparity. Perhaps the disparity has to do with a law student’s experience or their experience before law school. Instead of eliminating the bar exam, law schools can and should be doing more to prepare their students for the bar exam while in law school and to reduce the resource and time deficits that examinees experience in preparing for the bar exam. This may include reducing tuition costs; administering assessments in law school that better simulate the timed, closed-book nature of the bar exam; offering bar prep classes during law school; providing discounted commercial bar prep courses; or offering free bar tutoring services.
An objective measure like the UBE, or its successor the NextGen Bar Exam, also provides necessary accountability to the nearly 200 ABA-accredited law schools in this country and protection to consumers of legal education. Law schools have an interest in remaining ABA-accredited, and objective metrics like bar passage ensure that law schools admit and prepare students for competent law practice.
Indeed, the first gatekeepers to the practice of law in most jurisdictions are the law schools. So long as the law schools remain susceptible to the free market economy, their financial viability will necessarily reign supreme. For the sake of our profession, though, we must insist that competency to practice law continues to reign supreme. Right now, the UBE is holding the line and the NextGen Bar Exam is expected to do the same as well as to address the shortcomings of the UBE. Until we have a more reliable, fair, and objective alternative that measures minimum competence, we need to have a bar exam.
DENA SONBOL has served as dean of academic excellence and professor of law at Mitchell Hamline School of Law (MHSOL) since 2017. In her role, she prepares students to pass the bar exam in all 50 jurisdictions. Prior to joining the faculty at MHSOL, she was a legal analysis and writing professor for five years at Southern University Law Center, a historically Black university in Baton Rouge, Louisiana. She is a magna cum laude graduate of Hamline University School of Law and practiced real estate law in the Minneapolis office of Stinson LLP prior to joining academia.
‘An inadequate measure of minimum competence’
By Tyler Wessman-Conroy
Practicing law is an extraordinary privilege and responsibility. And it is imperative that we assess the minimum competence of attorneys before they enter the legal profession. But the bar exam is an inadequate measure of minimum competence.
I sat for the July 2022 bar exam in Minnesota. As a recent examinee, I was extraordinarily privileged. I had the help of my family and professional network throughout my exam preparation. My wife is expecting our third child very soon and, although she was limited in what she could give during my exam preparation, she was incredibly supportive. My mother lived with us and helped to care for our children while I was in law school and during my bar exam preparation.
I had to continue working while studying for the bar exam, but I was able to reduce my hours at work to free up more time to study. Because I took time off, I earned half as much to support our family during those months of preparation as I ordinarily would have. Fortunately, my wife also works and that lessened the financial impact on our family. Even though I reduced my hours, I still needed to work more than my bar preparation course recommended. Many bar preparation companies suggest not working at all, or at a minimum, not working during the two weeks leading up to the exam.
I had a job. I had others helping to support our family. I had skilled professional mentorship. I have a job waiting for me after law school that, if I do not pass the exam on the first try, will still be waiting for me while I take the bar exam again.
The bar exam is undoubtedly high stakes. I was fortunate that the stakes were not as high for me as they were for many of my law school classmates and fellow examinees who were equally as competent but who may not have had as much privilege. And yet, despite that, taking the bar exam was still one of the most difficult things I have ever experienced.
Physically and mentally, preparing for the bar exam was the unhealthiest two-three months of my life for numerous reasons. The lack of movement and exercise. The constant guilt that I should not be playing with my children (ages six and three) because my time would be better spent reading outlines and working through flash cards. The uncertainty about my future and the constant pressure of knowing that it was all riding on a two-day exam. The stress and anxiety of my fellow examinees on the day of the bar exam was palpable.
The bar exam purports to be a test of minimum competency. Yet the volume of information, the depth with which a successful examinee must memorize, understand, and analyze the information, and the unrealistic time constraints under which the exam is administered are not at all reflective of minimum competence or the actual practice of law.
I studied as hard as I could for the bar exam, within my circumstances, and completed nearly all the program materials provided by my commercial bar preparation company. If I or other examinees mastered 85 percent of the material in a particular area of law (some of which we may never practice in) and the National Conference of Bar Examiners tests heavily in the remaining 15 percent, we may not be considered minimally competent. Our efforts during law school and in preparing for the bar exam are meaningless in terms of licensure. Our futures as practicing attorneys depend entirely on whether we are deemed minimally competent during a two-day exam taken in an intensely controlled environment and under unreasonable time constraints, and whether we have the extensive resources (time, money, and mental and emotional support) needed to prepare for the exam.
In theology there exists the notion of sacramentality—that we use ritual, tangible experiences to encounter something greater and to become transformed through that encounter. In the practice of law, there should be a process or experience that transforms us to be ready to undertake the great privilege of representing clients’ legal rights and interests. Working hard and sacrificing to achieve a purpose is important for us as humans and lawyers. But when hard work and sacrifice are accompanied by barriers and gatekeeping measures that exclude qualified lawyers of all backgrounds from the practice of law, they no longer serve their purpose.
Every subject and skill tested on the bar exam can be evaluated in another way. Successful passage of the bar exam does not tell me whether someone is capable of practicing law. It tells me only that someone is adept at memorization and test-taking. And it risks excluding individuals who would be excellent attorneys simply because they are not.
As a former teacher, I understand pedagogy and how to help people learn. Much of my law school experience and much of my preparation for the bar exam failed to prepare me for the practice of law. I have worked with my father-in-law, a lawyer, for a little over a year now. I have learned more about the practice of law by shadowing him and being present in his law office.
An apprenticeship model that involves learning through experience and having trustworthy attorneys attest to my abilities after reviewing my legal work product and witnessing my interactions with clients is a much more accurate assessment of my competence and fitness to practice law.
I am also in full support of subject-based competency exams—shorter exams that objectively test a lawyer’s competence in a particular area or areas of the law in which the lawyer intends to practice.
After I completed the bar exam in July, I reached out to several attorneys who had been mentors to me in law school, and I asked them how they felt after finishing the exam. They all said that they walked out of the exam feeling terrible about themselves and uncertain about how they had performed. Not one said they walked out of the exam feeling competent or confident that they had passed.
These people were all high-achieving students in law school who are now highly successful and competent attorneys. That was telling to me. If the bar exam was a test of minimum competency, most of these attorneys should have left the exam tossing their hats in the air like Mary Tyler Moore and feeling hopeful and accomplished.
I am glad to see that Minnesota is evaluating the efficacy of the bar exam and considering alternative paths to attorney licensure.
TYLER WESSMAN-CONROY graduated magna cum laude from the University of St. Thomas School of Law and sat for the Minnesota bar exam in July 2022. He and his wife have two (soon to be three) young children. Prior to law school, Tyler graduated from St. John’s University and went on to earn a Master of Divinity degree from Boston College and taught theology, religion, and social ethics at the high-school level for seven years. After becoming licensed as an attorney, Tyler plans to practice at Conroy Law Office, Ltd., in Monticello, Minnesota.