While acres of text have been devoted to controls on how students use AI, the LLM habits of much of the rest of society – including the nation’s law makers – remains relatively unscrutinised.
Should Parliamentarians be allowed to rely on AI to do some of their job by scrutinising legislation, at the same time as they trumpet efforts to stop students from using AI to do some of their assessment tasks?
La Trobe University Law Dean and External Legal Adviser to the Australian Senate Standing Committee for the Scrutiny of Delegated Legislation Lorne Neudorf asks (in a far more polite fashion) whether our Parliamentarians can (and should) be assisted by gen AI in the task of reviewing legislation.
In his research for a recent paper on the topic, Professor Neudorf compared the outputs of several leading LLMs against real tasks of legislation scrutiny undertaken by two committees of real life Parliamentarians.
Wise counsel for whether and how politicians should use AI bears a striking resemblance to advice on how law students use it.
“Law firms also expect law graduates to be familiar with artificial intelligence tools as part of
their professional skillset.” Professor Neudorf writes.
“This expectation has, in turn, prompted law schools to consider how artificial intelligence should be incorporated into legal education in a way that supports, instead of undermines, the development of core competencies. The challenge is to ensure that law graduates understand how to use artificial intelligence tools effectively, while preserving their ability to operate independently of them, particularly in relation to legal reasoning and the application of judgment.”
He also provides a valuable insight into the limitations of AI in legal argument and reasoning.
“The careful synthesis of multiple, and sometimes conflicting, authorities is essential
to legal reasoning. Constructing a defensible interpretation requires fidelity to existing law and an appreciation of where uncertainty exists and to what degree,” Professor Neudorf writes.
The paper concludes that while AI is already being used to support parliamentary administration and also in drafting legislation, but that humans are integral to legislative scrutiny.
“Committees provide a forum for more detailed examination than is typically possible during parliamentary debate. They also serve as the principal conduit through which the evidence and perspectives of experts and members of the public are brought directly before parliamentarians,” he writes.
Limitations with existing AI models included not testing explanatory materials, proposing advice that did not align with practice and an inability to provide judgement that would match that of humans.