Generative AI: The importance of human oversight in the law

All generative artificial intelligence (AI) systems are fallible. Machine learning (ML) finds patterns in huge data sets and produces results based on those patterns. The AI system needs no human direction to produce outputs. And, if lacking human direction, if relying solely on big data and ML, the AI system will produce results that are misleading, bias, or simply wrong.

Human oversight is necessary throughout the entire lifecycle of an AI system, from inputting the first piece of data to monitoring the last output. Humans need to have ownership and accountability over the development, use, and outcomes of AI systems. Effective oversight will improve quality, enhance performance, build trust with users, and ultimately ensure success.

Below we explore how human oversight reduces risk, amplifies benefits, and boosts overall accountability when practicing the law, and we look at the best ways to increase human oversight of AI systems.

The importance of human oversight in AI systems

A common complaint about AI systems is the lack of transparency. Some solutions do not reveal data sources, do not disclose copyright, provide no information about algorithms, and fail to highlight people responsible for building and maintaining the system. If anything goes wrong, accountability is absent. If accountability is absent, things are more likely to go wrong.

That’s why human oversight is so important: it creates accountability. AI systems are complex, multifaceted, and occasionally unpredictable. They can make mistakes. Accountability limits such mistakes and ensures people correct the mistakes to prevent any reoccurrence.

Accountability also brings moral judgment into an amoral system, which is particularly important when people use generative AI in sensitive areas, such as law. Consider, for example, that a lawyer may build a case using AI. If the information is factually incorrect, the lawyer has created ethical and legal risk, not to mention the potential reputational damage.

Human oversight also mitigates other prominent AI risk. It can minimise the introduction of bias through simple steps. It can ensure inputs are carefully curated and of a much higher quality, while simultaneously ensuring effective data governance. It can make sure outputs are more accurate, more reliable, free from hallucinations, and more up-to-date. In short, human oversight drastically improves the results of the AI and boosts user trust.

Ensuring human oversight in AI systems

It is vital, for specific AI systems built for specific purposes, that organisations rely upon the knowledge of experts. In the legal sector, for example, people with an in-depth understanding of law should continuously monitor and improve the performance of AI systems, inserting their specialist knowledge to build much improved AI solutions.

Human oversight should start with the inputting of data. Experts should carefully curate the data used to train systems and then constantly evaluate that data, looking for any outliers or anomalies and correcting sources to maintain high quality. They can minimise inaccuracies and bias through bias-reducing procedures and algorithmic detection tools.

Data quality assessments can also help with monitoring. The assessments highlight missing values, outliers, and general issues within data. The results allow humans to clean data sets and handle missing data. Perhaps most importantly, data assessments ensure that the data is representative and reflects the real-world scenario in which the AI operates.

Organisations are responsible for the results of the AI and should always consider the real-world impact. That’s why AI systems should also apply human oversight through auditing. At the highest level, AI systems can perform model performance evaluation, developing a series of metrics – accuracy, precision, speed, relevance, so on – and judging outputs against the metrics. Shortcomings should become clear through the process.

AI systems should judge outputs in real-world scenarios, inviting feedback from the people using the system. Lawyers should provide feedback on legal AI systems, for example, as they are most likely to spot issues, understand context, and notice errors.  

Human oversight depends, then, on the knowledge of the expert and the irreplaceable shared wisdom of the community. The best AI systems are accountable systems, with individuals providing human oversight and accountability, minimising risks and amplifying the benefits.

Practicing human oversight at law firms

Lawyers and law firms should also apply human oversight. They can perform their own audits, ensuring the AI systems meet the suggested standards. Law firms and lawyers can dig deeper in terms of auditing systems, adopting a more technical approach.

Tools like local interpretable model-agnostic explanations (LIME) and Shapley Additive exPlanations (SHAP) explore explainability and interpretability of AI systems, for example. These are techniques that approximate any black box machine learning model with a local, interpretable model to reveal what is happening within systems and identify potential issues.

LIME and SHAP should not often prove necessary, as lawyers, in the pursuit of minimal risk, should use AI systems that offer greater transparency – and greater accountability. But AI systems are not static – they evolve, they improve, and they also deteriorate – so a once transparent system can become increasingly opaque. That’s why regular auditing is necessary.

Training and guidance also allows lawyers to practice effective human oversight. Law firms should train staff on the best ways to apply human oversight to outputs. That means lawyers should be able to pinpoint potentially flawed outputs, fact-check essential and high-risk information, note data breaches and exposures, and so on. And lawyers should be aware of how to report that information to the systems themselves, allowing the systems to improve future outputs, minimising future risk, and creating better models for everyone.