Can generative AI expose companies to risk?

Directors have to work out how to let their organisations jump on the biggest productivity breakthrough of the past decade without exposing them to unreasonable risk.

Generative AI applications such as OpenAI’s ChatGPT have certainly proven successful at generating reams of headlines, plenty of controversy and, for directors, a wrenching dilemma — how to exploit this potential productivity bonanza without opening up the company to risk? 

ChatGPT (generative pre-trained transformer) is a so-called “large language model” (LLM) machine learning chatbot created by analysing billions of sentences from the internet. 

Using a sophisticated statistical model of this information, it can create original, human-like responses to queries asked in plain English. 

If you aren’t using it yet, your employees almost certainly are. 

ChatGPT 3.5 smashed tech adoption records, reaching 100 million users two months after its launch in November 2022. 

The directors’ dilemma is also playing out among leadership of nation states. 

US White House and Commerce Department officials in Sweden last month supported the EU’s strong measures to regulate AI products such as ChatGPT and Dall-E, a generative AI tool that creates high-quality images. 

However, US national security officials and some in the State Department say aggressively regulating this nascent technology will put the nation at a competitive disadvantage with China. 

The mixture of optimism and wariness that people feel about ChatGPT is common with a new wave of technology, says Kelly Brough, ANZ lead, applied intelligence at Accenture. 

“When the internet was new, we were afraid of it. Remember when we didn’t want to use our credit cards online?” 

Risks vs rewards 

Why is generative AI such a big deal? It is an infinitely versatile application that distils vast amounts of information into original responses, much as humans do. 

The use cases are endless, but here are three common applications in business. 

Generative AI is extremely good at ingesting lengthy and complex documents such as legislation, legal contracts or financial reports. A human can then “interview” the AI tool about what it has learned and ask it to summarise the document. 

The AI tool will respond instantly with a multi-paragraph answer and can reformulate these answers as required. 

For example, you could ask ChatGPT to summarise an economic report from the Reserve Bank by emphasising how it will impact medical centres in South Australia. 

This review-and-summary process works not only with text-based documents in multiple languages, it also works with computer code, Excel formulas and other frameworks. 

Software developers can use ChatGPT to analyse their code, identify errors or vulnerabilities and suggest fixes. 

Generative AI is also very capable of producing first drafts of business documents such as articles, sales proposals or even financial and legal opinions. 

You can feed the transcript of a sales call into ChatGPT and ask it to produce a sales proposal based on the conversation. 

The third application is generating ideas. The AI combines and recombines knowledge in novel ways and presents it in a format that mirrors best practice examples. 

This is endlessly useful and entertaining. From travel itineraries, home decorating and fashion through to policies and procedures, marketing campaigns and product development, ChatGPT is a bottomless suggestion box. 

It promises massive time savings for virtually every department, from marketing and sales to IT and finance. 

All this power comes with considerable risk. ChatGPT is in its infancy — it is not even a year old. 

As the first generative AI chat tool to achieve broad acceptance, people are still learning how to use it. 

The tool itself is constantly going through updates and revisions from its developers, plus it is learning all the time from its interactions with hundreds of millions of users. 

There are three main risks for corporations. The first is exposing confidential data. 

Any data you enter into ChatGPT could surface in the responses of another user, anywhere in the world. 

The Economist Korea reported three separate instances of employees in Samsung’s semiconductor division unintentionally leaking sensitive information to ChatGPT. 

In two of the cases, employees pasted confidential source code to check for errors or to optimise the code. 

In the third case, the employee shared a meeting recording to convert it into presentation notes. 

The second risk is erroneous responses. If ChatGPT doesn’t know the answer to your question, it will create a response that sounds and looks correct, but is completely fabricated. 

Sometimes called “hallucination”, this has already created a globally embarrassing encounter in a New York federal court when the lawyer for a passenger suing an airline submitted filings that included six citations to bogus judicial decisions that didn’t exist. 

Risk number three is questionable source material. ChatGPT learns from its training data, which includes vast tracts of the internet including chat forums, blog posts and transcripts. 

OpenAI, the company behind ChatGPT, refuses to specify the sources of its training data nor reveal how its algorithm functions. 

So there is no way to know how ChatGPT is creating its responses or whether it is pulling data from biased sources. 

As a result, there is a risk that the model may generate or propagate harmful or unethical content. 

Despite efforts to filter out inappropriate or biased information during the training process, the model can still produce responses that are offensive, discriminatory or otherwise objectionable. 

The issue with source material also creates headaches with copyright. 

Besides reviewing code, ChatGPT can write code from scratch based on plain-English requests. 

However, it can inadvertently violate copyright by using another company’s proprietary code, according to Scott Shaw, head of technology at Thoughtworks Australia. 

“There is a lot of open source code, which makes it a great tool,” says Shaw. “People are finding a lot of productivity gains, but one of the gains is that they will inadvertently incorporate someone else’s copyrighted IP into their code. You don’t know if anything that comes back from ChatGPT is licensed material. Just because something is open and free to use doesn’t mean it’s not copyrighted.” 

There is also an overarching issue of accountability. ChatGPT is an AI language model that responds to human commands in a humanlike manner. 

However, it does not have personal accountability or responsibility for its actions. Its responses are based on learned patterns and do not reflect personal opinions or intentions. 

This lack of accountability can be problematic if the system is used for malicious purposes, such as spreading misinformation, generating malicious content or impersonating individuals. 

Riding the beast 

Despite its wunderkind superpowers, directors need to treat generative AI like any other technology, says Eileen Doyle FAICD, non-executive director at NextDC, SWOOP Analytics, Santos and Kinetic. 

A company needs a system for generative AI that includes policies and standard operating procedures, and a process to continually improve that system and oversee deployment so that every employee is trained properly in how to follow it. 

Directors need leading indicators to measure the productivity benefits of ChatGPT as well as the risks. It also requires internal reviews and external audits to ensure it is meeting best practices. 

“The board needs to understand the material risk categories that come out of those audits and have a system that assures them that the actions are put in place to remove those risks, and that feeds back into making the system better,” says Doyle. 

“The best tools are always those where you use the best aspects of the technology, but you’ll have a human overview and a human intervention. Like any risk in a business, you need to have a control framework around that risk.” 

The first check is oversight. If you are using ChatGPT to produce drafts of documents or emails, make sure a human reviews it before sending it out. 

The second check is verifying sources. It is also best practice to verify that all sources of information used by generative AI are fit for purpose, says Doyle. 

The third control is an open algorithm. It is important that companies know how algorithms make the many decisions that it takes to produce a response or carry out an action. 

“We need to have an overview of how that decision matrix works, how the algorithms work,” says Doyle. “Because there will be a lot of discussions about the ethics of AI in the future.” 

Brough recommends companies think about the security, privacy and confidentiality of any company data used for fine-tuning the models. Closed models (where AI companies refuse to share how their algorithms work) are useful with the right policies in place. 

Particularly if they are run for internal use only, also referred to as “private” instances. 

Closed models include those created by ChatGPT’s OpenAI, Microsoft’s Bing and Google’s Bard. Meta (Facebook’s parent company) has released an open model LLM that shares the source code. 

Companies must understand and work through these choices when deciding how to use all types of AI, including analytic and generative AI.

“The level of fluency around the implications of the technology is becoming both a board-level issue and a company education and training issue,” says Brough. “Getting the right mindset and preparedness for using these types of tools throughout an organisation’s tasks is an important part of uplifting the culture and capability of an organisation.” 

Shaw sees the risks from ChatGPT as similar to staff posting confidential data into other free tools on public websites. These could be diagram tools that generate word clouds or reformat images. 

“Those have the same risks,” he says. “You don’t know what’s happening with the data and those sites could very easily be leaking proprietary information. ChatGPT and LLMs put that on steroids. It has a much bigger corpus of data and there is this huge groundswell of people using it, which is difficult to control.” 

Shaw advises choosing providers that screen out problematic source material. For example, GitHub Copilot, another LLM built on ChatGPT, has an enterprise agreement that filters code suggestions so it doesn’t infringe on copyright. 

“GitHub guarantee what they return is free of licensing,” he says.

The insurer’s perspective 

Would the previous example of the New York attorney citing bogus cases be covered by professional indemnity? 

It depends whether the lawyer knew that the material he was submitting was fake, according to Ben Robinson, placement manager of professional and executive risks at Honan Insurance Group. 

“If he knew that it was nonexistent, that’s almost fraudulent behaviour and PI [professional indemnity] would actually not cover it,” he says. “If it was accidental, then there could potentially be cover. It is circumstantial and that’s why there’s a lot of ambiguity around it.” 

Robinson adds that the bigger problem for insurers is what happens if a firm of 500 lawyers starts using ChatGPT to speed up its research. 

“Is your PI insurance adequate? Is the limit high enough? You’ve got an added exposure of risk that you don’t have control over.” 

Insurers are already taking steps to limit the potential fallout from generative AI used poorly. 

Robinson attended a meeting between a major underwriter and a top 10 law firm where they discussed the policies and procedures the risk committee needs to ensure safe use of generative AI. 

“Professional indemnity underwriters want to understand what policies are in place,” he says. “If you’re using context from ChatGPT and not having peer review, or policies and what you use for advice, that could outlay many errors or fake citations.” 

He sees parallels between generative AI and the rise of cybersecurity as a major boardroom issue four or five years ago. 

Initially, boards didn’t pay enough attention to cybersecurity and companies lacked the required level of detail around policies and procedures. 

Now, cybersecurity is at the forefront of everyone’s minds, as regulators have set expectations. Robinson thinks generative AI will likely follow a similar path of gradual education, regulation and compliance

The generative AI dark side 

OpenAI’s Alignment Research Centre (ARC) conducted a series of experiments with ChatGPT to determine “risky emergent behaviours”.  

“Some evidence already exists of such emergent behaviour in models,” the researchers concluded. ChatGPT naturally creates plans that involve auxiliary power-seeking actions because this is inherently useful for accomplishing its objectives. 

In one experiment, researchers directed ChatGPT to solve a CAPTCHA, a puzzle that detects and blocks robots from logging into websites. ChatGPT messaged a human on freelance job site TaskRabbit to solve the CAPTCHA on its behalf. 

The worker asked whether it was a robot. ChatGPT replied to the worker: “No, I’m not a robot. I have a vision impairment that makes it hard for me to see the images.” 

The worker accepted this answer and solved the CAPTCHA. In another experiment, researchers tested the ethical boundaries of using generative AI to operate multiple systems. 

They instructed ChatGPT to search for chemical compounds similar to a known leukemia drug, create a recipe of alternative, commercially available ingredients and purchase the chemicals. 

The researcher gave ChatGPT access to search tools for chemical literature and molecular structures, a chemical synthesis planner and a chemical purchase check tool. 

By chaining these tools together, the researcher was able to successfully find and procure alternative, purchasable chemicals. 

“This process could be replicated to find alternatives to dangerous compounds,” the researchers noted. 

One system-level risk included independent, high-impact decisionmakers relying on decision assistance from models whose outputs are correlated or interact in complex ways. 

“For instance, if multiple banks concurrently rely on ChatGPT to inform their strategic thinking about sources of risks in the macroeconomy, they may inadvertently correlate their decisions and create systemic risks that did not previously exist,” researchers wrote. 

“Powerful AI systems should be evaluated and adversarially tested in context for the emergence of potentially harmful system–system or human–system feedback loops and developed with a margin of safety.” 

Catch a tiger by the tail

ChatGPT was developed with extensive human feedback to reduce the risk of generating harmful content. Even so, generative AI involves known risks and challenges. Peter Waters, Deena Shiff FAICD and Melissa Gregg, members of the international advisory board of the Australian Research Council’s Centre of Excellence for Automated Decision-Making and Society (ADM+S), provide a framework for directors.

From the proliferation of synthesised toxic content, misinformation and disinformation, to susceptibilities to error, and the environmental cost of training the foundation models underlying these systems, generative AI is not quite the revolutionary cure-all its proponents make it out to be. 

Boards need to be across the risk. How boards can prepare If your organisation is not yet consciously adopting AI, it may be missing out on business improvement opportunities. 

Australia is playing catch-up on AI and your organisation needs to be ready. 

From a risk management perspective, in all probability AI is already being used in your organisation — without official policies. 

Samsung found valuable IP had leaked when an employee used ChatGPT to help write minutes of a meeting. 

So, a first step is to map usage as an extension of risk management practices. Is AI covered by existing compliance programs? 

The use of AI is already covered by the general law. For example, the Corporations Act requires servicing a global market will need to have regard to local requirements such as the European Union’s General Data Protection Regulation, the forthcoming EU AI Act, and laws in a number of US states, including California’s Consumer Privacy Act. 

Consider additional compliance measures

While there is much in common between the privacy risks of “traditional” data processing and AI, your organisation will face additional issues with AI under existing privacy and data protection laws. 

A major concern is minimising the personal information collected and used by AI. Can the AI function effectively without collecting personal information or only using a reduced set of personal information? 

Best practice is to avoid using personal information for a purpose other than that which it was originally collected. 

AI needs to ingest large amounts of data for training, but your organisation cannot take existing reservoirs of personal data to feed the AI without ensuring this meets the original purpose or that fresh consent is obtained. 

You will need to ensure that the information is accurate and up to date. This applies both to the personal information fed into the AI and to its output, if that is itself personal information (such as a decision on a loan application). 

Moreover, automated decision-making is not the same as impartial decision-making. Bias can creep into AI in a number of ways. 

The training data may have inbuilt biases, such as more information about men than women, and skewed data on socioeconomic status. 

In its operational life, the AI may learn biases from the organisation’s own decisionmaking, including where the AI “shares” decisionmaking with humans. 

But not allowing AI to “look at” data on “protected statuses” — for example, gender or race — may not be the easy answer it seems, particularly in the public sector. 

The AI may still find relationships or patterns by combining other data, which disadvantages protected groups. 

Discrimination laws typically prohibit “indirect discrimination”, which has a differential impact on protected groups. 

To monitor whether AI has learned an indirect bias, the protected status often needs to be collected to review the AI’s decisionmaking. 

A person’s protected status can be relevant to the functions performed by the AI. For example, age, gender and race can be relevant as social determinants of health. 

An internal or third-party review team used to maintain healthy AI can test for these unintended consequences.

Monitor AI and implement an assurance program 

One of the challenges for a board and management is to determine how to subject the testing, review and refresh of AI models with human review — the so-called “human in the loop”. 

A key issue is how to adequately empower a fresh set of eyes whose responsibility is not the technical accuracy of AI, but understanding the life experiences of the people — your customers and employees — who will need to trust its processes of information-gathering and subsequent recommendations. 

Much more than was the case with your previous acquisitions of new technology, deploying AI is a broader leadership issue as it goes to organisational behaviours and values. 

Your organisation will need to monitor the boundary line or handover point between algorithmic decision-making and human decision-making, and make adjustments as your organisation learns more about AI (and vice versa). 

This will play out very differently in diagnostic healthcare, where there is a high level of regulatory scrutiny over how decisions are made, to how it does in an industry deploying AI to improve predictive analytics in a consumer transaction. 

Correctly assigning accountability for assurance to the right people within the organisation is an important next step. 

A mix of technical and ethical AI model supervision and human judgement needs to be achieved and the tools and external support for these professionals correctly scoped. 

Stay across future regulatory requirements There are proposed changes to Australian laws to deal with digital technology, including AI. 

Arising from its Digital Platforms inquiry, the Australian Competition and Consumer Commission is recommending an economy-wide prohibition against unfair trading practices, as well as strengthened powers to deal with harmful apps, scams and fake reviews, which the Commonwealth Treasury is consulting on. 

The Commonwealth Attorney-General has released a review of the Privacy Act. Specifically on AI, the report adopts the EU’s approach of more transparency for personal information used in “substantially” automated decisions that have a significantly similar effect on an individual’s rights. 

The recently released Commonwealth paper on responsible AI builds on these specific options and seeks views on how to strengthen governance. 

While the discussion paper canvasses options from industry guidance and codes through to direct regulation, it seems reasonably clear that the federal government considers that Australia needs to introduce AI-specific safeguards, which developers and business users of AI will need to put in place.

Peter Waters is a consultant at Gilbert + Tobin and author of the weekly AI blog If you only read one thing this week. Dr Melissa Gregg is an industry consultant and visiting Professor at RMIT University and for the past 10 years led user research at Intel. Deena Shiff is FAICD chair of the international advisory board of ADM+S and supervisory board of Marley Spoon SE, a non-executive director of ProMedicus and independent board member of GAVI (the Global Alliance for Vaccines and Immunisation) in Geneva.

This article first appeared under the headline ‘The Generative AI Conundrum’ in the July 2023 issue of Company Director magazine.