The Curious Case of AI Hallucination

So, earlier today, I was having coffee with a friend who’s a technology program manager at a large financial services firm and fellow nerd – let’s call him Steve (since that’s his name). Given our shared interests, the discussion eventually turned to generative AI and, in particular, Large Language Models (LLM) like ChatGPT. Steve is a technologist like me but hasn’t had a lot of exposure to LLMs beyond what he’s seen on Medium and in YouTube videos, and he was asking the predictable questions – “Is this stuff for real? Can it do all the things they’re saying it can? What’s up with the whole hallucination thing?”

Overall, it was a great discussion, and we covered a broad range of topics. In the end, I think we both came away from the conversation with interesting perspectives on the potential for generative AI as well as some of the potential shortcomings.

I plan to share my perspective on our discussion in a series of posts over the next few weeks, beginning with the first topic we discussed — AI Hallucination. I hope you find value in these perspectives and, as always, comments are most welcome.

So, what’s up with AI hallucination?

For AI, hallucination refers to when an LLM generates text that sounds plausible but is factually incorrect or unsupported by evidence. For example, if you prompted an LLM to “write about the first American woman in space,” it might pull plausible-sounding information from its vast training data but get the facts wrong by hallucinating fictional details and attributing them to the first female American astronaut. This tendency of large language models to confidently generate fake details and pass them off as truthful accounts when prompted for topics outside its actual training data is extremely problematic, especially if the user is unaware of this tendency and takes the output at face value.

When I say “tendency”, I mean this is a very common issue that arises frequently with large language models today. The propensity to hallucinate false details with high confidence is extremely prevalent in modern LLMs, even sophisticated models trained on huge datasets. For example, a 2021 study from Anthropic found that LLMs hallucinated over 40% of the time when asked simple factual questions from datasets they were not trained on. And openAI has warned that its GPT models “sometimes write plausible-sounding but incorrect or nonsensical text” and should not be relied upon for factual accuracy without oversight.

This is especially dangerous in high-stakes fields like medicine or law, and in fact, there is a recent story of a lawyer using an LLM to prepare a court filing, inadvertently referencing fake cases (fortunately, the court caught this prior to the case proceeding).

As to why LLMs hallucinate, there are several potential reasons:

They are trained on limited data that does not cover all possible topics, so they try to fill in gaps.
Their goal is to generate coherent, fluent text, not necessarily accurate
They lack grounding in common sense or the real world.
Their statistical nature means they will occasionally sample incorrect or imaginary information.

An important point is that the LLM does not intentionally construct false information (unless asked to do so); rather it builds its responses based on available data (the data it was trained on). The models attempt to continue patterns and maintain internal coherence in their generated text, which can result in persuasive but false details when their knowledge is imperfect or they are asked to extrapolate beyond their training. In some ways, this exacerbates the problem, as the model can respond with high confidence while, in fact, having no factual basis for the response. Perhaps more worrisome, with further scaling up of models, this tendency may only become more pronounced as they get better at producing persuasive human-like text.

Clearly, better techniques are needed to detect and reduce hallucination.

There are some approaches that are being explored to reduce the occurrence of hallucination and/or to correct it prior producing generated responses. Here are some of the techniques being explored by researchers:

Human Feedback with Reinforcement Learning (HFRL): Having humans flag hallucinated text during the training process, then using Reinforcement Learning (RL) to adjust the model to reduce false information.
Incorporating Knowledge Bases: Connecting the LLM to an external knowledge base like Wikipedia can ground its output in facts.
Causal Modeling: Modeling cause-and-effect relationships helps the LLM better understand interactions in the real world.
Self-Consistency: Penalizing the model when its predictions contradict each other can minimize internal inconsistencies.
Robust Question Answering: Training the model to carefully consider a question before answering reduces speculative responses.
Hallucination Detection Systems: Separate classifiers can be developed specifically to detect hallucinated text.
Retrieval Augmented Generation (RAG): Retrieving relevant text and data before generating from them improves grounding.
Human-in-the-Loop: Letting humans interactively guide the model during text generation can steer it away from hallucination.

Which solution(s) will perform best is determined in part by the particular use case (for example, HFRL might not be practical for very large datasets) and more than likely a combination of techniques will be required to achieve desired levels of confidence in responses.

Even with these additional controls and safeguards in place, it will continue to be important to perform some level of quality control prior to using LLM output.

As a thought experiment, let’s take a private equity firm — the firm wishes to use LLMs to streamline the summarization and analysis of corporate data for acquisition targets. Indeed, LLMs can provide significant productivity lift in consuming and condensing large volumes of structured and unstructured data, and the firm can certainly use an appropriately fine-tuned LLM to facilitate the process of analyzing an organization’s fitness for acquisition. Having said that, fact-checking any specific conclusions produced by the LLM must be scrutinized closely to ensure its veracity prior to use in decision-making and, where necessary, adjustments made. Note that this should be no different than the same level of scrutiny that would be applied to human-generated analysis; the point is to not make the assumption that because the analysis is ‘computer generated’ that it is somehow more reliable – in fact, the opposite is true.

All said, hallucination remains a significant obstacle to leveraging the full power and potential of large language models. But proper controls, along with continued research into techniques like the ones discussed here provides a pathway for leveraging LLMs to generate accurate, trustworthy text as easily as they currently produce fluent, creative text.

If you’re ready to take advantage of AI in a meaningful way but want to avoid the growing pains and pitfalls (including hallucinations), we should talk! Our 5-day AI assessment takes the guesswork out of maximizing the value of AI while minimizing the risks associated with LLMs. You can find out more about this offering here or connect with me on LinkedIn.

(Note: Artwork for this and subsequent posts in this series are part of my collection, produced by MidJourney. Lined here.)

Today’s business leaders find themselves navigating a world in which artificial intelligence (AI) plays an increasingly pivotal role. Among the various types of AI, generative AI – the kind that can produce novel content – has been a game changer. One such example of generative AI is OpenAI’s ChatGPT. Though it’s a powerful tool with significant business applications, it’s also essential to understand its limitations and potential pitfalls.

1. What are Generative AI and ChatGPT?
Generative AI, a subset of AI, is designed to create new content. It can generate human-like text, compose music, create artwork, and even design software. This is achieved by training on vast amounts of data, learning patterns, structures, and features, and then producing novel outputs based on what it has learned.

In the realm of generative AI, ChatGPT stands out as a leading model. Developed by OpenAI, GPT, or Generative Pre-training Transformer, uses machine learning to produce human-like text. By training on extensive amounts of data from the internet, ChatGPT can generate intelligent and coherent responses to text prompts.

Whether it’s crafting detailed emails, writing engaging articles, or offering customer service solutions, ChatGPT’s potential applications are vast. However, the technology is not without its drawbacks, which we’ll delve into shortly.

2. Strategic Considerations for Business Leaders
Adopting a generative AI model like ChatGPT in your business can offer numerous benefits, but the key lies in understanding how best to leverage these tools. Here are some areas to consider:

2.1. Efficiency and Cost Savings
Generative AI models like ChatGPT can automate many routine tasks. For example, they can provide first-level customer support, draft emails, or generate content for blogs and social media. Automating these tasks can lead to considerable time savings, freeing your team to focus on more strategic, creative tasks. This not only enhances productivity but could also lead to significant cost savings.
2.2. Scalability
One of the biggest advantages of generative AI models is their scalability. They can handle numerous tasks simultaneously, without tiring or requiring breaks. For businesses looking to scale, generative AI can provide a solution that doesn’t involve a proportional increase in costs or resources. Moreover, the ability of ChatGPT to learn and improve over time makes it a sustainable solution for long-term growth.
2.3. Customization and Personalization
In today’s customer-centric market, personalization is key. Generative AI can create content tailored to individual user preferences, enhancing personalization in your services or products. Whether it’s customizing email responses or offering personalized product recommendations, ChatGPT can drive customer engagement and satisfaction to new heights.
2.4. Innovation
Generative AI is not just about automating tasks; it can also stimulate innovation. It can help in brainstorming sessions by generating fresh ideas and concepts, assist in product development by creating new design ideas, and support marketing strategies by providing novel content ideas. Leveraging the innovative potential of generative AI could be a game-changer in your business strategy.

3. The Pitfalls of Generative AI
While the benefits of generative AI are clear, it’s essential to be aware of its potential drawbacks and pitfalls:

3.1. Data Dependence and Quality
Generative AI models learn from the data they’re trained on. This means the quality of their output is directly dependent on the quality of their training data. If the input data is biased, inaccurate, or unrepresentative, the output will likely be flawed as well. This necessitates rigorous data selection and cleaning processes to ensure high-quality outputs.
Employing strategies like AI auditing and fairness metrics can help detect and mitigate data bias and improve the quality of AI outputs.
3.2. Hallucination
Generative AI models can sometimes produce outputs that appear sensible but are completely invented or unrelated to the input – a phenomenon known as “hallucination”. There are numerous examples in the press regarding false statements or claims made by these models, sometimes funny (like claiming that someone ‘walked’ across the English Channel) to the somewhat frightening (claiming someone has committed a crime, when in fact, they did not). This can be particularly problematic in contexts where accuracy is paramount. For example, if a generative model hallucinates while generating a financial report, it could lead to serious misinterpretations and errors. It’s crucial to have safeguards and checks in place to mitigate such risks.
Implementing robust quality checks and validation procedures can help. For instance, combining the capabilities of generative AI with verification systems, or cross-checking the AI outputs with trusted data sources, can significantly reduce the risk of hallucination.
3.3. Ethical Considerations
The ability of generative AI models to create human-like text can lead to ethical dilemmas. For instance, they could be used to generate deepfake content or misinformation. Businesses must ensure that their use of AI is responsible, transparent, and aligned with ethical guidelines and societal norms.
Regular ethics training for your team, and keeping lines of communication open for ethical concerns or dilemmas, can help instill a culture of responsible AI usage.
3.4. Regulatory Compliance
As AI becomes increasingly pervasive, regulatory bodies worldwide are developing frameworks to govern its use. Businesses must stay updated on these regulations to ensure compliance. This is especially important in sectors like healthcare and finance, where data privacy is paramount. Not adhering to these regulations can lead to hefty penalties and reputational damage.
Keep up-to-date with the latest changes in AI-related laws, especially in areas like data privacy and protection. Consider consulting with legal experts specializing in AI and data to ensure your practices align with regulatory requirements.
3.5 AI Transparency and Explainability
Generative AI models, including ChatGPT, often function as a ‘black box’, with their internal workings being complex and difficult to interpret.
Enhancing AI transparency and explainability is key to gaining trust and mitigating risks. This could involve using techniques that make AI decisions more understandable to humans or adopting models that provide an explanation for their outputs.

4. Navigating the Generative AI Landscape: A Step-by-Step Approach
As generative AI continues to evolve and redefine business operations, it is essential for business leaders to strategically navigate this landscape. Here’s an in-depth look at how you can approach this:

4.1. Encourage Continuous Learning
The first step in leveraging the power of AI in your business is building a culture of continuous learning. Encourage your team to deepen their understanding of AI, its applications, and its implications. You can do this by organizing workshops, sharing learning resources, or even bringing in an AI expert (like myself) to educate your team on the best ways to leverage the potential of AI. The more knowledgeable your team is about AI, the better equipped they will be to harness its potential.
4.2. Identify Opportunities for AI Integration
Next, identify the areas in your business where generative AI can be most beneficial. Start by looking at routine, repetitive tasks that could be automated, freeing up your team’s time for more strategic work. Also, consider where personalization could enhance the customer experience – from marketing and sales to customer service. Finally, think about how generative AI can support innovation, whether in product development, strategy formulation, or creative brainstorming.
4.3. Develop Ethical and Responsible Use Guidelines
As you integrate AI into your operations, it’s essential to create guidelines for its ethical and responsible use. These should cover areas such as data privacy, accuracy of information, and prevention of misuse. Having a clear AI ethics policy not only helps prevent potential pitfalls but also builds trust with your customers and stakeholders.
4.4. Stay Abreast of AI Developments
In the fast-paced world of AI, new developments, trends, and breakthroughs are constantly emerging. Make it a point to stay updated on these advancements. Subscribe to AI newsletters, follow relevant publications, and participate in AI-focused forums or conferences. This will help you keep your business at the cutting edge of AI technology.
4.5. Consult Experts
AI implementation is a significant step and involves complexities that require expert knowledge. Don’t hesitate to seek expert advice at different stages of your AI journey, from understanding the technology to integrating it into your operations. An AI consultant or specialist can help you avoid common pitfalls, maximize the benefits of AI, and ensure that your AI strategy aligns with your overall business goals.
4.6. Prepare for Change Management
Introducing AI into your operations can lead to significant changes in workflows and job roles. This calls for effective change management. Prepare your team for these changes through clear communication, training, and support. Help them understand how AI will impact their work and how they can upskill to stay relevant in an AI-driven workplace.
In conclusion, navigating the generative AI landscape requires a strategic, well-thought-out approach. By fostering a culture of learning, identifying the right opportunities, setting ethical guidelines, staying updated, consulting experts, and managing change effectively, you can harness the power of AI to drive your business forward.

5. Conclusion: The Promise and Prudence of Generative AI
Generative AI like ChatGPT carries immense potential to revolutionize business operations, from streamlining mundane tasks to sparking creative innovation. However, as with all powerful tools, its use requires a measured approach. Understanding its limitations, such as data dependency, hallucination, and ethical and regulatory challenges, is as important as recognizing its capabilities.

As a business leader, balancing the promise of generative AI with a sense of prudence will be key to leveraging its benefits effectively. In this exciting era of AI-driven transformation, it’s crucial to navigate the landscape with a keen sense of understanding, responsibility, and strategic foresight.

If you have questions or want to identify ways to enhance your organization’s AI capabilities, I’m happy to chat. Feel free to reach out to me at jfuqua@ccpace.com or connect with me on LinkedIn