Artificial Intelligence, Hallucinations, and Bias

In order to make decisions about artificial intelligence, it is important that students learn how generative AI affects us and our world. Previously, I discussed teaching students about how AI affects learning and mental health, but increasingly AI is also affecting our information landscape. As some students turn to generative AI as a search tool, it is important to teach them about the hallucinations and bias found in AI generated content.

To access slides I used to teach students about generative AI, please click here.

Hallucinations

Hallucinations are when an AI model gives incorrect information or appears to fabricate information. For instance, in early 2025 if you made up an idiom, like "there's always more potatoes in the stew," Google's AI Overview would confidently explain the meaning of the fake idiom, as pictured below.

Hallucinations are a significant issue that AI companies have had trouble resolving. For instance, OpenAI has reported that ChatGPT-5 generates incorrect information about 10% of the time. AI performs even worse when it comes to news content. An October 2025 study by the BBC tested several AI assistants and found the AI models misrepresented news content 45% of the time. This included issues with sources, inappropriate tone, altered direct quotations, and more. In my own experiences, I have caught Google's AI assistant making false claims, incorrectly attributing the claims to a source that does not contain that information, and then doubling down on the claim when challenged.

Hallucinations have significant real world impacts. Lawyers have faced discipline for citing cases that AI models had made up. Medical professionals have also advised caution when using AI Overview for medical advice, since it can give health advice that is misleading and at times dangerous. More generally, the reliance on generative AI for research and information makes for a less reliable and more precarious information landscape. Understanding that hallucinations occur and why they happen is important for students and adults alike.

Hallucinations can happen for several reasons. Errors or inaccuracies may be part of an AI model's training data, which means these same errors might surface in the AI's outputs. However, hallucinations can also occur because of how AI works. AI generates content using probability. It predicts what the next word could plausibly be, which is not the same thing as determining a correct answer. Moreover, AI models are incentivized to give confident answers. During the training process, benchmarks are often used to assess an AI model's answers, where an incorrect answer and an "I don't know" generally receive a score of zero. As a result, AI models are incentivized to guess, rather than admit uncertainty. It's possible that AI researchers might adjust how AI models are trained in the future, but at the moment we are stuck with models that are more likely to confidently hallucinate than indicate that the answer is unclear.

Because there is a tendency to personify AI, it is important for students to know that AI does not understand what it is generating, not like a person would. Consequently, sometimes AI might generate text that is inconsistent or self-contradictory. For instance, in March 2025 I searched "Carney 97" in Google to see why Prime Minister Mark Carney was wearing a hockey jersey with the number 97 on it. In the same response, Google's AI Overview initially indicated that Mark Carney turned 97 years old on March 20th, while also coming to the conclusion that he is 60 years old and not 97. Showing the example bel0w to students and spending time deducing how AI Overview strung together disconnected sources to generate this answer helps students better understand how AI models can generate contradictory information.

Considering Sources of Information

Google's AI Overview is different from many other AI assistants because typically the sources are listed (even if the generated information and the sources don't always quite match up). This doesn't seem to be typical for lots of other AI models. Often AI models act as black boxes, where they generate information, but it is not always clear where the information is coming from or how conclusions were reached. Unfortunately, when an AI model confidently presents information to a user, this can give a false sense of accuracy. It is important for us to know where our information is coming from and to use reliable sources.

A good starting point for teaching students about reliable sources of information is CTRL-F's Online Search Skills lessons. While students frequently use the Internet, they are often unaware of a lot of basic facts about the Internet. Teaching them the difference between an Internet browser, a search engine, and an online source of information is important. Similarly, if students find that Google's AI Overview is providing them with AI generated summaries, it is useful to practice assessing the sources the AI is drawing from. In the example below, when asked if a grizzly bear or a gorilla would win in a fight, AI Overview drew information from Xavier News, a student newspaper, and a website for fitness enthusiasts. Neither of these sources are particular authorities on the topic, which calls the AI generated summary into question.

The more you practice these skills, the more students will learn to question whether information is reliable. For instance, when we were practicing some of our search skills, one of my classes chose to search for grizzly bear, only to find that Google's AI Overview offered facts about the size of grizzlies that differ from its previous statement depicted above. Seeing these inconsistencies teaches us to opt for more reliable and authoritative online sources rather than generative AI sources.

Beyond issues of reliability and accuracy, when generative AI doesn't cite its sources, this can be disrespectful to people who wrote the original content. AI models function by scraping information from the Internet and other sources, generally without permission from the people who created that content. This process is exploitative and respecting the voices of others means not letting their words, stories and lived experiences be obscured and buried by generative AI.

Bias

AI also has significant issues with bias. This has been demonstrated by different machine learning algorithms well before generative AI hit the mainstream. For instance, in 2018, Amazon stopped using an AI pr0gram in their hiring process after it was found to discriminate against female applicants. In 2019, a risk prediction algorithm prioritized the treatment of White patients over Black patients. In both cases, race or gender were not explicit parts of the training data and they weren't explicitly programmed to discriminate. So what happened?

AI uses its training data to perform tasks or generate information. However, when training data demonstrates bias, AI models will reflect and reproduce that bias. For example, if Amazon trained its AI model on current employees' résumés and most current employees are male, résumés that are different from this training data will be downgraded. This could include résumés where the applicants attended women's colleges, participated in female-dominated clubs or activities or other more subtle factors. Often it is difficult to predict how training data might be biased. As a result, people must continually root out unconscious bias in the training data as new problematic content is generated.

Examples of bias are easiest to notice when it comes to AI image generators. Since AI image generators are trained on images that are scraped from the Internet, that means that AI will replicate racial and gender biases that exist in the training data. It will replicate ablest assumptions. It will generate information that is grounded in Western perspectives, given the Internet is dominated by that content. As author Shannon Vallor states, AI is a mirror that reflects our biases back at us.

To demonstrate this with students, I pulled images from this article by T.J. Thomson and Ryan J. Thomas. These images were generated using Midjourney using the attached prompts. When students were asked what they noticed, they were able to point out that the men were portrayed as older, while the women were younger. They also noticed that the men seemed to have the more complicated job titles, and that no people of colour were shown. From these images, a very specific vision of the news industry is illustrated, one which does not represent reality. To further explore these ideas, students also read the article "AI image generators tend to exaggerate stereotypes," which outlined other examples relating to ableism, racial and gender biases, and the difficulty tech companies face in addressing these biases.

Generative AI is often presented as a time saving measure that can help people quickly find the information they need. However, students need to understand that the information AI generates is prone to being inaccurate and biased. Understanding this drawback lets them proceed with caution as they consider whether they would recommend the use of AI, as I will discuss in part 4 of this series.