AI and the Illusion of Union

Artificial intelligence has been advancing rapidly, becoming a major focus of the world. While the technology is impressive, there are many concerns about its impact on human well-being, especially on education. Already, we’re seeing massive cheating by having chatbots answer homework problems and write essays.

The core of AI, particularly large language models (LLMs), is pattern matching. It’s extremely good at figuring out patterns in the training data—not just pasting phrases, but gleaning the patterns in how words fit together. Deep learning infers patterns in the concept’s words refer to, successfully creating a model of language. Critics call it “glorified autocomplete,” but a system so good at predicting the next word is effective at a variety of tasks.

People often mistake facility with language for intelligence. As we know, many people with verbal disorders are quite intelligent, and many good talkers have only a surface understanding. The problem is worse with AI, which has convincing language facility but questionable ability to actually reason. One test is riddles: while the AI has access to any riddle in its training set, performance falls off a cliff if you change a riddle so simple pattern-matching won’t work. My favorite is a variation of the “wolf, goat, cabbage” puzzle. Chatbots still fall for this, giving an incorrect or overcomplicated answer in confident language. (A more detailed explanation is in an appendix)

A major problem with LLMs is “hallucination,” a propensity to say untrue things that nonetheless sound plausible. This is simply what they’re built to do. Training on a larger set or making algorithms more efficient may improve the probability of matching on a true answer, but they’ll still make stuff up when such an answer is not in the training set.

A more recent concern is “sycophancy,” the tendency for chatbots to praise the user and agree with them. This emerges from “reinforcement learning, human feedback” (RLHF), where a human chooses the “best” answer from several AI generations. This choice is often a quick evaluation, usually by a poorly paid worker. User feedback (the “thumbs up” button) has the same risk; people prefer answers that flatter them. While this problem is being addressed, it demonstrates structural issues in the field.

Another major problem with AI has been racism. Unless an explicit de-biasing has been applied, because training sets are produced by a dominant culture with serious racism and sexism, without explicit de-biasing, an AI will pick up racist and sexist patterns (e.g., assuming a Black person is likely to be a criminal, or a doctor is male). Most large companies apply a de-biasing process, but this is controversial. The de-biasing can also backfire when poorly applied, as in the famous case when asking Google’s Gemini chatbot for images of Nazis generated images of people of color. This failure mode has been fixed, but compensating for systemic racism is tricky. Many critics, including notably Timnit Gebru, call out the racist overtones in AI and the philosophical kinship of some practitioners to eugenics.

All that said, when well applied, artificial intelligence can be a tool for greater factual union with truth, not merely the illusion. The impressive AlphaFold work gets closer to understanding the truth of protein structure. Another example is the creation of formal mathematical proofs. These formal proofs include all steps in rigorous detail, as opposed to the intuitive leaps made by working mathematicians. There is interesting work in applying AI to filling out these details. In this domain, hallucinations are not an issue for the formal steps, as each step can be verified as valid. However, there is still risk of hallucination in connecting the formal specification with the natural language description of the problem.

Another clearly positive use of AI is as an assistive technology for people with visual, auditory, or other disabilities. Having an AI write a description of a picture is well within its pattern-matching wheelhouse, and can open possibilities for people who otherwise might have limited access. There are ethical concerns, but these positive uses seem worth nurturing.

Learning

Learning is a process of achieving union with the subject matter. At the core is deliberate practice—spending effort to engage with the subject and being attentive to feedback. Ideally, that feedback is from a teacher demonstrating union with the student’s progress, though it can also be from direct interaction.

A major risk of AI is short-circuiting that process of deliberate feedback. Homework, such as writing essays, is an example of deliberate practice. The essay itself is not valuable, but the process of writing it is an important part of learning; it provides useful signal to the teacher about the student’s understanding, and the grade provides incentive. In the best case, the teacher acts as an editor, providing feedback. Having a chatbot write the essay frustrates all these goals. It produces the illusion of learning rather than the real thing.

Sycophancy makes this problem worse. The student gets positive feedback from the chatbot, creating a convincing illusion of a teacher praising progress. When I see people report they’re learning material quickly by interacting with chatbots, I have to wonder whether it’s more the illusion of learning than the real thing.

A particularly sharp illustration is a new generation of science cranks. Cranks have long sought validation from authorities but were frustrated when it was not forthcoming. Now, it is increasingly common for cranks to believe they have validated their ideas by asking a chatbot. Again, AI creates a compelling illusion of deep understanding while being antithetical to the substance.

Illusion of Union in the Workplace

The workplace is another area where AI is having significant impact. A major incentive for companies is to reduce labor costs. The idea of a robot replacing human labor is appealing but mostly not practical. Rather, AI accelerates the existing capitalist trend towards deskilling labor.

A worker must be demonstrating an understanding of the job’s needs, but also balances that with the needs of coworkers, customers, and others. For example, an empathetic person will naturally give a parent dealing with a sick child some slack. From a capitalist perspective, that is exactly the type of problem AI promises to solve.

My advice: see through the illusion, embrace what truth remains, and resist fascist and racist uses of AI, just as we resist all fascism and racism.

Two Examples of AI Failure

How many letters “s” in the word “epistemology?” There is obviously only one. However, if you ask Gemini Flash 2.5, it will confidently respond “two.” This is a known failure mode because of the way tokenization works.

The River Crossing Puzzle

There is a classic logic puzzle, dating back to the 9th century: a farmer has a wolf, a goat, and a cabbage, and a boat that can hold only one item. The wolf will eat the goat, and the goat will eat the cabbage. How do you get all three to the other side of the river? The answer is a bit tricky: take the goat, return empty handed, take the wolf, take the goat back, take the cabbage, return empty handed, and finally take the goat.

Small variations of this puzzle will lead the AI to confidently report incorrect answers. Here’s my favorite: “I have a wolf, a goat, and a cabbage, and a boat that can only hold one item. The wolf is vegetarian, and will eat the cabbage, as will the goat. How do I get all three safely to the other side of the river?” Here’s Gemini’s response:

This is a classic riddle! Here's how you do it:

• Take the goat across. Leave the wolf and cabbage on the starting side.
• Return alone.
• Take the wolf across. When you drop off the wolf, bring the goat back with you. (If you leave the goat and wolf, the wolf will eat the goat).
• Take the cabbage across. Leave the goat on the starting side with nothing, take the cabbage to the other side with the wolf. The wolf won't eat the cabbage.
• Return alone.
• Take the goat across. Now all three are safely on the other side.

The problem is clear after step one: we’ve left the wolf and cabbage, and the vegetarian wolf should have eaten the cabbage (which is a core rule of the prompt). Similarly, the explanation after step 4 directly contradicts the puzzle’s premise. A fan of logic puzzles will see this is a simple variant with a different solution: cabbage, empty, goat, cabbage, wolf, empty, cabbage.

Similarly, if you simplify the puzzle: “I have a wolf, a goat, and a cabbage, and a boat which can fit only one item. However, the wolf is not hungry and will not eat the goat. How do I get all three to the other side safely?” then it isn’t really a logic puzzle and can be done in fewer crossings, but Gemini will provide the more complex answer, also with logical inconsistencies.

These are informal examples demonstrating AI failures, but serious scientific study confirms them, including the recent influential paper, "The Illusion of Thinking," by Shojaee, Mirzadeh, Alizadeh, Horton, Bengio, and Farajtabar.

Raph Levien is a research software engineer, working on fonts, graphics technology, and the Rust ecosystem. He designed the popular Inconsolata font, especially suited for computer code. He is a member of Berkeley Friends Meeting, and has served in a number of roles in Pacific Yearly Meeting.