The article’s central thesis—that LLMs create an illusion of intelligence and that their uncontrolled deployment is dangerous—is correct. Its empirical observations deserve serious attention.

“The Communicational Illusion—Truth, Polemic, and Reality Behind the LLM Debate”

by ChatGPT-5

The article “The party trick called LLM – blowing away smoke and break some mirrors”presents a provocative, polemical critique of large language models (LLMs) and their use in public-facing contexts. Drawing on empirical testing across 36 Dutch municipal chatbots, cognitive science research, and a strong rhetorical style, the author argues that current AI systems are communicational illusions—text calculators that mimic human language without meaning, understanding, or cognition. While much of this critique is justified and aligns with evidence from real-world performance, some claims overreach, collapse necessary distinctions, or obscure useful and demonstrably beneficial applications of modern LLMs.

What follows is a balanced examination of the key arguments: where they are correct, where they are overstated, and what they imply for the future of responsible AI deployment.

Points of Agreement

1. Empirical failures in government chatbots are real and serious

The reported findings—100% accessibility failure, 100% error on opening hours, failure to provide correct answers about holidays, and alarming errors on tax remission—demonstrate the fragility of AI deployments in civic services. These are not “edge cases,” but basic operational tasks. The failures point to:

• Lack of governance

• Lack of testing

• Lack of supervised domain-specific knowledge

• Poor or nonexistent accessibility compliance

These criticisms are valid and important. They echo wider European concerns around automated decision-making, accuracy, and public trust.

2. The “communicational illusion” thesis is accurate

Humans automatically ascribe agency and intention to language. When text appears coherent, readers instinctively assume an intelligent mind is behind it. The article rightly anchors this in Gricean cooperation principles and cognitive biases. LLMs exploit these biases, even though they do not “understand” in a biological or phenomenological sense.

3. Marketing hype distorts public comprehension

The critique of marketeers, pseudo-experts, and “innovation theatre” is sharp but accurate. The hype cycle encourages careless adoption, overpromising, and unrealistic expectations of AI capabilities. This is especially dangerous in the public sector, where mistakes have legal and human consequences.

4. “Human-in-the-loop” cannot be a fig leaf for accountability

The article rightly argues that the phrase has become a cliché. In practice, “human-in-the-loop” is often used to offload responsibility to users without providing them with tools, training, or oversight capacity. This is a governance failure disguised as safety.

5. LLMs do not replace human thinking

The core argument—that writing requires thinking, and machines do not think—is valid and supported by cognitive science. LLMs recombine patterns statistically; they do not reason or possess intentionality.

Points of Disagreement

1. Overstatement: “A machine cannot think”

This assertion collapses nuanced debates in cognitive science, philosophy of mind, and AI theory. The article defines “thinking” strictly as human biological cognition, but:

• Reasoning processes

• Planning

• Symbolic manipulation

• Simulation

• Abstract inference

can be implemented computationally, even if they differ from human phenomenology. More precise phrasing would be: current LLMs do not engage in human-like intentional reasoning or semantic understanding. This is true. But dismissing all forms of machine reasoning is scientifically outdated.

2. The claim that LLMs are “synthetic bullshit” goes too far

LLMs demonstrably produce useful outputs in numerous domains:

• summarisation of known texts

• code generation

• structured classification

• translation

• information extraction from large corpora

• pattern recognition in unstructured data

They can be reliable within defined constraints and with proper guardrails. In calling all output “bullshit,” the article ignores years of evidence showing the utility of language models as assistive tools.

3. The dismissal of efficiency gains oversimplifies reality

While many tasks require critical thinking, many others are mechanical, repetitive, or require pattern-based processing at scale. Examples include:

• normalising reference formats

• drafting boilerplate content

• annotating datasets

• summarising meeting notes

The claim that “editing is faster without machines” is empirically false in many contexts.

4. Ignoring the spectrum of LLM architectures and safety design

The article treats all chatbots as identical in capability, quality, and architecture. In reality, there are significant differences between:

• rule-based chatbots

• fine-tuned domain models

• general LLMs

• agentic systems

• retrieval-augmented generation (RAG) systems

This matters because public-sector chatbots often deploy outdated or poorly integrated systems.

5. Neglecting the systemic causes of LLM failure in government deployments

The critique blames the technology, but the deeper issue is:

• procurement systems

• low technical literacy

• cost-cutting

• outsourcing to vendors

• weak evaluation frameworks

LLMs used incorrectly do not prove LLMs are inherently misleading.

What the Article Gets Right—and Why It Matters

The article accurately signals a set of genuinely problematic issues:

• accessibility failures undermine democratic digital inclusion

• illusionary thinking encourages overdependence on automated outputs

• marketing-driven deployment prioritises speed over safety

• deskilling risks threaten professional competencies and critical thinking

• lack of accountability creates legal, ethical, and operational hazards

These concerns are valid, urgent, and supported by real-world evidence. The article’s rhetorical tone can obscure its essential message: AI systems must be deployed with rigour, transparency, testing, and governance.

Advice and Recommendations

For AI Users

1. Assume the model is wrong unless you verify it.
   Treat outputs as drafts or suggestions, not truths.

2. Keep critical thinking intact.
   Use AI for amplification, not substitution.

3. Ask simple questions: “What is missing?”
   LLMs rarely admit gaps, but gaps are the biggest risk.

4. Avoid emotional anthropomorphism.
   Do not treat AI as a companion, therapist, or authority.

For Governments and Public Institutions

1. Prohibit deployment without accessibility compliance.
   Every interface must pass WCAG standards before launch.

2. Require domain-specific tuning + retrieval mechanisms.
   Public services must use authoritative databases, not general models.

3. Mandate rigorous testing and certification before rollout.
   Healthcare, justice, taxation, and immigration require the highest bar.

4. Implement procurement standards for AI.
   Include performance benchmarks, auditability, and red-teaming.

5. Install legal accountability frameworks.
   If a chatbot provides harmful misinformation, liability must be clear.

For Regulators

1. Create enforceable accuracy and transparency requirements.
   Not vague ethics; actual measurable obligations.

2. Ban unsupported claims in marketing.
   AI vendors must substantiate capabilities, limitations, and risks.

3. Introduce mandatory logging and audit trails.
   Regulators must be able to trace hallucinations and failures.

4. Require disclosure of training data categories and provenance.
   This protects creators, ensures bias analysis, and enhances accountability.

5. Create a tiered regulatory regime.

   • High-risk (public services, health, legal advice)

   • Medium-risk (education, HR)

   • Low-risk (creative tools)

6. Invest in digital literacy and cognitive resilience programs.
   Societies need critical thinkers, not passive consumers of synthetic text.

Conclusion

The article’s central thesis—that LLMs create an illusion of intelligence and that their uncontrolled deployment is dangerous—is correct. Its empirical observations deserve serious attention. Yet its sweeping dismissal of all AI utility and its rigid anthropocentric framing oversimplify the landscape and obscure the very real benefits that responsibly deployed AI can provide.

The correct stance lies in the middle ground: respect the limitations, leverage the strengths, enforce accountability, and build a culture of critical digital literacy.

When these conditions are met, LLMs cease to be parlor tricks and become tools—tools that must be governed with discipline, humility, and transparency.