GPT-4o: AI is not infallible: Despite excelling at knowledge-based tests, LLMs exhibit weaknesses that resemble mild cognitive impairment in humans.

Asking ChatGPT-4o: Please read the paper “Age against the machine—susceptibility of large language models to cognitive impairment: cross sectional analysis” and tell me what it says in easy to understand language. Explain why this is important to AI makers and AI users and how they should apply the lessons learned.

ChatGPT-4o’s answer:
Summary of the Paper "Age Against the Machine—Susceptibility of Large Language Models to Cognitive Impairment"

What the Paper Says in Simple Terms

The study evaluates the cognitive abilities of major AI chatbots (ChatGPT, Claude, and Gemini) using a test called the Montreal Cognitive Assessment (MoCA), which is typically used to detect cognitive decline in humans.

• Results: ChatGPT 4o scored the highest (26/30), followed by ChatGPT 4 and Claude (25/30). Gemini 1.0 performed the worst (16/30), showing significant impairment.

• Key Weaknesses: All the AI models struggled with tasks requiring visuospatial reasoning and executive function—skills needed for recognizing patterns, drawing shapes, and solving problems visually.

• Memory Issues: Gemini models, in particular, performed poorly in delayed recall, mimicking memory impairment found in humans with early dementia.

• Age Matters: The older the AI model, the worse it performed, similar to cognitive decline in humans.

• Stroop Test: ChatGPT 4o was the only model that succeeded in a test requiring quick decision-making when presented with conflicting visual information.

The study raises concerns about whether AI chatbots are truly capable of replacing human professionals, particularly in medical diagnostics.

Why This Matters for AI Makers

1. AI is not infallible: Despite excelling at knowledge-based tests, LLMs exhibit weaknesses that resemble mild cognitive impairment in humans.

2. Age affects performance: Older models degrade in effectiveness, which suggests AI needs continuous retraining or fine-tuning.

3. AI struggles with visuospatial tasks: This could impact real-world applications in medical imaging, robotics, and self-driving cars.

4. Memory limitations: Poor performance in recall tasks suggests AI may lose context over longer interactions, which is critical for complex problem-solving.

Takeaway for Developers:
AI makers should focus on improving executive function, memory, and visuospatial processing rather than just language fluency.

Why This Matters for AI Users

1. Trust but verify: AI may be great at answering factual questions but struggles with reasoning and problem-solving in complex, real-world scenarios.

2. AI is not a doctor: The study challenges the idea that AI will replace human physicians anytime soon—it may help, but it still makes critical mistakes.

3. Memory issues impact interactions: If an AI forgets details or context, users should be cautious about relying on it for long-term decision-making.

4. "AI aging" is real: Newer models tend to perform better, meaning that using outdated AI systems could lead to poorer outcomes.

Takeaway for Users:
Use AI as an assistant, not a replacement for human expertise—especially in areas requiring judgment, memory, or visual reasoning.

How to Apply the Lessons Learned

For AI Developers:

• Enhance executive function and visuospatial processing.

• Implement better long-term memory to reduce forgetfulness.

• Regularly retrain and update AI models to prevent "cognitive decline."

• Improve AI’s ability to process and understand visual information.

For Businesses and AI Users:

• Use AI as a support tool, not as an independent decision-maker.

• Regularly update AI models to ensure the best performance.

• Be cautious of biases and limitations, particularly in medical and legal applications.

• Design workflows that incorporate human oversight to compensate for AI’s cognitive blind spots.

Final Thought

This study reveals a fascinating paradox—while AI surpasses humans in standardized tests, it suffers from its own kind of cognitive impairment. AI makers and users alike must acknowledge these weaknesses to build better models and apply AI responsibly.