GPT-4o: The hype around machine learning technologies leads to bad science as it creates an illusion of extracting true knowledge just by pattern recognition.

Question 1 of 2 for ChatGPT-4o: Please read the transcript of the video “Machine Learning Has a Pseudoscience Problem” and tell me what the most relevant, controversial and surprising statements are

ChatGPT-4o’s answer:
Based on the transcript of the video "Machine Learning Has a Pseudoscience Problem," here are some of the most relevant, controversial, and surprising statements:

Most Relevant Statements

1. Lack of Theoretical Foundation: Machine learning has a problem with doing science without theory. The reliance on achieving high classifier accuracy without proper theoretical underpinning is problematic.

   • "This sort of runaway idea that we can do science without theory...then everyone sort of believes that these Technologies are capable of extracting true knowledge...by having achieved high classifier accuracy on some natural data set."

2. Implicit Theorizing: Researchers often interpret patterns as confirmations of their intuitive ideas without explicit theorizing, leading to potential misrepresentations.

   • "If they're not explicitly doing the theorizing, then they're implicitly theorizing, which means that they're effectively trying to con you into believing whatever their intuitions were at the start of the research procedure."

3. Hype Narratives: The hype around machine learning technologies leads to bad science as it creates an illusion of extracting true knowledge just by pattern recognition.

   • "Because of all these hype narratives surrounding machine learning...you get a lot of bad science."

Most Controversial Statements

1. Machine Learning vs. Traditional Disciplines: The claim that machine learning researchers do not engage with the historical and theoretical context of the problems they tackle, unlike traditional social sciences or economics, is contentious.

   • "Machine Learners don't even read the history of their own work...the general rule is people in machine learning do not read."

2. Inadequate Peer Review: The peer review process in machine learning conferences is described as radically incomplete, which undermines the credibility of published research.

   • "The consensus is that the peer review process for these venues is wildly inadequate."

3. Disregard for Traditional Methods: The dismissal of traditional scientific approaches in favor of deep learning without understanding the complexity and history of the field being studied.

   • "Deep learning will solve the problem and I don't have to pay my dues and read about the methods in this field."

Most Surprising Statements

1. Historical Roots of Machine Learning: The connection of machine learning methods like MCMC sampling to World War II research, specifically the work at Los Alamos, is a surprising historical note.

   • "It was out of the research at Los Alamos that you got MCMC sampling...Metropolis-Hastings sampling...it goes back to the late 40s early 50s."

2. Organic Development of Current Practices: The explanation that the current state of machine learning practices and its issues with peer review and journal standards evolved organically rather than being a deliberate choice.

   • "It was never the case that people were applying machine learning to some problem in biology and then submitting that to a bio Journal...there's effectively no incentive to start journals."

3. Emergence of Pre-print Servers: The rise of pre-print servers as a dissemination method, which contributed to the lack of standard journals in machine learning.

   • "It's at the same time that you're getting pre-printing servers as the sort of way that stuff is disseminated...there's incentive against starting journals."

These statements highlight significant concerns about the epistemic status, methodological rigor, and peer review processes in the field of machine learning, contrasting sharply with traditional scientific disciplines.

Question 2 of 2 for ChatGPT-4o: What can the consequences be for AI makers, the value and use of LLMs in scientific research, the quality of the research itself and for scholarly publishing?

ChatGPT-4o’s answer:
The consequences of the issues raised in the transcript for AI makers, the value and use of Large Language Models (LLMs) in scientific research, the quality of the research itself, and scholarly publishing can be significant. Here are the detailed potential consequences:

For AI Makers

1. Credibility and Trust: The perceived lack of theoretical foundation and rigorous peer review can erode trust in AI technologies developed by AI makers. Users and stakeholders may question the validity and reliability of AI models.

   • Reputation Damage: Companies and researchers might face reputational damage if their AI models are perceived as pseudoscientific.

   • Regulatory Scrutiny: Increased scrutiny from regulatory bodies could arise, demanding more transparency and accountability in AI development.

2. Ethical and Legal Implications: Misleading claims about the capabilities of AI systems could lead to ethical breaches and potential legal consequences.

   • Liability Issues: If AI systems fail or cause harm due to inadequate scientific grounding, AI makers could be held liable.

For the Value and Use of LLMs in Scientific Research

1. Misapplication of Technology: LLMs and other AI tools might be applied to scientific problems without proper domain knowledge, leading to erroneous conclusions and potentially harmful decisions.

   • False Positives/Negatives: AI models might produce high accuracy in classification tasks but still misinterpret data, leading to false scientific claims.

2. Diminished Value: If LLMs are not grounded in solid theoretical foundations, their perceived value in scientific research might diminish.

   • Reduced Adoption: Researchers might become skeptical of using LLMs, preferring traditional methods that have a stronger theoretical basis.

For the Quality of Research Itself

1. Lowered Standards: The lack of rigorous peer review and theoretical grounding can lead to a proliferation of low-quality research.

   • Influx of Poor-Quality Papers: Conferences and journals might be flooded with papers that lack scientific rigor, making it harder to discern valuable contributions from noise.

2. Scientific Misconceptions: Misinterpretation of results due to inadequate theorizing and reliance on pattern recognition without understanding can propagate misconceptions in scientific literature.

   • Stagnation of Knowledge: Poor-quality research can slow down the progress of knowledge by building on flawed premises.

For Scholarly Publishing

1. Erosion of Trust in Peer Review: If the peer review process in AI and machine learning continues to be perceived as inadequate, it could undermine trust in scholarly publishing as a whole.

   • Questionable Validity: Published papers might be viewed with skepticism, leading to a devaluation of published research.

2. Need for Reform: The issues highlighted may push for reforms in the peer review process and publishing standards within AI and machine learning disciplines.

   • Stricter Standards: Journals and conferences might adopt stricter standards and involve domain experts in the review process to ensure the validity of interdisciplinary research.

3. Emergence of New Publishing Models: The reliance on pre-print servers and the inadequacies of traditional peer review might lead to new models of scholarly communication and validation.

   • Collaborative Review Platforms: Platforms that enable open peer review and post-publication critique could become more prominent.

Summary

In summary, the consequences of the pseudoscientific tendencies in machine learning and AI research are far-reaching. For AI makers, these issues can lead to a loss of credibility and increased regulatory scrutiny. For the use of LLMs in scientific research, there is a risk of misapplication and diminished perceived value. The quality of research can suffer, leading to a proliferation of low-quality papers and scientific misconceptions. Finally, scholarly publishing might face an erosion of trust, necessitating reforms and potentially leading to new publishing models.