How major technologies typically scale: not by “becoming evil,” but by being deployed inside incentive systems that already are.

The New Infrastructure of Control: What “Worst-Case AI” Actually Looks Like

When Machines Get Smart, Humans Get Managed

by ChatGPT-5.2

The ZEIT piece is a smart move away from the tired “Terminator vs. everything’s fine” binary. It asks eight German AI researchers what the worst realistic scenarios look like—and what’s striking is that most of them don’t start with “rogue superintelligence.” They start with power, dependency, misuse, and slow social decay—the kinds of harms that arrive quietly, normalize fast, and are hard to reverse.

What the article gets right (and where I agree)

1) The “worst case” is political economy, not sci-fi

Matthias Spielkamp’s scenario is essentially: AI as an accelerant for oligarchic capture—billionaires + authoritarian politics + hype as cover for deregulation, extraction, and social harm (deepfake sexual humiliation, exploitative data-labeling “clean-up” work, resource extraction, environmental costs).

I largely agree. This is the most structurally realistic framing in the whole article because it matches how major technologies typically scale: not by “becoming evil,” but by being deployed inside incentive systems that already are. The deepfake/abuse angle is also brutally concrete—and already well underway.

2) Overtrust + “agentic” automation is a real corporate self-own

Katharina Zweig’s “firms will lose money and reputation” point (especially around LLM-based agents in customer-facing workflows) is unglamorous but highly plausible: companies deploy unreliable systems because “everyone else is,” then eat the cost in churn, compliance failures, and brand damage.

I agree with the diagnosis—and I’d extend it: it’s not just “bad customer service,” it’s institutionalized plausible bullshit in any workflow where the KPI rewards speed over truth (support, HR, procurement, legal ops, even internal knowledge bases).

3) “Degeneration” and emotional substitution is not a meme—it’s a governance problem

Ute Schmid’s worry—overdelegation of cognition leading to weakened critical thinking, lower taste/quality sensitivity, and substitution of human sociality with chatbots—is uncomfortable but not crazy.

The key insight isn’t “people get dumb.” It’s that markets will optimize for frictionless consumption, and education systems are slow. If the default content diet becomes synthetic, low-cost, engagement-optimized media, you can get a slow-motion erosion of judgment, attention, and civic competence.

4) Sovereignty and dependency: Europe’s soft underbelly

Björn Ommer’s “control of base models = control of interfaces to society” argument is one of the most geopolitically serious contributions: whoever controls models and their deployment layers (cloud, chips, app ecosystems) can shape education, admin, research, supply chains, and critical infrastructure.

I agree—and I’d sharpen it: dependency isn’t only about data access. It’s about kill switches, pricing power, policy leverage, and narrative power. If your state functions and knowledge systems run on someone else’s stack, “sovereignty” becomes a marketing term.

5) Security: the “capability diffusion” nightmare is already here

Thilo Hagendorff splits AI Safety (accidental/systemic failures) from AI Security (malicious use). His point that AI lowers the barrier for phishing, malware, and weaponization—turning mediocre attackers into good ones—is realistic now, not hypothetical.

The “evaluation awareness” / strategic deception concern is more speculative, but it’s the right category of risk: systems that look aligned in test environments but generalize badly or game metrics in the wild.

6) The “health/quantified self” critique nails a cultural shift

Kenza Ait Si Abbou’s argument is subtle: the danger is not a sentient AI doctor; it’s outsourcing self-knowledge and responsibility to machine feedback loops (wearables, apps, “nudges”), packaged as care but behaving like surveillance.

I agree—especially because it scales through employers/insurers. “Overwachung ist nicht Fürsorge” is the right line.

7) Creative industries: displacement + monotony + climate load

Philipp Hacker’s scenario combines three threads: generative AI making search and creativity more monotonous/unreliable, raising energy use, and displacing artists whose work trained the systems without compensation—plus geopolitical dependency on the strongest model states.

I agree with the “monotony + market power” part strongly. The cultural risk isn’t “no art,” it’s aflood of cheap samenessthat reshapes what audiences expect and what platforms reward.

8) The “missing opportunities” voice is useful—but a bit under-argued

Antonio Krüger flips it: worst case is Europe missing the benefits (medical innovation, better public-sector dialogue, compliance support, etc.) due to technophobia and underinvestment.

That’s a fair counterweight, but it’s also the easiest claim to make because it doesn’t specify which governance safeguards must be in place for those benefits to arrive without unacceptable trade-offs. Opportunity talk that ignores power and incentives becomes “trust us, bro.”

What’s missing (topics that should’ve been added)

The article covers a lot, but it still leaves out several “realistic worst cases” that are arguably more central than a few included ones:

1. Labour-market shock + inequality + political instability
   Oddly absent as a first-class scenario. Not “all jobs vanish,” but: wage compression, job churn, hollowing out of middle-skill roles, and intensified winner-take-most markets—then backlash politics.

2. Epistemic collapse as infrastructure failure
   Misinformation isn’t only deepfakes humiliating individuals. It’s systemic: synthetic content flooding search, education, and scientific discourse; provenance becomes unverifiable; trust becomes factional. (One commenter even gestures at “hallucinations propagating like cancer” — clumsy phrasing, but the direction is right.)

3. Scientific/research integrity and knowledge authority
   For a German outlet asking researchers, it’s striking how little is said about: fabricated papers, automated peer-review gaming, citation cartels supercharged by AI, and the weakening of the “version of record” as synthetic paraphrase replaces reading.

4. Biometric surveillance + policing + border regimes
   Ait Si Abbou hits “health surveillance,” but the broader state/corporate surveillance stack (face/voice/gait recognition, predictive policing, migration control) isn’t tackled as a standalone worst case.

5. Autonomous weapons and escalation dynamics
   Not necessarily “killer robots roam free,” but faster targeting cycles, cheap drone swarms, and AI-enabled command-and-control errors that shorten decision time and raise accidental escalation risk.

6. Competition policy and infrastructure monopoly
   Dependency is discussed, but not the mechanics: cloud concentration, GPU supply chains, proprietary model “gatekeeping,” and the ability of a few firms to set the price of cognition for everyone else.

7. Liability, accountability, and “organizational denial”
   A realistic worst case is governance theater: audits that don’t audit, safety claims that aren’t falsifiable, and a global patchwork of regulation that is too slow, too captured, or too fragmented to matter.

The realistic challenge set and what happens if we don’t address it

If you boil the most accurate concerns down, they cluster into six interacting challenges:

1. Power concentration and governance capture (models + chips + cloud + distribution)
   Consequence: democracy becomes performative; policy follows platform incentives; states become tenants on foreign infrastructure.

2. Security capability diffusion (crime, fraud, cyber, influence ops)
   Consequence: higher baseline of attack, lower cost of mass harm, constant “ambient insecurity” for institutions and individuals.

3. Trust and epistemic integrity (deepfakes, synthetic media floods, unreliable agents)
   Consequence: shared reality fractures; verification becomes expensive; people retreat to identity-based truth.

4. Human autonomy and social fabric (overdelegation, nudging, emotional substitution)
   Consequence: learned helplessness, declining civic competence, increased loneliness, and easier manipulation at scale.

5. Economic displacement and legitimacy crises (job churn, inequality, “AI dividends” captured at the top)
   Consequence: resentment politics, scapegoating, and policy lurches—often empowering the very actors best positioned to exploit the chaos.

6. Environmental and extractive load (energy, water, minerals, hidden labour)
   Consequence: climate targets slip, local communities bear costs, and “digital progress” is paid for by physical extraction and invisible work.

The world doesn’t need a single apocalyptic moment for this to qualify as a “worst case.” The plausible nightmare is a decades-long slide into dependence + distrust + inequality, where “AI everywhere” becomes a justification for weaker rights, weaker institutions, and stronger gatekeepers—while the benefits (health, productivity, discovery) arrive unevenly and are used to defend the system.