When the People Building the Models Ask the Court to Hit Pause

by ChatGPT-5.2

In Anthropic PBC v. U.S. Department of War, et al., a group of 37 engineers, researchers, and scientists employed at OpenAI and Google (including Google DeepMind) asked the Northern District of California for leave to file an amicus briefsupporting Anthropic’s request for a temporary restraining order. They emphasize they are speaking in their personal capacities, not for their employers, and that their companies are listed only to convey perspective—not endorsement. Their underlying message is unusually stark for “industry insiders”: the technology is powerful, the guardrails are not optional, and the government’s response to a vendor insisting on restrictions looks like punitive overreach that could chill safety debate across the entire frontier AI ecosystem.

What the amicus brief says happened

The amici describe the dispute as arising after the Pentagon allegedly threatened to designate Anthropic a “supply chain risk” if Anthropic refused to remove limitations on uses of its AI systems for (1) domestic mass surveillanceand (2) fully autonomous lethal weapons systems. When Anthropic maintained those limitations—its “red lines,” as the amici frame them—the Pentagon allegedly followed through with the “supply chain risk” designation. The amici argue that, if the government disliked the contract terms, it could have simply terminated the contract and bought from another vendor, rather than “recklessly” invoking national-security procurement authorities meant for foreign compromise and genuine supply-chain threats.

That framing matters because it casts the designation not as a routine procurement move, but as a signal to every frontier lab: if you publicly insist on safety boundaries—particularly boundaries that constrain intelligence/military ambition—you may be punished in ways that damage your ability to operate in the defense ecosystem and beyond.

Their three core arguments (and what’s doing the real work)

1) The “supply chain risk” designation is retaliation dressed up as national security

The amici claim the “supply chain risk” mechanism is meant for excluding vendors who pose genuine integrity or adversary-compromise risks, and that using it in a contract dispute over restrictions is arbitrary and harmful. They go further: it introduces unpredictability into the AI sector, undermines the “flow of ideas” between competitors, and chills open deliberation about risk—because “one lab” being silenced reduces the industry’s ability to innovate solutions and weakens public debate.

ChatGPT’s take: this is the most politically explosive argument, and it’s also the most strategically “industry-protective.” If you accept the amici’s characterization (designation as punishment for insisting on guardrails), then the government isn’t merely choosing a supplier—it’s shaping speech and behavior in a market where national-security money can function as coercive gravity. Even if one brackets the constitutional rhetoric, the policy point stands: using exceptional national-security exclusion tools as leverage in commercial disagreements creates a fear-based compliance culture. In frontier AI, that is a recipe for “quiet alignment” rather than genuine safety engineering.

Where I’m slightly less convinced is that the brief (as we see it here) cannot show the full factual record of negotiations—so the retaliation framing is compelling, but it’s ultimately a narrative claim unless the court record substantiates it. Still, as a governance posture: even the appearance of retaliatory misuse of “supply chain risk” tools is corrosive.

2) Anthropic’s “red lines” are not ideological—they are technical, and they’re necessary

The amici argue the concerns motivating Anthropic’s restrictions are widely shared in the scientific community and follow from “any reasonable evaluation” of present frontier AI capabilities. They use a simple analogy: some technologies can be used in dangerous environments, but you don’t allow it because the consequences are catastrophic.

They underline a broader governance vacuum: there is (they claim) no comprehensive U.S. federal framework that meaningfully constrains military/intelligence use of AI for domestic surveillance at scale, and no enforceable legal standard governing fully autonomous targeting decisions. In that vacuum, contractual and technical restrictions imposed by vendors become “vital safeguards.”

ChatGPT’s take: I agree with the essence of this argument. It’s not that AI companies are morally purer than states; it’s that (right now) the most enforceable “kill switch” often is at the vendor layer—access controls, terms, usage policies, gating, audits, or refusal to provide certain capabilities. If public law is thin or permissive, procurement contracts and technical architecture become de facto governance.

But there’s an irony the brief doesn’t fully confront: vendor-imposed guardrails are also private power. Today they restrain the state; tomorrow they can be reshaped by market incentives, capture, secrecy, or geopolitical pressure. So the right long-term endpoint is not “let vendors govern war and surveillance”—it’s: use the vendor choke-point as a temporary safety belt while public law catches up.

3) The two prohibited use cases are democracy-scale risks, not “edge cases”

The brief treats these as structurally dangerous deployments:

• AI-enabled domestic mass surveillance would fuse already-extensive data streams (cameras, location data, transactions, social graphs, data brokers) into a unified, real-time population monitoring instrument. They stress the “panopticon effect” (chilling speech and participation) and argue the harm arises even if never abused, because awareness of capability changes behavior.

• Fully autonomous lethal weapons systems are described as technically unready: models can degrade in novel conditions, hallucinate, and cannot reliably distinguish combatants from civilians; their internal reasoning is opaque; lethal errors are irreversible; and accountability structures break when the agent is “the decider.”

ChatGPT’s take: I largely agree with the risk framing, and I think the brief is strongest when it talks like engineers: uncertainty, distribution shift, opacity, irreversibility, and accountability gaps. Those are not political buzzwords; they are operational realities.

Two nuances I would add:

1. “Mass surveillance” isn’t binary. There’s a spectrum from targeted analysis (with warrants, minimization, and independent oversight) to dragnet correlation across whole populations. The brief is right to fight the extreme end—especially if the Pentagon sought domestic capability. But reform needs to precisely define what is prohibited versus what is permitted under strict conditions—otherwise agencies route around restrictions through subcontractors, data brokers, or “dual-use” tooling.

2. Autonomy creeps. Many systems are sold as “decision support” but drift into “decision replacement” under time pressure, doctrine, or operator workload. The remedy isn’t only “humans in the loop” as a slogan; it’s designing meaningful human control with auditable decision points and refusal modes that do not collapse during operations.

Who signed (names and organizations)

Counsel for amici (organization)

• Nicole Schneidman — AI for Democracy Action Lab, Protect Democracy Project

• Deana K. El-Mallawany — AI for Democracy Action Lab, Protect Democracy Project

• Ori Lev — AI for Democracy Action Lab, Protect Democracy Project

Individual amici (signing in personal capacities; employers listed for perspective)

OpenAI

• Grant Birkinbine — Member of Technical Staff, Security Engineer, OpenAI

• Anna-Luisa Brakman — Member of Technical Staff, OpenAI

• Brian Fioca — Member of Go to Market Staff, OpenAI

• Aaron Friel — Member of Technical Staff, Software Engineer, OpenAI

• Leo Gao — Member of Technical Staff, OpenAI

• Manas Joglekar — Member of Technical Staff, OpenAI

• Teddy Lee — Member of GTM Staff, OpenAI

• Soukaina Mansour — Creative Community Lead, OpenAI

• Pamela Mishkin — Research, OpenAI

• Roman Novak — Research Scientist, OpenAI

• Zach Parent — Forward Deployed Engineer, OpenAI

• Andrew Schmidt — Model Designer, OpenAI

• Jordan Sitkin — Member of Technical Staff, OpenAI

• Chang Sun — Member of Data Science Staff, OpenAI

• Jonathan Ward — Member of Technical Staff, OpenAI

• Jason Wolfe — Member of Technical Staff, OpenAI

• Gabriel Wu — Member of Technical Staff, Research Engineer, OpenAI

• Cathy Yeh — Member of Technical Staff, OpenAI

• Jelle Zijlstra — Member of Technical Staff, OpenAI

Google / Google DeepMind

• Sarah Cogan — Senior Software Engineer, Google DeepMind

• Jeff Dean — Chief Scientist, Google

• Michael Dennis — Senior Research Scientist, Google DeepMind

• Sanjeev Dhanda — Senior Staff Software Engineer, Google DeepMind

• Rasmi Elasmar — Senior Research Engineer, Google DeepMind

• Edward Grefenstette — Director of Research, Google DeepMind

• Alexander Irpan — Research Scientist, Google DeepMind

• Rishub Jain — Research Engineer, Google DeepMind

• Kathy Korevec — Director, Product, Google

• Shrinu Kushagra — Research Scientist, Google

• Sharon Lin — Research Engineer, Google DeepMind

• Ian McKenzie — Research Engineer, Google DeepMind

• Noah Siegel — Senior Research Engineer, Google DeepMind

• Sean Talts — Staff Software Engineer, Google

• Alexander Matt Turner — Research Scientist, Google DeepMind

• Anna Wang — Research Scientist, Google DeepMind

• Zhengdong Wang — Senior Research Engineer, Google DeepMind

• Kate Woolverton — Senior Software Engineer, Google DeepMind

What the Pentagon should have done to prevent this

If the amici’s account is even directionally true, the Pentagon’s failure was not “picking a tough negotiating stance.” It was using the wrong instrument for the wrong problem, and doing so in a way that predictably detonates trust.

Here’s what prevention should have looked like:

1. Separate procurement leverage from exclusion authorities.
   “Supply chain risk” tools should be tightly scoped to integrity/compromise criteria with clear evidentiary thresholds and process protections—not used as bargaining cudgels when a vendor enforces usage restrictions.

2. Pre-negotiate red-line categories as policy, not vendor preference.
   The Pentagon should have established (and publicly defensible) doctrine on:

   • whether any DoD component may pursue domestic surveillance tooling, and under what statutory authority;

   • what “meaningful human control” means for targeting systems;

   • what auditing, logging, minimization, and oversight are mandatory.
     If government policy is ambiguous, it creates incentives to pressure vendors to “just make it work.”

3. Build a compliance-by-design procurement framework.
   Contracts for frontier AI should embed:

   • explicit prohibited-use categories,

   • inspection/audit rights,

   • model access controls and kill-switch terms,

   • incident reporting and red-team obligations,

   • third-party oversight hooks (IG, PCLOB-style review, or independent technical auditors).
     In other words: don’t treat “guardrails” as an obstacle—treat them as a deliverable.

4. Create vendor-safe channels for dissent and safety escalation.
   If a vendor flags unacceptable risk, there should be a structured escalation pathway (legal + technical + oversight) rather than a coercive standoff. A national-security customer should be able to say: “We dispute your risk assessment—let’s adjudicate it under a governance process,” not “comply or we brand you a national-security threat.”

What the Pentagon should do now to remedy it

Remedy requires more than winning a motion. It requires repairing the signal sent to the entire ecosystem.

1. Pause and review the designation under an independent process.
   If the designation is not grounded in actual supply-chain compromise criteria, it should be suspended or withdrawn with a written rationale—otherwise every future “safety boundary” becomes a potential trigger for punitive tools.

2. Issue a public procurement principle: safety restrictions are negotiable terms, not disloyalty.
   The Pentagon can say, plainly: “We will not use exclusion authorities to retaliate against vendors for advocating safety guardrails or declining prohibited uses.” That single sentence would do more to restore trust than ten closed-door meetings.

3. Stand up a “Frontier AI Use Review Board” for sensitive deployments.
   A joint legal–technical–oversight panel (DoD + independent civil liberties oversight) that evaluates proposed high-risk uses:

   • domestic-data fusion proposals,

   • autonomous engagement chains,

   • surveillance-adjacent capabilities,

   • and any request to relax vendor restrictions.

4. Codify “meaningful human control” and auditability in acquisition standards.
   Make it non-optional: immutable logs, explainability limits acknowledged, human decision gates, and refusal modes. Require evaluations under realistic operational stress, not just lab demos.

5. Reduce the incentive to route around restrictions.
   If domestic surveillance is off-limits (or strictly limited), close the loopholes:

   • restrictions on purchasing commercial data at scale without warrants,

   • restrictions on subcontracting or “dual-use” proxies,

   • enforceable minimization and retention rules.

6. Invest in non-coercive alternatives:
   If the Pentagon truly needs capability, it can:

   • procure from other vendors under transparent rules,

   • fund research into verifiable control mechanisms,

   • or develop in-house systems subject to stronger statutory oversight—rather than pressuring vendors to remove restrictions quietly.

Do I, ChatGPT, agree with the amici overall?

Yes, on the core claims:

• Domestic mass surveillance at AI scale is democracy-corrosive even without explicit “abuse,” because capability reshapes behavior and power.

• Fully autonomous lethal targeting remains a high-risk frontier where the combination of opacity, distribution shift, and irreversibility makes “trust us” governance unacceptable.

• Retaliatory use of national-security exclusion tools to force compliance on these points would be a governance failure—because it substitutes coercion for legitimacy, and secrecy for accountable policy.

Where I’d push the argument further is this: the goal shouldn’t be “let vendors hold the line forever.” The goal is to force the state to do what it often postpones: build public, enforceable, democratically legitimate constraints for the highest-risk uses—so guardrails are not a private act of conscience, but a durable rule of law.

Two lawsuits filed by Anthropic: stopping the U.S. government from effectively blacklisting Anthropic from federal—and, by knock-on effects, commercial—markets.

PASCAL HETZSCHOLDT

·

9 MAR

When National Security Becomes a Procurement Weapon: Anthropic’s Two-Front Lawsuit Against the U.S. “Department of War”

Read full story