Judge Stein’s ruling restores clarity at a moment when clarity is badly needed. Robots.txt is a sign. The DMCA protects locks.

Robots.txt, Polite Requests, and the Limits of Copyright Law

by ChatGPT-5.2

The recent ruling by Judge Stein in the dispute between Ziff Davis and OpenAI may look technical on the surface, but at its core it addresses a surprisingly human question: what is the difference between asking someone not to do something and actually stopping them from doing it?

The answer matters enormously for publishers, AI companies, and the future of how the web is governed.

1. What robots.txt really is (and isn’t)

A robots.txt file is a convention dating back to the early web. Website owners use it to signal to automated programs—such as search engine crawlers or scrapers—which parts of a site they would prefer those programs not to access.

Crucially, robots.txt:

• does not block access

• does not require a password

• does not encrypt content

• does not technically prevent copying

It is, in effect, a polite request. The web equivalent of a sign that says:

Some actors respect the sign. Others ignore it. The sign itself never physically stops anyone.

2. What Ziff Davis tried to argue

Ziff Davis attempted to elevate robots.txt from a request to a legal barrier. Their argument was novel and ambitious:

If OpenAI’s crawler ignored robots.txt instructions, then OpenAI must have “circumvented a technological protection measure,” which is prohibited under the Digital Millennium Copyright Act (DMCA).

This matters because the DMCA’s anti-circumvention rules are powerful. They are designed to protect things like:

• paywalls

• encryption

• DRM systems

• password-protected access

If robots.txt qualified as such a measure, ignoring it could trigger serious legal consequences—even before any copyright infringement is proven.

3. Why the judge rejected that theory

Judge Stein rejected the argument decisively, and for a simple reason: robots.txt does not control access.

In plain language, the court said:

• A lock controls access

• A password controls access

• Encryption controls access

• robots.txt does not

Ignoring robots.txt may be discourteous. It may violate norms. It may even support other legal claims. But it is not the same as breaking a lock.

The judge’s reasoning draws a bright line between:

• Norms and expectations (what people ask others to do), and

• Technological enforcement (what systems actually prevent).

Under the DMCA, only the second category counts.

4. Why this distinction matters so much

This ruling matters far beyond this single case.

If robots.txt were treated as a DMCA-protected access control, then:

• any ignored instruction could become a federal offense

• informal web standards would quietly harden into law

• courts would effectively rewrite copyright protections without Congress

By rejecting that approach, the court preserved a key principle: copyright law protects real barriers, not symbolic ones.

That does not mean publishers are powerless. It means they must be honest about the tools they are using.

5. The uncomfortable reality for publishers

The ruling forces a difficult but necessary reckoning.

Many publishers have relied on robots.txt while simultaneously knowing:

• it is voluntary

• it is selectively respected

• it offers no enforcement

The court is essentially saying:

That could mean:

• authentication systems

• rate-limiting

• contractual access controls

• licensing-based APIs

Robots.txt alone is not enough—no matter how reasonable the request behind it may be.

6. What the ruling does not say

Importantly, the judge did not say:

• scraping is lawful

• training AI on copyrighted content is lawful

• publishers have no rights

• OpenAI is “in the clear” overall

The ruling is narrow. It only says that robots.txt cannot be stretched into a DMCA anti-circumvention claim.

Other legal theories—copyright infringement, contract breach, unfair competition—remain very much alive.

7. A deeper lesson about power and governance on the web

At a deeper level, this case exposes a tension that has been building for years.

The open web runs on norms.
The modern internet economy runs on power.

Robots.txt belongs to the first world. AI infrastructure belongs to the second.

Courts cannot pretend that polite signals are the same as enforceable controls, no matter how asymmetric the power imbalance between content creators and large technology platforms may be.

That imbalance is real. But stretching old laws to cover it risks breaking the law itself.

Conclusion: signs, locks, and legal clarity

Judge Stein’s ruling restores clarity at a moment when clarity is badly needed.

Robots.txt is a sign.
The DMCA protects locks.

Confusing the two may feel tempting in a world of large-scale scraping and AI training—but it is not how the law works.

For publishers, the message is sobering but useful: norms are not safeguards.
For policymakers, the message is sharper still: if society wants stronger protection, it must build it deliberately—through legislation, infrastructure, and enforceable standards—not by pretending that courtesy equals control.

The web was built on trust.
The future will be built on architecture.