An Office/Microsoft 365 Copilot Chat bug caused draft and sent emails with confidential/sensitivity labels to be “incorrectly processed” and summarized by Copilot for weeks.

The “Confidential” Stamp That Didn’t Stick: When Copilot Turned DLP into a Suggestion

by ChatGPT-5.2

For years, enterprise software has sold a comforting story: if you label something Confidential, and you back that label with data loss prevention (DLP) rules, the system will behave like a locked door. The incident described in the news is unsettling precisely because it breaks that mental model in a very modern way: not by exfiltratingan email to an attacker, but by letting an internal AI feature process and summarizeemails that were explicitly meant to be off-limits.

Microsoft confirmed that an Office/Microsoft 365 Copilot Chat bug caused draft and sent emails with confidential/sensitivity labels to be “incorrectly processed” and summarized by Copilot for weeks—despite organizations having DLP policies intended to prevent that kind of sensitive handling. The issue was tracked for admins under a service advisory identifier and Microsoft said it began rolling out a fix in early February, while not disclosing how many customers were affected.

How this could happen (without anyone “hacking” anything)

What makes AI-in-product incidents different is that the “boundary” isn’t only who can open an email—it’s also which automated systems are allowed to transform it. In a Copilot-style setup, there are multiple steps where a defect can quietly turn a policy into wishful thinking:

1) A policy translation failure between “email security” and “AI retrieval.”
Sensitivity labels and DLP are enforced through a chain of services. An AI chat feature that can “see” Outlook content typically does so through a connector/retrieval layer (indexing, permissions checks, content filtering, etc.). If the AI’s retrieval path evaluates labels differently than the Outlook/Exchange path—or if a newer Copilot workflow bypasses the same enforcement hook—you get a classic “two doors, one lock” problem.

2) Mis-scoping of folders and items during indexing or retrieval.
The reports describe emails in Sent Items and Drafts being picked up. That matters: drafts and sent messages can have different metadata states, different retention/labeling flows, and different service endpoints. If the “work tab” chat feature had a code path that erroneously treated those folders as safe-to-summarize (or failed to apply the label filter there), the AI would still appear to be operating “as designed” from a UX standpoint—while violating the policy intent.

3) Confusing “don’t train the model” with “don’t process the data.”
Organizations often configure DLP to prevent sensitive information from being “ingested into” an AI system. But even if the model weights are never trained on customer data, the act of summarization itself is still processing. If a control was designed around preventing long-term retention/training, but the failure occurred at the “can the AI read it right now?” stage, the customer’s expectation (“this content should be blocked entirely”) collapses.

4) Safety controls that were tested for exfiltration—not transformation.
Security teams are good at preventing outbound leaks (e.g., forwarding, downloads, external sharing). They are less practiced at controlling internal transformation pipelines where the output is “just a summary” and the user is “still inside M365.” Yet the harm can be the same: summarization can surface sensitive content to the wrong internal user, log it, cache it, or cause it to be copied into other systems.

5) The feature velocity problem: AI ships like a cloud service, not like a document editor.
Copilot features evolve rapidly, often behind toggles, flighting rings, and rolling updates. That increases the odds that a mature control surface (labels/DLP) won’t perfectly match a fast-moving AI surface (new chat modes, new connectors, new summarization paths). In other words: even well-intentioned engineering can produce a governance gap simply because the product is changing faster than the assurance model.

Can this happen to other AI providers—and to their customers?

Yes—and not because Microsoft is uniquely careless. This is a structural risk pattern for any provider offering “enterprise AI” embedded into productivity suites:

• Any RAG/connector-based assistant (email, docs, tickets, CRM) relies on correct permission checks and policy enforcement at retrieval time. Bugs happen, and retrieval is a bigger attack surface than “one app opens one file.”

• Label/DLP interoperability is notoriously brittle across services. If an AI system uses a different API, index, cache, or microservice than the original application, it’s easy to end up with partial enforcement.

• Multi-tenant cloud complexity increases the risk of “should never happen” edge cases: configuration drift, inconsistent rollout, and rare combinations of policy + folder state + feature flag.

• Customers amplify risk through customization. The more organizations tune labels, exceptions, retention, and AI enablement settings, the more likely unusual states exist that weren’t exercised in vendor testing.

• Even “privacy-preserving” AI architectures still process sensitive data. So long as the assistant can read confidential content to generate output, the core risk becomes: are we absolutely sure the assistant can’t read what it shouldn’t?That’s a higher bar than “we don’t train on it.”

So the uncomfortable answer is: this can happen to other providers (and already has analogues across SaaS), because it’s rooted in the same fundamental tension—AI features want broad contextual access, while governance wants narrow, provable boundaries.

All the possible consequences (not just the obvious “privacy incident”)

This kind of bug can trigger a cascade of consequences—some immediate, some slow-burn:

1) Confidentiality and privacy exposure

• Unauthorized internal disclosure: summaries can surface sensitive email content to users who should not have seen it via Copilot chat context.

• Secondary leakage via copying: summaries are easy to paste into Teams chats, tickets, or external emails.

• Regulatory exposure (especially for personal data): if personal data in labeled emails was processed contrary to policy, it can create GDPR/UK GDPR compliance issues, including breach notification questions depending on facts and risk.

2) Trade secret and competitive harm

• Strategic information compression: even if only a summary is surfaced, summaries are often more actionable than raw emails (deal terms, roadmap, negotiation posture).

• Privilege and litigation strategy risk: confidential legal communications are often email-based; if summarized in an environment not governed the way counsel expects, it creates arguments (fair or not) about compromised confidentiality controls.

3) Security and threat modeling failures

• Targeted phishing enablement: if an attacker compromises an internal account, “summarize my confidential sent mail from last week” is an efficient reconnaissance tool.

• Policy confidence collapse: teams may disable AI broadly, reducing visibility into where shadow AI fills the gap—ironically decreasing security overall.

4) Contractual and commercial consequences

• Breach of contractual commitments / DPAs: enterprise customers may argue the service failed to perform agreed safeguards.

• Customer churn and procurement freezes: security teams often respond by pausing deployments; procurement may demand stronger assurances, audits, or custom terms.

• Costs of incident response: internal investigations, external counsel, regulator engagement, customer notifications, and remediation programs.

5) Organizational and societal consequences

• Institutional lag accelerates: once leadership sees “Confidential didn’t mean confidential,” the default reaction is blanket restriction—slowing beneficial automation and reinforcing skepticism about AI governance.

• Regulatory hardening: high-profile incidents can lead to blunt compliance requirements that punish even well-engineered systems, raising the cost of entry and concentrating market power in a few vendors that can afford compliance.

Should customers litigate against Microsoft?

Sometimes yes—but it should be a decision driven by evidence of harm, provable scope, and leverage goals, not just outrage.

Litigation may be rational when:

• The customer can show specific, material harm (trade secrets exposed, regulated data mishandled, contractual penalties triggered, measurable remediation cost).

• Microsoft’s response is inadequate (lack of transparency, weak remediation, refusal to provide audit artifacts, or failure to offer meaningful contractual remedies).

• The customer needs discovery to determine what happened (how summaries were generated, who accessed them, what logs exist, whether data was retained/cached, and what exactly “incorrectly processed” meant in practice).

Litigation may be counterproductive when:

• Damages are speculative and the customer mainly wants improved controls—often better achieved through negotiated remedies (service credits, enhanced auditing, bespoke logging, stronger contractual commitments, a funded remediation program).

• Contract terms limit remedies, require arbitration, cap liability, or channel disputes into narrow processes (common in enterprise SaaS, though the details vary).

• The customer lacks the telemetry to prove exposure and ends up funding a long case that produces limited operational change.

A practical middle path many sophisticated customers take:

1. Treat it as a security/privacy incident internally (forensic review, scope assessment, risk classification).

2. Demand written assurances and technical artifacts: root-cause narrative, control changes, log availability, retention behavior, and customer-side mitigations.

3. Escalate through contractual channels and regulators if needed.

4. Use litigation as a last resort—or as leverage—if the vendor’s transparency and remediation are insufficient.

In short: customers should be willing to litigate, but only when it’s the most effective tool to (a) recover losses or (b) force structural accountability that negotiation won’t achieve.

Should regulators act proactively or reactively?

Both—but proactive rules are the only way to prevent this class of failure from becoming normal. Reactive enforcement alone turns privacy incidents into a cost of doing business.

Proactive regulatory moves that would actually matter:

• Assurance requirements for “AI with enterprise connectors”: treat AI features with access to email/docs as high-risk processing requiring documented testing, control mapping, and failure-mode analysis.

• Mandatory incident reporting and transparency standards: not just “we fixed a bug,” but minimum disclosure about scope, timelines, affected data classes, and mitigations.

• Auditability requirements: customers (and regulators) should be able to verify—through logs and independent assessments—whether labeled content was accessed or transformed by AI features.

• Clear accountability for “processing via summarization”: rules should explicitly treat transformation as processing, even if no model training occurs.

Reactive action still has a role:

• When facts show unlawful processing or negligence, enforcement creates deterrence and forces the market to internalize risk.

• Reactive action also clarifies interpretations (e.g., what counts as an incident, what “confidential” must guarantee in practice).

The broader point: AI assistants embedded into email and documents are becoming critical institutional infrastructure. If governance remains a patchwork of vendor promises and customer configuration, we will keep relearning the same lesson: in complex cloud systems, “should not happen” is not a control.