Researchers at Permiso warn that threat actors can plant phishing messages within Copilot AI summaries. Notably, the researchers found that attackers can trick Copilot into including internal information to craft a more targeted message.
In a proof-of-concept attack, Permiso outlined the following attack:
- “An attacker sends a benign-looking email.
- The attacker includes hidden or low-visibility “instruction text” intended for Copilot.
- The recipient clicks Summarize (a normal productivity workflow).
- The Copilot summary includes an ‘Action Required’ section that looks like a legitimate Microsoft security notification.
- The summary can include a clickable link presented with safe-looking anchor text.”
The researchers note, “At that point, the phishing content is no longer ‘just an email.’ It’s presented as assistance generated by an AI tool that the organization may have endorsed. This is a form of model-mediated phishing: the attacker doesn’t need Copilot to execute code they only need it to speak with Copilot’s voice.”
Since Copilot and other AI tools have access to internal information, the attackers can trick the tools into using this information to create a targeted message.
“Phishing through AI summaries is concerning, but the bigger question is: what happens when these assistants can pull from your entire digital workspace?” the researchers write. “Microsoft 365 Copilot doesn't just read emails, it can access Teams conversations, OneDrive files, SharePoint documents, and meeting notes, all depending on licensing, configuration, and permissions.
“This attack can start simple: an injected prompt that just makes the summary say something alarming. But it can escalate quickly. If Copilot has access to your Teams chats, OneDrive files, or SharePoint docs, an attacker can craft prompts that pull from that context to build more convincing output or quietly exfiltrate sensitive information outside.”
Permiso adds that this technique isn’t unique to Copilot; a similar tactic has been documented with Google’s Gemini for Workspace. Users should be aware that AI tools can be manipulated in this fashion.
KnowBe4 empowers your workforce to make smarter security decisions every day. Over 70,000 organizations worldwide trust the KnowBe4 HRM+ platform to strengthen their security culture and reduce human risk.
Permiso has the story.
