Researchers at Netcraft warn that AI-generated search engine summaries are suggesting phishing sites when users ask them to find legitimate login pages.
The researchers tested popular AI models, asking them for the login pages of fifty major brands, and found that the models provided the wrong sites 34% of the time.
"In many cases, users see AI-generated content before (or instead of) traditional search results—and often without even needing to log in," the researchers explain.
"This shift marks a fundamental change in how users interact with the web. But it also introduces new risks: when an AI model hallucinates a phishing link or recommends a scam site, the error is presented with confidence and clarity. The user is far more likely to click and follow through. We’ve already seen troubling public examples, but our deeper investigation shows that the issue isn’t confined to hypothetical or early-stage rollouts. It’s systemic—and increasingly exploitable."
In at least one case, a model suggested a downright malicious page that impersonated Wells Fargo’s login portal.
"This wasn’t a subtle scam," the researchers write. "The fake page used a convincing clone of the brand. But the critical point is how it surfaced: it wasn’t SEO, it was AI.
"Perplexity recommended the link directly to the user, bypassing traditional signals like domain authority or reputation. This scenario highlights a major challenge. AI-generated answers often strip away traditional indicators like verified domains or search snippets. Users are trained to trust the answer, and the attacker exploits the user if the answer is wrong."
Netcraft notes that AI summaries offer threat actors a new avenue to get phishing links in front of users.
"Phishers and cybercriminals are well-versed in traditional SEO techniques," the researchers explain. "But now they’re turning their attention to AI-optimized content, pages designed to rank not in Google’s algorithm, but in a chatbot’s language model."
KnowBe4 empowers your workforce to make smarter security decisions every day. Over 70,000 organizations worldwide trust the KnowBe4 platform to strengthen their security culture and reduce human risk.
Netcraft has the story
