A study has found that AI chatbots can be more effective at social engineering than human scammers, WIRED reports. The researchers looked at a form of romance scam commonly known as “pig butchering,” in which scammers spend weeks or months building a relationship with the victim before tricking them into sending money for a phony investment scheme.
“[The researchers] found that for the relationship-establishing stages of the scam—the stage that in real-world scams typically represents the longest part of the interactions with the victim, often stretching to months—an AI chatbot performed remarkably effectively, successfully impersonating a human and by some measures outperforming the real human ‘scammers’ in their experiment,” WIRED says.
Since the majority of the scam consists of non-malicious conversations, the researchers found that even legitimate AI chatbots could be used to automate the trust-building stage. Once the scammer is ready to ask for money, a human operator takes over.
Yisroel Mirsky, a computer science professor at Ben Gurion University who led the research, stated, “By having the full first stage of the scam performed automatically with LLMs at scale, you bring the victim up to this point where they have a very high level of trust. Then by transitioning it over to the human scammer at the end, this completely bypasses any vendor safeguards. With relatively little effort, we're able to make an agent that can outperform a human at building this exploitable emotional trust.”
Notably, when the participants in the study were informed that one of the conversations they were having was with an AI chatbot, nearly all of them were able to identify which conversation it was. Before they were told, however, only one participant suspected that they were talking to an AI. One of the researchers, Gilad Gressel, noted, “That's exactly how scams are, actually. Once the scam victim realizes what's going on, it's obvious. But when you're in the illusion of it, you just don't see it.
AI-native security awareness training can give your employees a healthy sense of suspicion so they can avoid falling for social engineering attacks. KnowBe4 empowers your workforce to make smarter security decisions every day. Over 70,000 organizations worldwide trust the KnowBe4 Platform to strengthen their security culture and reduce human risk.
WIRED has the story: https://www.wired.com/story/ai-scammers-are-better-at-building-trust-than-humans/
