The Govinfosecurity site just reported on some very interesting scientific research that finds so-called "embedded training" is ineffective. Let's quickly define "embedded", they chose this word to describe what we in the industry call "point-of-failure" training where the employee in their normal work environment gets instant corrective feedback, often via a short training module after they fail a simulated phishing attack. The title of the article is a bit misleading and creates the false impression that training does not work.
The headlines states: "Why Training Doesn't Mitigate Phishing" - combined with a subheader that reads "embedded training is ineffective". They are referring to brand new published research done by a team from Vanderbilt University, MITRE and I3P Dartmouth College. There is an audio interview with Eric Johnson, a Vanderbilt University professor who is a co-author of the study.
Let's go in some more depth regarding this new MITRE/Vanderbilt study about embedded training and finish off showing that there are other approaches to security awareness training that do work.
The study's executive summary stated: "To explore the effectiveness of embedded training, researchers conducted a large-scale experiment that tracked employees’ reactions to spear phishing emails and immediate training activities. The results showed that many participants did not read the training, and clicked either all links or none across three trials."
They ran three tests, with a 90-day period. They based their research on earlier studies to see if they could replicate their results. Their final sample included 1,359 participants and they were able to identify three groups: people that never clicked, people that almost always clicked, and the ones that only clicked sometimes.
Their so called treatment group landed on a page that showed them their error:
Afterward, researchers interviewed users and asked them what their reaction was after seeing this one-page corrective instruction. Many participants misinterpreted the training page when it popped up immediately after clicking a spear phish and feared it represented a further attempt to entrap them. Most all-clickers did remember a page popping up when they clicked, but almost none remembered specifics.
Most interviewees had taken only the company’s annual training, plus one or two courses required by their projects. Although they assumed that spear phishing awareness was part of this standard training, few could remember it with any certainty. Almost all non-clickers considered annual company training ineffective; it repeated information that they already knew and instructed them to act in ways they already did.
"Unteachable Moments - Incorrigible Users?"
In their study, they tested the retention effects over approximately 90 days instead of 28 days, because training every month in a corporate setting isn’t manageable. The study commented: "Company security might offer a false sense of confidence. Almost all interviewees believed that company firewall and information security programs prevented malicious emails from reaching them."
Research Catch and Takeaway
They stated: "Making embedded training effective in a corporate setting is more difficult than earlier studies suggest. Our results indicate that immediate feedback and tailored framing don’t suffice to reduce click rates or increase reporting." Another note was "This article reports on only one corporate study. The researchers conducted a second study, to be reported in the future, using MBA students with business experience. The study yielded similar results, reinforcing our conclusions. Nevertheless, for widespread validity, the study must be replicated at other institutions". One of their hypotheses for the surprising results was: "Repetition might be required to achieve the training effects; participants might need repeated exposure to the training before their behavior is changed."
Why Are We Not Surprised?
These results are no surprise here at KnowBe4. We have known for quite a while that this type of security awareness computer-based training does not work all that well. There are methodologies of training end-users that are extremely effective though.
The results of the 2023 KnowBe4 Phishing by Industry Benchmarking Report clearly show where organizations’ Phish-proneTM Percentages started and where they ended up after at least 12 months of regular testing and security awareness training. The study analyzed a data set that included more than 12.5 million users across 35,681 KnowBe4 customers with more than 32.1 million simulated phishing security tests across nineteen different industries.
The overall industry initial Phish-prone Percentage benchmark turned out to be a troubling 33.2%. Fortunately, the data showed that this 33.2% can be brought down almost in half to 18.5% within 90 days of deploying new-school security awareness training. The one-year results show that by following these best practices, the final Phish-prone Percentage can be minimized to 5.4% on average.
The initial baseline percentage was often an unpleasant surprise for both IT and non-IT management, and cause for concern. The result though was support from the C-level on down for a sustained year-round security awareness program, with feedback on a regular basis, creating an effective security culture.
We think that the article's title is needlessly confusing. Our position is that security awareness training programs done the right way can be extremely effective.
New-school Security Awareness Training is critical to enabling you and your IT staff to connect with users and help them make the right security decisions all of the time. This isn't a one and done deal, continuous training and simulated phishing are both needed to mobilize users as your last line of defense. Request your one-on-one demo of KnowBe4's security awareness training and simulated phishing platform and see how easy it can be!
