The Govinfosecurity site just reported on some very interesting scientific research that finds so-called "embedded training" is ineffective. Let's quickly define "embedded", they chose this word to describe what we in the industry call "point-of-failure" training where the employee in their normal work environment gets instant corrective feedback, often via a short training module after they fail a simulated phishing attack. The title of the article is a bit misleading and creates the false impression that training does not work.
The headlines states: "Why Training Doesn't Mitigate Phishing" - combined with a subheader that reads "embedded training is ineffective". They are referring to brand new published research done by a team from Vanderbilt University, MITRE and I3P Dartmouth College. There is an audio interview with Eric Johnson, a Vanderbilt University professor who is a co-author of the study.
Let's go in some more depth regarding this new MITRE/Vanderbilt study about embedded training and finish off showing that there are other approaches to security awareness training that do work.
The study's executive summary stated: "To explore the effectiveness of embedded training, researchers conducted a large-scale experiment that tracked employees’ reactions to spear phishing emails and immediate training activities. The results showed that many participants did not read the training, and clicked either all links or none across three trials."
They ran three tests, with a 90-day period. They based their research on earlier studies to see if they could replicate their results. Their final sample included 1,359 participants and they were able to identify three groups: people that never clicked, people that almost always clicked, and the ones that only clicked sometimes.
Their so called treatment group landed on a page that showed them their error:
Afterward, researchers interviewed users and asked them what their reaction was after seeing this one-page corrective instruction. Many participants misinterpreted the training page when it popped up immediately after clicking a spear phish and feared it represented a further attempt to entrap them. Most all-clickers did remember a page popping up when they clicked, but almost none remembered specifics.
Most interviewees had taken only the company’s annual training, plus one or two courses required by their projects. Although they assumed that spear phishing awareness was part of this standard training, few could remember it with any certainty. Almost all non-clickers considered annual company training ineffective; it repeated information that they already knew and instructed them to act in ways they already did.
"Unteachable Moments - Incorrigible Users?"
In their study, they tested the retention effects over approximately 90 days instead of 28 days, because training every month in a corporate setting isn’t manageable. The study commented: "Company security might offer a false sense of confidence. Almost all interviewees believed that company firewall and information security programs prevented malicious emails from reaching them."
Research Catch and Takeaway
They stated: "Making embedded training effective in a corporate setting is more difficult than earlier studies suggest. Our results indicate that immediate feedback and tailored framing don’t suffice to reduce click rates or increase reporting." Another note was "This article reports on only one corporate study. The researchers conducted a second study, to be reported in the future, using MBA students with business experience. The study yielded similar results, reinforcing our conclusions. Nevertheless, for widespread validity, the study must be replicated at other institutions". One of their hypotheses for the surprising results was: "Repetition might be required to achieve the training effects; participants might need repeated exposure to the training before their behavior is changed."
Why Are We Not Surprised?
These results are no surprise here at KnowBe4. We have known for quite a while that this type of security awareness training does not work all that well. There are methodologies of training end-users that are extremely effective though.
In the latest graph you see results on 30,173 organizations, over 23.4 simulated phishing tests, and 9.5 million users over 12 months. The numbers speak for themselves. The initial baseline phish-prone average percentage is just over 32%, meaning these employees clicked on our initial simulated phishing test. Twelve months later, after having been trained and been sent regular simulated phishing attacks, this dropped to 5%.
The initial baseline percentage was often an unpleasant surprise for both IT and non-IT management, and cause for concern. The result though was support from the C-level on down for a sustained year-round security awareness program, with feedback on a regular basis, creating an effective security culture.
We think that the article's title is needlessly confusing. Our position is that security awareness training programs done the right way can be extremely effective.