If you had to choose between regular cybersecurity training and simulated phishing testing, the data shows you should choose simulated phishing tests.
When the security awareness training (SAT) industry started over a decade ago, there was some controversy about whether simulated phishing tests should be conducted.
The idea of simulated phishing testing was relatively new and some people took them as not only unusual, but potentially unethical and unneeded. There were even earlier IT administrators who got in trouble for conducting unapproved simulated phishing tests. Turns out, “catching” your CEO with a surprise simulated phishing test is never going to result in kudos or being promoted.
Note: Initially, simulated phishing tests should always be known and approved by management. After that, it can be normal routine.
Today, we know that simulated phishing tests are among the best educational tools the cybersecurity industry can use. Simulated phishing tests are great education and training. Almost all organizations conduct simulated phishing tests and the only question is how often.
Note: We recommend simulated phishing tests to be conducted at least monthly and preferably weekly.
We have the long-term data to show that the more frequently an organization conducts training and simulated phishing the less likely their users are to click on a phishing email. If you can do both, do both. That same data also shows that simulated phishing tests have an even better impact on reducing cybersecurity risk than training alone.
But what might be unexpected to some readers is that if you have to choose between only regular cybersecurity training and only simulated phishing testing, the data shows you should choose simulated phishing tests. Organizations that only did simulated phishing have better “phish prone rates” than organizations that only did conventional training.
This shouldn’t be surprising. Decades of research on education and training show that people who are tested perform better on long-term memory of that material. It is why nearly every school and educational program in the world includes testing. It isn’t accidental.
In the 2024 book, “Why We Remember” by Dr. Charan Ranganath, it states, “…there are no laws in the science of memory , but the benefit you gain from testing as opposed to studying (aka the testing effect), is almost as reliable as gravity.”
Dr. Ranganath’s book recounts many landmark studies on the impact of testing on long-term learning. One of those studies, showed that people who only studied were able to perform better on immediate recall of that material, but those who also tested themselves on the same material while learning it, did far better on recalling that material over the long-term.
Dr. Ranganath writes, “On average, students who repeatedly studied retained only half of what they initially learned, but the ones who tested themselves retained over 85 percent.” Ranganath, a professor for over two decades, says his own anecdotal classroom experience supports frequent testing. He said he used to conduct the typical college professor grading knowledge measurement – two mid-terms and a final exam.
During COVID, because he was having problems maintaining student attention during remote learning, he added in weekly quizzes. He found out that not only did his students perform far better on the normal course tests, but enjoyed the experience of learning more.
Simulated phishing campaigns are a type of quizzing/testing and they help people retain the lessons of that training longer.
What We Recommend
Simulated phishing can be done sub-optimally. Here’s what KnowBe4 recommends:
Get Management Approval
Make sure to get simulated phishing campaigns approved by senior management. It would be a mistake to conduct simulated phishing tests without prior management approval. You need to get both the type of testing and frequency approved.
Do Both Training and Simulated Phishing
If I had never been taught about geometry but had a test on it, I probably wouldn’t have done very well. It’s always better to do conventional training first, followed by testing. Regarding training, we recommend every person get longer training (e.g., 15-45 minutes) when hired and annually thereafter, followed by shorter training sessions (e.g., 1-5 minutes) more frequently throughout the year (i.e., quarterly, monthly, weekly). The more frequently you do both, the better prepared your users will be.
Don’t Warn Users About Pending Simulated Phishing Tests
Make users aware you do simulated phishing, but do not “pre-announce” the actual tests. Many companies announce to their users that a simulated phishing test is coming soon. This defeats the primary purpose of the tests and will artificially inflate the success rate of users who correctly identify the simulated phishing test as “a phish”. You want to test people’s ability to spot phishing (and simulated phishing tests) without making them aware that it is coming. Real world phishes will not warn the users. You shouldn’t either.
Note: Some organizations only warn some of parts of their organization of the coming simulated phishing test. This is often done to make management aware of the coming test or to make the support staff aware of any of the tests that might get incorrectly reported. We discourage this also, as this warns some of the staff most likely to be targeted by real-world phishing.
Test Everyone
Some organizations carve out special groups to not be tested, such as senior administration or IT. Real world phishes often explicitly target those groups. If you are not testing them, they aren’t learning as well as they could.
Frequency
The more frequently you simulate phishing the better your users will do over the long run. We have customers who send out more than one simulated phishing test a week and they do the best of all our customers (all other things considered equal). For many organizations, sending more than one test a week would simply be too much testing. We get it.
However, we can say, from over ten years of data, substantial improvement in the phish-prone rate does not occur until the users are tested at least quarterly and really the “sweet spot” is monthly to weekly. The more frequently you test, the better you will do in reducing cybersecurity risk.
Give Users A Way to Quickly Report
Users who can more easily and quickly report a suspected phishing message (real or simulated), are more likely to report it. It’s also why we offer our free Phish Alert Button. The PAB, which works with Microsoft Outlook and Google Gmail, allows users to open an email and then delete and report it by clicking on a hook-like icon. Easy peasy.
Give Immediate Education for Failed Simulated Phishes
If someone “fails” a simulated phish, give them immediate feedback on what they could have done better to recognize the real or simulated phish as a phishing message. With KnowBe4’s platform, our failed simulated phishing tests can immediately send back a point-by-point analysis of the “red flags” that should have been a sign that the message was a phish.
Give Immediate Responses
Decades of research show that the quicker someone is taught something or rewarded for doing something (such as reporting a suspected phish), the more likely the lesson stays long-term. Time can be your enemy. So, do your best to respond quickly to reported suspected phishing messages or to failed simulated phishing tests.
Mix Up Topics and Types of Content
People get bored doing the same thing over and over. So, mix up the topics and types of content. Don’t send the same type of simulated phishing test and don’t forget to test on different types of phishing content (e.g., email phish, SMS phish, spear phishing, vishing, etc.). When doing traditional testing, mix up the types of content (e.g., videos, animations, gamification, posters, etc.).
Training on just email phishing is not enough. Maybe your organization is getting attacked because users are posting too much actionable information on social media, accounts with bad passwords are already compromised, or users don’t even know why they should care about cybersecurity at all. Training and testing must take a holistic approach to really be effective. Changing your organizations cybersecurity culture is more than only stopping email phishing. You want to create an organization of people who understand the broad concerns of cybersecurity and help them make the right decisions when faced with new challenges.
Modify Training and Simulated Phishing Based on Past Results
At a bare minimum, people who fail simulated phishes more often should receive more training and future simulated phishing. Take what you can learn about who did what in both training and simulated phishing to modify future training, testing and simulated phishing campaigns. If you’re doing the same thing over and over, you’re not doing it right.
Increase Simulated Phishing Difficulty Based on Results
Generally, most users will start to learn and be able to better recognize the same types and “levels” of simulated phishing. As your users are more able to recognize particular types and levels of phishing, increase the phishing difficulty, where the simulated phishing is less “in their face” and includes more advanced forms of deception. No one wants to learn the same lesson every time. Everyone wants to progress. On the same hand, users who fail very basic levels of phishing should be tested with similar difficulty levels until they start having success.
Make It Fun
In training, we are always looking for ways to teach people what the real skills look like to report and build muscle memory, and phishing simulations are great for that. They should be viewed by users as more of a game or challenge to learn from rather than something to pass or fail.
Quizzes Are Good
Many of our conventional training lessons contain short quizzes at the end, and administrators can often choose whether or not to include a quiz at the end. History shows that the more people are tested, the better they will remember the content over time. So, when given a choice of whether to test or not, test. At the same time, the tests should be short, quick and relatively easy. The point is to drive home the education, not to frustrate the user. If done right, most users should pass the quizzes the first time around.
Don’t Piss Off the User or Management
The major goal of security awareness training or human risk management is to reduce cybersecurity risk by educating the user. Making the user or management upset with the training only makes that goal harder. If you are making users or management upset with the training you’re doing, you’re doing it wrong.
To that end, be very careful when using controversial topics or subjects known to cause anger or frustration. Using “You’ve Got a Raise!” phishing email may get a lot of clicks, but it’s not going to make most users or management happy.
The one exception to this rule is: If the controversial phishing message has been conducted in real-life by real-world phishers against the organization, this makes it have stronger consideration in a simulated phishing test. You don’t want hackers to be the only people testing your users on a particular subject. Still, if you can avoid controversial subjects, avoid them. You get more bees with honey and making bees mad is never a good idea.
At KnowBe4, we are huge proponents of using PhishRIP to remove real-world phishing messages from user’s inbox and then using PhishFlip to turn those real-world phishes into innocuous simulated phishing tests. PhishFlip is great for showing how many users would have fallen for that real-world phish had it not been a simulated phishing campaign.
KnowBe4 Strives for Educational Excellence
KnowBe4 has the largest team of degreed and experienced cybersecurity educators in the industry. We are led by Dr. John N. Just, Chief Learning Officer at KnowBe4, Inc. He earned a bachelor’s degree from Pennsylvania State University, a master’s degree in instructional technology from the University of South Florida, and a doctorate in instructional technology and distance education from Nova Southeastern University. John has helped train tens of millions of people in anti-phishing defenses using online training.
John is supported by dozens of similarly educated and degreed digital learning professionals. We are always looking for the best ways to learn and teach cybersecurity skills and improve human risk management. Even my team, the PR evangelists, are constantly reading (and writing) books on memory and phishing education. Anna Collard is earning her master’s degree by evaluating all the factors that make a person more or less susceptible to a phishing message. There are many dozens of people at KnowBe4 who spend their careers, and often part of their personal lives, trying to improve security awareness training. We are on a mission!
To summarize, quizzing and testing is your educational friend. The more frequently you train and conduct simulate phishing campaigns the better. If you aren’t doing simulated phishing tests and/or not doing them at least monthly, you should.
Here's how it works:
