In short
- A test taken straight after training measures what is still fresh in memory at that moment, and that is exactly the knowledge that fades in the months that follow. Employees were still demonstrably better at recognising phishing four months after training, but after six months that gain had gone. A high score on the day of the training therefore says little about what an employee still knows six months later.
- Three things make knowledge last, and the evidence for all three is strong: have employees retrieve the material from memory themselves, repeat it at intervals, and match the difficulty to what someone already knows. Adaptive learning does all three in a single design.
- Measure again weeks or months later, with questions of the same level that the employee has not seen before. Only then can you show that knowledge has really increased. Put that knowledge measurement next to a measure of behaviour, because knowing more does not automatically mean behaving differently.
About this study
- Type
- Literature study based on peer-reviewed research.
- Sources
- Five meta-analyses and several review studies from cognitive psychology and educational science, supplemented with experimental research on memory and learning and with field research on phishing training in organisations.
- Cut-off date
- November 2026.
01 · FindingThe test straight after training comes too early
In most awareness programmes, a training course ends with a test. Whoever passes is done, and the pass rate goes into the report as proof that knowledge is up to standard. That reasoning has one weak spot, which is the moment at which you measure. What people learn fades quickly. Ebbinghaus described this as early as 1885, and when researchers repeated his experiment in 2015 they found the same forgetting curve, which drops steeply at first and then gradually levels off (Murre and Dros, 2015).
An early measurement can even mislead. Roediger and Karpicke had students read a text, after which one group read the text again and the other group took a test on it, without being told what they had got right or wrong. After five minutes the students who had reread the text knew the most, namely 81% of the content against 75% in the tested group. Two days later the picture had reversed (54% against 68%), and after a week the gap was just as wide (42% against 56%) (Roediger and Karpicke, 2006). What produces the highest score straight after learning is therefore not what lasts longest. Anyone who measures only on the day itself even risks picking the wrong approach as the best one.
Rereading puts you ahead at once, but behind after a week
Share of a text that students remembered, in per cent
Figure 1 The share of the text that students recalled five minutes, two days and one week after learning. Source: Roediger and Karpicke (2006), experiment 1.
Security awareness shows the same pattern in practice. Researchers at the Karlsruhe Institute of Technology followed 409 employees of a German public-sector organisation who had received phishing training. Straight after the training, and also four months later, these employees were demonstrably better than before at telling phishing emails from genuine ones. After six months that difference could no longer be shown, which led the researchers to recommend refreshing the training after six months (Reinheimer et al., 2020).
The test at the end of a training course therefore shows that the training was completed and that the material was understood at that moment, but not what remains of it afterwards. On top of that, a test on which almost everyone gets almost everything right leaves no room to see differences or to measure growth.
02 · FindingRetrieving knowledge from memory is itself a form of learning
Roediger and Karpicke's experiment does not stand alone. The phenomenon is known as the testing effect, and it means that people who have to retrieve knowledge from memory remember it better afterwards than people who read the material once more. In a meta-analysis of 61 studies, testing produced a medium-sized advantage over rereading on average, with an effect size of 0.50 (Rowland, 2014). A later meta-analysis, of 272 effects from 118 articles, arrived at 0.61 compared with other ways of studying (Adesope, Trevisan and Sundararajan, 2017).
A widely cited review that compared ten learning techniques reaches the same conclusion. Only self-testing and distributed practice received the highest rating, while rereading and highlighting, the techniques people use most, scored low (Dunlosky et al., 2013).
For an awareness programme this means that a question is more than a way of measuring. A question about material from weeks ago shows what has stuck and at the same time helps that knowledge last, because the employee has to retrieve it again.
03 · FindingSpacing works better than everything at once
The second insight backed by a large body of evidence is that learning pays off more when it is spread over time than when the same study time is packed into one block. A review of 317 experiments from 184 articles confirmed this spacing effect (Cepeda et al., 2006).
A follow-up study examined how long the interval should ideally be. More than 1,350 people learned a set of facts, reviewed them after an interval of up to three and a half months, and then took a final test up to a year later. The longer the knowledge had to last, the longer the best interval turned out to be. For a test after one week it was 20 to 40% of that week, and for a test after one year 5 to 10% of that year (Cepeda et al., 2008). If you want knowledge to last a year, the best time to repeat the material is therefore after roughly three to five weeks.
A yearly training course in a single block does exactly the opposite, because all the material arrives at once and then nothing happens for a year. The field research on phishing training points in the same direction. A short refresher, in the form of a video or of examples the employee clicks through, kept the effect in place there for at least another six months (Reinheimer et al., 2020).
04 · ExplanationWhy the difficulty has to move with the learner
The third insight concerns who the material is meant for. Explanation and guidance that help a beginner along are superfluous for someone with experience and can even get in the way of learning. Educational science calls this the expertise reversal effect (Kalyuga et al., 2003). A single module for the whole organisation therefore suits nobody well, because one person cannot keep up while another clicks through material every year that they have known for a long time.
Learning goes fastest on a task that is neither too easy nor too hard. For a broad class of learning models, the optimum has been calculated at around 85% correct answers (Wilson et al., 2019). That result holds for models that learn to make simple choices between two answers. The percentage therefore indicates a direction, but it is not a norm for training employees.
A Dutch example shows how this works out in practice. Math Garden (Rekentuin), developed at the University of Amsterdam, estimates at the same time how able a child is and how hard an exercise is, using a calculation method from the world of chess (the Elo system). The programme selects exercises so that a child gets about three out of four right. In the study that describes the method, 3,648 children together completed more than 3.5 million exercises (Klinkenberg, Straatemeier and van der Maas, 2011). The same method has since been used in other adaptive learning systems (Pelánek, 2016).
The question remains whether that adaptation also delivers more. A meta-analysis of fifty controlled evaluations of systems that tailor instruction to the learner found a median effect of 0.66 standard deviations compared with conventional teaching. That takes an average learner to the level of one who outperforms three quarters of the group (Kulik and Fletcher, 2016). An earlier review found that systems which guide the learner step by step work almost as well as a human tutor, with an effect of 0.76 against 0.79, in both cases compared with learning without tutoring (VanLehn, 2011). One caveat applies, because the gain was larger on tests that belonged to the lessons themselves than on general, standardised tests.
05 · ApproachWhat adaptive learning does with these insights
Adaptive learning brings these three insights together in one design. For an awareness programme that comes down to five choices.
- Keep one fixed standard for passing.Some of the questions are equally hard for everyone and decide whether someone passes. That keeps the outcome fair and makes results comparable, for an auditor too.
- Let the remaining questions match what someone already knows.Those who have mastered the basics get harder questions, and those who are not there yet get questions that are within reach, so that everyone learns at their own level (Kalyuga et al., 2003; Wilson et al., 2019).
- Come back to earlier material later.A question about training from weeks or months ago makes the employee retrieve the knowledge just as it starts to fade (Roediger and Karpicke, 2006; Cepeda et al., 2008).
- Use a question the employee has not seen yet.With a question they have seen before, you measure whether they remembered the question, not whether they have mastered the material.
- Derive the difficulty of a question from the answers.The author of a question can misjudge its difficulty. Systems such as Math Garden therefore continuously derive the difficulty from the answers people actually give (Klinkenberg, Straatemeier and van der Maas, 2011).
How 2LRN4 does this
In 2LRN4 every test has ten questions. Seven of them are equally hard for everyone and decide whether the employee passes, so the standard stays the same across the whole organisation. The other three match the employee's knowledge level. The platform tracks that level per topic and adjusts it after every test. One of the three questions is about a course the employee took at least a month earlier, and it is a question they have not seen before. In this way they retrieve earlier material, and the result shows what has stuck. The platform calculates how hard a question is from employees' answers as soon as there are enough of them. The employer does not get to see the knowledge level of individual employees.
06 · ApproachHow to show that knowledge is increasing
A rising line in a report is not yet proof. If a score goes up because employees take more tests, that says something about activity and nothing about knowledge. To show that knowledge is increasing, you need a measurement that meets four conditions.
- Measure at a later moment.Do so weeks or months after the training, not on the day itself.
- Measure with questions of the same level.If the later questions are as hard as the questions straight after the training, you can put the two results side by side.
- Measure with questions the employee has not seen yet.Only then does a correct answer say something about the material.
- Report on groups.The evidence is about the organisation, not about one person. So show a result only when the group is large enough that nobody in it can be identified.
Measuring twice with questions of the same level
Design of a measurement that shows what sticks
Figure 2 The test straight after the training shows what was understood. A new question about the same material, weeks to months later and equally hard, shows how much of it has stuck.
Two caveats apply here. The first is that a knowledge level derived from colleagues' answers mainly compares employees with one another. If everyone improves at the same time, such a level hardly changes. The evidence therefore lies in the percentage of correct answers to questions of a fixed level, and the knowledge level provides background at most.
The second caveat is that knowledge is not yet behaviour. In a meta-analysis of 69 studies, Leiden University found a large effect of training on knowledge and attitude (d = 1.02) and a small effect on behaviour (d = 0.36) (Prümmer, van Steen and van den Berg, 2025). A knowledge measurement therefore belongs next to a measure of behaviour, such as the number of suspicious messages employees report.
07 · ConclusionWhat counts is what sticks
The test at the end of a training course answers the question of whether the material has landed. What matters is what an employee still knows at the moment they need that knowledge, and that moment lies weeks or months later. Research on learning is clear on this. Knowledge fades, but it lasts better when people regularly have to retrieve what they learned from memory, when repetition is spread over time and when the material builds on what they already know.
Adaptive learning turns those insights into a way of working. Those who also measure again later, with new questions of the same level, no longer have to assume that knowledge is increasing, because they can show it.
Limitations
- This report is a literature study that summarises existing scientific research, and it contains no new research of our own.
- Most research on testing, spacing and adapting was done in education, with pupils and students and with material such as texts, facts and sums. It is reasonable to expect the results to hold for employees in an awareness programme as well, but there is still little direct research on that.
- How much systems that tailor instruction to the learner gain depends on the test used to measure it. On standardised tests the gain is smaller.
- The optimum of 85% correct answers was calculated for learning models and simple tasks, and has not been studied in employee training.
- More knowledge does not automatically lead to different behaviour. This report is about knowledge and makes no claims about behaviour change.
Sources
- Adesope, O. O., Trevisan, D. A., and Sundararajan, N. (2017). Rethinking the Use of Tests: A Meta-Analysis of Practice Testing. Review of Educational Research, 87(3), 659–701. doi.org/10.3102/0034654316689306
- Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., and Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354–380. doi.org/10.1037/0033-2909.132.3.354
- Cepeda, N. J., Vul, E., Rohrer, D., Wixted, J. T., and Pashler, H. (2008). Spacing Effects in Learning: A Temporal Ridgeline of Optimal Retention. Psychological Science, 19(11), 1095–1102. doi.org/10.1111/j.1467-9280.2008.02209.x
- Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., and Willingham, D. T. (2013). Improving Students' Learning With Effective Learning Techniques: Promising Directions From Cognitive and Educational Psychology. Psychological Science in the Public Interest, 14(1), 4–58. doi.org/10.1177/1529100612453266
- Kalyuga, S., Ayres, P., Chandler, P., and Sweller, J. (2003). The Expertise Reversal Effect. Educational Psychologist, 38(1), 23–31. doi.org/10.1207/S15326985EP3801_4
- Klinkenberg, S., Straatemeier, M., and van der Maas, H. L. J. (2011). Computer adaptive practice of Maths ability using a new item response model for on the fly ability and difficulty estimation. Computers & Education, 57(2), 1813–1824. doi.org/10.1016/j.compedu.2011.02.003
- Kulik, J. A., and Fletcher, J. D. (2016). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research, 86(1), 42–78. doi.org/10.3102/0034654315581420
- Murre, J. M. J., and Dros, J. (2015). Replication and Analysis of Ebbinghaus' Forgetting Curve. PLOS ONE, 10(7), e0120644. doi.org/10.1371/journal.pone.0120644
- Pelánek, R. (2016). Applications of the Elo rating system in adaptive educational systems. Computers & Education, 98, 169–179. doi.org/10.1016/j.compedu.2016.03.017
- Prümmer, J., van Steen, T., and van den Berg, B. (2025). Assessing the effect of cybersecurity training on end-users: A meta-analysis. Computers & Security, 150, 104206. doi.org/10.1016/j.cose.2024.104206
- Reinheimer, B., Aldag, L., Mayer, P., Mossano, M., Duezguen, R., Lofthouse, B., von Landesberger, T., and Volkamer, M. (2020). An investigation of phishing awareness and education over time: When and how to best remind users. Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020), 259–284. usenix.org/conference/soups2020/presentation/reinheimer
- Roediger, H. L., and Karpicke, J. D. (2006). Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention. Psychological Science, 17(3), 249–255. doi.org/10.1111/j.1467-9280.2006.01693.x
- Rowland, C. A. (2014). The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin, 140(6), 1432–1463. doi.org/10.1037/a0037559
- VanLehn, K. (2011). The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems. Educational Psychologist, 46(4), 197–221. doi.org/10.1080/00461520.2011.611369
- Wilson, R. C., Shenhav, A., Straccia, M., and Cohen, J. D. (2019). The Eighty Five Percent Rule for optimal learning. Nature Communications, 10, 4646. doi.org/10.1038/s41467-019-12552-4