The Attention Problem

Phishing is a persistent problem, and the most common response to it is the phishing simulation: a fake message that tests whether employees fall for it. In practice this gives rise to a pattern in which organisations that only simulate want to send ever harder messages. The success of such a programme is then measured mainly by how convincing your own phishing email was.

The question of whether employees actually learn to recognise phishing better as a result remains unanswered. This report sets out what the scientific research shows about this, and which approach follows from it.

01 · FindingThe click rate does not measure what it appears to measure

A falling click rate is quickly interpreted as proof that employees have improved. A large-scale reproduction study shows, however, that it is mainly the difficulty level of the phishing email that determines click behaviour (Rozema and Davis, 2025). Measured with the NIST Phish Scale, the click rate rose from about 7 percent for the easy messages to about 15 percent for the difficult ones. A harder simulation that catches more people therefore primarily shows that the message was harder, not that the employees have got worse.

On top of this, in the largest study to date the share of employees who fell for the messages actually increased over time rather than decreasing. This was a study in which participants were assigned at random, a so-called randomised controlled trial, among more than 19,500 employees of UC San Diego Health (Ho et al., 2025). Repeated simulations can teach employees to recognise the test emails, without their also recognising real, new phishing any better. This is precisely where the core of the problem lies: recognising a test email is something other than recognising real phishing.

02 · FindingFrom knowledge to behaviour: the design is decisive

Training demonstrably increases employees' knowledge and attitude. This emerges strongly from the meta-analysis of 69 studies by Leiden University (Prümmer, van Steen and van den Berg, 2024): the effect on knowledge and attitude is large. On actual behaviour, however, the average effect is small. That difference is no coincidence, but a well-known finding from behavioural science: knowledge is a precondition for safe behaviour, but not enough on its own. The Behaviour Change Wheel shows that besides knowledge, motivation and the opportunity to do the right thing are also needed (Michie, van Stralen and West, 2011).

The average also conceals large differences, and these lie above all in the design of the training. Awareness campaigns that merely disseminate information change little (Bada, Sasse and Nurse, 2015). Training that is active, repeated and personalised, by contrast, does affect behaviour. In a controlled experiment, a game format improved not only attitude and intention but also reported behaviour, compared with a game without security content (van Steen and Deeleman, 2021). The conclusion is therefore not that training does not work, but that the form determines whether knowledge is also translated into behaviour.

03 · ExplanationPhishing is an attention problem, not a knowledge problem

The explanation for why more knowledge does not always help comes from two studies by a research group at ETH Zürich. In their largest experiment, which lasted fifteen months and involved nearly 15,000 employees, it turned out that the embedded training did not make employees more resilient and in some cases even had a counterproductive effect. By embedded training we mean the short explanation someone is shown after clicking on a test email (Lain, Kostiainen and Čapkun, 2022). In the follow-up study, which was awarded a prize at the ACM conference on computer and communications security, the authors investigated where that small effect came from (Lain et al., 2024). The conclusion was sobering: the effect came from the periodic reminder of the threat, and not from the content of that short explanation, which most employees barely read.

This insight changes the whole approach. If the problem is not that employees know too little, but that at the wrong moment they briefly fail to pay attention, then there is little point in teaching them even more. Someone who is distracted at such a moment will not recognise even a good phishing email, however many modules they have completed. The solution therefore lies less with the alertness of the individual employee, and more with the technical environment and the working practices around it. By this we mean systems that catch a mistake before it causes damage, and processes that let a suspicious message be reported quickly and rendered harmless.

04 · ApproachA different approach: layered and evidence-based

When the vigilance and knowledge of the employee are not the decisive factors, the research points to a layered approach. The idea behind it is to make the attack fail before a mistake can cause damage, and to treat human alertness as the thinnest and last layer rather than as the first line of defence.

1. Make the attack technically impossible.Phishing-resistant authentication, such as FIDO2 or WebAuthn and keys based on PKI, binds the login to the real domain. A fake site therefore receives no valid response, regardless of who clicks on it. NIST places this form at the highest assurance level and CISA calls it the standard (NIST SP 800-63B; CISA, 2022).
2. Let fewer phishing emails arrive and warn about the rest.Measures against sender spoofing (DMARC, DKIM and SPF), filtering and warnings on suspicious email relieve the burden on the employee's attention. The ETH study showed that such warnings work well (Lain et al., 2022).
3. Use employees as a collective sensor.A reporting button turns employees together into a detection layer that quickly makes new campaigns visible (Lain et al., 2022). Phishing simulations are ideally suited to practising that reporting reflex, provided you steer on the reporting rate and on the time between seeing and reporting, and not on the click rate.
4. Design the training for behaviour rather than for knowledge.Short, repeated and context-tailored reminders work better than long courses, and interactive and playful forms have a greater effect on behaviour (Prümmer, van Steen and van den Berg, 2024). Effective training is, moreover, continuous and tailored to the employee, rather than one-off and generic (Jampen et al., 2020).
5. Limit the damage if something does go wrong.Assume that some clicks will succeed, and ensure with least privilege, segmentation, monitoring and a rapid response that one mistake does not grow into a full breach.

Finally, a point about measurement. The researchers behind the largest experiment argue for measuring, as in medical research, with random assignment which measure removes the most risk, instead of steering on click figures or on the difficulty of your own phishing email (Ho et al., 2025). The question is therefore not how convincing you can make a fake message, but how much real resilience a measure delivers per euro invested.

05 · ConclusionNot more and harder, but different

Phishing simulations do not, in themselves, demonstrably make employees better at recognising phishing, and ramping up the difficulty level shifts attention to the wrong metric. The scientific research does not point to more or harder simulations, but to a layered approach. That approach makes the attack fail technically, detects it quickly through reporting, and supports behaviour with well-designed, repeated and active training instead of passive knowledge transfer. Training and simulations retain a clear role within this, as long as they are aimed at behaviour and do not grow into an arms race with your own employees.