Why does announcing a simulation change what it measures?

Two groups of people received the same inbox, with fifty emails, half legitimate and half phishing. The task was also identical, because each participant had to decide what to do with each email. The only thing that changed was the initial instruction. The first group was told the study was about email management. The second was told their ability to detect phishing would be assessed.

That is how Parsons and colleagues designed their experiment in 2013. The first group distinguished phishing barely better than if they had answered at random, with an index of 0.52 on a scale where 0.5 is chance and 1 is perfect discrimination. The group that had been told reached 0.72.

The emails were the same. What had changed was what each group knew about the test.

In psychology and in experimental work, that change is known as measurement reactivity, the change that knowing a behavior is being measured produces in that behavior. In a security awareness program it appears every time someone recognizes a phishing simulation for what it is.

What changes when people know they are being assessed?

With a warning, the test shows how well a person can discriminate when security is their main task. Without one, it shows what they do while busy with something else, which is the situation in which a real attack arrives.

The experiment also left a less obvious finding. In the group that did not know what the study was about, people with more education handled phishing better, as if they were thinking about security without being prompted. In the informed group, that difference disappeared. The whole group improved and the differences in education stopped showing.

An announced test stops showing who thinks about security on their own. A program that plans simulations has to decide whether that is exactly what it wants to know.

The authors draw the practical conclusion themselves. The results of the uninformed group, they write, better represent the performance of real-world users, because in everyday life people are seldom reminded of the risk of phishing.

Something similar happens when the question changes. In a 2016 study, people distinguished phishing much better when asked whether an email was fraudulent than when asked what they would do with it. A test that only asks the first question keeps the most favorable version.

The 2013 experiment is a single study, run with university students and screenshots instead of a real inbox, and I found no direct replication of that manipulation. I use it for what it shows about the mechanism, and the exact magnitude remains open.

Is reactivity always a problem?

No. A 2024 field study with more than four thousand employees found that much of the effect of simulations comes from the periodic reminder of the threat, more than from the training content that accompanies them. Someone checking their inbox more carefully after a campaign is part of what the program is after.

Measurement is, in part, the intervention. The same reactivity that makes someone more attentive also makes the number describe a person on alert rather than their habitual behavior, and the dashboard does not separate the two unless someone decides to when designing the campaign.

How do you read a simulation without confusing the best case with an ordinary day?

A dashboard showing a falling click rate invites celebration. Before celebrating, three decisions help read that number correctly.

1. Keep capability and behavior in separate indicators. Exams and any announced test tell you what a person can do when they know they are being assessed. Simulations nobody anticipates tell you what they do. The exams in SMARTFENSE’s security awareness tools measure the first, which is why it makes sense to report the two results separately rather than averaging them.
2. Look for what gives the simulation away before reading its result. Any marker that appears only on simulations, such as a warning banner real emails never carry or a delivery that depends on an allowlist, teaches people to recognize the test rather than the attack. The technical setup that gives a simulation away can be reviewed before the first campaign. The same applies to the reporting rate, because if the simulation can be told apart from real email, reporting it measures how many people recognize the simulation.
3. Do not read a falling click rate as learning without ruling out recognition. With frequent, predictable campaigns, a falling rate is compatible with two explanations, that staff are better at detecting phishing or better at detecting simulations. Varying days, times and difficulty reduces the second. Smart Path lets you configure which days of the week a campaign can start, at what time, at what difficulty level and with what monthly load per person.

No configuration removes reactivity. What can be managed is what the person knows about the test at the moment of responding to it, which complements what I worked through in the “repeat-clicker” list, where the variable was the difficulty of the email.

A simulation everyone expects mostly measures that everyone expected it.