When you can compare your click rate with last year’s

Suppose an organization ran three phishing simulations over the last two years and got click rates of 18%, 11% and 6%. On the slide for the board, the three bars look like a downward curve, until someone asks whether they were measured on the same population and under the same conditions. If the answer is no, those three bars don’t show a trend.

I have argued for a long time that the only benchmarking worth doing is the one an organization does against itself. That comparison only works if your phishing simulation click rate is always measured under the same conditions.

What makes two phishing simulation click rates impossible to compare?

There are five conditions that change easily between one simulation and the next.

1. The population. If the simulation goes from part of the organization to all of it, or from one department to another, the rate measures a different group of people. Normal turnover doesn’t break the series, but it is recorded in every round.
2. The window. A simulation records activity until it expires. One that stays open for three days and one that stays open for seven don’t capture the same clicks.
3. The difficulty. A generic held-parcel notice and an email impersonating a well-known supplier, with internal context, don’t measure the same thing. According to NIST, when it presented its Phish Scale, a low rate can come from an email that is too easy, lacks relevant context or resembles a previous exercise, and reading it without knowing the difficulty can create a false sense of security.
4. The denominator. Dividing by the people assigned doesn’t give the same result as dividing by the emails that actually arrived, and delivery can change the click rate without any change in behavior. With sample sending, the denominator should include only the people who received the simulation.
5. False positives. Antivirus tools and sandboxes that open links to analyze them generate false positives, clicks no person made. If one measurement discounts them and another doesn’t, or if the tool that scans email changes, the two rates count different clicks.

If any of the five changes, the difference between two rates mixes that change with the progress of the program, and the two can no longer be separated.

Why isn’t a single measurement enough?

Even with the five conditions fixed, two consecutive measurements don’t come out exactly the same, because people’s behavior varies from one week to the next. That is why the difference between two rounds says little on its own, and what you read is the trend across several.

Sample sending lets you measure frequently and build that series. In each round, the simulation reaches a fixed percentage of the campaign’s recipients, chosen at random on each send. That way you can measure, for example, every quarter without sending every simulation to the whole organization, and each round adds a point to the series.

To read that series correctly, keep two things in mind:

• Because people are chosen at random, the sample’s rate is an estimate of the rate for all recipients. Drawing again in every round spreads the simulations around and keeps the series from always depending on the same group.
• The margin of that estimate depends on how many people the sample includes. 5% of an organization of 20,000 people is 1,000, and the margin is around 2 points. 20% of one with 200 people is 40, each click moves the rate 2.5 points and the margin is around 9. A small organization needs a higher percentage, or has to send the simulation to everyone.

What is a phishing baseline and what has to stay fixed?

A phishing baseline is the first measurement of how many people fall for a simulation, taken with its conditions recorded so they can be repeated. Later measurements form a series that starts from it, and if the conditions hold, the trend of that series reflects the progress of the program.

When you measure with sample sending, the sample percentage becomes a sixth condition. Before the first simulation of the series, record all six and keep them in every round.

Condition	What stays the same	What changes in each round
Population	The same set of recipients	Normal turnover, which is recorded
Sample	The same sample sending percentage	The people chosen at random
Window	The same number of days open	The start date
Difficulty	An equivalent level	The scenario
Denominator	The same calculation	Nothing
False positives	The same filter	Nothing

The scenario is what changes on purpose. If you reuse it, the second round measures people’s memory of the first round.

How do you repeat a measurement in SMARTFENSE?

To repeat a measurement, duplicate the campaign from the campaign list or from its detail page. SMARTFENSE’s Simulation Tools copy the original campaign’s configuration, with its scenarios, its recipients, the sample sending percentage and the duration. Before launching it, replace the scenarios with others of equivalent difficulty.

To check that two rounds had the same difficulty, look at the “Content difficulty” column in the campaign detail, which shows the level each person received. The platform also detects and hides false positives, so the filter stays the same between rounds. If the tool that scans email changes, record it along with the other conditions.

What should be presented to the board?

With the conditions recorded, the board’s question has an answer. What you present is the series of equivalent measurements, with their conditions, the sample size and its margin. For example, the slide can say that over the last four rounds the rate was 14%, 12%, 10% and 7%, in medium-difficulty simulations open for five days, on a random sample of 20% of recipients (about 600 people per round, with a margin of about 2.5 points), with the same denominator and false positives filtered out.

Comparing with last year works if both measurements belong to the same series. The same slide includes the report rate and how long reports take, measured under the same conditions, because the click rate alone isn’t enough to know whether the program works.

If the population changed substantially, the series breaks, and you say so. A merger, a restructuring or the addition of a subsidiary justifies taking a new baseline, and comparing it with the old series as if nothing had changed undermines whatever you present afterwards. The same applies if previous measurements weren’t taken this way, and in that case the next simulation is the new baseline.

If you want to see how a campaign is duplicated and sample sending is configured, you can request a platform demo.