A phishing simulation went out to ten thousand people. The report closed with nine thousand five hundred opens and nine thousand clicks. A 95% open rate and a 90% click rate are not numbers a group of people produces.
That result showed up more than once in SMARTFENSE’s early years, when simulations were launched without preparing the environment. What it showed was an email security tool that opened and visited the messages before they reached their recipients. That report measured the organization’s email defenses.
When a result doesn’t add up, the conversation lands on my team. And the first question we ask points at phishing simulation delivery in that organization rather than at the people, where the email came in, what touched it before the mailbox, and what the mail client was allowed to load when the employee opened it. The explanation is almost always there.
What does a phishing simulation actually measure?
Every metric in a simulation measures two things at once, the person’s decision and the state of the environment the email travelled through. Being able to separate the two is the condition for reading the result at all.
A 4% click rate can mean the organization learned to be suspicious. It can also mean half the emails never arrived. Both scenarios produce the same number and call for opposite decisions.
It is worth being clear about what is under test. Email security tools have their own evaluation and a team of their own watching them. The simulation exists to measure the layer that remains when those tools fall short, which is the person facing the message. If the barriers stop the lure, the exercise stops reporting on what it went looking for, and it is worth remembering that a campaign is designed to change behavior rather than to audit email security.
Why doesn’t the simulation reach the inbox?
Several layers of control sit between the platform and the employee’s mailbox, and any one of them can stop the message. Preparing them is what is usually called allowlisting, and it comes with a detail almost everyone forgets: it splits into two halves.
The first half is email delivery, handled by declaring the sending IP addresses and domains in the tools that scan inbound mail. The second is browsing, because the employee who clicks ends up on a simulated phishing site that can be blocked too. That half means enabling the browsing domains in the browser, in its extensions and in the endpoint security tools. A campaign with the first half solved and the second one pending delivers the emails and loses every click.
There are three ways to declare it, by IP address, by domain and by message header. The third one comes up when a security tool sits in front of the mail server and rewrites the source IP. In that case declaring our addresses isn’t enough, because the one arriving at the server is already a different address, and the email has to be identified by its header.
The current list of IPs and domains lives inside the platform, under Settings › Security › Whitelist, and it is also part of the published minimum requirements. Check it on the day of the campaign rather than on the day the project started, because it changes when the sending servers change.
Does direct injection replace the allowlist?
Direct Message Injection is an alternative delivery method. Instead of travelling over SMTP and crossing every layer, the message is inserted straight into the inbox through a secure API connection with the mail provider, available for organizations running Microsoft or Google.
It solves three things. The first is delivery, to the point where it removes the need for an email allowlist in some cases. The second is the set of warnings some clients add automatically on top of the message, which tip off the employee before they get a chance to decide. The third is the fragility of the setup, because an update to a security tool can invalidate an allowlist that was working.
What it does not solve is the second half. The simulated phishing site still sits behind the browsing tools, and post-delivery filtering can move the message once it has landed. That is the same conclusion the piece that introduced DMI on this blog already reached, and it still holds. Direct injection shrinks the configuration surface without making it disappear.
Why is the open rate the most fragile metric?
The platform counts an open when the recipient’s mail client displays the message and loads a transparent pixel carrying a unique reference to that person. Open statistics for newsletter, phishing, ransomware and educational moment campaigns all depend on that pixel.
In a new deployment the symptom always looks the same. Emails arrive in the inbox, employees open them, and the open statistic is never recorded. The cause sits in the mail client, which is not allowed to display images, and since the pixel is an image it never loads. Our own minimum requirements guide says it plainly, if the client blocks images the open statistics will be lower than the real ones.
The fix belongs at the level of the whole organization, with a safe sender list applied through central policy or the command line, rather than asking each person to enable images in their own account.
The deviation runs in the opposite direction too. Email privacy protections load remote content on their own. Apple documents that, with that protection on, remote content is downloaded in the background when the message is received rather than when it is opened. The pixel loads without anyone having read anything, and the open is counted all the same.
With both deviations in play, the open rate is only good enough to confirm that delivery worked. The metrics that can carry a decision are the ones that require a deliberate action from the person, the click on the link, the data entered on the simulated site and the report of the message. Those are the ones worth putting on the dashboard when you define what the program is measuring.
When is the number going up not a person?
A false positive is a statistic generated by software and recorded under a user’s name. It is what sat behind the case in the opening paragraph, and it arrives through three routes, the corporate tools that scan the organization’s email, the personal tools installed on the devices that access that email, and other software that touches the message, such as browser extensions or link preview.
Anyone who administers email for an organization knows the list of suspects: spam filters, antivirus, DLP, security gateways, IDS/IPS, continuous link analysis and inbox plugins. None of them is doing anything wrong. They are doing their job on an email that looks like an attack, because it was built to look like one.
The platform detects and filters these cases with an algorithm that flags the affected campaigns, reports the origin of software-generated statistics and lets you tune its parameters to the tools each organization has in place. Detected interactions stay out of the final results, so the statistics and the audit trail contain only what people actually did. The origin and the fixes for false positives are covered on this blog.
The point matters downstream. That same data feeds the risk score for each person. An unfiltered false positive does not stay inside the monthly report. It moves the score of someone who did nothing, and that score later decides who gets additional training.
What should you check before the first campaign?
There are six items, in the order worth resolving them.
- Delivery declared in the tools that scan inbound mail, by IP address, by domain or by header depending on what sits in front of the mail server.
- Browsing enabled for the platform domains in the browser, in its extensions and in the endpoint security tools.
- Current list checked under Settings › Security › Whitelist on the day of the campaign, not on the day the project started.
- Images visible in the mail client, applied through central policy across the organization instead of user by user.
- False positive detection active, with its parameters tuned to the tools the organization has installed.
- A pilot campaign to a small group before the first organization-wide measurement, reviewing delivered, opened and clicked for suspicious matches.
The review is not a one-time task. Every update to the security stack can invalidate a setup that was working, and the best moment to find that out is a fifty-person pilot, not the campaign whose results go to the board.
Frequently asked questions
Why don’t phishing simulations reach the inbox?
Because several layers of control sit between the platform that sends them and the employee’s mailbox, and any of them can stop the message. Preparing those layers means declaring the sending IP addresses and domains, and doing it by IP address, by domain or by message header depending on which tool sits in front of the mail server.
Do I still need an allowlist if I use direct message injection?
Yes, though a smaller one. Direct injection solves email delivery, and in some cases it removes the need for an allowlist on that half. Browsing to the simulated phishing site still requires the domains to be enabled in the browser, in its extensions and in the endpoint security tools.
Why isn’t the open recorded when the employee did open the message?
Because an open is counted when the mail client loads a transparent pixel, and that pixel is an image. If the client does not display images, the message gets read but the open is never recorded. The fix is to enable image display through central policy across the whole organization.
What is a false positive in a phishing simulation?
It is a statistic generated by software and recorded under a user’s name. It comes from corporate tools that scan the organization’s email, personal tools installed on the devices that access that email, and other software that touches the message, such as browser extensions or link preview.
Which simulation metrics are good enough to decide on?
The ones that require a deliberate action from the person, the click on the link, data entered on the simulated site and the report of the message. The open rate is only good enough to confirm that delivery worked.
If you are about to launch a first campaign, the minimum requirements are published and the current list of IPs and domains lives inside the platform. Half an hour of review before sending saves the awkward conversation about whether the number in the report can be trusted.
Leave a Reply