Mea Culpa. When you make a mistake, admit you made a mistake.
This week, we sent out a marketing email to many of our existing customers about our forthcoming, annual, very popular KB4-CON event (https://www.knowbe4.com/kb4-con). We have been sending out these types of emails since day one of operations back in 2010. We have sent many thousands of these types of emails in the past without error. We have it down to an exact science…or so we thought.
Turns out even the best of intentions can have errors, and we made a big one.
When you send large audience marketing emails, you create a message and use your database of email addresses and names to send it out. Our email content was correct and so was our email address database, but we used an incorrect email address field that resulted in every sent email appearing as if it was a possible phishing attack. That is not a good look for an organization dedicated to defeating social engineering and phishing. Let me explain more.
We had intended to send the email to every involved customer as being sent by that customer’s existing Customer Success Manager (CSM), so the recipient could contact the appropriate KnowBe4 person if they had questions or needed more details. Our customers are familiar with their CSM and often have an ongoing relationship. If this marketing email was done appropriately, the email would have had the CSM’s name in what is known as the “Friendly From” and CSM’s email address in the visible DISPLAY FROM field.
The email’s “FROM” email address should have looked something like, assuming Roger A. Grimes was the involved CSM:
Roger A. Grimes <firstname.lastname@example.org>
The ‘Roger A. Grimes’ part is the “Friendly From” email address and the email@example.com is known as the DISPLAY FROM email address (also officially known as the RFC 5322 address).
Out of the two types of “FROM” email addresses discussed here so far, the second one is more important. That is because the “Friendly From” email address is meant to be very customizable by the sender and the 5322 email address should be the email address of who the email is really from.
That is how I can send an email that appears to be from Roger Grimes from one email client and Roger A. Grimes or Roger Is An Idiot from another. The “Friendly From” email address portion can be updated to be different things while staying attached to the same email address. The 5322 address is supposed to be the real involved email address regardless of the “Friendly From” portion.
Many email servers will not accept an email from a sending email server unless the sending email server’s domain matches the domain indicated in the 5322 email address. Most receiving email servers do not check or care about the “Friendly From” address (although too many users take the opposite approach).
That is why you will often see a spam or phishing email with a pretend “Friendly From” name that matches the brand they are trying to impersonate, but the 5322 email address is something completely different (and often not related to the brand they are trying to impersonate). Here is an example of a real phishing email I received:
The Apple@Service[.]com is the “Friend From” name that is easy to fake, but the 5322 is where the email is claiming to be from, and in this case, it is the entertainingworkshop.com domain. This phishing email was definitely not from apple.com, as it would have been had it been legitimate.
Back to Our Mistake
The customer’s name and email address should have been in the email’s displayed To field (both the customer’s name and email address)…if everything went according to plan. Our email should have been formatted similar to this, assuming Roger A. Grimes was the involved KnowBe4 CSM:
From: Roger A. Grimes <firstname.lastname@example.org>
To: Customer’s Email Address
Instead, what happened was the CSM’s name was included in the “Friendly From” email address (as desired), but the CSM’s DISPAY FROM email address was replaced with the customer’s local KnowBe4 account admin email address. In some cases, the customer’s local account admin could be the same customer. In other cases, it was someone else, usually someone else within their same company, but it could have been someone in a parent company, a tech reseller, an employee of a managed service provider, etc. In any case, the CSM would never have the email address of the customer’s account admin, but that’s what happened.
This is what the errored email looked like logically:
CSM Name <Customer’s Account Admin Email Address>
To: Recipient’s Email Address
But it looked more like this:
From: Roger A. Grimes <MarySmith@customer.com>
To: Customer’s Email Address
The CSM’s name was on the same line as the customer’s local KnowBe4 account admin email address in the FROM field.
That is not right! Mea Culpa!
To compound the mistake further, many email clients will automatically show the sender’s picture or another type of associated image (based on the 5322 email address) if the recipient has previously saved an image to the related contact listing in their email address book. For those “lucky” recipients receiving our incorrect email sent to their own email address with their own image associated, they received an email halfway pretending to be from them AND it was auto-populated with their own image (or their account admin’s). Either way, the image shown would not have been the CSM’s image.
You can imagine how our otherwise safe and legitimate email with mixed up FROM names and addresses looked to the recipients. Our good intentions of sending an email about our special annual event that is enthusiastically attended by many thousands of people each year turned into an unintended, embarrassing “Can You Spot the Phish” test.
While we were happy to see that many of our customers saw something potentially suspicious and first verified its legitimacy before clicking on the included links, it expectedly and rightly created a fair amount of confusion and grief from many others.
We thank all recipients who received the incorrect emails for their patience and understanding. We will endeavor not to make these sorts of email mistakes in the future, as we had not with all the previous emails sent over the last 13 years. We do not like even making one mistake. We have updated our checks and processes to try our best to make sure it does not happen again.
But How Did the Mistaken Email Make It Past Existing Anti-Phishing Filters?
Many of our customers wondered how an email with mixed up email addresses could have made it past their own anti-phishing filters and ended up in their inboxes. Some questioned had we done something to make our emails, configured correctly or not, to make them slip past their anti-phishing filters?
Whether or not your receiving email system would or would not reject an incoming email which contains the recipient’s own email address and/or domain in the FROM field depends on the recipient’s email defenses. We think it is always good to have defenses that reject external emails that claim to be from the recipient’s own domain.
We even have many free email server testing tools you can use to determine if your email server will take and pass an email appearing to be from your own domain: https://www.knowbe4.com/free-it-security-tools#free-email-security-tools.
Some of the customer email servers have policies that will accept any email coming from one of our many legitimate domains. Our official corporate domain is knowbe4.com, but we have many others that we use for sending our simulated phishing tests.
We encourage our customers when allowing emails from our legitimate domains to be accepted (and not blocked) to make sure that our corporate emails, training reminders and simulated phishing emails are seen. Our education and simulated phishing tests cannot do their best if they cannot reach your users.
Domain-based Message Authentication, Reporting and Conformance (DMARC)
Some recipients wondered how our incorrect email could have gotten past their DMARC filters? This part is going to get a little technical, so you can stop reading here if you do not care about DMARC.
We are big fans of DMARC and think every organization should have it enabled. We frequently talk and present about DMARC (and SPF and DKIM), including here: https://info.knowbe4.com/implementing-dmarc and here: https://blog.knowbe4.com/understanding-dmarc-better.
DMARC, Sender Policy Framework (SPF) and Domain Keys Identified Mail (DKIM) are related global anti-phishing standards that allow email receivers to verify if an email that claims to be from a particular domain is really from the domain it claims. In short, it helps to prevent domain spoofing. Email senders can use DMARC to protect their email domains from spoofing by spammers and phishers and email receivers can use DMARC to verify that emails were sent from the domains claimed. Every organization sending or receiving email should enable DMARC. If you are not sure if your organization has enabled DMARC for both sending and receiving, verify it.
All emails can have multiple sender email addresses (or types) attached to it in different places (some are easy to see, some are not). The “Friendly From” and 5322 email addresses were covered above, but there are more address types that can be sent along with an email. Some are visible to anyone opening an email and others require that a recipient open up and view the email’s “header”. The 5321 email address is an example of one that might be buried in the email header until the receiver performs an involved action.
SPF verifies what is known as the RFC 5321 MAIL FROM/RECPT To email address domain. The 5321 email address is placed in the email header by the sending email server and when you hit reply it auto-populates the “Return” email address field (see example below).
DKIM verifies the RFC 5322 DISPLAY FROM email address domain (using the same example shown previously above).
It is important to note that neither SPF or DKIM verifies the “Friendly From” email address which is often what a receiver might see and think is the “real email address”.
A sender’s DMARC policy tells a participating receiver detecting an email from a domain other than what is in its 5322 and 5321 email addresses, how to handle (e.g., reject, quarantine, pass along without modification); plus, it defines reports and diagnostic information. You do not need DMARC to have SPF or DKIM, but you must have either SPF or DKIM enabled, or both, in order for DMARC to work.
When DMARC is enabled for a participating sending domain, involved email servers and clients can check for DMARC-identifying information and then place a “Pass” or “Fail” in the incoming email’s header. Recipients’ email servers and/or client will look at the incoming email’s 5321 (SPF) and/or 5322 (DKIM) addresses, check to see if the sender has enabled a DMARC (or SPF and/or DKIM) policy for the involved domains, and if enabled, do a verification check. If the domain(s) claimed in the email match the sender’s DMARC policy, the email is considered to have passed the DMARC checks. If the domain(s) in the email addresses do not match the addresses indicated in the sender’s DMARC (or SPF and/or DKIM) policies, then it will be considered a Fail. There will be one Pass or Fail for SPF and one Pass or Fail for DKIM (if both are enabled). Usually, a failure of either SPF or DKIM is considered a DMARC failure and handled appropriately.
If an incoming email is identified as having failed an SPF or DKIM check, the sender’s DMARC policy will instruct the sender (really the receiver’s email server or client on behalf of the recipient) on how to treat that failed email. The DMARC policy failure handling instructions can be REJECT (i.e., delete the email and do not display it to the user), QUARANTINE (i.e., place the email in the user’s or an admin’s Spam or Quarantine folder for further manual inspection), or NONE (i.e., allow the email to be processed and displayed like any other email, failure or not). The receiving domain can follow the sender’s DMARC policy recommendation or handle the email however they like depending on the receiver’s policy. If they conflict, the receiver’s handling policies win.
Note: Receiving email servers can be configured with other unrelated policies and rules that could take precedence over anything set in DMARC/SPF/DKIM.
A mature receiving DMARC environment will reject and delete failed emails. Unfortunately, for various reasons, many legitimate emails end up with a Fail rating and if receivers automatically rejected all failures, there could be a lot of angry users who do not get the legitimate email they were expecting. Instead, many DMARC receivers will quarantine failures instead. That is the safest choice. Regrettably, too many organizations set their DMARC policy to NONE. Currently, there are more DMARC policies set to NONE than the two, better options. This is usually because senders are overly worried about too many unexpected rejections of otherwise legitimate emails or they placed their DMARC policy in that state early on for testing and have since never updated it. Either way, there are a lot of sender DMARC policies set to NONE and so any email failing a DMARC check will still end up in user’s inboxes.
If you are interested in learning more about DMARC, SPF or DKIM, watch this webinar: https://info.knowbe4.com/implementing-dmarc.
Back To Our Errored Email and DMARC
Some of the recipients wondered why these incorrect emails did not trigger a “Fail” answer and automatically reject the email since the email’s 5322 address was incorrect. Well, the 5322 address was incorrect, but it was not necessarily a guaranteed failure.
You have to remember that the errored email placed the recipient’s KnowBe4 account admin’s domain in the 5322 position. This means any DKIM check (which verifies the 5322 address) would have pointed to the recipient’s account admin’s DMARC policy (and not ours). The errored email did not come from the recipient’s account admin’s domain and thus would have generated a DKIM failure. But in the recipient cases we have reviewed, the recipient’s account admin’s domain did not have DMARC enabled or had DMARC set to NONE. This means any DKIM failures would have been ignored.
The errored email also had a correct 5321 domain address (checked by SPF) as it did come from one of KnowBe4’s authorized SPF domains. Systems checking for SPF would have seen a Pass rating. If DKIM was not enabled (in this case by the receiver’s account admin’s DMARC policy), which was often the case as well, then DMARC can only rely on the SPF finding, which would have been a Pass. This is to say that in the cases we have seen, DMARC worked as designed (but perhaps not as recipients expected).
We think all organizations should enable DMARC and both SPF and DKIM, and set their DMARC policy to Quarantine or Reject. All email servers should be configured to reject any incoming email that claims to be from an internal domain, by default. We encourage all organizations to ensure that all their DMARC and anti-phishing tools are up to date and configured as expected.
And mostly, we apologize for any frustration we caused due to our error. We will continue to endeavor to be one of your most trusted partners in the battle against cybercrime.