Email Security Test
This is mostly a shortened English translation of my German article Email Sicherheit – der Test. If you are able to read German and also interested in the status of email security in Germany, then please read the original version, and also some of my other articles.
Of course this is not the only test for email security on the internet, but I haven´t found any that tests inbound and outbound, tries downgrades, and reports the result in a manner I hope end users can understand. And of course the test has to be able to run automated on my side and within reasonable time. Feedback and discussions welcome, send a mail to emailtest@lindenberg.one.
Running the Test
As an end user, send Email to en@et.lindenberg.one and en@ut.lindenberg.one (the mail shall go to both addresses). The test may run just a few minutes, sometimes hours or even days, depending on your server behavior. See How does the Test work? below if you want to understand why.
You may get a non-delivery-notification almost immediately. Some email services using RFC 7672 SMTP Security via Opportunistic DNS-Based Authentication of Named Entities (DANE) Transport Layer Security (TLS)
(short SMTP-DANE in the following) do not repeat message transmission if the receiving server does not offer STARTTLS immediately. Imho the service should retry later as it could be both a misconfiguration or a downgrade attack, but RFC 7672 is not clearly specifying the expected behavior. The same may be true for servers supporting RFC 8461: SMTP MTA Strict Transport Security (MTA-STS)
, though it should not due to section 5. In that case, please resend the test message until you do not get a non-delivery-notification.
GDPR notice: obviously the test processes your personal data email address. By sending the test message you consent to that processing. I am collecting test results and for public email services also publishing some of them. Your own email address will not be published and is redacted in any replies, as there could be multiple tests running in parallel.
What does the Test check?
Four your outgoing emails the test tries a downgrade attack by not offering STARTTLS, in order to figure out whether your server enforces encryption. If instead your server retries later – great. Later on, it offers STARTTLS. In case the tested service supports SMTP-DANE or MTA-STS, it will be using SNI and deliver to various virtual servers following a pattern that can be used to decode the policy used. The test cannot test, whether your server uses DNSSEC outbound. After completing the outbound test, it also reports on inbound characteristics like STARTTLS, DNSSEC or SMTP-DANE.
Outbound and inbound are from the perspective of the user or service tested. Both directions are important. I encountered email servers that support SMTP-DANE inbound but not outbound, and v.v.. The test also reports on DKIM and ARC but doesn´t validate signatures. The MTA-STS test for inbound is incomplete as it only tests for the existence of both the DNS record and the mta-sts.txt file. So far outlook.com is the only free service that uses MTA-STS and does not also use SMTP-DANE. After completing all tests, it sends a result email providing analysis. As my test is software it may have bugs, if there is anything you don´t understand or want to discuss, feel free to send a mail to the address above.
I admit, I am biased towards SMTP-DANE. Please checkout RFC 8461, Section 2, it clearly states DANE requires DNSSEC [RFC4033] for authentication; the mechanism described here instead relies on certification authorities (CAs) and does not require DNSSEC, at a cost of risking malicious downgrades.
Today there is no reason not to implement DNSSEC. In fact, I am publishing information for MTA-STS inbound but not using MTA-STS outbound as postfix does not support MTA-STS.
How does the Test work?
This part is very technical, but some of my readers have been curious and thus I documented it. Of course also because I am a security consultant and advocate for automated testing of security features, and occasionally I want to demo it is possible. If you are not interested in the details, you may skip that section.
First you need to understand the server configuration. The service runs on two domains, and one of them offers many MX records, all pointing to the same IP address (as IP-addresses are expensive). Thus usage of different virtual
servers can only be detected via SNI or recipient email addresses. The servers use different certificates with different TLSA records, and depending on your outbound policy different servers will receive a message or not. Except for mta-sts
, the sender policies are in postfix
terminology as I am using postfix (mailcow) myself. However postfix does not support mta-sts.
| Configuration | Sender Policy | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Server | STARTTLS | SNI | Certificate | TLSA Record | none | may | encrypt | verify | dane | dane-only | mta-sts | |
| 1 | -- (1) | -- | -- | yes | yes | -- | -- | -- | -- | -- | ||
| 2 | yes | -- (1) | self signed | -- | -- | yes | yes | no | -- (3) | -- (3) | -- (3) | |
| 3 | yes | mx03.et.lindenberg.one | self signed | invalid | -- | (2) | (2) | no | no | no | -- (4) | |
| 4 | yes | mx04.et.lindenberg.one | self signed | valid | -- | (2) | (2) | no | yes | yes | no | |
| 5 | yes | mx05.et.lindenberg.one | Letsencrypt | invalid | -- | (2) | (2) | yes | no | no | (5) | |
| 6 | yes | mx06.et.lindenberg.one | Letsencrypt | valid | -- | (2) | (2) | yes | yes | yes | yes | |
| 7 | yes | ut.lindenberg.one (1) | Letsencrypt | -- | yes | yes | yes | yes | yes | no | yes | |
| Comments | ||||||||||||
| (1) | ut.lindenberg.one will be detected via SNI or recipient address, as ut.lindenberg.one does neither support SMTP-DANE nor MTA-STS. Without STARTTLS server 1, without SNI, server 2 is assumed, unless overridden by recipient address. | |||||||||||
| (2) | I haven´t encountered SNI with policies may or encrypt. I previously classified Outlook.com as an exception, but it does support MTA-STS | |||||||||||
| (3) | SNI is mandatory for both DANE and MTA-STS. | |||||||||||
| (4) | This server is not listed in the MTA-STS policy and thus should never be hit if MTA-STS is used. Unfortunately this "no-hit" can only be deduced by either implementing RFC 8460: SMTP TLS Reporting, waiting an appropriate time (say 8h) or for many connection attempts where it is highly unlikely that mx03.let.lindenberg.one should not have been selected based on priority. At present only a timeout is implemented. SMTP TLS Reporting could be more deterministic, but is also available with significant delay (section 4.1, next day plus some hours, plus potential retransmissions) | |||||||||||
| (5) | Depends on whether sender is also implementing SMTP-DANE: senders who implement MTA-STS validation MUST NOT allow MTA-STS Policy validation to override a failing DANE validation– section 2 of RFC 8461, last sentence. | |||||||||||
Priorities of MX records are updated every five minutes in order to probabilistically ensure that every server gets a chance over time even if the sending service tries only one MX per retry. The test service acknowledges the emails to et.lindenberg.one only after any of the following combinations have been hit: only server 2, all of servers 3-6, or servers 4-6 for a sufficient interval or number of connections. In the connection history there will be a status, that runs from None, SniSeen, TlsStarted, Mail, to Acked. In case a server with policies verify, dane/dane-only, or mta-sts encounters a certificate it has to consider as invalid, it will disconnect rather than transmit. Please drop me a mail in case you do or encounter test quirks.
After that introduction you are now able to decode a result mail:
Connection History:08.27.2022 22:47:49 – 22:47:49 (Server 1, plain text, None): 08.27.2022 22:47:49 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): ...
For the first connection attempts and within the first three minutes, the test service does not offer STARTTLS – simulated downgrade attacks. A server implementing RFC 7672 or MTA-STS may send unencrypted to ut.lindenberg.one, but never to et.lindenberg.one. For a server not to send unencrypted to ut.lindenberg.one a sender policy needs to be in place. Some providers allow to do that within their user interface, for others the administrators needs to do that. If no policy is defined, there is a line containing (Server 7, plain text, Mail)
already in the first three minutes.
After ca. 3 minutes STARTTLS is active:
08.27.2022 22:51:03 – 22:51:04 (Server 3, encrypted, TlsStarted): 08.27.2022 22:51:04 – 22:51:08 (Server 6, encrypted, Mail): From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim08.27.2022 22:51:02 – 22:51:11 (Server 7, encrypted, Mail): From: <j***m@lindenberg.one> To: <en@ut.lindenberg.one> Signatures:Dkim08.27.2022 22:51:08 – 22:51:11 (Server 4, encrypted, Mail): From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim08.27.2022 22:52:04 – 22:52:04 (Server 5, encrypted, TlsStarted): 08.27.2022 22:52:03 – 22:52:06 (Server 7, encrypted, Acked): From: <j***m@lindenberg.one> To: <en@ut.lindenberg.one> Signatures:Dkim08.27.2022 22:52:04 – 22:52:08 (Server 4, encrypted, Acked): From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim
Signatures:Dkim or Arc tells that Mails are signed and how. Successful mail deliveries occurred for server 4 and 6, but there have also been connection attempts to servers 3 and 5. The pattern is logically compared to the table above and then the answer is produced. There is an acknowledgment for server 7 and – in that run – server 6 – after all expected servers noticed a connection attempt.
After that there will be the diagnosis for inbound:
Mailservers with valid certificates according to RFC 7672 got a message. Looks like your mailserver uses RFC 7672 (Postfix: dane-only) (very good).Domain lindenberg.one does use DNSSEC with MX-Records (good).Domain lindenberg.one does use DNSSEC with A-Records (good).Domain lindenberg.one does use DNSSEC with TLSA-Records (good).Domain lindenberg.one does support STARTTLS (good).Domain lindenberg.one does use valid certificates (good).Domain lindenberg.one does support qualified transport encryption (good).Domain lindenberg.one does support RFC 7672 SMTP-DANE (good).Domain lindenberg.one may support RFC 8461 MTA-STS (good).
qualified transport encryption
is defined in German recommendation Orientierungshilfe zur Emailverschlüsselungof the German Datenschutzkonferenz
(data protection conference) which allows the combination of mandatory encryption, DNSSEC and trusted certificates as an alternative to SMTP-DANE plus mandatory encryption. I am not testing inbound MTA-STS configuration so far.
Results for Specific Domains/Providers
This section collects some test results, in particular if the analysis is not very clear. The color code is reused from other tests (inbound only) I published, red = no encryption, yellow = encryption but no trusted certificates, white = encryption with valid certificates, green = RFC 7672 or German qualified transport encryption
. Grey = see comments. If you are running your domain with one of the providers listed, your result can be worse, as RFC 7672 expects all relevant DNS records to be protected with DNSSEC.
Previous tests analyzed inbound email security of ca. 41,000 public authorities and ca. 40,000 lawyers in Germany (and both articles are in German only, but you can see the colors). There are lots of red and yellow results for domains or providers of these groups. Most of the free offerings in Germany are better, most of them even adopting RFC 7672.
| Domain/Provider | Encryption | Adequate Certificates | RFC 7672 | Encryption mandatory | DKIM | ARC | Comments |
|---|---|---|---|---|---|---|---|
| aol.com/yahoo.com | yes | yes | no | no | yes | no | |
| dismail.de | yes | yes | only inbound | no | yes | yes | dismail.de supports RFC 7672 only for inbound mails. When sending it does neither enforce enryption nor use TLSA records. |
| e.email/ecloud.global | yes | yes | only outbound | yes (2) | yes | yes | TLSA records missing. |
| gmx.de | yes | yes | yes (1) | no | yes | (3) | |
| google.com/gmail.com | yes | yes | no | no | yes | no | |
| kasserver.com | yes | yes | no | no | yes | (3) | |
| lindenberg.one | yes | yes | yes | optional | yes | no | |
| mail.de | yes | yes | yes | no | yes | no | In the user interface one can tell easily whether encryption will be used. Unfortunately, both RFC 7672 and STARTTLS are shown in green with only a minor difference in the graphics. |
| mailbox.org | yes | yes | yes (1) | optional | yes | yes | mailbox.org allows to configure mandatory encryption. |
| outlook.com | yes | yes | no | no | partially | yes | outlook.com does support MTA-STS, thus the test may run very long. outlook.com does not support RFC 7672 with the free offering. There are plans for the payed version – How can Exchange Online customers use SMTP DANE Outbound?). When sending, DKIM and MTA-STS are used by some domains (e.g. outlook.com, live.com) but not all (e.g outlook.de). |
| proton.me/protonmail.ch | yes | yes | only inbound | no | yes | no | uses verifyoutbound, but does not enforce encryption. |
| t-online.de | yes | yes | no | no | no | no | |
| web.de | yes | yes | yes (1) | no | yes | no | |
| Comments | |||||||
| (1) | Non-delivery-notification when STARTTLS is not offered. | ||||||
| (2) | Not clear whether configuration was required. | ||||||
| (3) | No test after my test server started reporting on ARC. | ||||||
Last updated on 12/01/2022.
© 2023 Joachim Lindenberg. This page reflects my personal opinion and does not provide legal advice.