Email Security Test

This is mostly a shortened English translation of my German article Email Sicherheit – der Test. If you are able to read German and also interested in the status of email security in Germany, then please read the original version, and also some of my other articles.

Of course this is not the only test for email security on the internet, but I haven´t found any that tests inbound and outbound, tries downgrades, and reports the result in a manner I hope end users can understand. And of course the test has to be able to run automated on my side and within reasonable time. Feedback and discussions welcome, send a mail to emailtest@lindenberg.one.

Running the Test

As an end user, send Email to en@et.lindenberg.one and en@ut.lindenberg.one (the mail shall go to both addresses). The test may run just a few minutes, sometimes hours or even days, depending on your server behavior. See How does the Test work? below if you want to understand why.

You may get a non-delivery-notification almost immediately. Some email services using RFC 7672 SMTP Security via Opportunistic DNS-Based Authentication of Named Entities (DANE) Transport Layer Security (TLS) (short SMTP-DANE in the following) do not repeat message transmission if the receiving server does not offer STARTTLS immediately. Imho the service should retry later as it could be both a misconfiguration or a downgrade attack, but RFC 7672 is not clearly specifying the expected behavior. The same may be true for servers supporting RFC 8461: SMTP MTA Strict Transport Security (MTA-STS), though it should not due to section 5. In that case, please resend the test message until you do not get a non-delivery-notification.

You might also get a non-delivery-notification in case the test service is classifying your message as spam.

GDPR notice: obviously the test processes your personal data email address. By sending the test message you consent to that processing. I am collecting test results and for public email services also publishing some of them. Your own email address will not be published and is redacted in any replies, as there could be multiple tests running in parallel.

What does the Test check?

Four your outgoing emails the test tries a downgrade attack by not offering STARTTLS, in order to figure out whether your server enforces encryption. If instead your server retries later – great. Later on, it offers STARTTLS. In case the tested service supports SMTP-DANE or MTA-STS, it will be using SNI and deliver to various virtual servers following a pattern that can be used to decode the policy used. The test cannot test, whether your server uses DNSSEC outbound. After completing the outbound test, it also reports on inbound characteristics like STARTTLS, DNSSEC or SMTP-DANE.

Outbound and inbound are from the perspective of the user or service tested. Both directions are important. I encountered email servers that support SMTP-DANE inbound but not outbound, and v.v.. The test also reports on DKIM and ARC but doesn´t validate signatures. The MTA-STS test for inbound is incomplete as it only tests for the existence of both the DNS record and the mta-sts.txt file. So far outlook.com is the only free service that uses MTA-STS and does not also use SMTP-DANE. After completing all tests, it sends a result email providing analysis. As my test is software it may have bugs, if there is anything you don´t understand or want to discuss, feel free to send a mail to the address above.

I admit, I am biased towards SMTP-DANE. Please checkout RFC 8461, Section 2, it clearly states DANE requires DNSSEC [RFC4033] for authentication; the mechanism described here instead relies on certification authorities (CAs) and does not require DNSSEC, at a cost of risking malicious downgrades. Today there is no reason not to implement DNSSEC. In fact, I am publishing information for MTA-STS inbound but not using MTA-STS outbound as postfix does not support MTA-STS.

How does the Test work?

This part is very technical, but some of my readers have been curious and thus I documented it. Of course also because I am a security consultant and advocate for automated testing of security features, and occasionally I want to demo it is possible. If you are not interested in the details, you may skip that section.

First you need to understand the server configuration. The service runs on two domains, and one of them offers many MX records, all pointing to the same IP address (as IP-addresses are expensive). Thus usage of different virtual servers can only be detected via SNI or recipient email addresses. The servers use different certificates with different TLSA records, and depending on your outbound policy different servers will receive a message or not. Except for mta-sts, the sender policies are in postfix terminology as I am using postfix (mailcow) myself. However postfix does not support mta-sts.

Priorities of MX records are updated every five minutes in order to probabilistically ensure that every server gets a chance over time even if the sending service tries only one MX per retry. The test service acknowledges the emails to et.lindenberg.one only after any of the following combinations have been hit: only server 2, all of servers 3-6, or servers 4-6 for a sufficient interval or number of connections. In the connection history there will be a status, that runs from None, SniSeen, TlsStarted, Mail, to Acked. In case a server with policies verify, dane/dane-only, or mta-sts encounters a certificate it has to consider as invalid, it will disconnect rather than transmit. Please drop me a mail in case you do or encounter test quirks.

After that introduction you are now able to decode a result mail:

Connection History:08.27.2022 22:47:49 – 22:47:49 (Server 1, plain text, None): 08.27.2022 22:47:49 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): ...

For the first connection attempts and within the first three minutes, the test service does not offer STARTTLS – simulated downgrade attacks. A server implementing RFC 7672 or MTA-STS may send unencrypted to ut.lindenberg.one, but never to et.lindenberg.one. For a server not to send unencrypted to ut.lindenberg.one a sender policy needs to be in place. Some providers allow to do that within their user interface, for others the administrators needs to do that. If no policy is defined, there is a line containing (Server 7, plain text, Mail) already in the first three minutes.

After ca. 3 minutes STARTTLS is active:

08.27.2022 22:51:03 – 22:51:04 (Server 3, encrypted, TlsStarted): 08.27.2022 22:51:04 – 22:51:08 (Server 6, encrypted, Mail):     From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim08.27.2022 22:51:02 – 22:51:11 (Server 7, encrypted, Mail):     From: <j***m@lindenberg.one> To: <en@ut.lindenberg.one> Signatures:Dkim08.27.2022 22:51:08 – 22:51:11 (Server 4, encrypted, Mail):     From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim08.27.2022 22:52:04 – 22:52:04 (Server 5, encrypted, TlsStarted): 08.27.2022 22:52:03 – 22:52:06 (Server 7, encrypted, Acked):     From: <j***m@lindenberg.one> To: <en@ut.lindenberg.one> Signatures:Dkim08.27.2022 22:52:04 – 22:52:08 (Server 4, encrypted, Acked):     From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim

Signatures:Dkim or Arc tells that Mails are signed and how. Successful mail deliveries occurred for server 4 and 6, but there have also been connection attempts to servers 3 and 5. The pattern is logically compared to the table above and then the answer is produced. There is an acknowledgment for server 7 and – in that run – server 6 – after all expected servers noticed a connection attempt.

Analysis Sending of EmailMailservers with valid certificates according to RFC 7672 got a message. Looks like your mailserver uses RFC 7672 (Postfix: dane) (very good).Your mailserver did not send a mail (FROM/RCPT/DATA) without using STARTTLS first. This is often special configuration by the administrator (very good).

After that there will be the diagnosis for inbound:

Qualified transport encryption is defined in German recommendation Orientierungshilfe zur Emailverschlüsselungof the German Datenschutzkonferenz (data protection conference) which allows the combination of mandatory encryption, DNSSEC and trusted certificates as an alternative to SMTP-DANE plus mandatory encryption.

Results for Specific Domains/Providers

This section collects some test results, in particular if the analysis is not very clear. The color code is reused from other tests (inbound only) I published, red = no encryption, yellow = encryption but no trusted certificates, white = encryption with valid certificates, green = RFC 7672 or German qualified transport encryption. Grey = see comments. If you are running your domain with one of the providers listed, your result can be worse, as RFC 7672 expects all relevant DNS records to be protected with DNSSEC.

Previous tests analyzed inbound email security of ca. 41,000 public authorities and ca. 40,000 lawyers in Germany (and both articles are in German only, but you can see the colors). There are lots of red and yellow results for domains or providers of these groups. Most of the free offerings in Germany are better, most of them even adopting RFC 7672.

Domain/Provider	Encryption	Adequate Certificates	RFC 7672	Encryption mandatory	DKIM	ARC	Comments
aol.com/yahoo.com	yes	yes	no	no	yes	no
dismail.de	yes	yes	only inbound	no	yes	yes	dismail.de supports RFC 7672 only for inbound mails. When sending it does neither enforce enryption nor use TLSA records.
dkinbox.com (posthub.me)	yes	no	no	no	no	no
e.email/ecloud.global	yes	yes	only outbound	yes (2)	yes	yes	TLSA records missing.
gmx.de/gmx.net	yes	yes	yes (1,4)	no	yes	(3)
google.com/gmail.com	yes	yes	no	no	yes	no	google.com/gmail.com does support MTA-STS but fails to implement section 5.1 no 2 of RFC 8461.
icloud.com	yes	yes	no	no	yes	no
kasserver.com	yes	yes	no	no	yes	(3)
kundenserver.de (Ionos)	yes	yes	no	no	no	no
lindenberg.one	yes	yes	yes	optional	yes	no
mail.de	yes	yes	yes	no	yes	no	In the user interface one can tell easily whether encryption will be used. Unfortunately, both RFC 7672 and STARTTLS are shown in green with only a minor difference in the graphics.
mailbox.org	yes	yes	yes (1)	optional	yes	yes	secure.mailbox.org allows to configure mandatory encryption at the expense of then not using RFC 7672 when sending. With the "expert settings" you can choose between encrypt, verify, and dane-only – see RFC 7672 mit Mailcow / Postfix (German). Test results of secure.mailbox.org may therefore vary, especially if multiple tests with different settings are running in parallel.
secure.mailbox.org	yes	yes	see comment, (1)	yes	yes	yes
manitu.net	yes	no	no	no	no	no
outlook.com	yes	yes	outbound only	no	partially	yes	outlook.com does support SMTP-DANE and MTA-STS outbound. Inbound outlook.com does neither support SMTP-DANE nor MTA-STS with the free offering. There are plans for the payed version – How can Exchange Online customers use SMTP DANE Outbound?). When sending, DKIM and MTA-STS are used by some domains (e.g. outlook.com, live.com) but not all (e.g outlook.de).
posteo.de/posteo.ch	yes	yes	inbound only (1)	optional	yes	no	posteo allows to configure mandatory encryption, c.f. TLS-sending guarantee, however RFC 7672 will then not be used.
proton.me/protonmail.ch/protonmail.com	yes	yes	only inbound	no	yes	no	uses verify outbound, but does not enforce encryption.
skiff.com	yes	yes	only inbound	no	yes	no	In one of my tests skiff failed to deliver or bounce a message when receiving server doesn´t accept it repeatedly, violating section 6.2 of RFC 5321 Simple Mail Transfer Protocol. Outbound doesn´t support RFC 7672 nor RFC 8461 nor does it verify certificates.
t-online.de	yes	yes	no	no	no	no	t-online does not even support SPF, no DKIM, no DMARC.
tuta.com/tutanota.com	yes	yes	yes	no	yes	no
vodafonemail.de	yes	yes	no	no	yes	no
web.de	yes	yes	yes (1,4)	no	yes	no
Comments
			(1)				Non-delivery-notification when STARTTLS is not offered.
				(2)			Not clear whether configuration was required.
						(3)	No test after my test server started reporting on ARC.
			(4)				Fails with certificates not generally trusted but trusted via DANE only.