Email Security Test

This is mostly a shortened English translation of my German article Email Sicherheit – der Test. If you are able to read German and also interested in the status of email security in Germany, then please read the original version, and also some of my other articles.

Of course this is not the only test for email security on the internet, but I haven´t found any that tests inbound and outbound, tries downgrades, and reports the result in a manner I hope end users can understand. And of course the test has to be able to run automated on my side and within reasonable time. Feedback and discussions welcome, send a mail to emailtest@lindenberg.one.

Running the Test

As an end user, send Email to en@et.lindenberg.one and en@ut.lindenberg.one (the mail shall go to both addresses). The test may run just a few minutes, sometimes hours or even days, depending on your server behavior. See How does the Test work? below if you want to understand why.

You may get a non-delivery-notification almost immediately. Some email services using RFC 7672 SMTP Security via Opportunistic DNS-Based Authentication of Named Entities (DANE) Transport Layer Security (TLS) (short SMTP-DANE in the following) do not repeat message transmission if the receiving server does not offer STARTTLS immediately. Imho the service should retry later as it could be both a misconfiguration or a downgrade attack, but RFC 7672 is not clearly specifying the expected behavior. The same may be true for servers supporting RFC 8461: SMTP MTA Strict Transport Security (MTA-STS), though it should not due to section 5. In that case, please resend the test message until you do not get a non-delivery-notification.

GDPR notice: obviously the test processes your personal data email address. By sending the test message you consent to that processing. I am collecting test results and for public email services also publishing some of them. Your own email address will not be published and is redacted in any replies, as there could be multiple tests running in parallel.

What does the Test check?

Four your outgoing emails the test tries a downgrade attack by not offering STARTTLS, in order to figure out whether your server enforces encryption. If instead your server retries later – great. Later on, it offers STARTTLS. In case the tested service supports SMTP-DANE or MTA-STS, it will be using SNI and deliver to various virtual servers following a pattern that can be used to decode the policy used. The test cannot test, whether your server uses DNSSEC outbound. After completing the outbound test, it also reports on inbound characteristics like STARTTLS, DNSSEC or SMTP-DANE.

Outbound and inbound are from the perspective of the user or service tested. Both directions are important. I encountered email servers that support SMTP-DANE inbound but not outbound, and v.v.. The test also reports on DKIM and ARC but doesn´t validate signatures. MTA-STS is tested outbound only at present, and so far I haven´t encountered a single (free) email service that uses MTA-STS and does not also use SMTP-DANE. After completing all tests, it sends a result email providing analysis. As my test is software it may have bugs, if there is anything you don´t understand or want to discuss, feel free to send a mail to the address above.

I admit, I am biased towards SMTP-DANE. Please checkout RFC 846, Section 2, it clearly states DANE requires DNSSEC [RFC4033] for authentication; the mechanism described here instead relies on certification authorities (CAs) and does not require DNSSEC, at a cost of risking malicious downgrades. Today there is no reason not to implement DNSSEC. In fact, I am publishing information for MTA-STS inbound but not using MTA-STS outbound.

How does the Test work?

This part is very technical, but some of my readers have been curious and thus I documented it. Of course also because I am a security consultant and advocate for automated testing of security features, and occasionally I want to demo it is possible. If you are not interested in the details, you may skip that section.

First you need to understand the server configuration. The service runs on two domains, and one of them offers many MX records, all pointing to the same IP address (as IP-addresses are expensive). Thus usage of different "virtual" servers can only be detected via SNI or recipient email addresses. The servers use different certificates with different TLSA records, and depending on your outbound policy different servers will receive a message or not. The sender policies are in postfix terminology as I am using postfix (mailcow) myself.

Priorities of MX records are updated every five minutes in order to probabilistically ensure that every server gets a chance over time even if the sending service tries only one MX per retry. The test service acknowledges the emails to et.lindenberg.one only after any of the following combinations have been hit: only server 2, all of servers 3-6, or server 11/12. In the connection history there will be a status, that runs from None, SniSeen, TlsStarted, Mail, to Acked. In case a server with policies verify, dane/dane-only, or mta-sts encounters a certificate it has to consider as invalid, it will disconnect rather than transmit. Connection attempts to mx11 and mx12 are possible, but likely only in case the sending server is very aggressive in trying MX records or it uses MTA-STS, and if it does, connections to servers 11/12 are the only connection attempts. As I wrote, I didn´t encounter any service using MTA-STS only. Please drop me a mail in case you do or encounter test quirks.

After that introduction you are now able to decode a result mail:

Connection History:08.27.2022 22:47:49 – 22:47:49 (Server 1, plain text, None): 08.27.2022 22:47:49 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): ...

For the first connection attempts and within the first three minutes, the test service does not offer STARTTLS – simulated downgrade attacks. A server implementing RFC 7672 or MTA-STS may send unencrypted to ut.lindenberg.one, but never to et.lindenberg.one. For a server not to send unencrypted to ut.lindenberg.one a sender policy needs to be in place. Some providers allow to do that within their user interface, for others the administrators needs to do that. If no policy is defined, there is a line containing (Server 7, plain text, Mail) already in the first three minutes.

After ca. 3 minutes STARTTLS is active:

08.27.2022 22:51:03 – 22:51:04 (Server 3, encrypted, TlsStarted): 08.27.2022 22:51:04 – 22:51:08 (Server 6, encrypted, Mail):     From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim08.27.2022 22:51:02 – 22:51:11 (Server 7, encrypted, Mail):     From: <j***m@lindenberg.one> To: <en@ut.lindenberg.one> Signatures:Dkim08.27.2022 22:51:08 – 22:51:11 (Server 4, encrypted, Mail):     From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim08.27.2022 22:52:04 – 22:52:04 (Server 5, encrypted, TlsStarted): 08.27.2022 22:52:03 – 22:52:06 (Server 7, encrypted, Acked):     From: <j***m@lindenberg.one> To: <en@ut.lindenberg.one> Signatures:Dkim08.27.2022 22:52:04 – 22:52:08 (Server 4, encrypted, Acked):     From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim

Signatures:Dkim or Arc tells that Mails are signed and how. Successful mail deliveries occurred for server 4 and 6, but there have also been connection attempts to servers 3 and 5. The pattern is logically compared to the table above and then the answer is produced. There is an acknowledgment for server 7 and – in that run – server 6 – after all expected servers noticed a connection attempt.

After that there will be the diagnosis for inbound:

Mailservers with valid certificates according to RFC 7672 got a message. Looks like your mailserver uses RFC 7672 (Postfix: dane-only) (very good).Domain lindenberg.one does use DNSSEC with MX-Records (good).Domain lindenberg.one does use DNSSEC with A-Records (good).Domain lindenberg.one does use DNSSEC with TLSA-Records (good).Domain lindenberg.one does support STARTTLS (good).Domain lindenberg.one does use valid certificates (good).Domain lindenberg.one does support qualified transport encryption (good).Domain lindenberg.one does support RFC 7672 SMTP-DANE (good).

qualified transport encryption is defined in German recommendation Orientierungshilfe zur Emailverschlüsselungof the German Datenschutzkonferenz (data protection conference) which allows the combination of mandatory encryption, DNSSEC and trusted certificates as an alternative to SMTP-DANE plus mandatory encryption. I am not testing inbound MTA-STS configuration so far.

Results for Specific Domains/Providers

This section collects some test results, in particular if the analysis is not very clear. The color code is reused from other tests (inbound only) I published, red = no encryption, yellow = encryption but no trusted certificates, white = encryption with valid certificates, green = RFC 7672 or German qualified transport encryption. Grey = see comments. If you are running your domain with one of the providers listed, your result can be worse, as RFC 7672 expects all relevant DNS records to be protected with DNSSEC.

Previous tests analyzed inbound email security of ca. 41,000 public authorities and ca. 40,000 lawyers in Germany (and both articles are in German only, but you can see the colors). There are lots of red and yellow results for domains or providers of these groups. Most of the free offerings in Germany are better, most of them even adopting RFC 7672.

Domain/Provider	Encryption	Adequate Certificates	RFC 7672	Encryption mandatory	DKIM	ARC	Comments
aol.com/yahoo.com	yes	yes	no	no	yes	no
dismail.de	yes	yes	only inbound	no	yes	yes	dismail.de supports RFC 7672 only for inbound mails. When sending it does neither enforce enryption nor use TLSA records.
e.email/ecloud.global	yes	yes	only outbound	yes (2)	yes	yes	TLSA records missing.
gmx.de	yes	yes	yes (1)	no	yes	(3)
google.com/gmail.com	yes	yes	no	no	yes	no
kasserver.com	yes	yes	no	no	yes	(3)
lindenberg.one	yes	yes	yes	optional	yes	no
mail.de	yes	yes	yes	no	yes	no	In the user interface one can tell easily whether encryption will be used. Unfortunately, both RFC 7672 and STARTTLS are shown in green with only a minor difference in the graphics.
mailbox.org	yes	yes	yes (1)	optional	yes	yes	mailbox.org allows to configure mandatory encryption.
outlook.com	yes	yes	no	no	partially	yes	outlook.com does not support RFC 7672 with the free offering. There are plans for the payed version – How can Exchange Online customers use SMTP DANE Outbound?). Outlook does use SNI, thus the test may run very long. DKIM is used by some domains (e.g. outlook.com, live.com) but not all (e.g outlook.de).
t-online.de	yes	yes	no	no	no	no
web.de	yes	yes	yes (1)	no	yes	no
Comments
			(1)				Non-delivery-notification when STARTTLS is not offered.
				(2)			Not clear whether configuration was required.
						(3)	No test after my test server started reporting on ARC.