Email Security Test

This is mostly a shortened English translation of my German article Email Sicherheit – der Test. If you are able to read German and also interested in the status of email security in Germany, then please read the original version, and also some of my other articles.

Of course this is not the only test for email security on the internet, but I haven´t found any that tests inbound and outbound, tries downgrades, and reports the result in a manner I hope end users can understand. And of course the test has to be able to run automated on my side and within reasonable time. Feedback and discussions welcome, send a mail to emailtest@lindenberg.one.

    Running the Test

    As an end user, send Email to en@et.lindenberg.one and en@ut.lindenberg.one (the mail shall go to both addresses). The test may run just a few minutes, sometimes hours or even days, depending on your server behavior. See How does the Test work? below if you want to understand why.

    You may get a non-delivery-notification almost immediately. Some email services using RFC 7672 SMTP Security via Opportunistic DNS-Based Authentication of Named Entities (DANE) Transport Layer Security (TLS) (short SMTP-DANE in the following) do not repeat message transmission if the receiving server does not offer STARTTLS immediately. Imho the service should retry later as it could be both a misconfiguration or a downgrade attack, but RFC 7672 is not clearly specifying the expected behavior. The same may be true for servers supporting RFC 8461: SMTP MTA Strict Transport Security (MTA-STS), though it should not due to section 5. In that case, please resend the test message until you do not get a non-delivery-notification.

    You might also get a non-delivery-notification in case the test service is classifying your message as spam.

    GDPR notice: obviously the test processes your personal data email address. By sending the test message you consent to that processing. I am collecting test results and for public email services also publishing some of them. Your own email address will not be published and is redacted in any replies, as there could be multiple tests running in parallel.

    What does the Test check?

    Four your outgoing emails the test tries a downgrade attack by not offering STARTTLS, in order to figure out whether your server enforces encryption. If instead your server retries later – great. Later on, it offers STARTTLS. In case the tested service supports SMTP-DANE or MTA-STS, it will be using SNI and deliver to various virtual servers following a pattern that can be used to decode the policy used. The test cannot test, whether your server uses DNSSEC outbound. After completing the outbound test, it also reports on inbound characteristics like STARTTLS, DNSSEC or SMTP-DANE.

    Outbound and inbound are from the perspective of the user or service tested. Both directions are important. I encountered email servers that support SMTP-DANE inbound but not outbound, and v.v.. The test also reports on DKIM and ARC but doesn´t validate signatures. The MTA-STS test for inbound is incomplete as it only tests for the existence of both the DNS record and the mta-sts.txt file. So far outlook.com is the only free service that uses MTA-STS and does not also use SMTP-DANE. After completing all tests, it sends a result email providing analysis. As my test is software it may have bugs, if there is anything you don´t understand or want to discuss, feel free to send a mail to the address above.

    I admit, I am biased towards SMTP-DANE. Please checkout RFC 8461, Section 2, it clearly states DANE requires DNSSEC [RFC4033] for authentication; the mechanism described here instead relies on certification authorities (CAs) and does not require DNSSEC, at a cost of risking malicious downgrades. Today there is no reason not to implement DNSSEC. In fact, I am publishing information for MTA-STS inbound but not using MTA-STS outbound as postfix does not support MTA-STS.

    How does the Test work?

    This part is very technical, but some of my readers have been curious and thus I documented it. Of course also because I am a security consultant and advocate for automated testing of security features, and occasionally I want to demo it is possible. If you are not interested in the details, you may skip that section.

    First you need to understand the server configuration. The service runs on two domains, and one of them offers many MX records, all pointing to the same IP address (as IP-addresses are expensive). Thus usage of different virtual servers can only be detected via SNI or recipient email addresses. The servers use different certificates with different TLSA records, and depending on your outbound policy different servers will receive a message or not. Except for mta-sts, the sender policies are in postfix terminology as I am using postfix (mailcow) myself. However postfix does not support mta-sts.

    ConfigurationSender Policy
    ServerSTARTTLSSNICertificateTLSA Recordnonemayencryptverifydanedane-onlymta-sts
    1-- (1)----yesyes----------
    2yes-- (1)self signed----yesyesno-- (3)-- (3)-- (3)
    3yesmx03.et.lindenberg.oneself signedinvalid--(2)(2)nonono-- (4)
    4yesmx04.et.lindenberg.oneself signedvalid--(2)(2)noyesyesno
    5yesmx05.et.lindenberg.oneLetsencryptinvalid--(2)(2)yesnono(5)
    6yesmx06.et.lindenberg.oneLetsencryptvalid--(2)(2)yesyesyesyes
    7yesut.lindenberg.one (1)Letsencrypt--yesyesyesyesyesnoyes
    Comments
    (1)ut.lindenberg.one will be detected via SNI or recipient address, as ut.lindenberg.one does neither support SMTP-DANE nor MTA-STS. Without STARTTLS server 1, without SNI, server 2 is assumed, unless overridden by recipient address.
    (2)I haven´t encountered SNI with policies may or encrypt. I previously classified Outlook.com as an exception, but it does support MTA-STS
    (3)SNI is mandatory for both DANE and MTA-STS.
    (4)This server is not listed in the MTA-STS policy and thus should never be hit if MTA-STS is used. Unfortunately this "no-hit" can only be deduced by either implementing RFC 8460: SMTP TLS Reporting, waiting an appropriate time (say 8h) or for many connection attempts where it is highly unlikely that mx03.let.lindenberg.one should not have been selected based on priority. At present only a timeout is implemented. SMTP TLS Reporting could be more deterministic, but is also available with significant delay (section 4.1, next day plus some hours, plus potential retransmissions)
    (5)Depends on whether sender is also implementing SMTP-DANE: senders who implement MTA-STS validation MUST NOT allow MTA-STS Policy validation to override a failing DANE validation – section 2 of RFC 8461, last sentence.

    Priorities of MX records are updated every five minutes in order to probabilistically ensure that every server gets a chance over time even if the sending service tries only one MX per retry. The test service acknowledges the emails to et.lindenberg.one only after any of the following combinations have been hit: only server 2, all of servers 3-6, or servers 4-6 for a sufficient interval or number of connections. In the connection history there will be a status, that runs from None, SniSeen, TlsStarted, Mail, to Acked. In case a server with policies verify, dane/dane-only, or mta-sts encounters a certificate it has to consider as invalid, it will disconnect rather than transmit. Please drop me a mail in case you do or encounter test quirks.

    After that introduction you are now able to decode a result mail:

    Connection History:08.27.2022 22:47:49 – 22:47:49 (Server 1, plain text, None): 08.27.2022 22:47:49 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): 08.27.2022 22:47:50 – 22:47:50 (Server 1, plain text, None): ...

    For the first connection attempts and within the first three minutes, the test service does not offer STARTTLS – simulated downgrade attacks. A server implementing RFC 7672 or MTA-STS may send unencrypted to ut.lindenberg.one, but never to et.lindenberg.one. For a server not to send unencrypted to ut.lindenberg.one a sender policy needs to be in place. Some providers allow to do that within their user interface, for others the administrators needs to do that. If no policy is defined, there is a line containing (Server 7, plain text, Mail) already in the first three minutes.

    After ca. 3 minutes STARTTLS is active:

    08.27.2022 22:51:03 – 22:51:04 (Server 3, encrypted, TlsStarted): 08.27.2022 22:51:04 – 22:51:08 (Server 6, encrypted, Mail):     From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim08.27.2022 22:51:02 – 22:51:11 (Server 7, encrypted, Mail):     From: <j***m@lindenberg.one> To: <en@ut.lindenberg.one> Signatures:Dkim08.27.2022 22:51:08 – 22:51:11 (Server 4, encrypted, Mail):     From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim08.27.2022 22:52:04 – 22:52:04 (Server 5, encrypted, TlsStarted): 08.27.2022 22:52:03 – 22:52:06 (Server 7, encrypted, Acked):     From: <j***m@lindenberg.one> To: <en@ut.lindenberg.one> Signatures:Dkim08.27.2022 22:52:04 – 22:52:08 (Server 4, encrypted, Acked):     From: <j***m@lindenberg.one> To: <en@et.lindenberg.one> Signatures:Dkim

    Signatures:Dkim or Arc tells that Mails are signed and how. Successful mail deliveries occurred for server 4 and 6, but there have also been connection attempts to servers 3 and 5. The pattern is logically compared to the table above and then the answer is produced. There is an acknowledgment for server 7 and – in that run – server 6 – after all expected servers noticed a connection attempt.

    Analysis Sending of EmailMailservers with valid certificates according to RFC 7672 got a message. Looks like your mailserver uses RFC 7672 (Postfix: dane) (very good).Your mailserver did not send a mail (FROM/RCPT/DATA) without using STARTTLS first. This is often special configuration by the administrator (very good).

    After that there will be the diagnosis for inbound:

    Analysis Reception of Email
    Mail Server (Preference)Address(es)Root Certificate IssuerDNSSECSTARTTLSTLS VersionCertificateMatches TLSAMTA-STS
    mx1.lindenberg.one (10)5.45.107.185, 212.83.56.199ISRG Root X1Mx, A, TlsaSuccessOptional1.2/1.3TrustedOkOk
    Qualified Transport Encryption   
    RFC 7672 SMTP-DANE  
    RFC 8461 MTA-STS  
     Summary
    Networks
     DE-1FIRE-HOSTING, 23M Hostmaster, Germany : 212.83.56.199
     NETCUP_NET-10, netcup GmbH, Germany : 5.45.107.185

    Qualified transport encryption is defined in German recommendation Orientierungshilfe zur Emailverschlüsselungof the German Datenschutzkonferenz (data protection conference) which allows the combination of mandatory encryption, DNSSEC and trusted certificates as an alternative to SMTP-DANE plus mandatory encryption.

    Results for Specific Domains/Providers

    This section collects some test results, in particular if the analysis is not very clear. The color code is reused from other tests (inbound only) I published, red = no encryption, yellow = encryption but no trusted certificates, white = encryption with valid certificates, green = RFC 7672 or German qualified transport encryption. Grey = see comments. If you are running your domain with one of the providers listed, your result can be worse, as RFC 7672 expects all relevant DNS records to be protected with DNSSEC.

    Previous tests analyzed inbound email security of ca. 41,000 public authorities and ca. 40,000 lawyers in Germany (and both articles are in German only, but you can see the colors). There are lots of red and yellow results for domains or providers of these groups. Most of the free offerings in Germany are better, most of them even adopting RFC 7672.

    (1)Non-delivery-notification when STARTTLS is not offered.
    ErstelltGeändertStatusBehördeThema
    18.09.202119.04.2022EingeschlafenBundesamt für Sicherheit in der InformationstechnikSichere Email nach BSI TR-03108
    Das BSI kann ganz offensichtlich keine klaren Empfehlungen geben und verbrennt damit Zeit und Geld aller die sich mit Sicherheit intensiv befassen wollen oder müssen. Mehr dazu auf https://blog.lindenberg.one/BundesamtUnsicherheit.Speziell hier: widersprüchliche Empfehlungen in Grundschutz und BSI TR 03108, keine Übernahme von Internet-Standards, so daß unklar bleibt welche Produkte geeignet sind, und Testanleitungen die unklar bleiben und offensichtlich nur selten praktiziert werden. So skaliert Sicherheit nicht. Das gefährdet unsere Grundrechte auf Privatheit (Artikel 7 EU-Grundrechtecharta) und Datenschutz (Artikel 8) – mehr dazu auf https://blog.lindenberg.one/AufsichtOhneOrientierung.
    24.09.202113.02.2023Teilweise erfolgreichKonferenz der unabhängigen Datenschutzbehörden des Bundes und der LänderOrientierungshilfe Emailverschlüsselung – Sichere Email beim BSI?
    https://blog.lindenberg.one/AufsichtOhneOrientierung
    26.11.202114.04.2022EingeschlafenBundesrechtsanwaltskammerVerzicht auf TOMs, insbesondere Verschlüsselung – BRAO §2?
    Die Anwälte erlauben sich, Artikel 32 DSGVO zu ignorieren, und der BfDI unternimmt nichts. Mehr auf https://blog.lindenberg.one/EmailsicherheitAnwalte und https://blog.lindenberg.one/AufsichtOhneOrientierung.
    08.12.202109.09.2022EingeschlafenKonferenz der unabhängigen Datenschutzbehörden des Bundes und der LänderAuftragsverarbeitung bei Email?
    Der BfDI hat also keine Unterlagen zur Auftragsverarbeitung bei Email die er nach dem IFG herausgeben kann. Aber er ist m.W. auch zuständig für die Beratung, mal sehen ob da etwas kommt. Oder vielleicht aus Baden-Württemberg? Die haben bisher auch nicht reagiert (https://blog.lindenberg.one/BeschwerdenLfdiBw).Weitere Informationen auf https://blog.lindenberg.one/AuftragsverarbeitungEmail und https://blog.lindenberg.one/AufsichtOhneOrientierung.
    18.12.202111.06.2022EingeschlafenBundesnetzagenturöffentlich zugängliche Telekommunikationsdienste?
    Diese Anfrage ist Teil von https://blog.lindenberg.one/AufsichtOhneOrientierung. Über ein Jahr nach Inkrafttreten des neuen TTDSG und TKG ist immer noch nicht klar, wer wieviel Sicherheit gewährleisten soll.
    18.12.202103.07.2024Warten auf Antwort (verspätet)BundesnetzagenturSchutzmaßnahmen und Sicherheitsanforderungen in TKG §§165ff
    Diese Anfrage ist Teil von https://blog.lindenberg.one/AufsichtOhneOrientierung. Über ein Jahr nach Inkrafttreten des neuen TTDSG und TKG ist immer noch nicht klar, wer wieviel Sicherheit gewährleisten soll.
    07.02.202224.05.2022EingeschlafenBundesamt für Sicherheit in der InformationstechnikPGP/S/MIME vs RFC 7672 und DKIM?
    Das BSI hat anscheinend keine (Er-)Kenntnisse, wie leider an so vielen anderen Stellen auch. Allen die hier vorbeilesen oder folgen daher zum Lesen empfohlen, vom speziellen (PGP) zum allgemeineren:https://blog.lindenberg.one/VergleichRfc7672PgpSmimehttps://blog.lindenberg.one/AufsichtOhneOrientierunghttps://blog.lindenberg.one/BundesamtUnsicherheitund natürlich freue ich mich auch über Feedback und Diskussionen.
    19.02.202221.06.2022EingeschlafenBundesnetzagenturFernmeldegeheimnis und Email-Anbieter?
    Diese Anfrage ist Teil von https://blog.lindenberg.one/AufsichtOhneOrientierung. Über ein Jahr nach Inkrafttreten des neuen TTDSG und TKG ist immer noch nicht klar, wer wieviel Sicherheit gewährleisten soll.
    04.03.202227.05.2022EingeschlafenLandesbeauftragter für den Datenschutz Sachsen-AnhaltDie kleinen hängt man, die großen lässt man laufen?
    28.03.202221.06.2022AbgelehntBundesministerium des Innern und für HeimatStand des Regierungsvorhabens „Recht auf Verschlüsselung“
    eigentlich traurig, Mitarbeiter unserer Behörden können nicht lesen. Ich habe nicht nach Entwürfen gefragt sondern nach Evaluierungen warum man ein neues Recht braucht. Diese Bedarfs-Evaluierungen fallen in meinen Augen nicht unter IFG §4.
    09.05.202204.10.2022EingeschlafenDie Bundesbeauftragte für den Datenschutz und die InformationsfreiheitVerhaltensregeln für Notare – ohne Verschlüsselung von Emails?
    Ist halt peinlich für den Bundesbeauftragten für den Datenschutz und die Informationsfreiheit: er fordert Verschlüsselung wenn es selbstverständlich ist oder wo man die Forderung nicht selbst durchsetzen muss. Aber warum die Webseite aber nicht Emails verschlüsselt werden sollen – das möchte ich schon gerne wissen.Siehe auch https://blog.lindenberg.one/AufsichtOhneOrientierung die Anfragen dort.

    Last updated on 03/19/2024.

    © 2024 Joachim Lindenberg. This page reflects my personal opinion and does not provide legal advice.