Case Study: IPng's mail servers

Intro

I have seen companies achieve great successes in the space of consumer internet and entertainment industry. I’ve been feeling less enthusiastic about the stronghold that these corporations have over my digital presence. I am the first to admit that using “free” services is convenient, but these companies are sometimes taking away my autonomy and exerting control over society. To each their own of course, but for the last few years, I’ve been more and more inclined to take back a little bit of responsibility for my online social presence, away from centrally hosted services and to privately operated ones.

First off - I love Google’s Workspace products. I started using GMail just after it launched, back in 2004. Its user interface is sleek, performant, and very intuitive. Its filtering, granted, could be a bit less … robotic, but that’s made up by labels and an incredibly comprehensive search function. I would dare say that between GMail and Photos, those are my absolute favorite products on the internet.

That said, I have been running e-mail servers since well before Google existed as a company. I started off at M.C.G.V. Stack, the computer club of the University of Eindhoven, in 1995. We ran sendmail back then, and until about two months ago, I have continuously run sendmail in production using the PaPHosting platform [ref] that I wrote with my buddies Paul and Jeroen.

However, two things happened, both of them somewhat nerdsnipe-esque:

1. Mrs IPngNetworks said “Well if you are going to use NextCloud and PeerTube and PixelFed and Mastodon, why would you not run your own mailserver?”
2. I added a forward for event@frys-ix.net on my Sendmail relays at PaPHosting, and was tipped by my buddy Jelle that his e-mail to it was bouncing due to SPF strictness.

I tried to resist …

My main argument against running a mailserver has been the mailspool. Before I moved to GMail, I had the misfortune of having my mail and primary DNS fail, running on bfib.ipng.nl at the time, a server 700km away from me, without redundancy. Even the nameserver slaves went beyond their zone refresh. It was not a good month for me, even though it was twenty years or so ago :)

Last year, during a roadtrip with Fred, he and I spent a few long hours restoring a backup after a catastrophic failure of a hypervisor at IP-Max on which his mailserver was running. Luckily, backups were awesome and saved the day, but having to go into a Red Alert mode and not being able to communicate, really can be stressful. I don’t want to run mailservers!!!1

.. but resistance is futile

After this nerdsnipe, I had a short conversation with Jeroen who mentioned that since I last had a look at this, Dovecot, a popular imap/pop3 server, had gained the ability to do mailbox synchronization across multiple machines. That’s a really nifty feature - but also it meant that there will be no more single points of failure, if I do this properly. Oh crap, there’s no longer an argument of resistance? Nerd-snipe accepted!

Let me first introduce the mail^W main characters of my story:

• Postfix: is Wietse Venema’s mail server that started life at IBM research as an alternative to the widely-used Sendmail program. After eight years at Google, Wietse continues to maintain Postfix.
• Dovecot: an open source IMAP and POP3 email server for Linux/UNIX-like systems, written with security primarily in mind. Dovecot is an excellent choice for both small and large installations.
• NGINX: an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server, originally written by Igor Sysoev.
• Rspamd: an advanced spam filtering system and email processing framework that allows evaluation of messages by a number of rules including regular expressions, statistical analysis and custom services such as URL black lists.
• OpenDKIM: is a community effort to develop and maintain a C library for producing DKIM-aware applications and an open source milter for providing DKIM service.
• Unbound: a validating, recursive, caching DNS resolver. It is designed to be fast and lean and incorporates modern features based on open standards.
• Roundcube: a web-based IMAP email client. Roundcube’s most prominent feature is the pervasive use of Ajax technology.

In the rest of this article, I’ll go over four main parts that I used to build a fully redundant and self-healing mail service at IPng Networks:

1. Green: smtp-in.ipng.ch which handles inbound e-mail
2. Red: imap.ipng.ch which serves mailboxes to users
3. Blue: smtp-out.ipng.ch which handles outbound e-mail
4. Magenta: webmail.ipng.ch which exposes the mail in a web browser

Let me start with a functional diagram, using those colors:

As you can see in this diagram, I will be separating concerns and splitting the design into three discrete parts, which will also be in three sets of redundantly configured backend servers running on IPng’s hypervisors in Zurich (CH), Lille (FR) and Amsterdam (NL).

1. Outbound: smtp-out

I’m going to start with a relatively simple component first: outbound mail. This service will be listening on the smtp submission port 587, require TLS and user authentication from clients, validate outbound e-mail using a spam detection agent, and finally provide DKIM signing on all outbound e-mails. It should spool and retry the delivery in case there is a temporary issue (like greylisting, or server failure) on the receiving side.

Because the only way to send e-mail will be using TLS and user authentication, the smtp-out servers themselves will not need to do any DNSBL lookups, which is convenient because it means I can put them behind a loadbalancer and serve them entirely within IPng Site Local. If you’re curious as to what this site local thing means, basically it’s an internal network spanning all IPng’s points of presence, with an IPv4, IPv6 and MPLS backbone that is disconnected from the internet. For more details on the design goals, take a look at the [article] I wrote about it last year.

Debian VMs

I’ll take three identical virtual machines, hosted on three separate hypervisors each in their own country.

pim@summer:~$ dig ANY smtp-out.net.ipng.ch
smtp-out.net.ipng.ch.	60	IN	A	198.19.6.73
smtp-out.net.ipng.ch.	60	IN	A	198.19.4.230
smtp-out.net.ipng.ch.	60	IN	A	198.19.6.135
smtp-out.net.ipng.ch.	60	IN	AAAA	2001:678:d78:50e::9
smtp-out.net.ipng.ch.	60	IN	AAAA	2001:678:d78:50a::6
smtp-out.net.ipng.ch.	60	IN	AAAA	2001:678:d78:510::7

I will give them each 8GB of memory, 4 vCPUs, and 16GB of bootdisk. I’m pretty confident that the whole system will be running in only a fraction of that. I will install a standard issue Debian Bookworm (12.5), and while my VMs by default have 4 virtual NICs, I only need one, connected to the IPng Site Local:

pim@smtp-out-chrma0:~$ ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128
enp1s0f0         UP             198.19.6.135/27 2001:678:d78:510::7/64 fe80::5054:ff:fe99:81b5/64
enp1s0f1         UP             fe80::5054:ff:fe99:81b6/64
enp1s0f2         UP             fe80::5054:ff:fe99:81b7/64
enp1s0f3         UP             fe80::5054:ff:fe99:81b8/64

pim@smtp-out-chrma0:~$ mtr -6 -c5 -r dns.google
Start: 2024-05-17T13:49:28+0200
HOST: smtp-out-chrma0             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- msw1.chrma0.net.ipng.ch    0.0%     5    1.6   1.5   1.3   1.6   0.1
  2.|-- msw0.chrma0.net.ipng.ch    0.0%     5    1.4   1.3   1.3   1.4   0.1
  3.|-- msw0.chbtl0.net.ipng.ch    0.0%     5    3.2   3.1   2.8   3.2   0.2
  4.|-- hvn0.chbtl0.net.ipng.ch    0.0%     5    1.5   1.5   1.4   1.5   0.0
  5.|-- chbtl0.ipng.ch             0.0%     5    1.6   1.7   1.6   1.7   0.0
  6.|-- chrma0.ipng.ch             0.0%     5    2.4   2.4   2.4   2.5   0.0
  7.|-- as15169.lup.swissix.ch     0.0%     5    3.2   3.8   3.2   5.6   1.0
  8.|-- 2001:4860:0:1::6083        0.0%     5    4.5   4.5   4.5   4.5   0.0
  9.|-- 2001:4860:0:1::12f9        0.0%     5    3.4   3.5   3.4   3.5   0.0
 10.|-- dns.google                 0.0%     5    3.8   3.9   3.8   4.0   0.1

One cool observation: these machines are not really connected to the internet - you’ll note that their IPv4 address is from reserved space, and their IPv6 supernet (2001:678:d78:500::/56) is filtered at the border. I’ll get to that later!

Postfix

I will install Postfix, and make a few adjustments to its config. First off, this mailserver will only be receiving submission mail, which is port 587. It will not participate or listen to the regular smtp port 25, nor smtps port 465, as such the master.cf file for Postfix becomes:

#smtp      inet  n       -       y       -       -       smtpd
#  -o smtpd_sasl_auth_enable=no
submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_client_restrictions=permit_sasl_authenticated,permit_mynetworks,reject
  -o milter_macro_daemon_name=ORIGINATING
#smtps     inet  n       -       y       -       -       smtpd
#  -o syslog_name=postfix/smtps
#  -o smtpd_tls_wrappermode=yes
#  -o smtpd_sasl_auth_enable=yes

The only thing I will make note of is that the submission service has a set of client restrictions. In other words, to be able to use this service, the client must either be SASL authenticated, or from a list of network prefixes that are allowed to relay. If neither of those two conditions are satisfied, relaying will be denied.

Now, I understand that pasting in the entire postfix configuration is a bit verbose, but honestly I’ve spent many an hour trying to puzzle together an end-to-end valid configuration, so I’m just going to swim upstream and post the whole main.cf, which I’ll try to annotate the broad strokes of, in case there’s anybody out there trying to learn:

myhostname = smtp-out.ipng.ch
myorigin   = smtp-out.ipng.ch
mydestination = $myhostname, smtp-out.chrma0.net.ipng.ch, localhost.net.ipng.ch, localhost
mynetworks = 127.0.0.0/8, [::1]/128

recipient_delimiter = +
inet_interfaces = all
inet_protocols = all
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no
readme_directory = no

# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 3.6 on fresh installs.
compatibility_level = 3.6

# SMTP Server
smtpd_banner = $myhostname ESMTP $mail_name (smtp-out.chrma0.net.ipng.ch)
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
smtpd_tls_cert_file = /etc/certs/ipng.ch/fullchain.pem
smtpd_tls_key_file = /etc/certs/ipng.ch/privkey.pem
smtpd_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtpd_tls_CApath = /etc/ssl/certs
smtpd_use_tls = yes
smtpd_tls_received_header = yes
smtpd_tls_auth_only = yes
smtpd_tls_session_cache_database = btree:$data_directory/smtpd_scache
smtpd_client_connection_count_limit = 4
smtpd_client_connection_rate_limit = 10
smtpd_client_message_rate_limit = 60
smtpd_client_event_limit_exceptions = $mynetworks

# Dovecot auth
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_authenticated_header = yes
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous, noplaintext
smtpd_sasl_tls_security_options = noanonymous

# SMTP Client
smtp_use_tls = yes
smtp_tls_note_starttls_offer = yes
smtp_tls_cert_file = /etc/certs/ipng.ch/fullchain.pem
smtp_tls_key_file = /etc/certs/ipng.ch/privkey.pem
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_tls_CApath = /etc/ssl/certs
smtp_tls_mandatory_ciphers = medium
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_security_level = encrypt

header_size_limit = 4096000
message_size_limit = 52428800
mailbox_size_limit = 0

# OpenDKIM, Rspamd
smtpd_milters = inet:localhost:8891,inet:rspamd.net.ipng.ch:11332
non_smtpd_milters = $smtpd_milters

# Local aliases
alias_maps = hash:/etc/postfix/aliases
alias_database = hash:/etc/postfix/aliases

Hostnames: The full (internal) hostname for the server is smtp-out.$(site).net.ipng.ch, in this case for chrma0 in Rümlang, Switzerland. However, when clients connect to the public hostname smtp-out.ipng.ch, they will expect that the TLS certificate matches that hostname. This is why I let the server present itself as simply smtp-out.ipng.ch, which will also be its public DNS name later, but put the internal FQDN for debugging purposes between parenthesis. See the smtpd_banner and myhostname for the destinction. I’ll load up the *.ipng.ch wildcard certificate which I described in my Let’s Encrypt [DNS-01] article.

Authorization: I will make Postfix accept relaying for those users that are either in the mynetworks (which is only localhost) OR sasl_authenticated (ie. presenting a username and password). This password exchange will only be possible after encryption has been triggered using the STARTTLS SMTP feature. This way, user/pass combos will be safe on the network.

Authentication: Those username and password combos can come from a few places. One popular way to do this is via a dovecot authentication service. Via the smtpd_sasl_path, I tell Postfix to ask these authentication questions using the dovecot protocol on a certain file path. I’ll let Dovecot listen in the /var/spool/postfix/private/auth directory. This is how Postfix will know which user to relay for, and which to deny.

DKIM/SPF: These days, most (large and small) mail providers will be suspicious of e-mail that is delivered to them without proper SPF and DKIM fields. DKIM is a mechanism to create a cryptographic signature over some of the E-Mail header fields (usually From/Subject/Date), which can be checked by the recipient for validity. SPF is a mechanism to use DNS to inform receiving mailservers of which are the valid IPv4/IPv6 addresses that should be used to deliver mail for a given sender domain.

Dovecot (auth)

The configuration for Dovecot is incredibly simple. The only thing I do is create a mostly empty dovecot.conf file which defines the auth service listening in the place where Postfix expects it. Then, I add a password file called sasl-users which will contain user:password tuples:

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    # Assuming the default Postfix user and group
    user = postfix
    group = postfix
  }
}

passdb {
  driver = passwd-file
  args   = username_format=%n /etc/dovecot/sasl-users
}

I can use doveadm pw to generate such passwords. I do this in an upstream Ansible repository and then push out the same configuration to any number of smtp-out servers, so they are all configured identically to this one.

OpenDKIM (signing)

Now that I can authorize (via SASL) and authenticate (via Dovecot backend) a user, it will be entitled to use the smtp-out Postfix to send e-mail. However, there’s a good chance that recipients will bounce the e-mail, unless it comes with a DKIM signature, and from the correct IP addresses.

To configure DKIM signing, I use OpenDKIM, which I give the following /etc/opendkim.conf file:

pim@smtp-out-chrma0:~$ cat /etc/opendkim.conf
Syslog              yes
LogWhy              yes
UMask               007
Mode                sv
AlwaysAddARHeader   yes
SignatureAlgorithm  rsa-sha256
X-Header            no
KeyTable            refile:/etc/opendkim/keytable
SigningTable        refile:/etc/opendkim/signers
RequireSafeKeys     false
Canonicalization    relaxed
TrustAnchorFile	    /usr/share/dns/root.key
UserID              opendkim
PidFile             /run/opendkim/opendkim.pid
Socket              inet6:8891

It opens a socket at port 8891, which is where Postfix expects it, based on its smtpd_milter configuration option. It will look at the so-called SigningTable to determine which outbound e-mail addresses it can sign. This table looks up From addresses, including wildcards, and informs which symbolic keyname in the KeyTable to use for the signature, like so:

pim@smtp-out-chrma0:/etc/opendkim$ cat signers
*@*.ipng.nl               ipng-nl
*@ipng.nl                 ipng-nl
*@*.ipng.ch               ipng-ch
*@ipng.ch                 ipng-ch
*@*.ublog.tech            ublog
*@ublog.tech              ublog
...

pim@smtp-out-chrma0:/etc/opendkim$ cat keytable
ipng-nl                   ipng.nl:DKIM2022:/etc/opendkim/keys/DKIM2022-ipng.nl-private
ipng-ch                   ipng.ch:DKIM2022:/etc/opendkim/keys/DKIM2022-ipng.ch-private
ublog                     ublog.tech:DKIM2022:/etc/opendkim/keys/DKIM2022-ublog.tech-private
...

This allows OpenDKIM to sign messages for any number of domains, using the correct key. Slick!

NGINX

Now that I have three of these identical VMs, I am ready to hook them up to the internet. On the way in, I will point smtp-out.ipng.ch to our NGINX cluster. I wrote about that cluster in a [previous article]. I will add a snippet there, that exposes these VMs behind a TCP loadbalancer like so:

pim@squanchy:~/src/ipng-ansible/roles/nginx/files/streams-available$ cat smtp-out.ipng.ch.conf
upstream smtp_out {
  server smtp-out.chrma0.net.ipng.ch:587 fail_timeout=10s max_fails=2;
  server smtp-out.frggh0.net.ipng.ch:587 fail_timeout=10s max_fails=2 backup;
  server smtp-out.nlams2.net.ipng.ch:587 fail_timeout=10s max_fails=2 backup;
}

server {
  listen [::]:587;
  listen 0.0.0.0:587;
  proxy_pass smtp_out;
}

I make use of the backup keyword, which will make the loadbalancer choose, if it’s available, the primary server in chrma0. If it were to go down, no problem, two connection failures within ten seconds will make NGINX choose the alternative ones in frggh0 or nlams2.

IPng Site Local gateway

When the smtp-out server receives the e-mail from the customer/client, it’ll spool it and start to deliver it to the remote MX record. To do this, it’ll create an outbound connection from its cozy spot within IPng Site Local (which, you will remember, is not connected directly to the internet). There are three redundant gateways in IPng Site Local (in Geneva, Brüttisellen and Amsterdam). If any of these were to go down for maintenance or fail, the network will use OSPF E1 to find the next closest default gateway. I wrote about how this entire european network is connected via three gateways that are self-repairing in this [article], in case you’re curious.

But, for the purposes of SMTP, it means that each of the internal smtp-out VMs will be seen by remote mailservers as NATted via one of these egress points. This allows me to determine the SPF records in DNS. With that, I’m ready to share the publicly visible details for this service:

_spf.ipng.ch.     3600 IN TXT "v=spf1 include:_spf4.ipng.ch include:_spf6.ipng.ch ~all"
_spf4.ipng.ch.    3600 IN TXT "v=spf1 ip4:46.20.246.112/28 ip4:46.20.243.176/28 ip4:94.142.245.80/29"
                              "ip4:94.142.241.184/29 ip4:194.1.163.0/24 ~all"
_spf6.ipng.ch.    3600 IN TXT "v=spf1 ip6:2a02:2528:ff00::/40 ip6:2a02:898:146::/48"
                              "ip6:2001:678:d78::/48 ~all"

smtp-out.ipng.ch. 3600 IN CNAME nginx0.ipng.ch.
nginx0.ipng.ch.   600  IN A     194.1.163.151
nginx0.ipng.ch.   600  IN A     46.20.246.124
nginx0.ipng.ch.   600  IN A     94.142.241.189
nginx0.ipng.ch.   600  IN AAAA  2001:678:d78:7::151
nginx0.ipng.ch.   600  IN AAAA  2a02:2528:ff00::124
nginx0.ipng.ch.   600  IN AAAA  2a02:898:146::5

To re-iterate one point: the inbound path of the mail is via the redundant cluster of nginx0 entrypoints, while the outbound path will be seen from gw0.chbtl0.ipng.ch, gw0.chplo0.ipng.ch or gw0.nlams3.ipng.ch, which are all covered by the SPF records for IPv4 and IPv6.

Bonus: opensmtpd on clients

By the way, every single server (VM, hypervisor, router) at IPng Neworks will all use smtp-out to send e-mail. I use opensmtpd for that, and it’s incredibly simple:

pim@squanchy:~$ cat /etc/mail/smtpd.conf
table aliases file:/etc/mail/aliases
table secrets file:/etc/mail/secrets

listen on localhost

action "local_mail" mbox alias <aliases>
action "outbound" relay host "smtp+tls://ipng@smtp-out.ipng.ch:587" auth <secrets> mail-from "@ipng.ch"

match from local for local action "local_mail"
match from local for any action "outbound"

pim@squanchy:~$ sudo cat /etc/mail/secrets
ipng bastion:<haha-made-you-look>

What happens here is, every time this server squanchy wants to send an e-mail, it will use an SMTP session with TLS, on port 587, of the machine called smtp-out.ipng.ch, and it’ll authenticate using the opensmtpd realm called ipng, which maps to a username:password tuple in the secrets file. It will also rewrite the envelope to be always from @ipng.ch. As a best practice I organize my SMTP users by Ansible group. Squanchy is in the group bastion, hence its username. By doing it this way, I can make use of the DKIM and SPF, which makes all mails properly formatted, routed, signed and delivered. I love it, so much!

2. Inbound: smtp-in

The smtp-out service I described in the previous section is completely standalone. That is to say, its purpose is only to receive submitted mail from humans and servers, sign it, spool it if need be, and deliver it. But users also want to deliver e-mail to me and my customers. For this, I’ll build a second cluster of redundant inbound mailservers: smtp-in.

Here, the base setup is not too different from above, so I won’t repeat it. I’ll take three identical VMs, in three different datacenters, and install them with Debian and Postfix as well. But, contrary to the outbound servers, here I will make them listen to the smtp port 25 and the smtps port 465, and I’ll turn off the ability to authenticate with SASL (and thereby, refuse to forward any e-mail that I’m not the MX record for), making master.cf look like this:

smtp      inet  n       -       y       -       -       smtpd
  -o smtpd_sasl_auth_enable=no
#submission inet n       -       y       -       -       smtpd
#  -o syslog_name=postfix/submission
#  -o smtpd_tls_security_level=encrypt
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=permit_sasl_authenticated,permit_mynetworks,reject
#  -o milter_macro_daemon_name=ORIGINATING
smtps     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=no

Many of the main.cf attributes are the same, unsurprisingly the myhostname configuration option is set to smtp-in.ipng.ch, which is going to be expected to match the wildcard SSL certificate from the smtpd_tls_cert_file config option. The banner is a bit more telling, as it shows also the FQDN hostname (eg. smtp-in.frggh0.net.ipng.ch), helpful when debugging.

# Impose DNSBL restrictions at SMTP time
smtpd_recipient_restrictions = permit_mynetworks,
        reject_invalid_helo_hostname,
        reject_non_fqdn_recipient,
        reject_unknown_recipient_domain,
        reject_unauth_pipelining,
        reject_unauth_destination,
        reject_rbl_client zen.spamhaus.org=127.0.0.[2..11],
        reject_rhsbl_sender dbl.spamhaus.org=127.0.1.[2..99],
        reject_rhsbl_helo dbl.spamhaus.org=127.0.1.[2..99],
        reject_rhsbl_reverse_client dbl.spamhaus.org=127.0.1.[2..99],
        warn_if_reject reject_rbl_client zen.spamhaus.org=127.255.255.[1..255],
        reject_rbl_client dnsbl-1.uceprotect.net,
        reject_rbl_client bl.0spam.org=127.0.0.[7..9],
        permit

# Milter for rspamd
smtpd_milters = inet:rspamd.net.ipng.ch:11332
milter_default_action = accept

# PostSRSd
sender_canonical_maps = tcp:localhost:10001
sender_canonical_classes = envelope_sender
recipient_canonical_maps = tcp:localhost:10002
recipient_canonical_classes= envelope_recipient,header_recipient

# Virtual domains
virtual_alias_domains = hash:/etc/postfix/virtual-domains
virtual_alias_maps = hash:/etc/postfix/virtual

The config arguably is quite compact. but I will hilight four specific pieces.

DNSBL: When connecting and receiving the envelope (ie. MAIL FROM and RCPT TO in the SMTP transaction), I’ll ask Postfix to do a bunch of DNS blocklist lookups. Many sender domains, and infected hosts/networks are mapped in public DNSBL zones, notably [Spamhaus], [UCEProtect], and [0Spam] do a great job at identifying malicious and spammy domain-names and networks. So I’ll tell Postfix to reject folks attempting to connect from these low-reputation places.

Rspamd: Here’s where I hook up a redundant cluster of rspamd servers. Each e-mail, once accepted, will be routed through this milter, and after thinking about it a little bit, the Rspamd server will either answer:

• greylisted: where Rspamd recommends a tempfail so the remote mailserver comes back after a few minutes after connecting for the first time, many spammers will not do this.
• blocked: if Rspamd finds the e-mail is egregious, it’ll simply recommend a permfail so Postfix immediately rejects it.
• tagged: if Rspamd is iffy about the e-mail, it may insert an X-Spam header, so that downstream mail clients like Thunderbird or Mail.app can decide for themselves to consider the e-mail junk or not.

PostSRS: This is a really useful feature which allows Postfix to safely forward an e-mail to another mailhost. Perhaps best explained with an example, notably the aforementioned nerdsnipe from my buddy Jelle:

1. Let’s say jelle@luteijn.email sends an e-mail to event@frys-ix.net for which IPng is the mailhost.
2. Jelle configured his SPF records to allow mail to come from either ip4:185.36.229.0/24 or ip6:2a07:cd40::/29, and if it comes from neither of those, to hard fail the SPF check -all.
3. My spiffy smtp-in.ipng.ch receives this e-mail and decides to forward it internally by rewriting it to foo@eritap.com.
4. The mailserver for eritap.com now sees an e-mail coming From: jelle@luteijn.email going to its foo@eritap.com. It does an SPF check and concludes: Yikes! That mailserver smtp-in.ipng.ch is NOT authorized to send e-mail on behalf of Jelle, so reject it! A kitten gets hurt, which is obviously unacceptable.

To handle this, PostSRSd detects when such a forward is about to happen, and rewrites the envelope From: header to be something that smtp-in.ipng.ch might be allowed to deliver mail for: something in the @ipng.ch domain! Using a secret (shared between the replicas of IPng’s smtp-in cluster), it can insert a little cryptographic signature as it does this rewrite.

In the example above, the e-mail from jelle@luteijn.email will be rewritten to an envelope such as SRS0=CCIM=MT=luteijn.email=jelle@ipng.ch and while hideous, it is in the @ipng.ch domain. If a bounce for this e-mail were to be generated, PostSRSd can also rewrite in reverse, re-assembling the original envelope From when sending the bounce on to Jelle’s mailserver.

I configure Postfix to do this using the sender and recipient canonical maps. I read these from a server running on localhost port 10001 and 10002 respectively. This is where PostSRSd does its magic.

Oh, what’s that I hear? The telephone is ringing! 1982 called, and it wants to change the title of [RFC821] from SMTP to CMTP (Convoluted Mail Transfer Protocol).

Virtual: With all of that out of the way, I can now receive and forward aliased e-mails. I won’t be using local mail delivery (to unix users on the local machine), but rather I will forward the mails for my local users onwards to what is called a redundant maildrop server. So for the virtualized part of the Postfix config, I have things like this:

pim@smtp-in-chrma0:~$ cat /etc/postfix/virtual-domains
ublog.tech	ublog.tech
frys-ix.net	frys-ix.net
ipng.nl	ipng.nl
ipng.ch	ipng.ch
...

pim@smtp-in-chrma0:~$ cat /etc/postfix/virtual
## Virtual domain: ipng.ch
postmaster@ipng.ch             pim+postmaster@maildrop.net.ipng.ch
hostmaster@ipng.ch             pim+hostmaster@maildrop.net.ipng.ch
abuse@ipng.ch                  pim+abuse@maildrop.net.ipng.ch
pim@ipng.ch                    pim+ipng@maildrop.net.ipng.ch
noreply@ipng.ch                /dev/null
...
## Virtual domain: ipng.nl
@ipng.nl                       @ipng.ch
## Virtual domain: frys-ix.net
postmaster@frys-ix.net         pim+postmaster@maildrop.net.ipng.ch
hostmaster@frys-ix.net         pim+hostmaster@maildrop.net.ipng.ch
abuse@frys-ix.net              pim+abuse@maildrop.net.ipng.ch
noc@frys-ix.net                pim+frysix@maildrop.net.ipng.ch,noc@eritap.com
pim@frys-ix.net                pim+frysix@maildrop.net.ipng.ch
arend@frys-ix.net              arend+frysix@eritap.com
event@frys-ix.net              someplace@example.com
...

The first file here virtual_alias_domains, simply explains to Postfix which domains it is to accept e-mail for. This avoids users trying to use it as a relay. If the domain is not listed in the lefthand side of the table, it’s not welcome here. But then once Postfix knows it’s supposed to be accepting e-mail for this domain, it will consult the virtual_alias_maps configuration. Here, I showed three domains, and a few features:

• I can simply forward along pim@ipng.ch to pim+ipng@maildrop.net.ipng.ch. Cool.
• I can toss the email by passing it to /dev/null (useful for things like noreply@ and nobody@)
• I can forward it to multiple recipients as well, for example noc@frys-ix.net goes to me and Eritap (hoi, Arend!)

When such a forward happens, PostSRSd kicks in, and for that e-mail, the envelope rewrite will happen such that smtp-in can safely deliver this to even the strictest of SPF users.

Why no NGINX ?

There’s an important technical reason for me not to be able to use an inbound loadbalancer, even though I’d love to frontend port 25 and 465 on IPng’s nginx cluster. I have enabled the use of DNSBL, which implies that Postfix needs to know the remotely connecting IPv4 and IPv6 addresses. While for domain-based blocklists this is not important, for IP based ones like zen.spamhaus.org it is critical. Therefore, I will assign a public IPv4 and IPv6 address to each of the machines in the cluster. They will be used in a round-robin way, and if one of them is down for a while, remote mail servers will automatically and gracefully use another replica.

With that, the public DNS entries:

ublog.tech.		86400	IN	MX	10 smtp-in.ipng.ch.
ipng.nl.		86400	IN	MX	10 smtp-in.ipng.ch.
ipng.ch.		86400	IN	MX	10 smtp-in.ipng.ch.

smtp-in.ipng.ch.	60	IN	A	46.20.246.125
smtp-in.ipng.ch.	60	IN	A	94.142.245.85
smtp-in.ipng.ch.	60	IN	A	194.1.163.141
smtp-in.ipng.ch.	60	IN	AAAA	2a02:2528:ff00::125
smtp-in.ipng.ch.	60	IN	AAAA	2a02:898:146:1::5
smtp-in.ipng.ch.	60	IN	AAAA	2001:678:d78:6::141

3. Dovecot: maildrop

Remember when I said that mail to pim@ipng.ch is forwarded to pim+ipng@maildrop.net.ipng.ch? Doing this allows me to have replicated, fully redundant, IMAP servers! As it turns out, Dovecot, a very popular open source pop3/imap server, has the ability to do realtime synchronization between multiple machines serving the same user.

On these servers, I’ll start with enabling Postfix only using the smtp and smtps transport in master.cf. The maildrop servers will be entirely within IPng Site Local, and cannot be reached from the internet directly, just the same as the smtp-out server replicas.

Postfix on the server receives mail from the smtp-in servers as the final destination for an e-mail. It does this very similar to the smtp-in server pool I described above, with two notable differences:

1. It does not need to do DNSBL lookups or spam analysis – those have already happened upstream from these maildrop servers by the smtp-in servers. That’s also why these can be safely tucked away in IPng Site Local.
2. Their virtual maps point to what is called an LMTP: Local Mail Transport Protocol, where I’ll ask Postfix to pump them into a redundalty replicated Dovecot pair.

# Completely virtual
virtual_alias_maps = hash:/etc/postfix/virtual-maildrop
virtual_mailbox_domains = maildrop.net.ipng.ch
virtual_transport = lmtp:unix:private/dovecot-lmtp

pim@maildrop0-chbtl0:$ cat /etc/postfix/virtual-maildrop
pim@maildrop.net.ipng.ch			pim

What I’ve done here is define only one virtual_mailbox_domains entry, for which I look up the users in the virtual_alias_maps and use a virtual_transport to deliver the enduser (pim) to a unix domain socket in /var/spool/postfix/private/dovecot-lmtp. Once again, mail servers are super simple after you’ve spent ten hours reading configuration manuals and RFCs and asked at least three other people how they did theirs…. Super… Simple!

Dovecot

By default, Dovecot ships with a very elaborate configuration file hierarchy. I decide to replace it with an autogenerated one from Ansible that has fewer includes (namely: none at all). Here’s the features that I want to enable in Dovecot:

• UserDB: To define username, password and mail directory for users.
• LMTP: To be a local recepticle for the Postfix delivery
• IMAP: To serve SSL enabled IMAP to mail clients like Mail.app, Thunderbird, Roundcube, etc.
• Replicator: To replicate mailboxes between pairs of Dovecot servers.
• Sieve: To allow users to create mail filters using the Sieve protocol.

Starting from the easier bits, here’s how I configure the User Database in dovecot.conf:

passdb {
  driver = passwd-file
  args   = username_format=%n /etc/dovecot/maildrop-users
}

userdb {
  driver = passwd-file
  args   = username_format=%n /etc/dovecot/maildrop-users
  default_fields = uid=vmail gid=vmail home=/var/dovecot/users/%n
}

mail_plugins = $mail_plugins notify push_notification replication
mail_location = mdbox:~/mdbox

I can add a user pim with an encrypted password from doveadm pw like so:

pim@maildrop0-chbtl0:/etc/dovecot$ sudo cat maildrop-users
...
pim:{CRYPT}$2y$<some encrypted password goes here>::::

Due to the passdb option, this user can authenticate with username and password, and due to the userdb option, this user receives a mailbox homedir in the specified location. One important observation is that the unix user is vmail:vmail for every mailbox. This is pretty cool as it allows the whole mail delivery system to be virtualized under Dovecot’s guidance. Slick!

There are two tried-and-tested mailbox formats: Maildir and mbox. mbox is one giant file per mail folder, and can be expensive to search and sort and delete mails out of. Maildir is cheaper to search and sort and delete, but is essentially one file per e-mail, which is bulky. Dovecot has its own high performance mailbox, which is the best of both worlds: an indexed append-only chunked mail format called mdbox. I learned more about the options and trade offs reading [this doc].

Dovecot: LMTP

The following dovecot.conf snippet ties Postfix into Dovecot:

protocols = $protocols lmtp
protocol lmtp {
  mail_plugins = $mail_plugins sieve
}

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0660
    user = postfix
    group = postfix
  }
}

Recall that in Postfix above, the virtual_transport field specified the same location. This is how user pim gets mail handed to Dovecot. One other tidbit here is that the LMTP protocol enables a plugin called sieve. What this does, is upon receipt of each e-mail, a list of filters is run through, on the server side! It is here that I can tell Dovecot that some mail goes to different folders and sub-folders, some might be forwarded, marked read or discarded entirely. I’ll get to that in a minute.

Dovecot: IMAP

Then, I enable SSL enabled IMAP in dovecot.conf:

disable_plaintext_auth = yes
auth_mechanisms = plain login

protocols = $protocols imap
protocol imap {
  mail_max_userip_connections = 50
  mail_plugins = $mail_plugins imap_sieve
}

service imap-login {
  inet_listener imap {
    port = 0  ## Disabled
  }
  inet_listener imaps {
    port = 993
  }
}

With this snippet, I instruct Dovecot to disable any plain-text authentication, and use either plain or login challenges to authenticate users. I’ll disable the un-encrypted IMAP listener by setting its port to 0, and I’ll allow for an IMAP+SSL listener on the common port 993, which will be presenting a *.ipng.ch wildcard certificate that’s shared between all sorts of services at IPng.

Dovecot: Replication

And now for something really magical. Dovecot can be instructed to replicate in multi-master (ie. read/write) mailboxes to remote machines also running Dovecot. This is called dsync and it’s hella cool! In reading the [docs], I take note that the same user should be directed to a stable replica in normal use, but changes do not get lost even if the same user modifies mails simultaneously on both replicas, some mails just might have to be redownloaded in that case. The replication is done by looking at Dovecot index files (not what exists in filesystem), so no mails get lost due to filesystem corruption or an accidental rm -rf, they will simply be replicated back.

This is amazing!! The configuration for it is remarkably straight forward:

mail_plugins = $mail_plugins notify replication

# Replication details
replication_max_conns = 10
replication_full_sync_interval = 1h

service aggregator {
  fifo_listener replication-notify-fifo {
    user  = vmail
    group = vmail
    mode  = 0660
  }
  unix_listener replication-notify {
    user  = vmail
    group = vmail
    mode  = 0660
  }
}

# Enable doveadm replicator commands
service replicator {
  process_min_avail = 1
  unix_listener replicator-doveadm {
    mode  = 0660
    user  = vmail
    group = vmail
  }
}

doveadm_port = 63301
doveadm_password = <some password here>
service doveadm {
  vsz_limit=512 MB
  inet_listener {
    port = 63301
  }
}

plugin {
  mail_replica = tcp:maildrop0.ddln0.net.ipng.ch
}

To try to explain this - The first service, the aggregator opens some notification FIFOs that will notify listeners of new replication events. Then, Dovecot will start a process called replicator, which gets these cues when there is work to be done. It will connect to a mail_replica on another host, on the doveadm_port (in my case 63301) which is protected by a shared password. And with that, all e-mail that is delivered via LMTP on this machine, is both retrievable via IMAPS but also gets copied to the remote machine maildrop0.ddln0.net.ipng.ch (and in its configuration, it’ll synchronize mail to maildrop0.chbtl0.net.ipng.ch). Nice!

Dovecot: Sieve

Having a flat mailbox is just no fun (unless you’re using GMail, in which case: tolerable). Enter Sieve, described in [RFC5228]. Scripts written in Sieve are executed during final delivery, when the message is moved to the user-accessible mailbox. In systems where the Mail Transfer Agent (MTA) does final delivery, such as traditional Unix mail, it is reasonable to filter when the MTA deposits mail into the user’s mailbox.

protocols = $protocols sieve

plugin {
  sieve = ~/.dovecot.sieve
  sieve_global_path = /etc/dovecot/sieve/default.sieve
  sieve_dir = ~/sieve
  sieve_global_dir = /etc/dovecot/sieve/
  sieve_extensions = +editheader
  sieve_before = /etc/dovecot/sieve/before.d
  sieve_after  = /etc/dovecot/sieve/after.d
}

plugin {
  sieve_plugins = sieve_imapsieve sieve_extprograms

  # From elsewhere to Junk folder
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_before = file:/etc/dovecot/sieve/report-spam.sieve

  # From Junk folder to elsewhere
  imapsieve_mailbox2_name = *
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_before = file:/etc/dovecot/sieve/report-ham.sieve

  sieve_pipe_bin_dir = /etc/dovecot/sieve

  sieve_global_extensions = +vnd.dovecot.pipe
}

This is a mouthful, but only because it’s hella cool. By default, each mailbox will have a .dovecot.sieve file that is consulted at each delivery. If no file exists there, the default sieve will be used. But also, some sieve filters might happen either sieve_before the users’ one is called, or sieve_after. Then, in the plugin I create two specific triggers:

1. if a file is copied to the Junk folder, I will run it through a script called report-spam.sieve.
2. similarly, if it is moved out of the Junk folder, I’ll run the report-ham.sieve script.

Using an rspamc client, I can wheel over the cluster of Rspamd servers one by one and offer them these two events (the train-spam and train-ham are similar, so I’ll only show one):

pim@maildrop0-chbtl0:~$ cat /etc/dovecot/sieve/report-spam.sieve
require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.email" "*" {
  set "email" "${1}";
}

pipe :copy "train-spam.sh" [ "${email}" ];
pim@maildrop0-chbtl0:~$ cat /etc/dovecot/sieve/train-spam.sh
logger learning spam
/usr/bin/rspamc -h rspamd.net.ipng.ch:11332 learn_spam

Users of Dovecot can now add their Sieve configs to their mailbox:

pim@maildrop0-chbtl0:~$ sudo ls -la /var/dovecot/users/pim/
lrwxrwxrwx 1 vmail vmail   19 Apr  2 17:27 .dovecot.sieve -> sieve/ipng_v1.sieve
-rw------- 1 vmail vmail 2113 Mar 29 11:35 .dovecot.sieve.log
-rw------- 1 vmail vmail 5001 May 14 14:34 .dovecot.svbin
drwx------ 4 vmail vmail 4096 May 17 16:02 mdbox
drwx------ 3 vmail vmail 4096 May 14 14:30 sieve

but seeing as (a) it’s tedious to have to edit these files on multiple dovecot replicas, and (b) my users will not receive access to vmail user in order to actually do that, as it would be a security risk, I need one more thing.

Dovecot: IMAP Sieve

Dovecot has an implementation of a replication-aware Sieve filter editor called managesieve:

service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}

service managesieve {
  process_limit = 256
}

protocol sieve {
}

It will use the IMAP credentials to allow users to edit their Sieve filter online. For example, Thunderbird has a plugin for it, which does syntax checking and what-not. When the filter is edited, it is syntax checked, compiled and replicated to the other Dovecot instance.

NGINX

I have an imap server and a mangesieve server, redundantly running on two Dovecot machines. I recall reading in the Dovecot manual that it is slightly preferable to have users go to a consistent replica and not bounce around between them. Luckily, I can do exactly that using the NGINX frontends:

upstream imap {
  server maildrop0.chbtl0.net.ipng.ch:993 fail_timeout=10s max_fails=2;
  server maildrop0.ddln0.net.ipng.ch:993 fail_timeout=10s max_fails=2 backup;
}

server {
  listen [::]:993;
  listen 0.0.0.0:993;
  proxy_pass imap;
}

upstream sieve {
  server maildrop0.chbtl0.net.ipng.ch:4190 fail_timeout=10s max_fails=2;
  server maildrop0.ddln0.net.ipng.ch:4190 fail_timeout=10s max_fails=2 backup;
}

server {
  listen [::]:4190;
  listen 0.0.0.0:4190;
  proxy_pass sieve;
}

I keep port 993 for maildrop as well as port 587 for smtp-out unfiltered on the NGINX cluster. I’m a little bit more protective of the managesieve service, so port 4190 is allowed only when users are connected to the VPN or the internal (office/home) network.

Now, you’ll recall that in the smtp-in servers, I forward mail to pim@maildrop.net.ipng.ch, for which the redundant Dovecot servers are both accepting mail. On the way in, I can see to it that the primary replica is used , by giving it a slightly lower preference in DNS MX records:

maildrop.net.ipng.ch.	300 IN	MX	10 maildrop0.chbtl0.net.ipng.ch.
maildrop.net.ipng.ch.	300 IN	MX	20 maildrop0.ddln0.net.ipng.ch.

imap.ipng.ch.           60  IN	CNAME   nginx0.ipng.ch.
nginx0.ipng.ch.         600 IN  A       194.1.163.151
nginx0.ipng.ch.         600 IN  A       46.20.246.124
nginx0.ipng.ch.         600 IN  A       94.142.241.189
nginx0.ipng.ch.         600 IN  AAAA    2001:678:d78:7::151
nginx0.ipng.ch.         600 IN  AAAA    2a02:2528:ff00::124
nginx0.ipng.ch.         600 IN  AAAA    2a02:898:146::5

This will make the smtp-in hosts prefer to use the chbtl0 maildrop replica when it’s available. If ever it were to go down, they will automatically fail over and use ddln0, which will replicate back any changes while chbtl0 is down for maintenance or hardware failure. On the way to out, the nginx cluster will prefer to use chbtl0 as well, as it has marked the ddln0 replica as backup.

4. Webmail: Roundcube

Now that I have all of the infrastructure up and running, I thought I’d put some icing on the cake with Roundcube, a web-based IMAP email client. Roundcube’s most prominent feature is the pervasive use of Ajax technology. It also comes with an online Sieve editor, and runs in Docker. What more can I ask for?

Installing it is really really easy in my case. Since I have an nginx cluster to frontend it and do the SSL offloading, I choose the simplest version with the following docker-compose.yaml:

version: '2'

services:
  roundcubemail:
    image: roundcube/roundcubemail:latest
    container_name: roundcubemail
    volumes:
      - ./www:/var/www/html
      - ./db/sqlite:/var/roundcube/db
    ports:
      - 9002:80
    environment:
      - ROUNDCUBEMAIL_DB_TYPE=sqlite
      - ROUNDCUBEMAIL_SKIN=elastic
      - ROUNDCUBEMAIL_DEFAULT_HOST=ssl://maildrop0.net.ipng.ch
      - ROUNDCUBEMAIL_DEFAULT_PORT=993
      - ROUNDCUBEMAIL_SMTP_SERVER=tls://smtp-out.net.ipng.ch
      - ROUNDCUBEMAIL_SMTP_PORT=587

There’s a small snag, in that by default the SMTP user and password are expected to be the same as for the IMAP server, which is not the case for my design. So, I create a user roundcube on the smtp-out cluster and give it a suitable password. I nose around a little bit, and decide my preference is to have threaded view by default, and I also enable the managesieve plugin:

    $config['log_driver'] = 'stdout';
    $config['zipdownload_selection'] = true;
    $config['des_key'] = '<this key of sorts>';
    $config['enable_spellcheck'] = true;
    $config['spellcheck_engine'] = 'pspell';
    $config['smtp_user'] = 'roundcube';
    $config['smtp_pass'] = '<something or other>';
    $config['plugins'] = array('managesieve');
    $config['managesieve_host'] = 'tls://maildrop0.net.ipng.ch:4190';
    $config['default_list_mode'] = 'threads';

I start the docker containers, and very quickly after, Roundcube shoots to life. I can expose it behind the nginx cluster, while keeping it accessible only for VPN + office/home network users:

server {
  listen [::]:80;
  listen 0.0.0.0:80;

  server_name webmail.ipng.ch webmail.net.ipng.ch webmail;
  access_log /var/log/nginx/webmail.ipng.ch-access.log;
  include /etc/nginx/conf.d/ipng-headers.inc;

  location / {
    return 301 https://webmail.ipng.ch$request_uri;
  }
}

geo $allowed_user {
 default 0;
 include /etc/nginx/conf.d/geo-ipng.inc;
}

server {
  listen [::]:443 ssl http2;
  listen 0.0.0.0:443 ssl http2;
  ssl_certificate /etc/certs/ipng.ch/fullchain.pem;
  ssl_certificate_key /etc/certs/ipng.ch/privkey.pem;
  include /etc/nginx/conf.d/options-ssl-nginx.inc;
  ssl_dhparam /etc/nginx/conf.d/ssl-dhparams.inc;

  server_name webmail.ipng.ch;
  access_log /var/log/nginx/webmail.ipng.ch-access.log upstream;
  include /etc/nginx/conf.d/ipng-headers.inc;

  if ($allowed_user = 0) { rewrite ^ https://ipng.ch/ break; }

  location / {
    proxy_pass http://docker0.frggh0.net.ipng.ch:9002;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

The configuration has one neat trick in it – it uses the geo module in NGINX to assert that the client address is used to set the value of allowed_user. It will be 1 if the client connected from any network defined in the geo-ipng.inc file, and 0 otherwise. I then use it to bounce unwanted visitors back to the main [website], and expose Roundcube for those that are welcome.

While the Roundcube instance is not replicated, it’s also non-essential. I will be using Thunderbird, Mail.app and other clients more regularly than Roundcube. It may just be handy in a pinch to either check mail using a browser, but also to edit the Sieve filters easily.

In my defense, considering roundcube is pretty much stateless, I can actually just run multiple copies of it on a few docker hosts at IPng – then in the nginx configs I might use a similar construct as for the maildrop and smtp-out services, with a primary and hot standby. But that will be for that one day that the docker host in Lille dies AND I decided I absolutely require Roundcube precisely on that day :)