Case Study: IPng's mail servers
Intro
I have seen companies achieve great successes in the space of consumer internet and entertainment industry. I’ve been feeling less enthusiastic about the stronghold that these corporations have over my digital presence. I am the first to admit that using “free” services is convenient, but these companies are sometimes taking away my autonomy and exerting control over society. To each their own of course, but for the last few years, I’ve been more and more inclined to take back a little bit of responsibility for my online social presence, away from centrally hosted services and to privately operated ones.
First off - I love Google’s Workspace products. I started using GMail just after it launched, back in 2004. Its user interface is sleek, performant, and very intuitive. Its filtering, granted, could be a bit less … robotic, but that’s made up by labels and an incredibly comprehensive search function. I would dare say that between GMail and Photos, those are my absolute favorite products on the internet.
That said, I have been running e-mail servers since well before Google existed as a company. I started off at M.C.G.V. Stack, the computer club of the University of Eindhoven, in 1995. We ran sendmail back then, and until about two months ago, I have continuously run sendmail in production using the PaPHosting platform [ref] that I wrote with my buddies Paul and Jeroen.
However, two things happened, both of them somewhat nerdsnipe-esque:
- Mrs IPngNetworks said “Well if you are going to use NextCloud and PeerTube and PixelFed and Mastodon, why would you not run your own mailserver?”
- I added a forward for
event@frys-ix.neton my Sendmail relays at PaPHosting, and was tipped by my buddy Jelle that his e-mail to it was bouncing due to SPF strictness.
I tried to resist …
My main argument against running a mailserver has been the mailspool. Before I moved to GMail, I had
the misfortune of having my mail and primary DNS fail, running on bfib.ipng.nl at the time, a
server 700km away from me, without redundancy. Even the nameserver slaves went beyond their zone
refresh. It was not a good month for me, even though it was twenty years or so ago :)
Last year, during a roadtrip with Fred, he and I spent a few long hours restoring a backup after a catastrophic failure of a hypervisor at IP-Max on which his mailserver was running. Luckily, backups were awesome and saved the day, but having to go into a Red Alert mode and not being able to communicate, really can be stressful. I don’t want to run mailservers!!!1
.. but resistance is futile
After this nerdsnipe, I had a short conversation with Jeroen who mentioned that since I last had a look at this, Dovecot, a popular imap/pop3 server, had gained the ability to do mailbox synchronization across multiple machines. That’s a really nifty feature - but also it meant that there will be no more single points of failure, if I do this properly. Oh crap, there’s no longer an argument of resistance? Nerd-snipe accepted!
Let me first introduce the mail^W main characters of my story:
 |
 |
 |
 |
 |
 |
- Postfix: is Wietse Venema’s mail server that started life at IBM research as an alternative to the widely-used Sendmail program. After eight years at Google, Wietse continues to maintain Postfix.
- Dovecot: an open source IMAP and POP3 email server for Linux/UNIX-like systems, written with security primarily in mind. Dovecot is an excellent choice for both small and large installations.
- NGINX: an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server, originally written by Igor Sysoev.
- Rspamd: an advanced spam filtering system and email processing framework that allows evaluation of messages by a number of rules including regular expressions, statistical analysis and custom services such as URL black lists.
- OpenDKIM: is a community effort to develop and maintain a C library for producing DKIM-aware applications and an open source milter for providing DKIM service.
- Unbound: a validating, recursive, caching DNS resolver. It is designed to be fast and lean and incorporates modern features based on open standards.
- Roundcube: a web-based IMAP email client. Roundcube’s most prominent feature is the pervasive use of Ajax technology.
In the rest of this article, I’ll go over four main parts that I used to build a fully redundant and self-healing mail service at IPng Networks:
- Green:
smtp-in.ipng.chwhich handles inbound e-mail - Red:
imap.ipng.chwhich serves mailboxes to users - Blue:
smtp-out.ipng.chwhich handles outbound e-mail - Magenta:
webmail.ipng.chwhich exposes the mail in a web browser
Let me start with a functional diagram, using those colors:
As you can see in this diagram, I will be separating concerns and splitting the design into three discrete parts, which will also be in three sets of redundantly configured backend servers running on IPng’s hypervisors in Zurich (CH), Lille (FR) and Amsterdam (NL).
1. Outbound: smtp-out
I’m going to start with a relatively simple component first: outbound mail. This service will be listening on the smtp submission port 587, require TLS and user authentication from clients, validate outbound e-mail using a spam detection agent, and finally provide DKIM signing on all outbound e-mails. It should spool and retry the delivery in case there is a temporary issue (like greylisting, or server failure) on the receiving side.
Because the only way to send e-mail will be using TLS and user authentication, the smtp-out servers themselves will not need to do any DNSBL lookups, which is convenient because it means I can put them behind a loadbalancer and serve them entirely within IPng Site Local. If you’re curious as to what this site local thing means, basically it’s an internal network spanning all IPng’s points of presence, with an IPv4, IPv6 and MPLS backbone that is disconnected from the internet. For more details on the design goals, take a look at the [article] I wrote about it last year.
Debian VMs
I’ll take three identical virtual machines, hosted on three separate hypervisors each in their own country.
pim@summer:~$ dig ANY smtp-out.net.ipng.ch
smtp-out.net.ipng.ch. 60 IN A 198.19.6.73
smtp-out.net.ipng.ch. 60 IN A 198.19.4.230
smtp-out.net.ipng.ch. 60 IN A 198.19.6.135
smtp-out.net.ipng.ch. 60 IN AAAA 2001:678:d78:50e::9
smtp-out.net.ipng.ch. 60 IN AAAA 2001:678:d78:50a::6
smtp-out.net.ipng.ch. 60 IN AAAA 2001:678:d78:510::7
I will give them each 8GB of memory, 4 vCPUs, and 16GB of bootdisk. I’m pretty confident that the whole system will be running in only a fraction of that. I will install a standard issue Debian Bookworm (12.5), and while my VMs by default have 4 virtual NICs, I only need one, connected to the IPng Site Local:
pim@smtp-out-chrma0:~$ ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
enp1s0f0 UP 198.19.6.135/27 2001:678:d78:510::7/64 fe80::5054:ff:fe99:81b5/64
enp1s0f1 UP fe80::5054:ff:fe99:81b6/64
enp1s0f2 UP fe80::5054:ff:fe99:81b7/64
enp1s0f3 UP fe80::5054:ff:fe99:81b8/64
pim@smtp-out-chrma0:~$ mtr -6 -c5 -r dns.google
Start: 2024-05-17T13:49:28+0200
HOST: smtp-out-chrma0 Loss% Snt Last Avg Best Wrst StDev
1.|-- msw1.chrma0.net.ipng.ch 0.0% 5 1.6 1.5 1.3 1.6 0.1
2.|-- msw0.chrma0.net.ipng.ch 0.0% 5 1.4 1.3 1.3 1.4 0.1
3.|-- msw0.chbtl0.net.ipng.ch 0.0% 5 3.2 3.1 2.8 3.2 0.2
4.|-- hvn0.chbtl0.net.ipng.ch 0.0% 5 1.5 1.5 1.4 1.5 0.0
5.|-- chbtl0.ipng.ch 0.0% 5 1.6 1.7 1.6 1.7 0.0
6.|-- chrma0.ipng.ch 0.0% 5 2.4 2.4 2.4 2.5 0.0
7.|-- as15169.lup.swissix.ch 0.0% 5 3.2 3.8 3.2 5.6 1.0
8.|-- 2001:4860:0:1::6083 0.0% 5 4.5 4.5 4.5 4.5 0.0
9.|-- 2001:4860:0:1::12f9 0.0% 5 3.4 3.5 3.4 3.5 0.0
10.|-- dns.google 0.0% 5 3.8 3.9 3.8 4.0 0.1
One cool observation: these machines are not really connected to the internet - you’ll note that their IPv4 address is from reserved space, and their IPv6 supernet (2001:678:d78:500::/56) is filtered at the border. I’ll get to that later!
Postfix
I will install Postfix, and make a few adjustments to its config. First off, this mailserver will only
be receiving submission mail, which is port 587. It will not participate or listen to the regular
smtp port 25, nor smtps port 465, as such the master.cf file for Postfix becomes:
#smtp inet n - y - - smtpd
# -o smtpd_sasl_auth_enable=no
submission inet n - y - - smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_reject_unlisted_recipient=no
-o smtpd_client_restrictions=permit_sasl_authenticated,permit_mynetworks,reject
-o milter_macro_daemon_name=ORIGINATING
#smtps inet n - y - - smtpd
# -o syslog_name=postfix/smtps
# -o smtpd_tls_wrappermode=yes
# -o smtpd_sasl_auth_enable=yes
The only thing I will make note of is that the submission service has a set of client
restrictions. In other words, to be able to use this service, the client must either be SASL
authenticated, or from a list of network prefixes that are allowed to relay. If neither of those two
conditions are satisfied, relaying will be denied.
Now, I understand that pasting in the entire postfix configuration is a bit verbose, but honestly
I’ve spent many an hour trying to puzzle together an end-to-end valid configuration, so I’m just
going to swim upstream and post the whole main.cf, which I’ll try to annotate the broad strokes
of, in case there’s anybody out there trying to learn:
myhostname = smtp-out.ipng.ch
myorigin = smtp-out.ipng.ch
mydestination = $myhostname, smtp-out.chrma0.net.ipng.ch, localhost.net.ipng.ch, localhost
mynetworks = 127.0.0.0/8, [::1]/128
recipient_delimiter = +
inet_interfaces = all
inet_protocols = all
biff = no
# appending .domain is the MUA's job.
append_dot_mydomain = no
readme_directory = no
# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 3.6 on fresh installs.
compatibility_level = 3.6
# SMTP Server
smtpd_banner = $myhostname ESMTP $mail_name (smtp-out.chrma0.net.ipng.ch)
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
smtpd_tls_cert_file = /etc/certs/ipng.ch/fullchain.pem
smtpd_tls_key_file = /etc/certs/ipng.ch/privkey.pem
smtpd_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtpd_tls_CApath = /etc/ssl/certs
smtpd_use_tls = yes
smtpd_tls_received_header = yes
smtpd_tls_auth_only = yes
smtpd_tls_session_cache_database = btree:$data_directory/smtpd_scache
smtpd_client_connection_count_limit = 4
smtpd_client_connection_rate_limit = 10
smtpd_client_message_rate_limit = 60
smtpd_client_event_limit_exceptions = $mynetworks
# Dovecot auth
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_authenticated_header = yes
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous, noplaintext
smtpd_sasl_tls_security_options = noanonymous
# SMTP Client
smtp_use_tls = yes
smtp_tls_note_starttls_offer = yes
smtp_tls_cert_file = /etc/certs/ipng.ch/fullchain.pem
smtp_tls_key_file = /etc/certs/ipng.ch/privkey.pem
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_tls_CApath = /etc/ssl/certs
smtp_tls_mandatory_ciphers = medium
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_security_level = encrypt
header_size_limit = 4096000
message_size_limit = 52428800
mailbox_size_limit = 0
# OpenDKIM, Rspamd
smtpd_milters = inet:localhost:8891,inet:rspamd.net.ipng.ch:11332
non_smtpd_milters = $smtpd_milters
# Local aliases
alias_maps = hash:/etc/postfix/aliases
alias_database = hash:/etc/postfix/aliases
Hostnames: The full (internal) hostname for the server is smtp-out.$(site).net.ipng.ch, in
this case for chrma0 in Rümlang, Switzerland. However, when clients connect to the public hostname
smtp-out.ipng.ch, they will expect that the TLS certificate matches that hostname. This is why I
let the server present itself as simply smtp-out.ipng.ch, which will also be its public DNS name
later, but put the internal FQDN for debugging purposes between parenthesis. See the smtpd_banner
and myhostname for the destinction. I’ll load up the *.ipng.ch wildcard certificate which I
described in my Let’s Encrypt [DNS-01] article.
Authorization: I will make Postfix accept relaying for those users that are either in the
mynetworks (which is only localhost) OR sasl_authenticated (ie. presenting a username and
password). This password exchange will only be possible after encryption has been triggered using
the STARTTLS SMTP feature. This way, user/pass combos will be safe on the network.
Authentication: Those username and password combos can come from a few places. One popular way
to do this is via a dovecot authentication service. Via the smtpd_sasl_path, I tell Postfix to
ask these authentication questions using the dovecot protocol on a certain file path. I’ll let
Dovecot listen in the /var/spool/postfix/private/auth directory. This is how Postfix will know
which user to relay for, and which to deny.
DKIM/SPF: These days, most (large and small) mail providers will be suspicious of e-mail that is delivered to them without proper SPF and DKIM fields. DKIM is a mechanism to create a cryptographic signature over some of the E-Mail header fields (usually From/Subject/Date), which can be checked by the recipient for validity. SPF is a mechanism to use DNS to inform receiving mailservers of which are the valid IPv4/IPv6 addresses that should be used to deliver mail for a given sender domain.
Dovecot (auth)
The configuration for Dovecot is incredibly simple. The only thing I do is create a mostly empty
dovecot.conf file which defines the auth service listening in the place where Postfix expects
it. Then, I add a password file called sasl-users which will contain user:password tuples:
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0660
# Assuming the default Postfix user and group
user = postfix
group = postfix
}
}
passdb {
driver = passwd-file
args = username_format=%n /etc/dovecot/sasl-users
}
I can use doveadm pw to generate such passwords. I do this in an upstream Ansible repository and
then push out the same configuration to any number of smtp-out servers, so they are all configured
identically to this one.
OpenDKIM (signing)
Now that I can authorize (via SASL) and authenticate (via Dovecot backend) a user, it will be
entitled to use the smtp-out Postfix to send e-mail. However, there’s a good chance that
recipients will bounce the e-mail, unless it comes with a DKIM signature, and from the correct IP
addresses.
To configure DKIM signing, I use OpenDKIM, which I give the following /etc/opendkim.conf file:
pim@smtp-out-chrma0:~$ cat /etc/opendkim.conf
Syslog yes
LogWhy yes
UMask 007
Mode sv
AlwaysAddARHeader yes
SignatureAlgorithm rsa-sha256
X-Header no
KeyTable refile:/etc/opendkim/keytable
SigningTable refile:/etc/opendkim/signers
RequireSafeKeys false
Canonicalization relaxed
TrustAnchorFile /usr/share/dns/root.key
UserID opendkim
PidFile /run/opendkim/opendkim.pid
Socket inet6:8891
It opens a socket at port 8891, which is where Postfix expects it, based on its smtpd_milter
configuration option. It will look at the so-called SigningTable to determine which outbound
e-mail addresses it can sign. This table looks up From addresses, including wildcards, and informs
which symbolic keyname in the KeyTable to use for the signature, like so:
pim@smtp-out-chrma0:/etc/opendkim$ cat signers
*@*.ipng.nl ipng-nl
*@ipng.nl ipng-nl
*@*.ipng.ch ipng-ch
*@ipng.ch ipng-ch
*@*.ublog.tech ublog
*@ublog.tech ublog
...
pim@smtp-out-chrma0:/etc/opendkim$ cat keytable
ipng-nl ipng.nl:DKIM2022:/etc/opendkim/keys/DKIM2022-ipng.nl-private
ipng-ch ipng.ch:DKIM2022:/etc/opendkim/keys/DKIM2022-ipng.ch-private
ublog ublog.tech:DKIM2022:/etc/opendkim/keys/DKIM2022-ublog.tech-private
...
This allows OpenDKIM to sign messages for any number of domains, using the correct key. Slick!
NGINX
Now that I have three of these identical VMs, I am ready to hook them up to the internet. On the way
in, I will point smtp-out.ipng.ch to our NGINX cluster. I wrote about that cluster in a [previous
article]. I will add a snippet there, that exposes these
VMs behind a TCP loadbalancer like so:
pim@squanchy:~/src/ipng-ansible/roles/nginx/files/streams-available$ cat smtp-out.ipng.ch.conf
upstream smtp_out {
server smtp-out.chrma0.net.ipng.ch:587 fail_timeout=10s max_fails=2;
server smtp-out.frggh0.net.ipng.ch:587 fail_timeout=10s max_fails=2 backup;
server smtp-out.nlams2.net.ipng.ch:587 fail_timeout=10s max_fails=2 backup;
}
server {
listen [::]:587;
listen 0.0.0.0:587;
proxy_pass smtp_out;
}
I make use of the backup keyword, which will make the loadbalancer choose, if it’s available, the
primary server in chrma0. If it were to go down, no problem, two connection failures within ten
seconds will make NGINX choose the alternative ones in frggh0 or nlams2.
IPng Site Local gateway
When the smtp-out server receives the e-mail from the customer/client, it’ll spool it and start to
deliver it to the remote MX record. To do this, it’ll create an outbound connection from its cozy
spot within IPng Site Local (which, you will remember, is not connected directly to the internet).
There are three redundant gateways in IPng Site Local (in Geneva, Brüttisellen and Amsterdam).
If any of these were to go down for maintenance or fail, the network will use OSPF E1 to find the
next closest default gateway. I wrote about how this entire european network is connected via three
gateways that are self-repairing in this [article], in case
you’re curious.
But, for the purposes of SMTP, it means that each of the internal smtp-out VMs will be seen by
remote mailservers as NATted via one of these egress points. This allows me to determine the
SPF records in DNS. With that, I’m ready to share the publicly visible details for this service:
_spf.ipng.ch. 3600 IN TXT "v=spf1 include:_spf4.ipng.ch include:_spf6.ipng.ch ~all"
_spf4.ipng.ch. 3600 IN TXT "v=spf1 ip4:46.20.246.112/28 ip4:46.20.243.176/28 ip4:94.142.245.80/29"
"ip4:94.142.241.184/29 ip4:194.1.163.0/24 ~all"
_spf6.ipng.ch. 3600 IN TXT "v=spf1 ip6:2a02:2528:ff00::/40 ip6:2a02:898:146::/48"
"ip6:2001:678:d78::/48 ~all"
smtp-out.ipng.ch. 3600 IN CNAME nginx0.ipng.ch.
nginx0.ipng.ch. 600 IN A 194.1.163.151
nginx0.ipng.ch. 600 IN A 46.20.246.124
nginx0.ipng.ch. 600 IN A 94.142.241.189
nginx0.ipng.ch. 600 IN AAAA 2001:678:d78:7::151
nginx0.ipng.ch. 600 IN AAAA 2a02:2528:ff00::124
nginx0.ipng.ch. 600 IN AAAA 2a02:898:146::5
To re-iterate one point: the inbound path of the mail is via the redundant cluster of nginx0
entrypoints, while the outbound path will be seen from gw0.chbtl0.ipng.ch, gw0.chplo0.ipng.ch
or gw0.nlams3.ipng.ch, which are all covered by the SPF records for IPv4 and IPv6.
Bonus: opensmtpd on clients
By the way, every single server (VM, hypervisor, router) at IPng Neworks will all use smtp-out to
send e-mail. I use opensmtpd for that, and it’s incredibly simple:
pim@squanchy:~$ cat /etc/mail/smtpd.conf
table aliases file:/etc/mail/aliases
table secrets file:/etc/mail/secrets
listen on localhost
action "local_mail" mbox alias <aliases>
action "outbound" relay host "smtp+tls://ipng@smtp-out.ipng.ch:587" auth <secrets> mail-from "@ipng.ch"
match from local for local action "local_mail"
match from local for any action "outbound"
pim@squanchy:~$ sudo cat /etc/mail/secrets
ipng bastion:<haha-made-you-look>
What happens here is, every time this server squanchy wants to send an e-mail, it will use an SMTP
session with TLS, on port 587, of the machine called smtp-out.ipng.ch, and it’ll authenticate
using the opensmtpd realm called ipng, which maps to a username:password tuple in the secrets
file. It will also rewrite the envelope to be always from @ipng.ch. As a best practice I organize
my SMTP users by Ansible group. Squanchy is in the group bastion, hence its username. By doing it
this way, I can make use of the DKIM and SPF, which makes all mails properly formatted, routed,
signed and delivered. I love it, so much!
2. Inbound: smtp-in
The smtp-out service I described in the previous section is completely standalone. That is to say,
its purpose is only to receive submitted mail from humans and servers, sign it, spool it if need be,
and deliver it. But users also want to deliver e-mail to me and my customers. For this, I’ll build a
second cluster of redundant inbound mailservers: smtp-in.
Here, the base setup is not too different from above, so I won’t repeat it. I’ll take three
identical VMs, in three different datacenters, and install them with Debian and Postfix as well.
But, contrary to the outbound servers, here I will make them listen to the smtp port 25 and the
smtps port 465, and I’ll turn off the ability to authenticate with SASL (and thereby, refuse to
forward any e-mail that I’m not the MX record for), making master.cf look like this:
smtp inet n - y - - smtpd
-o smtpd_sasl_auth_enable=no
#submission inet n - y - - smtpd
# -o syslog_name=postfix/submission
# -o smtpd_tls_security_level=encrypt
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_reject_unlisted_recipient=no
# -o smtpd_client_restrictions=permit_sasl_authenticated,permit_mynetworks,reject
# -o milter_macro_daemon_name=ORIGINATING
smtps inet n - y - - smtpd
-o syslog_name=postfix/smtps
-o smtpd_tls_wrappermode=yes
-o smtpd_sasl_auth_enable=no
Many of the main.cf attributes are the same, unsurprisingly the myhostname configuration option
is set to smtp-in.ipng.ch, which is going to be expected to match the wildcard SSL certificate
from the smtpd_tls_cert_file config option. The banner is a bit more telling, as it shows also the
FQDN hostname (eg. smtp-in.frggh0.net.ipng.ch), helpful when debugging.
# Impose DNSBL restrictions at SMTP time
smtpd_recipient_restrictions = permit_mynetworks,
reject_invalid_helo_hostname,
reject_non_fqdn_recipient,
reject_unknown_recipient_domain,
reject_unauth_pipelining,
reject_unauth_destination,
reject_rbl_client zen.spamhaus.org=127.0.0.[2..11],
reject_rhsbl_sender dbl.spamhaus.org=127.0.1.[2..99],
reject_rhsbl_helo dbl.spamhaus.org=127.0.1.[2..99],
reject_rhsbl_reverse_client dbl.spamhaus.org=127.0.1.[2..99],
warn_if_reject reject_rbl_client zen.spamhaus.org=127.255.255.[1..255],
reject_rbl_client dnsbl-1.uceprotect.net,
reject_rbl_client bl.0spam.org=127.0.0.[7..9],
permit
# Milter for rspamd
smtpd_milters = inet:rspamd.net.ipng.ch:11332
milter_default_action = accept
# PostSRSd
sender_canonical_maps = tcp:localhost:10001
sender_canonical_classes = envelope_sender
recipient_canonical_maps = tcp:localhost:10002
recipient_canonical_classes= envelope_recipient,header_recipient
# Virtual domains
virtual_alias_domains = hash:/etc/postfix/virtual-domains
virtual_alias_maps = hash:/etc/postfix/virtual
The config arguably is quite compact. but I will hilight four specific pieces.
DNSBL: When connecting and receiving the envelope (ie. MAIL FROM and RCPT TO in the SMTP
transaction), I’ll ask Postfix to do a bunch of DNS blocklist lookups. Many sender domains, and
infected hosts/networks are mapped in public DNSBL zones, notably
[Spamhaus], [UCEProtect], and
[0Spam] do a great job at identifying malicious and spammy domain-names and
networks. So I’ll tell Postfix to reject folks attempting to connect from these low-reputation
places.
Rspamd: Here’s where I hook up a redundant cluster of rspamd servers. Each e-mail, once accepted, will be routed through this milter, and after thinking about it a little bit, the Rspamd server will either answer:
- greylisted: where Rspamd recommends a tempfail so the remote mailserver comes back after a few minutes after connecting for the first time, many spammers will not do this.
- blocked: if Rspamd finds the e-mail is egregious, it’ll simply recommend a permfail so Postfix immediately rejects it.
- tagged: if Rspamd is iffy about the e-mail, it may insert an
X-Spamheader, so that downstream mail clients like Thunderbird or Mail.app can decide for themselves to consider the e-mail junk or not.
PostSRS: This is a really useful feature which allows Postfix to safely forward an e-mail to another mailhost. Perhaps best explained with an example, notably the aforementioned nerdsnipe from my buddy Jelle:
- Let’s say
jelle@luteijn.emailsends an e-mail toevent@frys-ix.netfor which IPng is the mailhost. - Jelle configured his SPF records to allow mail to come from either
ip4:185.36.229.0/24orip6:2a07:cd40::/29, and if it comes from neither of those, to hard fail the SPF check-all. - My spiffy
smtp-in.ipng.chreceives this e-mail and decides to forward it internally by rewriting it tofoo@eritap.com. - The mailserver for
eritap.comnow sees an e-mail coming From:jelle@luteijn.emailgoing to itsfoo@eritap.com. It does an SPF check and concludes: Yikes! That mailserversmtp-in.ipng.chis NOT authorized to send e-mail on behalf of Jelle, so reject it! A kitten gets hurt, which is obviously unacceptable.
To handle this, PostSRSd detects when such a forward is about to happen, and rewrites the envelope
From: header to be something that smtp-in.ipng.ch might be allowed to deliver mail for: something
in the @ipng.ch domain! Using a secret (shared between the replicas of IPng’s smtp-in cluster),
it can insert a little cryptographic signature as it does this rewrite.
In the example above, the e-mail from jelle@luteijn.email will be rewritten to an envelope such as
SRS0=CCIM=MT=luteijn.email=jelle@ipng.ch and while hideous, it is in the @ipng.ch domain. If
a bounce for this e-mail were to be generated, PostSRSd can also rewrite in reverse, re-assembling
the original envelope From when sending the bounce on to Jelle’s mailserver.
I configure Postfix to do this using the sender and recipient canonical maps. I read these from a server running on localhost port 10001 and 10002 respectively. This is where PostSRSd does its magic.
Oh, what’s that I hear? The telephone is ringing! 1982 called, and it wants to change the title of [RFC821] from SMTP to CMTP (Convoluted Mail Transfer Protocol).
Virtual: With all of that out of the way, I can now receive and forward aliased e-mails. I
won’t be using local mail delivery (to unix users on the local machine), but rather I will forward
the mails for my local users onwards to what is called a redundant maildrop server. So for the
virtualized part of the Postfix config, I have things like this:
pim@smtp-in-chrma0:~$ cat /etc/postfix/virtual-domains
ublog.tech ublog.tech
frys-ix.net frys-ix.net
ipng.nl ipng.nl
ipng.ch ipng.ch
...
pim@smtp-in-chrma0:~$ cat /etc/postfix/virtual
## Virtual domain: ipng.ch
postmaster@ipng.ch pim+postmaster@maildrop.net.ipng.ch
hostmaster@ipng.ch pim+hostmaster@maildrop.net.ipng.ch
abuse@ipng.ch pim+abuse@maildrop.net.ipng.ch
pim@ipng.ch pim+ipng@maildrop.net.ipng.ch
noreply@ipng.ch /dev/null
...
## Virtual domain: ipng.nl
@ipng.nl @ipng.ch
## Virtual domain: frys-ix.net
postmaster@frys-ix.net pim+postmaster@maildrop.net.ipng.ch
hostmaster@frys-ix.net pim+hostmaster@maildrop.net.ipng.ch
abuse@frys-ix.net pim+abuse@maildrop.net.ipng.ch
noc@frys-ix.net pim+frysix@maildrop.net.ipng.ch,noc@eritap.com
pim@frys-ix.net pim+frysix@maildrop.net.ipng.ch
arend@frys-ix.net arend+frysix@eritap.com
event@frys-ix.net someplace@example.com
...
The first file here virtual_alias_domains, simply explains to Postfix which domains it is to
accept e-mail for. This avoids users trying to use it as a relay. If the domain is not listed in
the lefthand side of the table, it’s not welcome here. But then once Postfix knows it’s supposed to
be accepting e-mail for this domain, it will consult the virtual_alias_maps configuration. Here, I
showed three domains, and a few features:
- I can simply forward along
pim@ipng.chtopim+ipng@maildrop.net.ipng.ch. Cool. - I can toss the email by passing it to
/dev/null(useful for things likenoreply@andnobody@) - I can forward it to multiple recipients as well, for example
noc@frys-ix.netgoes to me and Eritap (hoi, Arend!)
When such a forward happens, PostSRSd kicks in, and for that e-mail, the envelope rewrite will
happen such that smtp-in can safely deliver this to even the strictest of SPF users.
Why no NGINX ?
There’s an important technical reason for me not to be able to use an inbound loadbalancer, even though I’d love to frontend port 25 and 465 on IPng’s nginx cluster. I have enabled the use of DNSBL, which implies that Postfix needs to know the remotely connecting IPv4 and IPv6 addresses. While for domain-based blocklists this is not important, for IP based ones like zen.spamhaus.org it is critical. Therefore, I will assign a public IPv4 and IPv6 address to each of the machines in the cluster. They will be used in a round-robin way, and if one of them is down for a while, remote mail servers will automatically and gracefully use another replica.
With that, the public DNS entries:
ublog.tech. 86400 IN MX 10 smtp-in.ipng.ch.
ipng.nl. 86400 IN MX 10 smtp-in.ipng.ch.
ipng.ch. 86400 IN MX 10 smtp-in.ipng.ch.
smtp-in.ipng.ch. 60 IN A 46.20.246.125
smtp-in.ipng.ch. 60 IN A 94.142.245.85
smtp-in.ipng.ch. 60 IN A 194.1.163.141
smtp-in.ipng.ch. 60 IN AAAA 2a02:2528:ff00::125
smtp-in.ipng.ch. 60 IN AAAA 2a02:898:146:1::5
smtp-in.ipng.ch. 60 IN AAAA 2001:678:d78:6::141
3. Dovecot: maildrop
Remember when I said that mail to pim@ipng.ch is forwarded to pim+ipng@maildrop.net.ipng.ch?
Doing this allows me to have replicated, fully redundant, IMAP servers! As it turns out, Dovecot, a
very popular open source pop3/imap server, has the ability to do realtime synchronization between
multiple machines serving the same user.
On these servers, I’ll start with enabling Postfix only using the smtp and smtps transport in
master.cf. The maildrop servers will be entirely within IPng Site Local, and cannot be reached
from the internet directly, just the same as the smtp-out server replicas.
Postfix on the server receives mail from the smtp-in servers as the final destination for an
e-mail. It does this very similar to the smtp-in server pool I described above, with two notable
differences:
- It does not need to do DNSBL lookups or spam analysis – those have already happened upstream
from these maildrop servers by the
smtp-inservers. That’s also why these can be safely tucked away in IPng Site Local. - Their virtual maps point to what is called an LMTP: Local Mail Transport Protocol, where I’ll ask Postfix to pump them into a redundalty replicated Dovecot pair.
# Completely virtual
virtual_alias_maps = hash:/etc/postfix/virtual-maildrop
virtual_mailbox_domains = maildrop.net.ipng.ch
virtual_transport = lmtp:unix:private/dovecot-lmtp
pim@maildrop0-chbtl0:$ cat /etc/postfix/virtual-maildrop
pim@maildrop.net.ipng.ch pim
What I’ve done here is define only one virtual_mailbox_domains entry, for which I look up the
users in the virtual_alias_maps and use a virtual_transport to deliver the enduser (pim) to a
unix domain socket in /var/spool/postfix/private/dovecot-lmtp. Once again, mail servers are super
simple after you’ve spent ten hours reading configuration manuals and RFCs and asked at least three
other people how they did theirs…. Super… Simple!
Dovecot
By default, Dovecot ships with a very elaborate configuration file hierarchy. I decide to replace it with an autogenerated one from Ansible that has fewer includes (namely: none at all). Here’s the features that I want to enable in Dovecot:
- UserDB: To define username, password and mail directory for users.
- LMTP: To be a local recepticle for the Postfix delivery
- IMAP: To serve SSL enabled IMAP to mail clients like Mail.app, Thunderbird, Roundcube, etc.
- Replicator: To replicate mailboxes between pairs of Dovecot servers.
- Sieve: To allow users to create mail filters using the Sieve protocol.
Starting from the easier bits, here’s how I configure the User Database in dovecot.conf:
passdb {
driver = passwd-file
args = username_format=%n /etc/dovecot/maildrop-users
}
userdb {
driver = passwd-file
args = username_format=%n /etc/dovecot/maildrop-users
default_fields = uid=vmail gid=vmail home=/var/dovecot/users/%n
}
mail_plugins = $mail_plugins notify push_notification replication
mail_location = mdbox:~/mdbox
I can add a user pim with an encrypted password from doveadm pw like so:
pim@maildrop0-chbtl0:/etc/dovecot$ sudo cat maildrop-users
...
pim:{CRYPT}$2y$<some encrypted password goes here>::::
Due to the passdb option, this user can authenticate with username and password, and due to the
userdb option, this user receives a mailbox homedir in the specified location. One important
observation is that the unix user is vmail:vmail for every mailbox. This is pretty cool as it
allows the whole mail delivery system to be virtualized under Dovecot’s guidance. Slick!
There are two tried-and-tested mailbox formats: Maildir and mbox. mbox is one giant file per mail folder, and can be expensive to search and sort and delete mails out of. Maildir is cheaper to search and sort and delete, but is essentially one file per e-mail, which is bulky. Dovecot has its own high performance mailbox, which is the best of both worlds: an indexed append-only chunked mail format called mdbox. I learned more about the options and trade offs reading [this doc].
Dovecot: LMTP
The following dovecot.conf snippet ties Postfix into Dovecot:
protocols = $protocols lmtp
protocol lmtp {
mail_plugins = $mail_plugins sieve
}
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
mode = 0660
user = postfix
group = postfix
}
}
Recall that in Postfix above, the virtual_transport field specified the same location. This is how
user pim gets mail handed to Dovecot. One other tidbit here is that the LMTP protocol enables a
plugin called sieve. What this does, is upon receipt of each e-mail, a list of filters is run
through, on the server side! It is here that I can tell Dovecot that some mail goes to different folders
and sub-folders, some might be forwarded, marked read or discarded entirely. I’ll get to that in a
minute.
Dovecot: IMAP
Then, I enable SSL enabled IMAP in dovecot.conf:
disable_plaintext_auth = yes
auth_mechanisms = plain login
protocols = $protocols imap
protocol imap {
mail_max_userip_connections = 50
mail_plugins = $mail_plugins imap_sieve
}
service imap-login {
inet_listener imap {
port = 0 ## Disabled
}
inet_listener imaps {
port = 993
}
}
With this snippet, I instruct Dovecot to disable any plain-text authentication, and use either
plain or login challenges to authenticate users. I’ll disable the un-encrypted IMAP listener
by setting its port to 0, and I’ll allow for an IMAP+SSL listener on the common port 993, which will
be presenting a *.ipng.ch wildcard certificate that’s shared between all sorts of services at
IPng.
Dovecot: Replication
And now for something really magical. Dovecot can be instructed to replicate in multi-master (ie.
read/write) mailboxes to remote machines also running Dovecot. This is called dsync and it’s
hella cool! In reading the [docs], I
take note that the same user should be directed to a stable replica in normal use, but changes do
not get lost even if the same user modifies mails simultaneously on both replicas, some mails just
might have to be redownloaded in that case. The replication is done by looking at Dovecot index
files (not what exists in filesystem), so no mails get lost due to filesystem corruption or an
accidental rm -rf, they will simply be replicated back.
This is amazing!! The configuration for it is remarkably straight forward:
mail_plugins = $mail_plugins notify replication
# Replication details
replication_max_conns = 10
replication_full_sync_interval = 1h
service aggregator {
fifo_listener replication-notify-fifo {
user = vmail
group = vmail
mode = 0660
}
unix_listener replication-notify {
user = vmail
group = vmail
mode = 0660
}
}
# Enable doveadm replicator commands
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0660
user = vmail
group = vmail
}
}
doveadm_port = 63301
doveadm_password = <some password here>
service doveadm {
vsz_limit=512 MB
inet_listener {
port = 63301
}
}
plugin {
mail_replica = tcp:maildrop0.ddln0.net.ipng.ch
}
To try to explain this - The first service, the aggregator opens some notification FIFOs that will
notify listeners of new replication events. Then, Dovecot will start a process called replicator,
which gets these cues when there is work to be done. It will connect to a mail_replica on another
host, on the doveadm_port (in my case 63301) which is protected by a shared password. And with
that, all e-mail that is delivered via LMTP on this machine, is both retrievable via IMAPS
but also gets copied to the remote machine maildrop0.ddln0.net.ipng.ch (and in its
configuration, it’ll synchronize mail to maildrop0.chbtl0.net.ipng.ch). Nice!
Dovecot: Sieve
Having a flat mailbox is just no fun (unless you’re using GMail, in which case: tolerable). Enter Sieve, described in [RFC5228]. Scripts written in Sieve are executed during final delivery, when the message is moved to the user-accessible mailbox. In systems where the Mail Transfer Agent (MTA) does final delivery, such as traditional Unix mail, it is reasonable to filter when the MTA deposits mail into the user’s mailbox.
protocols = $protocols sieve
plugin {
sieve = ~/.dovecot.sieve
sieve_global_path = /etc/dovecot/sieve/default.sieve
sieve_dir = ~/sieve
sieve_global_dir = /etc/dovecot/sieve/
sieve_extensions = +editheader
sieve_before = /etc/dovecot/sieve/before.d
sieve_after = /etc/dovecot/sieve/after.d
}
plugin {
sieve_plugins = sieve_imapsieve sieve_extprograms
# From elsewhere to Junk folder
imapsieve_mailbox1_name = Junk
imapsieve_mailbox1_causes = COPY
imapsieve_mailbox1_before = file:/etc/dovecot/sieve/report-spam.sieve
# From Junk folder to elsewhere
imapsieve_mailbox2_name = *
imapsieve_mailbox2_from = Junk
imapsieve_mailbox2_causes = COPY
imapsieve_mailbox2_before = file:/etc/dovecot/sieve/report-ham.sieve
sieve_pipe_bin_dir = /etc/dovecot/sieve
sieve_global_extensions = +vnd.dovecot.pipe
}
This is a mouthful, but only because it’s hella cool. By default, each mailbox will have a
.dovecot.sieve file that is consulted at each delivery. If no file exists there, the default sieve
will be used. But also, some sieve filters might happen either sieve_before the users’ one is
called, or sieve_after. Then, in the plugin I create two specific triggers:
- if a file is copied to the Junk folder, I will run it through a script called
report-spam.sieve. - similarly, if it is moved out of the Junk folder, I’ll run the
report-ham.sievescript.
Using an rspamc client, I can wheel over the cluster of Rspamd servers one by one and offer them
these two events (the train-spam and train-ham are similar, so I’ll only show one):
pim@maildrop0-chbtl0:~$ cat /etc/dovecot/sieve/report-spam.sieve
require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];
if environment :matches "imap.email" "*" {
set "email" "${1}";
}
pipe :copy "train-spam.sh" [ "${email}" ];
pim@maildrop0-chbtl0:~$ cat /etc/dovecot/sieve/train-spam.sh
logger learning spam
/usr/bin/rspamc -h rspamd.net.ipng.ch:11332 learn_spam
Users of Dovecot can now add their Sieve configs to their mailbox:
pim@maildrop0-chbtl0:~$ sudo ls -la /var/dovecot/users/pim/
lrwxrwxrwx 1 vmail vmail 19 Apr 2 17:27 .dovecot.sieve -> sieve/ipng_v1.sieve
-rw------- 1 vmail vmail 2113 Mar 29 11:35 .dovecot.sieve.log
-rw------- 1 vmail vmail 5001 May 14 14:34 .dovecot.svbin
drwx------ 4 vmail vmail 4096 May 17 16:02 mdbox
drwx------ 3 vmail vmail 4096 May 14 14:30 sieve
but seeing as (a) it’s tedious to have to edit these files on multiple dovecot replicas, and (b) my
users will not receive access to vmail user in order to actually do that, as it would be a
security risk, I need one more thing.
Dovecot: IMAP Sieve
Dovecot has an implementation of a replication-aware Sieve filter editor called managesieve:
service managesieve-login {
inet_listener sieve {
port = 4190
}
}
service managesieve {
process_limit = 256
}
protocol sieve {
}
It will use the IMAP credentials to allow users to edit their Sieve filter online. For example, Thunderbird has a plugin for it, which does syntax checking and what-not. When the filter is edited, it is syntax checked, compiled and replicated to the other Dovecot instance.
NGINX
I have an imap server and a mangesieve server, redundantly running on two Dovecot machines. I recall reading in the Dovecot manual that it is slightly preferable to have users go to a consistent replica and not bounce around between them. Luckily, I can do exactly that using the NGINX frontends:
upstream imap {
server maildrop0.chbtl0.net.ipng.ch:993 fail_timeout=10s max_fails=2;
server maildrop0.ddln0.net.ipng.ch:993 fail_timeout=10s max_fails=2 backup;
}
server {
listen [::]:993;
listen 0.0.0.0:993;
proxy_pass imap;
}
upstream sieve {
server maildrop0.chbtl0.net.ipng.ch:4190 fail_timeout=10s max_fails=2;
server maildrop0.ddln0.net.ipng.ch:4190 fail_timeout=10s max_fails=2 backup;
}
server {
listen [::]:4190;
listen 0.0.0.0:4190;
proxy_pass sieve;
}
I keep port 993 for maildrop as well as port 587 for smtp-out unfiltered on the NGINX cluster.
I’m a little bit more protective of the managesieve service, so port 4190 is allowed only when
users are connected to the VPN or the internal (office/home) network.
Now, you’ll recall that in the smtp-in servers, I forward mail to pim@maildrop.net.ipng.ch,
for which the redundant Dovecot servers are both accepting mail. On the way in, I can see to it that
the primary replica is used , by giving it a slightly lower preference in DNS MX records:
maildrop.net.ipng.ch. 300 IN MX 10 maildrop0.chbtl0.net.ipng.ch.
maildrop.net.ipng.ch. 300 IN MX 20 maildrop0.ddln0.net.ipng.ch.
imap.ipng.ch. 60 IN CNAME nginx0.ipng.ch.
nginx0.ipng.ch. 600 IN A 194.1.163.151
nginx0.ipng.ch. 600 IN A 46.20.246.124
nginx0.ipng.ch. 600 IN A 94.142.241.189
nginx0.ipng.ch. 600 IN AAAA 2001:678:d78:7::151
nginx0.ipng.ch. 600 IN AAAA 2a02:2528:ff00::124
nginx0.ipng.ch. 600 IN AAAA 2a02:898:146::5
This will make the smtp-in hosts prefer to use the chbtl0 maildrop replica when it’s available.
If ever it were to go down, they will automatically fail over and use ddln0, which will replicate
back any changes while chbtl0 is down for maintenance or hardware failure. On the way to out, the
nginx cluster will prefer to use chbtl0 as well, as it has marked the ddln0 replica as backup.
4. Webmail: Roundcube
Now that I have all of the infrastructure up and running, I thought I’d put some icing on the cake with Roundcube, a web-based IMAP email client. Roundcube’s most prominent feature is the pervasive use of Ajax technology. It also comes with an online Sieve editor, and runs in Docker. What more can I ask for?
Installing it is really really easy in my case. Since I have an nginx cluster to frontend it and do
the SSL offloading, I choose the simplest version with the following docker-compose.yaml:
version: '2'
services:
roundcubemail:
image: roundcube/roundcubemail:latest
container_name: roundcubemail
volumes:
- ./www:/var/www/html
- ./db/sqlite:/var/roundcube/db
ports:
- 9002:80
environment:
- ROUNDCUBEMAIL_DB_TYPE=sqlite
- ROUNDCUBEMAIL_SKIN=elastic
- ROUNDCUBEMAIL_DEFAULT_HOST=ssl://maildrop0.net.ipng.ch
- ROUNDCUBEMAIL_DEFAULT_PORT=993
- ROUNDCUBEMAIL_SMTP_SERVER=tls://smtp-out.net.ipng.ch
- ROUNDCUBEMAIL_SMTP_PORT=587
There’s a small snag, in that by default the SMTP user and password are expected to be the same as
for the IMAP server, which is not the case for my design. So, I create a user roundcube on the
smtp-out cluster and give it a suitable password. I nose around a little bit, and decide my
preference is to have threaded view by default, and I also enable the managesieve plugin:
$config['log_driver'] = 'stdout';
$config['zipdownload_selection'] = true;
$config['des_key'] = '<this key of sorts>';
$config['enable_spellcheck'] = true;
$config['spellcheck_engine'] = 'pspell';
$config['smtp_user'] = 'roundcube';
$config['smtp_pass'] = '<something or other>';
$config['plugins'] = array('managesieve');
$config['managesieve_host'] = 'tls://maildrop0.net.ipng.ch:4190';
$config['default_list_mode'] = 'threads';
I start the docker containers, and very quickly after, Roundcube shoots to life. I can expose it behind the nginx cluster, while keeping it accessible only for VPN + office/home network users:
server {
listen [::]:80;
listen 0.0.0.0:80;
server_name webmail.ipng.ch webmail.net.ipng.ch webmail;
access_log /var/log/nginx/webmail.ipng.ch-access.log;
include /etc/nginx/conf.d/ipng-headers.inc;
location / {
return 301 https://webmail.ipng.ch$request_uri;
}
}
geo $allowed_user {
default 0;
include /etc/nginx/conf.d/geo-ipng.inc;
}
server {
listen [::]:443 ssl http2;
listen 0.0.0.0:443 ssl http2;
ssl_certificate /etc/certs/ipng.ch/fullchain.pem;
ssl_certificate_key /etc/certs/ipng.ch/privkey.pem;
include /etc/nginx/conf.d/options-ssl-nginx.inc;
ssl_dhparam /etc/nginx/conf.d/ssl-dhparams.inc;
server_name webmail.ipng.ch;
access_log /var/log/nginx/webmail.ipng.ch-access.log upstream;
include /etc/nginx/conf.d/ipng-headers.inc;
if ($allowed_user = 0) { rewrite ^ https://ipng.ch/ break; }
location / {
proxy_pass http://docker0.frggh0.net.ipng.ch:9002;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
The configuration has one neat trick in it – it uses the geo module in NGINX to assert that the
client address is used to set the value of allowed_user. It will be 1 if the client connected from
any network defined in the geo-ipng.inc file, and 0 otherwise. I then use it to bounce unwanted
visitors back to the main [website], and expose Roundcube for those that are
welcome.
While the Roundcube instance is not replicated, it’s also non-essential. I will be using Thunderbird, Mail.app and other clients more regularly than Roundcube. It may just be handy in a pinch to either check mail using a browser, but also to edit the Sieve filters easily.
In my defense, considering roundcube is pretty much stateless, I can actually just run multiple
copies of it on a few docker hosts at IPng – then in the nginx configs I might use a similar
construct as for the maildrop and smtp-out services, with a primary and hot standby. But that
will be for that one day that the docker host in Lille dies AND I decided I absolutely require
Roundcube precisely on that day :)