Setting up mailhubs

This page talks about what I call "the mailhub scenario". The idea is that you have one machine which handles the mailboxes, and one or more other machines (which I call "mailhubs") which accept incoming mail, handle things like RBL checking and virus and spam filtering, and then hand the messages off to the mailbox machine.

The mailbox machine should not run an SMTP server on port 25 at all. This will prevent random people on the Internet from trying to send mail there instead of to the mailhubs. However, it can run a TLS-required, AUTH-required SMTP server on port 587, along with an SSL, AUTH-required SMTP server on port 465, so that your users can use AUTH on the server to relay their outbound mail. It can also run a "normal" SMTP server (i.e. no AUTH, TLS, or SSL required) on some other port number (I normally use 925 for this), with a tcpserver access control list that allows the mailhubs to send mail but no other machines.

In a scenario like this, the same scripts we used on the mkvalidrcptto page for a single server can be used on the mailbox server, with one addition- after certain qmail control files change (including the validrcptto.cdb file) the new data needs to be "pushed" to the mailhub machines so that they know about any relevant changes in what mail they should and should not accept.

The qmail-updater script has a variable called $push_cmd which is normally empty, but if it contains the name of an executable (a program or script), qmail-updater will run this executable whenever these control files have been changed. The idea is to write a shell script which sends the files to each mailhub in such a way that they are safely installed for use by the qmail system on the mailhub.

Some people may be using the cron.qmail script (described on the mkvalidrcptto page as well as the scripts index page) instead of the qmail-updater service. The cron.qmail script contains comments which show where you would add commands to push the data out to other servers. The individual commands to push each file could be added at that point, but it's easier and more flexible to use the same "push-data" that you would use if you were using the qmail-updater service, and just add one line at the right point in your cron.qmail script which runs the push script.

How it works

The $push_cmd variable in the qmail-updater script will contain the name of a script. We will call this script push-data, because it will be pushing new data out to the mailhub machines. For simplicity's sake, the script will be located in the same directory with the qmail-updater script.

What this script will do is send the contents of several qmail control files- validrcptto.cdb, rcpthosts, morercpthosts.cdb, and smtproutes- to each mailhub. This transfer will be done using ssh, which not only makes the connection secure, but as you will see, it allows for the mailhub to verify that only one of the correct files are being sent, and to properly handle each kind of file as it is received.

This will be done by creating an ssh key pair, which a non-root user on each mailhub will trust for logging in, but will only be allowed to run one specific command- a script we will write, which will receive the data and rebuild the cdb file.

Setting up the mailbox server

This setup will assume that you are running the qmail-updater service on the mailbox server. We will be creating an ssh key pair and writing the push-data script.

Creating an ssh key pair

In order for the mailbox server to send data to the mailhubs within a script (i.e. without having to type a password) it will need to authenticate to each mailhub using an ssh key. Long ago I wrote a web page which explains this in more detail, the following is a quick walk-through of how to create and install the key on the various servers.

Note that an ssh key is actually a pair of keys- a private key that you keep to yourself, and a public key that you install on the servers you wish to have access to.

The ssh-keygen command creates a new key pair. The process looks like this:

On the mailbox server...
# cd /service/qmail-updater/root
# ssh-keygen -t dsa -b 2048 -f id_dsa_validrcptto -C 'validrcptto'
Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase): just hit ENTER
Enter same passphrase again: just hit ENTER
Your identification has been saved in id_dsa_validrcptto.
Your public key has been saved in id_dsa_validrcptto.pub.
The key fingerprint is:
f0:eb:82:ea:75:2c:1d:02:7f:f0:b4:7d:18:25:7d:c0 validrcptto

As you can see, it actually created two files: id_dsa_validrcptto is the private key, and id_dsa_validrcptto.pub is the public key. The public key is needed on each machine we will be logging in TO, while the private key is needed on the machine you are logging in FROM.

Installing the push-data script

The other necessity is the script which pushes the data out to the mailhubs. I have written a simple push-data script which you can download and either customize for your own servers, or use as an example to write your own script.

Before you do this, you will need to decide on a username, which we will be creating on each mailhub. This username should be something that you would not otherwise use. For the sake of the examples on this page, we are going to use the username qmailc.

The procedure will look like this:

On the mailbox server...
# cd /service/qmail-updater/root
# wget http://qmail.jms1.net/scripts/push-data
# chmod 755 push-data
# nano push-data (Feel free to use whatever text editor you like.)

You will need to edit the following lines:

# list of mailhub IP's, separated by spaces.
SERVERS="1.1.1.1 2.2.2.2 3.3.3.3"

# ssh port on mailhub machines. should be the same for all servers.
PORT="22"

# remote userid on target servers. should be the same for all servers.
USERID="qmailc"

# SSH private key file. should be the same for all servers.
IDFILE="/service/qmail-updater/root/id_dsa_validrcptto"

# the name and port number that the mailhubs' smtproutes file need to
# point to for domains listed in "locals" and "virtualdomains" on this
# server.
MYNAME=`cat /var/qmail/control/me`
MYPORT=""

At this point, everything should be set up on the mailbox machine. In order to test it, we will need to set up at least one mailhub.

Setting up a mailhub

Configuring sshd

In order for this to work, sshd (the process which answers incoming ssh connections) must be configured to allow public key authentication. This is normally already done when you install sshd, but it doesn't hurt to be sure.

We need to examine, and possibly edit, the sshd_config file. This is normally found in the /etc/ssh or /etc directory. Look for a line which starts with the word PubkeyAuthentication, and make sure it doesn't say "no" after it. If it does say "no", edit the file and change this line to say "yes". Then restart sshd.

Sending the public key to the mailhub

The next step is to copy the SSH public key (the id_dsa_validrcptto.pub file we created above) from the mailbox machine to the mailhub. This can be done using any mechanism you like- you can write it out to a floppy and manually copy it, you can use ssh or scp, or you can even throw it on a web server and use something like curl or wget on the mailhub to retrieve it. Since it is a public key, there is no danger if other people happen to get a copy of this file.

However you do this, you need to copy the id_dsa_validrcptto.pub file to root's home directory on the mailhub machine.

Creating the userid

The next step is to create the non-root user whose userid we chose above, which will handle the transfers. The transfers are not be done as root, just in case there is a bug in one of the scripts which might allow somebody with a copy of the private key to cause more damage than just messing with the validrcptto.cdb file. Also, this user should not have a valid password- there should be no way to log into the machine as this user, other than using the SSH key.

Again, for the sake of the examples on this page, we are using the username qmailc.

The commands used for this may be different for different systems. On my server (CentOS 4.2) the command is...

On the mailhub, as root...
# useradd qmailc

If your system's useradd command (or equivalent) asks you to assign a password to the new user, try hitting CONTROL-C to bypass the process of setting a password. If it won't let you, then choose something really ugly for a password- there is no need for you or anybody else to remember it, since it will never be used. In fact, if your system's passwd command has a -l option to "lock" an account by setting it to a non-usable password, you should use this after creating the account, just so that there is no real password for the account, and therefore no possibility of anybody using a brute-force attack to guess the password.

Once the userid has been created, chown the SSH public key file so that the new user owns it, and then mv the file into their home directory.

On the mailhub, as root...
# cd
# chown qmailc:qmailc id_dsa_validrcptto.pub
# mv id_dsa_validrcptto.pub ~qmailc/

Installing the script to process the data

Once the public key is present in the user's home directory, the rest of the setup should be done as the non-root user. Of course, we made sure that this user doesn't have a real password, so we can't log into it. What we can do, however, is use su to change from root to this user. This is possible because root is able to "become" any user on the machine without needing to know their password.

Once we "are" the non-root user, the first order of business is to write or install a script to handle the incoming data. If you are using my push-data script on the server, I have a script called update-files which correctly handles the data by writing the incoming stream of bytes to a temporary file, setting the permissions on that temporary file, and then renaming the temporary file to the actual name (so that no qmail process ever sees a half-done file, or a file with the wrong permissions.)

If you are writing your own scripts: when the ssh client (in this case, the mailbox machine) passes a command but also authenticates using a key with a forced command, the forced command attached to the key is actually executed and the command that the client requested is available in the SSH_ORIGINAL_COMMAND environment variable. My scripts use this to pass the name of the file we're updating.

On the mailhub, as the non-root user...
$ wget http://qmail.jms1.net/scripts/update-files
$ chmod 755 update-files

Of course you are free to write your own script to do this. Feel free to use my script as a guide for writing your own script if you like- but as simple as it is, there shouldn't really be any need to change it (unless you need to adjust the PATH, of course.)

Installing the public key with the forced command

The next step is to create the authorized_keys file in the non-root user's .ssh directory so that it includes this key with the forced command. The forced command goes at the beginning of the line in the authorized_keys file which contains the public key.

On the mailhub, as the non-root user...
$ cd
$ mkdir .ssh
$ mv id_dsa_validrcptto.pub .ssh/
$ cd .ssh
$ echo -n 'command="/home/qmailc/update-files" ' >> authorized_keys Home directory of the non-root user
$ cat id_dsa_validrcptto.pub >> authorized_keys

The file is created, this next command just shows you what the result looks like. Note that the block of garbage below is NOT a valid key file and should not be copied and used on any server- it just looks like one so you can see the format of what it looks like. Note that this is all one line of text, I have manualy wrapped it to 80 characters so that it doesn't force the page to be wide in some browsers.

$ cat authorized_keys
command="/home/qmailc/.ssh/update-files" ssh-dss AAAAB3NzaC1kc3MAAMKr1HxJzOtK5WR
QCm16SfWbKaS5fw4sYdio8LApp59RLATJ+SqYCFQzzcvppGtxdG8rPXFmi0yqb0Tkx8vQ00/e7QZKwmi
uvu2kubl2HcID3mGK8GdX817DOmYSO0lPDbTi+jbpG9bLAsaufhxdjimF5Zyv5uZEznHHzSIKQgsgPKa
cVj+LzaUsJ1/4zLw/JkOIZeps7xnJ25d6wGvZkfMrEBwLqNL1dFCvnZIK1oHEIXgnQxUtxoHA57DPIZF
GryBvVu6CybJD+uDv4Ia+jmtrxYCIqxocwuGW7gmGA== validrcptto

One more detail. sshd has a security feature where it will not trust an authorized_keys file if it, or the .ssh directory containing it, has any "group" or "other" permissions. We need to set the permissions so that these extra permissions are not present.

On the mailhub, as the non-root user...
$ chmod 600 authorized_keys
$ chmod 700 .

Testing from the mailbox machine

At this point we should be able to run the push-data script on the server. The first time you run it, if this is the first time you have ssh'd from the mailbox machine to the mailhub, it will ask you whether or not you want to accept the mailhub's host key. Say YES.

On the mailbox server...
# cd /service/qmail-updater/root
# ./push-data
/---- Start of messages from 1.1.1.1 for validrcptto.cdb
If it asks about accepting the mailhub server's public key here, say YES.
| Saving incoming data to validrcptto.cdb.new
| Setting permissions
| Renaming validrcptto.cdb.new to validrcptto.cdb
\---- End of messages from 1.1.1.1 for validrcptto.cdb
/---- Start of messages from 1.1.1.1 for rcpthosts
| Saving incoming data to rcpthosts.new
| Setting permissions
| Renaming rcpthosts.new to rcpthosts
\---- End of messages from 1.1.1.1 for rcpthosts
/---- Start of messages from 1.1.1.1 for morercpthosts.cdb
| Saving incoming data to morercpthosts.cdb.new
| Setting permissions
| Renaming morercpthosts.cdb.new to morercpthosts.cdb
\---- End of messages from 1.1.1.1 for morercpthosts.cdb
/---- Start of messages from 1.1.1.1 for smtproutes
| Saving incoming data to smtproutes.new
| Setting permissions
| Renaming smtproutes.new to smtproutes
\---- End of messages from 1.1.1.1 for smtproutes

On the mailhub, you should now have validrcptto.cdb, rcpthosts, and morercpthosts.cdb files which are identical to the ones on the mailbox server, as well as an smtproutes file which contains the contents of the smtproutes file (if any) from the mailbox machine, plus lines which direct any domains which are listed in locals or virtualdomains on the mailbox server, back to the mailbox server itself.

On the mailhub, as the non-root user...
$ cd
$ ls -l
total 17 -rw-r--r-- 1 qmailc qmailc 2798 Feb 15 16:07 morercpthosts.cdb -rw-r--r-- 1 qmailc qmailc 271 Feb 15 16:07 rcpthosts -rw-r--r-- 1 qmailc qmailc 1163 Feb 15 16:07 smtproutes -rwxr-xr-x 1 qmailc qmailc 1561 Feb 15 16:07 update-files -rw-r--r-- 1 qmailc qmailc 18578 Feb 15 16:07 validrcptto.cdb

Finishing the mailhub setup

So far, all we have accomplished is to securely get the necessary control files into the home directory of a non-root user on the mailhub. Of course, qmail doesn't look in this directory, so we still need some way to copy the files from this directory into /var/qmail/control... or so it would seem.

Another option is to have /var/qmail/control contain symbolic links to the files here. This way, when qmail-smtpd and qmail-remote try to read their control files from /var/qmail/control, they end up reading the files from the non-root user's home directory instead, without ever knowing (or caring) that anything different is happening.

Before we do this, we need to make sure that the non-root user's home directory is accessible to the userids that qmail-smtpd and qmail-remote run as. Some distributions may have created the user's home directory in a world-readable state, this is fine- the only things really needed are the "x" bits for "group" and "other", so that's all we're going to turn on. If they are already turned on, this command won't change anything:

On the mailhub, as the non-root user...
$ cd
$ chmod go+x .

And now we are done with the non-root user. The final commands (creating the symlinks) have to be run as root.

That's all! There is no need to reload or restart anything on the mailhub, because all of these files are used only by programs which start new copies for every message they process.

At this point, all new qmail-smtpd processes on the mailhub will use the validrcptto.cdb, rcpthosts, and morercpthosts.cdb files from the non-root user's home directory, all new qmail-remote processes on the mailhub will use the smtproutes file generated by the mailbox server, and all of these files will be updated whenever the push-data script is run on the mailbox server.

One last thing...

If you have just finished setting up your first mailhub machine, you probably still need to set the $push_cmd variable in the qmail-updater script. If you have to edit the script, you should shut the service down first, and make sure your editing didn't break something before starting it back up.

On the mailbox server...
# cd /service/qmail-updater
# svc -d .
# nano qmail-updater Use whatever text editor you like

Find this line, just below the copyright statement:

my $push_cmd = "" ;

Change it to contain the full path to the push-data script:

my $push_cmd = "/service/qmail-updater/push-data" ;

Save your changes, then do a perl syntax check to make sure you didn't break anything by accident.

On the mailbox server...
# perl -c qmail-updater
qmail-updater syntax OK

If the syntax check is okay, start the service back up.

On the mailbox server...
# cd /service/qmail-updater
# svc -u .

Contacting me for help

See this page for information.

Copyright 1995-2024 John M. Simpson KG4ZOW (email) (PGP key)
Last updated 2009-04-14 15:31:56 +0000