qmail-updater

One of the problems I used to run into when running ISPs was that we had up to four administrators working at the same time, and none of us were perfect- at one time or another, all of us would update files and forget to run the necessary commands afterward to make the changes active.

To help with this problem, I wrote a shell script run as a cron job which checked the timestamps of the files we commonly updated, along with the files built from each text file. If the timestamps showed that the "magic command" for that particular set of files hadn't been run, it would run the command automatically. Later versions also recognized when virtualdomains or locals had changed and would restart qmail-send automatically, and the last version added automatic maintenance of the validrcptto.cdb file (i.e. whenever a mailbox or domain had been added or removed, it would automatically run mkvalidrcptto to generated a new validrcptto.cdb file.

However, I also had a few servers where cron didn't work correctly. I never did figure out what the real problem was, but occasionally crond would just plain crash- and when this happens, none of the cron jobs would be executed.

Being a fan of daemontools, I decided to write a daemontools version of this script, called qmail-updater. This script would check the control files every minute, and whenever it saw something change, it would run the commands to rebuild or restart whatever needed to be done in order for the new information to become active.

A Better Way

The approach described above uses a lot of CPU time to continually check the files over and over, and it introduces an artificial delay of up to a minute before changes would become active. Both of these can be a problem for a busy ISP (which is my background.)

I have since come up with a better way to handle this. Basically, I have split the process into two programs- one which checks the files and rebuilds or restarts whatever is needed (basically the same thing that qmail-updater was doing once a minute) and a second program which controls WHEN the first script is run... by watching for input on a named pipe.

A named pipe is a way for two processes to communicate. The idea is that a "client" program will send data to the named pipe, as if it were a normal file, and a "server" program will read data from the pipe, also as if it were a normal file. If a server program tries to read from the pipe before a client has sent any data) then it will "block"- it will go to sleep until a client writes data to the pipe.

So what happens is that the daemontools service runs this "pipe watcher" program, which opens the named pipe and tries to read from it, resulting in it going to sleep. When it sees data on the pipe, it finishes reading the data, closes the pipe, runs the program to update the qmail files, and then starts over (opens the pipe and tries to read, thereby putting itself back to sleep.)

Which means that any process on the system which has the ability to write to the pipe can "trigger" the process of updating the qmail control files, simply by writing data to the pipe (or on some systems, just "touching" the pipe, which unblocks the server without sending any data.)

This also allows vpopmail (when compiled with the "onchange" patch) to trigger the service autmoatically, which means that programs like qmailadmin, vpopmaild, or the vpopmail command line programs, will all automatically trigger the update as needed, with no artificial delay and without running it unnecessarily. All you need is to create a script called ~vpopmail/etc/onchange which contains a line that sends data to, or touches, the pipe (as shown in the ~vpopmail/etc/onchange script I'm using on my own server.

Setting things up

The first step is to make sure you have the mkvalidrcptto and/or mkauth scripts installed on your system. I normally put them in the /usr/local/bin directory, if you choose to put them somewhere else you may need to adjust the paths in the update-qmail script below so it knows where to find them.

# cd /usr/local/bin/
# wget http://qmail.jms1.net/scripts/mkvalidrcptto
# wget http://qmail.jms1.net/scripts/mkauth
# chmod 755 mkvalidrcptto mkauth

The next step is to create the physical directory structure which will contain the service. On my own server, all daemontools service directories are stored in /var/service, you can store them anywhere you like on your own server, except within /service itself. The examples below will assume that you are using /var/service, if you are using something else you should adjust your paths accordingly.

# cd /var/service adjust the path as needed
# mkdir -m 1755 qmail-updater
# mkdir -m 755 qmail-updater/log
# cd qmail-updater/log
# wget http://qmail.jms1.net/scripts/service-any-log-run
...
# mv service-any-log-run run
# chmod 755 run
# cd ..
# wget http://qmail.jms1.net/scripts/pipe-watcher
...
# wget http://qmail.jms1.net/scripts/update-qmail
...
# wget http://qmail.jms1.net/scripts/service-qmail-updater-run
...
# mv service-qmail-updater-run run
# chmod 755 pipe-watcher update-qmail run

Configuring the scripts

The variables at the top of the "pipe-watcher" script control how it works. You should check these variables to make sure it works the way you need it to.

• $pfile contains the full pathname of the named pipe you will be watching. If you place the named pipe under the /tmp directory (as I have done), then any process will be able to write data to the pipe.

  Note that at the beginning of the code, there is a mkfifo command which actually creates the named pipe. The "-m u=rw,go=w" argument controls the permissions of the named pipe itself. This argument allows processes running as other userids to write to the pipe. If this is not what you want, you should adjust this parameter to reflect your wishes.

• $cmd contains the full pathname of the program you wish to run when data is received on the pipe. The "pipe-watcher" program was written to be fairly generic- it can be used for any number of other things which are not related to qmail.

• $cmd_needs_data tells whether or not the data received from the pipe should be given to the program you wish to run. For this application, the answer is no. If you set this variable to 1, the data received from the pipe will be present on the "stdin" channel of the program it runs.

• $min_delay sets a length of time (in seconds) which must pass between the program running. This is used to prevent a flood of requests causing the program to run over and over again.

There are also a few variables at the top of "update-qmail" which should be set to work with your system.

• $smtpd_user is the userid as which your qmail-smtpd process runs. If you're using my qmail-smtpd run script, this will be the same as the QUSER variable in that script.

• $mkvalidrcptto should hold the full path to where mkvalidrcptto is installed on your system. The normal location is /usr/local/bin/mkvalidrcptto.

• $mkauth should hold the full path to where mkauth is installed on your system. The normal location is /usr/local/bin/mkauth. If you are not using the AUTH_CDB feature, you can leave this as it is. The update-qmail script will only run mkauth if it sees an existing /var/qmail/control/auth.cdb file.

Creating the files

The update-qmail script will only run mkvalidrcptto if it sees an existing validrcptto.cdb file, and will only run mkauth if it sees an existing auth.cdb file. Obviously, this means you need to create one or both of these files before running update-qmail.

# mkvalidrcptto -c /var/qmail/control/validrcptto.cdb
# mkauth -c /var/qmail/control/auth.cdb

Starting the service

The last step is to tell daemontools to run the service. This is done by creating a symbolic link from /service to the physical service directory. A program called svscan checks the /service directory every five seconds, and when it sees the new link there, it will start up a supervise process for the new service- and because the service directory has the "sticky bit" set (the "1" in the mkdir -m 1755 command above) it will also start up a supervise process for the log directory within the new service, and create a set of pipes so that the primary service's output is routed to the log service's input.

# ln -s /var/service/qmail-updater /service/ adjust the path as needed

Wait about ten seconds...

# svstat /service/qmail-updater
/service/qmail-updater: up (pid 5087) 6 seconds

For the svstat command, you should see the service up for two seconds or more. If it says up for 0 or 1 seconds, run the same svstat command again. If it still doesn't want to go above one second (or if it doesn't want to start at all), check the log/main/current file, or if that has no output at all, check the command line of the readproctitle process.

# ps axww | grep readproctitle
347 ? S 0:00 readproctitle service errors: .......................
................................................................................
................................................................................
................................................................................
................................................................................
.........................................................

Where the example here shows a line of ".", you may see error messages from the service you're trying to set up, or from other services which may have had problems since the machine was booted. New messages are added to the end of this string, with older messages being pushed off of the beginning. There are unfortunately no timestamps or definite ways to identify which particular service sent a message- but the information here may give you a clue as to why the service isn't starting up correctly.

Testing the service

Testing is easy... in one window (or on one console) you should watch the log file, and in another window/console, simply "touch" the named pipe (or send data to it.) You should make sure that the service reacts correctly to "touches" and data from both root and non-root users.

Notes

2007-06-18 The mkvalidrcptto script was updated, it now knows how to build a validrcptto.cdb file by itself. I have updated the update-qmail script to take advantage of this. If you are installing this service, make sure your mkvalidrcptto script is dated 2007-06-18 or later.

Copyright 1995-2024 John M. Simpson KG4ZOW (email) (PGP key)
Last updated 2008-08-08 03:43:26 +0000