qmail Scripts

I do lots of scripts... scripts rock.

Major scripts

These scripts are documented on pages of their own.

• The qmail Rules Interface (I don't have any better name for it) is a system which allows users, domain owners, and system administrators to manage the kinds of filtering that qmail-smtpd does during the SMTP conversation. It consists of a processing script (which qmail-smtpd calls for every RCPT command sent by a client), a database containing the rules, and a web interface for editing the database. It requires the RCPTCHECK patch, which is part of my combined patch and can also be installed separately.

• jgreylist is a script which does "greylisting". Spammers use programs which, when faced with a rejection, simply move on to the next victim. A legitimate mail server will try the message again in a few minutes. With greylisting, clients from "unknown" IP addresses will be rejected, but "remembered". After a few minutes, mail from that IP will be accepted as normal. This way, legitimate mail servers are allowed to send mail after one retry, while spammers who don't retry at all are blocked.

  One thing to be aware of is that some spammers' programs are getting smarter- they're keeping a list of which messages did not go through and trying them again after a short time, just as a legitimate mail server would. Greylisting is not the ultimate solution to spam, but it is a good tool to help control the volume of spam you receive.

  Note that the rules interface includes the same functionality as jgreylist.

• migrate-domain reads the files which make up a vpopmail domain, create a tarball of each mailbox, and generate a shell script to create the same mailboxes and expand those tarballs back into place. It also handles autoresponders and ezmlm lists generated with qmailadmin.

  The idea is to run this script on the system you're migrating "from", copy the script and the tarballs en masse to the new machine, and run the generated script to "restore the domain" on the new server, by creating the same mailboxes with the same passwords and content they had on the old server.

• mkvalidrcptto is a perl script which reads the various files controlling mail delivery on the system to build a list of all valid email addresses on the system. This list is used by my validrcptto.cdb patch to allow qmail-smtpd to check recipient addresses and refuse mail for addresses which don't exist.

• mkauth is a script which can be used to generate a .cdb file for use with the AUTH_CDB patch. It scans the qmail control files (and system account files, if you have any true "local" domains) and produces a list of the valid userid and encrypted password pairs which should be accepted for the AUTH command. The qmail-updater script uses this script.

  Because of its complexity, I have moved the download link to the AUTH_CDB patch web page, which also has better documentation for the script.

• qfixq is a script that I wrote to "clean up" a damaged qmail queue. I have used it to recover from out-of-disk errors (where files would be created as zero bytes but no data could be written to them) as well as a "chown -R" typo and a few queue migrations from one disk to another (it sets the correct names to match the inode numbers.)

• qmail-updater is a script which runs mkvalidrcptto and/or mkauth, and if the output has changed, updates the live validrcptto.cdb and/or auth.cdb files, as well as sending a HUP signal to qmail-send whenever the control/virtualdomains or control/locals file changes. This is designed to be run within a pipe-watcher service.

Service Control Scripts

I run qmail under daemontools on all of my systems. Because the qmail that I run has an entire collection of patches applied to it, and because I run several different SMTP services on the same machine, I need a more customizable set of "run" scripts to start the services, especially the SMTP services.

I have found "run" scripts for qmail on other web sites, and while the ones on Life with qmail come close, they aren't clear enough or customizable enough for my needs.

qmail-send

The qmail-send process is the queue manager for a qmail system. It looks at each new message added to the queue and decides whether the message should be handled locally or sent to a remote server, and updates the queue's status files to reflect that choice. It also triggers local and remote deliveries as needed, including retrying unsuccessful deliveries.

Many people, myself included, call this "/service/qmail-send", although it can have any name you want. Whatever name you give it, the run script is fairly simple.

SMTP service

I am in the process of re-organizing the script, and am making a separate web page which documents all of the options. The page isn't finished yet, but it already contains a LOT more information than you will find here.

courier-imap services

I already have a web page which tells how to run courier-imap services under daemontools. These are simply download links for the "run" scripts needed for each service. Note that the courier-imap services can use the same "log" script as the qmail services (listed below.)

2008-05-08 I've added a line at the end of the script which makes it log the final command line just before exec'ing it, just as I did with the SMTP service script.

Logging services

service-any-log-run is a generic "run" script for any logging service managed by daemontools. When you create a service, you should also create a log service with it- otherwise your logs will disappear without a trace.

This script sets the auto-rotation parameters so that each log file is allowed to grow to 1MB, and the program is allowed to keep up to 1024 old log files laying around. These changes were made for a high-volume server at an ISP, and I use it everywhere because it doesn't really hurt anything (as long as the logs are rotated out of the way regularly, keep reading...) djb's multilog instructions explain how to change these numbers, and I have written a page which explains more detail about how daemontools log files work.

And speaking of rotating logs out of the way, I have written a program called convert-multilog which finds the log files created by multilog, converts their timestamps to a human-readable format, and adds them to log files in "/var/log" whose filenames are the service plus the date (i.e. all "/service/qmail-send/log/main/@*" files are combined into /var/log/qmail-send.YYYY-MM-DD.)

2007-08-14 Received a bug report. If an input file starts with a line which is all whitespace, no output file is opened. Changed the logic so that, unless multilog starts spitting out NUL bytes (which I don't think it can, since it's written in C) an output file will still be created, even if the name is a bit strange. Thanks to John Coryat for letting me know about the problem.

2007-09-04 Added the @exclude array, which holds the names of any services for which you may not want the logs to be rotated. Note that if you wish to exclude "/service/blah", the word in this array should just be "blah". Thanks to Patrick "marlowe" McDonald for the idea.

2009-11-15 Receieve a bug report. Apparently, the output from tai64nlocal doesn't always start with YYYY-MM-DD. I'm not sure what causes it, but I suspect it's embedded CR characters within the raw log lines sent to multilog. If the script sees a line like this, it will assume that the line is a continuation of the previous line, and just write it to whatever output file is currently open. If there is no output file (i.e. if the log file starts with a mis-formed line) then that line will be discarded.

2012-08-24 Found a bug on my own. Apparently it wasn't parsing the output from tai64nlocal correctly, and the copy on the web site wasn't working.

courierpassd service

service-courierpassd-run is a "run" script which runs the courierpassd program on localhost port 106. This program is used on some systems to allow users to change their passwords from a webmail interface. This page explains in more detail.

vpopmaild service

service-vpopmaild-run is a "run" script for a vpopmaild service. vpopmaild is a server which allows properly authorized clients to perform most of the actions you can do with the vpopmail command line tools. A very simple use of such a service might be to support the SMTP AUTH command (which is something I plan to write a patch for, but haven't done yet.)

2012-07-19 Changing the service to run as the vpopmail user, and use port 8900.

courierauthd service

service-courierauthd-run is a "run" script for a courierauthd service. courierauthd is a program I wrote to support the subset of vpopmaild's functions which are necessary to support the SMTP AUTH command (i.e. the login, quit, and help commands.) The idea is that this allows the (as yet un-written) SMTP AUTH patch for vpopmaild to be useful to people who may not be using vpopmail, but who are using courier-authlib (since courier-authlib can be used with any number of authentication methods.)

I originally used a shell script called cron.qmail to update my validrcptto.cdb file, and send a HUP to qmail-send when needed, and I relied on cron to run it periodically. However, on some systems cron doesn't always work correctly, so I wrote a perl script called qmail-updater which ran all the time and kept the files up to date. (Note that neither of these scripts is being updated. You are welcome to use them, but they will not be kept "up to date" when I come up with new features.)

Secure qmail-pop3d service

I don't use the POP3 server which comes with qmail. To me it always seemed easier to use the POP3 server which came with courier-imap, and then later with dovecot. However, people still seem to be asking how to set it up to run under daemontools.

At one point, Niamh Holding sent a sample of a qmail-pop3d "run" script to the mailing list. I took her example, added a few comments, and made it a bit easier to configure. So here's an example of what the "run" script for a qmail-pop3d service should look like.

Note that I have not personally tested this myself, so if anybody does try it, please let me know if it works or not, and what problems (if any) you ran into while getting it to work.

#!/bin/sh # any script running as root should always specify its own PATH PATH="/var/qmail/bin:/usr/bin:/bin" # basic configuration IP="1.2.3.4" # IP on which to listen PORT="995" # TCP port number, 995 is standard LOCAL="secure.jms1.net" # your local hostname CHECKPW="/bin/checkpassword" # checkpassword program to verify ID/PW # for vpopmail this would be "vchkpw" # SSL stuff (exported so sslserver can use it) export CERTFILE="/var/qmail/control/servercert.pem" export KEYFILE="" export DHFILE="" # do the deed exec sslserver -e -vRH -l $LOCAL $IP $PORT \ qmail-popup $LOCAL $CHECKPW qmail-pop3d Maildir 2>&1

You may notice that there are no "-u" or "-g" options for sslserver, as there would normally be for other services. This is because in order for it to work, a "checkpassword" program needs to start as root. It executes the "setuid()" function before exec()'ing the qmail-pop3d program (which actually works with the user's mailbox.) If qmail-popup were to run as a non-root user, and the owner of the mailbox happens to be some other user, then the checkpassword program wouldn't be able to setuid() to that other user, and therefore the mailbox wouldn't be accessible through the service.

Other Scripts

Over the past several years of building, running, and consulting for ISPs I have written and/or found a collection of scripts which have proven useful when working with qmail servers. They are presented below, in more-or-less alphabetical order.

Note: Any scripts listed below with a red background are dangerous. Make sure you read the directions for these scripts before running them, and understand what they do and when it's safe to use them.

• listfilter is a script I wrote to verify that incoming messages to a given mailbox are actually from a certain mailing list. I was subscribed to a list under a different email address, in order to keep the mailing list separate from my personal email, and some spammer found out the other address and started spewing stuff to it directly. This script dropped all of this spam- basically, any messages which want to be stored in that mailbox have to have a certain header which is set by the mailing list.

  I later discovered that it could be combined with qmail's condredirect program to automatically filter mailing list traffic to other dummy addresses. (Instructions are in the script itself.)

  I have since started using maildrop to sort my incoming mailing list traffic to different folders based on the headers. This is a lot more flexible, but it also has a steeper learning curve.

  File:	listfilter
  Size:	2,914 bytes
  MD5:	528b6658a913ca8fb3bd84e16f5f2131
  SHA-1:	0fcdf359284d57bc5d927149ca489686598b5f97
  RIPEMD-160:	5071631a996d9bc64c9bc3f4afd28d060ff4a673
  PGP Signature:	listfilter.asc

• maybe-recordio is a little script which makes it possible for an SMTP service to selectively enable recordio for certain IPs, so you can see what they are doing without filling up your log files with a lot of extra stuff you don't need to see from other addresses.

  Unfortunately, because recordio needs to run before qmail-smtpd starts, it is not possible to enable this using the AUTH_SET mechanism.

  To install it:

  • Download or copy it to a directory so you know where it is (such as /usr/local/bin) and give it suitable permissions (i.e. "chmod 755 /usr/local/bin/maybe-recordio".)

    # cd /usr/local/bin
    # wget http://qmail.jms1.net/scripts/maybe-recordio
    # chmod 755 maybe-recordio

  • Edit the "run" script for your SMTP service. If you are using the "run" script from my web site, simply un-comment the RECORDIO= line and make sure the path matches where maybe-recordio is installed on your system (/usr/local/bin/recordio is the standard location.) If not, you need to find the line at the bottom which runs the final command- it usually looks something like this:

    exec /usr/local/bin/softlimit -m 30000000 \
    /usr/local/bin/tcpserver -v -R -l "$LOCAL" -x /etc/tcp.smtp.cdb -c "$MAXSMTPD" \
    -u "$QMAILDUID" -g "$NOFILESGID" "$IP" smtp \
    /var/qmail/bin/qmail-smtpd mail.example.com \
    /home/vpopmail/bin/vchkpw /usr/bin/true 2>&1

    You want to add the full path to maybe-recordio just before qmail-smtpd on this command line:

    exec /usr/local/bin/softlimit -m 30000000 \
    /usr/local/bin/tcpserver -v -R -l "$LOCAL" -x /etc/tcp.smtp.cdb -c "$MAXSMTPD" \
    -u "$QMAILDUID" -g "$NOFILESGID" 0 smtp \
    /usr/local/bin/maybe-recordio \
    /var/qmail/bin/qmail-smtpd mail.example.com \
    /home/vpopmail/bin/vchkpw /usr/bin/true 2>&1

    Then restart the service, using "svc -t /service/qmail-smtpd" (or whatever name your service has.)

  Then, for the IPs you want to monitor, add ",USE_RECORDIO="1"" to their line in the tcpserver access control file (usually /etc/tcp/smtp) and rebuild the .cdb version of the file. For example, if you want to monitor 10.0.0.6, you would add a line like this:

  10.0.0.6:allow,USE_RECORDIO="1"

  Or, if a matching line already existed, you would add the new item to the end of the line- or if the existing line covers an entire network and you only want to monitor one specific address, copy the existing line, change the IP, and add the new item.

  If this line already exists ...
  10.0.0.:allow,RELAYCLIENT=""
  
  ... you would add this to watch 10.0.0.6:
  10.0.0.6:allow,RELAYCLIENT="",USE_RECORDIO="1"

  Then rebuild the .cdb file, using tcprules (or a script or make command which calls tcprules.)

  Once this is done, all client connections from this IP will be logged.

  File:	maybe-recordio
  Size:	1,233 bytes
  Date:	2008-07-30 21:57:48 +0000
  MD5:	81d5952ed8d7cced38cf988fbd4eca8a
  SHA-1:	fdbf6faf9fccc5aea9707f0ae2defb711d131eff
  RIPEMD-160:	b0450c15f59dfcc3793a04d76a9e63501324d55e
  PGP Signature:	maybe-recordio.asc

• mtrack is a script which I wrote back in 1998 to help understand the log output from qmail-send, by grouping together all of the log lines which pertain to a given message. People have been occasionally looking for something like this, and every time they have asked, I've thought about this old script. I kept meaning to dig it out and put it up here, but there always seemed to be something else more important going on...

  I finally dug it out and looked at it... not only did it not work very well, but it looked like really amateur code- which it was. 1998 is about the time I started using Perl for scripting. So, rather than try to remember my own thought process from seven years ago and fix the script, I figured it was easier to just rewrite it from scratch.

  2009-11-15 Back in June I added pattern-matching within the program itself, basically because I needed it. I just realized that I didn't put the changes on the web site back then, so I did a little bit of clean-up on the code and added a "-h" option to show the command line options. The newer version is below.

  strack is another log file tracking script. It does for qmail-smtpd logs what mtrack does for qmail-send logs. In addition, if you are using version 6cd or newer of the combined patch, this script will recognize the new log line formats and track them along with the tcpserver and rblsmtpd lines, making the output a LOT more useful.

  File: mtrack Size: 6,669 bytes Date: 2009-11-15 09:35:39 +0000 MD5: edbfcc9ba15e415e3652d1ab7bb8d03f SHA-1: 485179f6fe31e54991b3c4e627c45953af6b5482 RIPEMD-160: 219ef5427b558386e5012a61768b86b74416d1d7 PGP Signature: mtrack.asc

  File: strack Size: 3,158 bytes Date: 2007-03-22 15:36:33 +0000 MD5: 1a0ab364ba6c0f9f99c3ceed71c07857 SHA-1: 88e5895148738664471a4ea43671753ce1aa98a1 RIPEMD-160: 7a53ca4b68b60e3176c6339fea6d43388655c8e9 PGP Signature: strack.asc

• push-data handles building and sending qmail control files from a central mailbox server to a "mailhub", and update-files handles receiving and installing those files on the mailhub machine. The mailhub page explains how to set up both scripts.

  File: push-data Size: 4,119 bytes Date: 2007-01-26 05:03:21 +0000 MD5: 2fd5cf7833b999cc0ec2dce86a9aedee SHA-1: e3972df293fb24b685e60302ace315bc454ea79e RIPEMD-160: a030fd20c6899616bf282d04b59f5e66688ba3d8 PGP Signature: push-data.asc

  File: update-files Size: 2,061 bytes Date: 2008-04-16 16:12:34 +0000 MD5: 5a8bad9334d7891c5001f599486b9c5c SHA-1: 3fe6f1be4f35e82b382ddf27e2f2f8ad39360dd4 RIPEMD-160: 955e8d8c902b2171632192fb53fcd77e10d270be PGP Signature: update-files.asc

• qbonk is a script which changes the timestamp on a message to "a long time ago". This causes qmail-send to give up on the message after making one more attempt at delivery. This is a safe way to force a particular message to be deleted from the queue.

  ISPs receive a lot of spam, most of which has forged "From" addresses. If there is a problem delivering the message (user does not exist, user is over quota, etc.) the server tries to "bounce" the message, but if the "From" address is no good, the bounce can't go anywhere. The qbonk script made it possible to prevent those "hanging bounces" from sticking around in the queue for a week, eating up disk space. (Of course, since I wrote the validrcptto.cdb patch, this doesn't happen to my servers anymore.)

  I can't take credit for the idea, it's been floating around on the internet for a while- I just packaged it. The script takes one or more message numbers on the command line- the message numbers are the "#12345" numbers you see on "qmail-qread" output.

  qbonkfrom and qbonkto are extensions of qbonk which search for a given regular expression within the envelope sender or recipient list, and does "qbonk" on them. Note that they do NOT automatically do "qkick" afterwards. Also note that the regular expression IS case-sensitive.

  qbonkns is an extension of qbonk. It searches the queue for messages with no sender address, does a "qbonk" on them, and then does "qkick" afterwards. This deletes spam-bounces which otherwise would just sit in the queue until they time out on their own. One of my ISP clients runs this as an hourly cron job on their servers...

  File: qbonk Size: 1,880 bytes Date: 2006-06-29 00:24:35 +0000 MD5: d536714e890e9af47d87049b859461c1 SHA-1: c8213cf9d31415e5eb0367ec81d30dc8d2240b1a RIPEMD-160: 40be3eeaa842b116978d3c79500867b6520ac814 PGP Signature: qbonk.asc	File: qbonkfrom Size: 1,470 bytes Date: 2007-02-17 08:30:47 +0000 MD5: e2d714f65a2eaa879f55ca48e7798289 SHA-1: 4bc7ea9cb9aef592f4fcf1d6fec4fa3ac9c1e7e7 RIPEMD-160: e9a2dffb1cec1006ebea5ef063328550e5b77476 PGP Signature: qbonkfrom.asc
  File: qbonkns Size: 2,152 bytes Date: 2007-09-25 18:29:59 +0000 MD5: 232cf45bcb91ab25145659a059d51a04 SHA-1: cc7ab64367397a03c59ace906033c941d7b743e9 RIPEMD-160: 1d7e725b87fb425bbf532863c8272caf863bf2c6 PGP Signature: qbonkns.asc	File: qbonkto Size: 1,480 bytes Date: 2007-02-17 08:26:17 +0000 MD5: b1664c7ccd853e4920266f95c7c200d4 SHA-1: 6fb531cc7487327bce6b79c7e4230c01386ec897 RIPEMD-160: 357da32c6e590cac01264ddd2734962b3c995bb4 PGP Signature: qbonkto.asc

• qdomaincheck does a quick sanity check on the domains listed in your control files. It catches common errors like domains missing from the rcpthosts file, or domains listed in both locals and virtualdomains. Running the script gives you a help message, running it with "-a" gives you a list of every domain on the system, and running it with "-c" gives you a list of just the domains which might be causing problems.

  If anybody has ideas about how to make this script better (i.e. other common problems it might be able to check for) please let me know.

  File:	qdomaincheck
  Size:	6,332 bytes
  MD5:	af96f8f75fa94011955a8c78501c6a3d
  SHA-1:	482f9b6eb61ad117ead0dd86f3abac6ce155b35c
  RIPEMD-160:	fa228c8a9d020663d775ae1362498442d2c513a9
  PGP Signature:	qdomaincheck.asc

• qdomto is a script I wrote back when I was running an ISP and had to deal with a queue full of bogus bounce messages. It scans the queue, counts how many messages are waiting to be delivered to each domain name, and prints the totals. It's not somthing I need anymore (thanks to the validrcptto.cdb patch) but if you haven't upgraded and find yourself facing a queue full of garbage, this script may be helpful. The output is sorted alphabetically by the domain name, if you would rather have it sorted numerically (i.e. which domains are the biggest offenders) you should pipe the output through something like "sort -n".

  File:	qdomto
  Size:	1,690 bytes
  MD5:	0abee78b8e3a63cf2272501d0628e111
  SHA-1:	ccafb23a6a3fe3f74e47a005ab33f529a14d648a
  RIPEMD-160:	8dabdcefe4ed03c74bc3be7b0b352fe07eda54a8
  PGP Signature:	qdomto.asc

• qfixpermissions sets the ownership and permissions of all of the standard qmail directories and files to reasonable defaults. Note that before you use this script, you will need to edit the file and set the userids which the qmail-smtpd and qmail-scanner programs run as on your server.

  File:	qfixpermissions
  Size:	7,300 bytes
  Date:	2008-01-01 02:30:02 +0000
  MD5:	b6752534a19d28cff32b684197662966
  SHA-1:	343bda0a17e1c38d5209905128541cc15b56409b
  RIPEMD-160:	77546acb4791b6d4e430a8592dc15c830850c6b7
  PGP Signature:	qfixpermissions.asc

• qkick is a script which runs qmail-tcpok to clear the IP address timeout list that qmail-remote uses to tell when remote machines are non-responsive, and then sends an ALRM signal to qmail-send which forces it to attempt another delivery on every message in the queue as soon as possible. This is useful when you have made changes to the messages in the queue, or if you are recovering from a network-down situation and want to empty the queue as soon as possible. (The script relies on the "killall" command, which is present in Linux- it should be fairly simple to write some kind of "kill -ALRM `ps | grep | awk`" thingy for other operating systems to accomplish the same thing.)

  File:	qkick
  Size:	1,274 bytes
  MD5:	e669143f8c43df66796966ecfd5c51c4
  SHA-1:	612e4d5595de213d8d57ce87b38fbae4e27a6969
  RIPEMD-160:	1459e5a0c93fa0ab38afe5be16f5e690e6163b62
  PGP Signature:	qkick.asc

• qkill removes a given message number from the queue, by manually deleting all files in the queue which are involved with that message.

  qkillfrom and qkillto are extensions of qkill which search for a given regular expression within the envelope sender or recipient list, and removes the messages from the queue. Note that the regular expression IS case-sensitive.

  qkillns is to qkill what qbonkns is to qbonk. It searches the queue for messages with no sender address and removes them from the queue.

  NONE OF THESE SCRIPTS SHOULD BE USED WHILE qmail-send IS RUNNING. I figure as long as I put this warning here, if somebody uses it and it destroys their queue because they didn't stop qmail-send first, it's their own fault.

  File: qkill Size: 2,395 bytes Date: 2006-06-29 00:40:03 +0000 MD5: dd932f7f51ae91502a97ac09fcc05042 SHA-1: b8233540903a8861b06037b8c83c7af33f3b6e1e RIPEMD-160: 3bb2256ac4d34e93463d17b13d5609958b3f0d19 PGP Signature: qkill.asc	File: qkillfrom Size: 2,329 bytes Date: 2009-11-30 22:31:52 +0000 MD5: 363f498da5f511c63835e12a5d33e670 SHA-1: b16924319c06d9bec8c65ab0dda7bfeb213f9c85 RIPEMD-160: 8ef2c3517ae6f693265ee1bb5a0eae8be6bc01bc PGP Signature: qkillfrom.asc
  File: qkillns Size: 2,405 bytes Date: 2006-06-29 00:39:22 +0000 MD5: 271f72b2f8be80618f80b0007d715a72 SHA-1: 7b8ef2834832568f5811215ab729e489803d964a RIPEMD-160: 92bae46079d34b7affa4186c861c710e711ab68f PGP Signature: qkillns.asc	File: qkillto Size: 2,346 bytes Date: 2009-11-30 22:31:52 +0000 MD5: 8abf4da2490112b257d27c98e811010a SHA-1: acec22fccdb13facc19f150e06ee800a89e091ba RIPEMD-160: 5ea7f39ea8055368fbf183f62bfc74f8c1e4cc1b PGP Signature: qkillto.asc

• qlanalyze is a quick and dirty log file analyzer. It reads qmail-send and qmail-smtpd logs, either from standard input or from files named on the command line, extracts the date from the timestamp, counts certain text patterns in the lines, and shows a list of per-date totals when it's done.

  File:	qlanalyze
  Size:	6,854 bytes
  Date:	2008-04-22 16:27:44 +0000
  MD5:	1a6bf2f1fe69333dbd3c03760d30a86b
  SHA-1:	01e5dcaa1fa8d25a21c03329d2ee6b11e2735e0f
  RIPEMD-160:	1c23cf434ee5df5077b42d9769832182d0de6303
  PGP Signature:	qlanalyze.asc

  Note that, while this script does work with the logs from my own server, it's not meant as a final solution for anybody. I'm putting it up here as an example of how to write your own log analysis script. Different people use different sets of programs, particularly on the qmail-smtpd side, and will want or need to count different patterns in their own logs.

  Which means that I fully expect people to customize this, and to be quite honest, I'm not interested in trying to fold other peoples' custom changes back into the minimal version here on the web site. If you happen to find any real bugs, that's one thing- but if you figure out how to make it count something else in the logs, I don't really want or need to include those changes on the web site. My intent here is for this to serve as an example, a starting point for people to write their own log file analyzers.

  2008-04-22 Niamh Holding did find one real bug- the line which counts rblsmtpd rejecting messages was only counting "hard" rejects (i.e. 553 responses) because that's how I've always done it, and I had forgotten that it may also do a "soft" reject on a message (i.e. 451) as well. Thanks!

  Niamh also added a few more counters to her version, counting MFCHECK and validrcptto.cdb failures. She has posted her version of the script on her web site. By comparing her version to my original version here, you can get an idea of where to look within the script in order to make your own changes.

• qmail-nospam has been removed from the page. You can achieve the same effect, while using less CPU time, by creating a ".qmail-whatever" file containing a single "#" character. This makes qmail-local consider the message delivered without actually storing it somewhere. Example:

  # cd `vdominfo -d domain.xyz`
  # chmod +t . holds incoming deliveries while you're editing .qmail-* files
  # echo '#' > .qmail-nospam
  # chown vpopmail:vchkpw .qmail-nospam
  # chmod -t . releases the earlier "hold"

• qmail-spamassassin is an old script I wrote to let spamassassin be called on every message as it was being added to the queue. This was before I started using qmail-scanner (which is able to call spamassassin on its own.)

  File:	qmail-spamassassin
  Size:	1,450 bytes
  MD5:	a0ce161bb032b3a5557ca8d32cf318b4
  SHA-1:	f7b4840b6616b73bd61bf183183b43d7f06464d8
  RIPEMD-160:	5f79fd7d3bc798391fb869999db7ea144cc7241f
  PGP Signature:	qmail-spamassassin.asc

• qrdate is a quick little script which shows the output of qmail-remote in chronological order. It converts the dates from "19 Jul 2007" to "2007-07-19" internally, which makes it possible for a normal sort to give you chronological order. The script normally shows the output in the same format that qmail-remote uses, but if you run it with the "-d" option, it will output the dates in the converted format instead.

  File:	qrdate
  Size:	1,981 bytes
  MD5:	f30bbe7a6741822e4a9572cf1701b953
  SHA-1:	5d5e3d953007bff12df5f2862f6596cf5a04afaf
  RIPEMD-160:	e08b4d867d1019c96d7e0b210a79ccd87872edef
  PGP Signature:	qrdate.asc

• rbl-output is used as part of an automatic honeypot, or other system where you want to block connections from a list of IP addresses stored in a database. The script reads a list of IP blocks from a database and generates output in one of several formats, depending on what mechanism you use to reject connections.

  The script is documented on the honeypot page.

  File:	rbl-output
  Size:	10,909 bytes
  Date:	2007-11-12 02:19:14 +0000
  MD5:	6e5aeddab7760ead7285a980f2e84706
  SHA-1:	19b224a71839af4f3f7fef1b85f25f3da88b57c7
  RIPEMD-160:	be0b8d2cc833386219163712b90acc72ea89636a
  PGP Signature:	rbl-output.asc

• rblstat is a script which reads your dnscache log files, looking for queries relating to specific RBL domain names. If you want to use this, you should set the @rbls array to contain the exact list of domain suffixes you use with rblsmtpd.

  File:	rblstat
  Size:	2,692 bytes
  MD5:	db7d3375814f9d64f7b7bc3d64140d7a
  SHA-1:	7450cfdd744f0e06ad0e648708d56359cf3f19eb
  RIPEMD-160:	6ecb5dfadef6b13d050f044594f1f2f407a1d9e8
  PGP Signature:	rblstat.asc

• report-spam is used as part of an automatic honeypot, or other system where you want to block connections from a list of IP addresses stored in a database. The script reads a message from standard in, finds the IP address which sent it to your server (bypassing the Received: headers of any "trusted" servers) and adds the IP to an "rbl" table.

  See the honeypot page for more details.

  File:	report-spam
  Size:	13,359 bytes
  Date:	2007-11-15 02:36:33 +0000
  MD5:	059726f1cdfdf4244fb3181475bc88c8
  SHA-1:	748c26ff7b660a49d5dca564ff5a029946d446a2
  RIPEMD-160:	c4e6588c068c9668445feef7b0634f5cb59c42ed
  PGP Signature:	report-spam.asc

• smtpcheck is a wrapper around the tcprulescheck program. It checks one or more IP addresses against your /etc/tcp/smtp.cdb file, telling you whether a connection from that IP would be allowed or denied, and if it is allowed, what environment variables (if any) would be set for such a connection.

  If your tcpcontrol file has a different name or is in a different location, you should edit the script accordingly.

  File:	smtpcheck
  Size:	1,735 bytes
  MD5:	39a2dd83f6d7f6c65c3c412b1dda0f83
  SHA-1:	2dbdf17ed68db2d91e626315abe3606e22427335
  RIPEMD-160:	917a5906c2d62ff4e4fbd4f8ee78d6bd877206cf
  PGP Signature:	smtpcheck.asc

  Note that the script assumes you have djbdns installed on the system- it uses the "dnsname" command to reverse-resolve the IP address into a name, just as tcpserver would do before performing its check. If you haven't already done so, you should install the djbdns package, even if you don't plan on running a dnscache or tinydns server.

  $ cat /etc/tcp/smtp
  127.:allow,RELAYCLIENT="",SPFBEHAVIOR=0
  10.:deny
  
  $ smtpcheck 127.0.0.1 10.1.2.3 192.168.1.5
  (checking 127.0.0.1)
  rule 127.:
  set environment variable RELAYCLIENT=
  set environment variable SPFBEHAVIOR=0
  allow connection
  
  (checking 10.1.2.3)
  rule 10.:
  deny connection
  
  (checking 192.168.1.5)
  default:
  allow connection

• ss and sss are a script which shows the status of all services running under daemontools, in a format which is a bit more human-readable.

  And no, that's not a grammatical error. It's one script with two names- it adjusts its behaviour based on what name it has when it runs. The normal way to install this is to place it somewhere with the name "ss", and then create a symlink or hardlink called "sss" which points to "ss".

  File:	ss
  Size:	2,863 bytes
  MD5:	55b67aa78165ebcda298e217134c207c
  SHA-1:	1442af569bd6780c4a8e61051e524c2aa33f6e09
  RIPEMD-160:	d77857ed03ecae19185b9a0ca62139f745308ef0
  PGP Signature:	ss.asc

• vfixpermissions does what I call "the standard vpopmail permissions fix". It resets the permissions and ownership of the vpopmail directory, including any mailboxes it contains. The only thing it doesn't handle is that if you create domains with custom directories and UID's (i.e. the "-u", "-i", "-g", and "-d" options for vadddomain) it won't handle those custom settings correctly- but since I've never heard of anybody using them, I'm not too worried about it. Let me know if there's a need for it, and I'll find a way to add it into the program.

  2006-03-21 The script now fixes the permissions of any "parents" of the vpopmail user's home directory. It simply adds the "x" permission to the parent, the parent's parent, and so forth... out to eight levels "up" in the tree. If the full path to vpopmail's home directory ismore than eight layers deep (it has more than eight "/" characters in it), you may need to add enough lines to the script to reach all the way back to the root directory. And if the directory is less than eight layers deep, the extra lines at the end are harmless- the ".." entry in the root directory of the system points to itself, which means you end up adding the "x" bit to the root directory a few times instead of just once... which is harmless.

  2007-10-05 I have changed the script so that you need to specify the "-s" option if you want to make vchkpw setuid/setgid. This is only needed if you're using vchkpw to validate SMTP AUTH commands within qmail-smtpd. I don't recommend you do this anymore, I now recommend the AUTH_CDB patch to validate AUTH commands from a .cdb file.

  2007-12-13 Fixed a bug in processing the "-s" option. Thanks to Ingo Claro for pointing it out.

  File:	vfixpermissions
  Size:	5,619 bytes
  Date:	2007-12-14 01:51:42 +0000
  MD5:	2e3b738af1f4cdb8a79a9112e82a49e8
  SHA-1:	d710ae369bf5a78cc87a379a9c28503aeca0dec9
  RIPEMD-160:	8eb6b78a5dbb8458961cbf4fc156b6e1e9270b31
  PGP Signature:	vfixpermissions.asc

• Here is a sed one-liner which fixes the "errno problem" in all of djb's packages. The command is...

  $ sed -i '/extern int errno/{s/^/\/* /;s/$/ *\//;G;s/$/#include <errno.h>/;}' error.h

• fake-pop3 is a quick and dirty script I threw together, based on a question which appeared on the dovecot mailing list. Somebody needed a simple POP3 server which would accept any userid and password, and make the MUA believe that the mailbox was empty, in order to reduce the number of tech support calls they receive while working on a mail server.

  This reminded me of the fake SMTP conversation which rblsmtpd gives to clients which aren't authorized to connect to the real qmail-smtpd procss. I wrote the same thing into my jgreylist program, so I figured it couldn't be too hard to do the same thing, but using POP3 instead of SMTP. This script literally took fifteen minutes to write and test.

  File:	fake-pop3
  Size:	1,931 bytes
  Date:	2008-08-30 22:57:36 +0000
  MD5:	744db5be469d0c46a69fae9a0fab79d7
  SHA-1:	b514fe7c9125306aebe1f78cf60064a519058f2d
  RIPEMD-160:	92ec709ead3d0d1993553b28888ac30bd6bffdbc
  PGP Signature:	fake-pop3.asc

• recordio-split is a quick and dirty script I threw together, based on a question which appeared on djb's qmail mailing list. Somebody needed to split the log entries created by "recordio" into separate files, based on which SMTP transaction it was relating to.

  The PID is on each line, which allows us to correlate the lines by message- all they need is a script to do the correlation. That's easy enough to write, so here it is.

  File:	recordio-split
  Size:	3,162 bytes
  Date:	2008-09-17 19:51:18 +0000
  MD5:	e2dcca58358277b8771cce2d821aeef4
  SHA-1:	4bbb2608ea515090195e3bf4769a80fbf1bf02c0
  RIPEMD-160:	c61ba6ecf649ce6719aac1e54c8a93ff95d6bc23
  PGP Signature:	recordio-split.asc

• expunge-mailboxes is a script I wrote years ago, and had never thought about until there was a question about it on the mailing list.

  Under IMAP, when you delete a message, what you're really doing is setting a flag on the message which says "this message has been deleted". This does not actually delete the message. This allows you to "un-delete" messages if you delete them by mistake. Most clients hide these messages from view, but some clients will continue to show them, usually with a horizontal line through them (one of the few things Outlook actually got right.)

  The term "expunge" means to physically remove the messages which are flagged for deletion, so they cannot be "un-deleted" and are no longer taking up disk space on the IMAP server.

  Some clients handle this differently- some will automatically expunge a folder when you change your view to a different folder, some will expunge the current folder (or the main INBOX folder) when you exit the program, some have a menu command or toolbar button to manually expunge the current folder, and some just don't bother with it.

  Courier-imap has a setting which can automatically expunge certain folders. However, there are a few problems with it:

  • It only works for folders which are explicitly named in the config file. If a user creates their own folders and a machine administrator doesn't add those folder names to the config file, they will not be automatically expunged.
  • The expunge only happens when an IMAP client logs into the mailbox. If somebody deletes a bunch of messages and then never logs back into the mailbox, they just sit there.

  This script physically goes through the vpopmail mailbox storage area, finds any messages which are flagged for deletion AND which haven't been touched in seven days, and physically removes them.

  Note that it also prints the filenames while removing them- the idea is that, in a cron job, you can run this script and pipe the output through "wc -l" to tell the machine administrator (or whoever reads it) how many messages were deleted. If you don't want this output, you can redirect it to /dev/null in your cron script (or you can edit this script itself, obviously.)

  File:	expunge-mailboxes
  Size:	1,545 bytes
  Date:	2008-10-21 19:34:25 +0000
  MD5:	54160fa67058ed6cbb6d31c871fc3b94
  SHA-1:	a421c3309369edc43718846f4a89ef279ecf6c5b
  RIPEMD-160:	0dd18ad3feb9551af7fd9c31928cbd8b23e3eb61
  PGP Signature:	expunge-mailboxes.asc

• log-qmail-inject is another one of those scripts I wrote a while back and then forgot about because I hardly ever use it any more. It's a replacement for qmail-inject which can be used on systems which have a compromised or malicious program originating messages, such as poorly written scripts on a web server. It logs (hopefully) enough information to identify the script, and then exec()'s the real qmail-inject.

  Note that on Linux systems, it will also log the chain of parent processes, all the way back to the init process. It does this by using entries in the /proc filesystem. I don't know how to trace a chain of parent processes on other systems, and I don't have any of those other systems to play with to figure it out, so if you're not using Linux, your log files won't have the parent process listing.

  To use it:

  • Save this script as /var/qmail/bin/qmail-inject.log, with the same ownership and permissions as the normal qmail-inject binary.

  • Rename your qmail-inject binary to qmail-inject.real.

  • Create a symbolic link from qmail-inject which points to qmail-inject.log.

  Once you have done this, any process on the machine which uses qmail-inject directory, or indirectly using the sendmail wrapper program, will have its information logged.

  You probably don't want to leave this running all the time, especially on a system which originates a lot of mail. There are two options for removing it:

  • Remove the qmail-inject symlink and replace it with a symlink to qmail-inject.real. This allows you to enable and disable logging at any time in the future. (This is how I've always done it on my own servers.)

    # cd /var/qmail/bin
    
    To DISABLE the logging, point qmail-inject to the real binary.
    # rm qmail-inject ; ln -s qmail-inject.real qmail-inject
    
    To ENABLE the logging, point qmail-inject to the logging script.
    # rm qmail-inject ; ln -s qmail-inject.log qmail-inject

  • Remove the qmail-inject symlink and rename the qmail-inject.real binary back to qmail-inject. Once this is done, you can remove the qmail-inject.log script there if you like, although it shouldn't hurt anything to leave it there.

  File:	log-qmail-inject
  Size:	2,428 bytes
  Date:	2009-12-14 18:34:06 +0000
  MD5:	6e029b044d1141d13e0e710db0e9f02a
  SHA-1:	8c08740a4bd6f691795171f0e3c8270b380c3289
  RIPEMD-160:	08f41b57563b3bb93745ac43489b8182b2eef4d5
  PGP Signature:	log-qmail-inject.asc

• kill-qmail-smtpd-zombies ... I've seen cases where qmail-smtpd will "hang", and processes will stick around for more than a few minutes... in some cases, several days. This script looks at your system, and if it sees any qmail-smtpd processes which have been running for more than one hour, it kills them.

  This originally said "one day". Thanks to John Halladay for pointing out the typo.

  Note that this script depends on the format of the output of the "ps" command. It works on CentOS 5, so I'm pretty sure it will work on any Linux machine, however you may want to test it first:

  $ ps ax -o etime,pid,comm
  You should get a list of times, process IDs, and process names.

  File:	kill-qmail-smtpd-zombies
  Size:	230 bytes
  Date:	2010-02-07 07:01:28 +0000
  MD5:	4ea9ffda6d5bee4b6488004a599e4a8b
  SHA-1:	44041cb4d6be814e2b11bc87450f5421bbc147e5
  RIPEMD-160:	baf40cf9ae85e7115521d08d8faa3b579317d365
  PGP Signature:	kill-qmail-smtpd-zombies.asc

Copyright 1995-2024 John M. Simpson KG4ZOW (email) (PGP key)
Last updated 2012-08-25 03:13:18 +0000