![[hacker emblem]](/hacker.png)
![[donate]](/donate.png)
Copyright
1995-2024
John M. Simpson KG4ZOW
(email)
(PGP key)Last updated 2012-07-24 07:17:12 +0000
This page documents how to install the qmail rules interface system on your own server.
The web service's CGI scripts and the email processing script are all written in Perl. I tried to make the code as clear as I could, and have included enough comments that anybody who is at least halfway familiar with Perl should be able to read and understand them. There is also one simple shell script, used for setting the permissions on the files.
There are a few things which will need to be done on the server ahead of time in order to set up this system.
Database engine. When I wrote the system I was using PostgreSQL, however I tried to not make the system dependent on any PostgreSQL-specific features (such as the inet data type.) I haven't tested it yet, but the system should work with any database engine for which there is a Perl DBI module.
Apache. I recommend you have an SSL-secured site, since the login process involves the browser sending passwords to the server. The rest of these directions will assume that you have a working web site, running under Apache. It may be possible to run this system under a different web server, so long as it supports standard CGI scripts, however I have not tested this.
Configure apache to run .cgi files as CGI scripts. Your httpd.conf file may need to be changed. Look for (and possibly fix) the following items:
The mod_cgi module needs to be loaded. This must be done in the httpd.conf file (or in a file that it <Include>s), in the main server config, outside of any <VirtualHost> blocks.
LoadModule cgi_module modules/mod_cgi.so
Look for any AllowOverride directives in the main httpd.conf file (or in files that it <Include>s) which might prevent the "Options +ExecCGI" line in the .htaccess file in the application directory from working. You may wish to add a block like this, just to be safe.
<Directory "/www/secure.domain.xyz/docs/rules">
(The directory where the system is installed)
AllowOverride All
</Directory>
Note that if you do this, it must be in the main httpd.conf file. This cannot reliably be done in a .htaccess file, since another directive in a higher-level .htaccess file or in the main httpd.conf file can prevent it from working.
The cgi-script handler needs to be associated with the .cgi file extension. This is done in the .htaccess file which is part of the download, but could also be done within a <VirtualHost> or <Directory> block in the main httpd.conf file.
AddHandler cgi-script .cgi
The DirectoryIndex line needs to have index.cgi as one of the choices, preferably the first choice. This is done in the .htaccess file which is part of the download, but could also be done within a <VirtualHost> or <Directory> block in the main httpd.conf file.
DirectoryIndex index.cgi index.shtml index.html index.html.var index.txt
The directory needs to have permission to run CGI scripts. This can be done in a "<Directory>" block in the main httpd.conf file, but I find it's easier to do this in a .htaccess file.
Options +ExecCGI
If you made any changes to the httpd.conf file, you should restart Apache. You do not need to restart Apache if you changed the .htaccess file.
Perl, and a few modules from CPAN. The following modules are needed:
The standard modules are included with the normal Perl distribution. The others may be available as OS packages (i.e. CentOS has "perl-DBI" and "perl-DBD-Pg" modules) or they can be installed from CPAN, the Comprehensive Perl Archive Network.
You can check whether or not each module is installed by trying to include it in a perl command. For a module which IS installed, your script should run without any problems:
$ perl -MCGI -e 'print "ok\n"'
ok
For a module which is missing, you will see an error messge:
$ perl -MDBI -e 'print "ok\n"'
Can't locate DBI.pm in @INC (@INC contains: ...
After installing any modules, you should also make sure that the permissions within the Perl module directories are correct. My pfix script may be helpful for this.
qmail with Jay Soffian's RCPTCHECK patch applied. This patch is part of my combined patch, version 7.07 and later. It may also be part of some of the other combined patches out there (I haven't looked at them in a while, so I don't know for sure which ones do and don't include it.)
vpopmail, with a vpopmaild service which is accessible from the machine where the web interface will be running (i.e. no firewalls blocking access.) The web interface connects to vpopmaild in order to validate users' passwords and find each user's access level.
My own vpopmaild service runs under daemaontools, on the same machine as the web interface and the mailboxes. It listens on 127.0.0.1, so it cannot be reached from other machines. This page explains how to set up a vpopmaild service under daemontools.
Once you have verified these items, you should be ready to start installing the system.
There are two ways to download the system.
Download a tarball for the current release, and expand that into place on your web site.
| File: | qmail-rules-136.tar.gz |
| Size: | 33,379 bytes |
| Date: | 2012-07-24 07:17:12 +0000 |
| MD5: | 63367c48d3bb119dc7de33555102a26c |
| SHA-1: | 62959d2f200d5af4f08e2a10696cefbd7ad2aec7 |
| RIPEMD-160: | 552e4fd5cf0df2f3e4d477deef6a3eb02a349354 |
| PGP Signature: | qmail-rules-136.tar.gz.asc |
To expand the files, create the directory within your DocumentRoot where you want the system to live, and untar the file there.
$ cd /www/secure.domain.xyz/docs
(the web site's DocumentRoot)
$ mkdir rules
$ cd rules
$ tar xzf (wherever)/qmail-rules-x.x.tar.gz
Check the system out of my subversion repository. This will result in you running the latest bleeding-edge version of the code, including any changes which I may be working on but haven't released yet. You have been warned.
$ cd /www/secure.domain.xyz/docs
(the web site's DocumentRoot)
$ mkdir rules
$ cd rules
$ svn co https://secure.jms1.net/svn/rules/trunk .
(Don't forget the extra "." at the end of this command!)
Whichever option you choose, there are two more things which need to be done before you can use the system:
The first thing is to copy the .htaccess.dist file to .htaccess. This is because certain things about the web site (i.e. how to reach the database, how to reach vpopmaild, and a few tweaks on the visual appearance of the site) are configured using SetEnv lines in the .htaccess file, and I don't want updates (whether via tarball or subversion) to overwrite your configuration.
The other thing is to run the fix-permissions.sh script. As the name implies, this will set the permissions on the files so that Apache will be able to read them.
If you are using the subversion repo, any time you do an svn update command in the future, you should run the fix-permissions.sh script again.
$ cd /www/secure.domain.xyz/docs/rules
(the web site's DocumentRoot)
$ cp .htaccess.dist .htaccess
$ sh fix-permissions.sh
At this point, we can start setting up the application itself.
The process of setting up the database will be different, depending on which database engine you're using. Rather than trying to cram it all into one page, I have written separate pages for each database engine on which I have tested the system.
PostgreSQL is the engine I used when writing the system, and which I use on my own server. Because of this, it will probably be the most well-tested version (at least in terms of being tested by myself.)
MySQL is another popular database engine. Many people are more familiar with it. I have tested the system using MySQL as the engine. It was recently bought by Oracle, who claims they will continue to keep it available as open source software, however they have a history of being very protective of their "property", so some people are starting to look for other alternatives.
SQLite is a bit different. It's not a real database "engine", in that there is no server process running on a "server" machine somewhere. Instead, it's a library of functions which interpret most SQL commands, and store the entire database in a single flat file. It doesn't enforce data types nearly as well as most other SQL engines, and it has no concept of a "userid" within the server.
It should be possible to make the system work with other database engines (i.e. Oracle, Sybase, etc.) however I have no way to write or test the code to make that happen. The system uses the Perl DBI interface to connect to whatever engine you're using, so in theory any of the database engines with a DBD module should be usable.
Once the database is set up, you should be able to configure and use the web interface to be used to start creating rules.
I have also written a page which describes the database schema and the codes used within the database.
The configuration of the web interface is mostly done by changing environment variables in the .htaccess file. This file is not directly included with the download, however there is a file called .htaccess.dist which you should copy to .htaccess, and then customize it for your needs.
I did this for two reasons:
A future update might include changes to what needs to be in the .htaccess file, but it might not. I don't want to cause people a bunch of extra work by having every single update overwrite the .htaccess file, and forcing them to re-do their configurations. Future updates might include a change to the .htaccess.dist file, but they will not update the actual .htaccess file that Apache uses.
I use subversion to track the entire contents of the "rules" directory (from which I build the distribution tarballs) on the servers where I've been writing and testing this thing, and I got tired of seeing messages telling me that the .htaccess file had been modified on every machine (they all have different database connection information, different PERL5LIB values, and at some points other changes as well.) I ended up setting a property on the directory which tells the svn command to ignore the .htaccess file itself. (The .htaccess.dist file is tracked normally.)
# cd /www/secure.domain.xyz/docs/rules
(your web site's DocumentRoot)
# cp .htaccess.dist .htaccess
# ./fix-permissions.sh
Once you have created the .htaccess file itself, you will need to edit the file. The variables you can configure are:
PERL5LIB should contain the absolute pathname to the "rules/lib" directory within the web site. This will allow the CGI scripts to find the Rules.pm module without a bunch of ugly pathname mangling.
SetEnv PERL5LIB /www/secure.domain.xyz/docs/rules/lib
G4DBCONN contains a string which tells Perl which DBD module to use, and any parameters it needs (such as a database name, a hostname and port, or a filename, depending on the DBD module you're using to access your database engine.) The syntax depends on the module. For example, this string works for PostgreSQL on my server:
SetEnv G4DBCONN dbi:Pg(PrintError=>0):dbname=rules
G4DBUSER and G4DBPASS contain the userid and password used to connect to the database. Depending on how you set up the database, you may not need these variables. If you don't need a userid and password to connect (which is normal for SQLite, and may be normal if you don't care about security on PostgreSQL or Mysql) you can simply comment out the line(s) you don't need by putting a "#" in front of them.
SetEnv G4DBUSER g4web
SetEnv G4DBPASS web
G4SQLLOG can be set to 1 if you want the script to log every single SQL query. The logs end up in the Apache error_log file for the site on which the software is running. This information can be useful in debugging the system, however it will make your log files much larger than they probably need to be.
VPOPMAILD_HOST and VPOPMAILD_PORT tell the script how to connect to vpopmaild in order to validate logins and find each user's access level. Note that if the script cannot reach vpopmaild, nobody will be able to log in.
SetEnv VPOPMAILD_HOST 127.0.0.1
SetEnv VPOPMAILD_PORT 89
SESSION_TIMEOUT should contain an integer telling how long, in seconds, a web session will remain active without any activity. Whenever the browser makes a request using the session cookie, it updates the session's "last used" timer. If no hits come in before this timeout, the session will be exipred and can no longer be used. If this value is not set, the default is 1800 (i.e. 30 minutes.)
SetEnv SESSION_TIMEOUT 1800
Assuming everything goes well, you should now be able to log in through the web interface and start entering rules.
Enter the site's URL in your browser. Assuming you created a directory called "/rules" on a web site called "https://secure.domain.xyz/", the URL will be "https://secure.domain.xyz/rules/".
You will see a login screen, asking for a userid and password. You should be able to log in using the email address and password of any vpopmail mailbox on the system.
The users' access levels are controlled by vpopmail. You can look at a user's access level using vpopmail's "vuserinfo" command:
# vuserinfo admin@domain.xyz name: admin passwd: $1$xxxxxxxx$xxxxxxxxxxxxxxxxxxxxxx clear passwd: comment/gecos: admin uid: 1 gid: 69632 flags: 69632 gecos: admin limits: has qmailadmin administrator access has system administrator access dir: /home/vpopmail/domains/domain.xyz/admin quota: NOQUOTA usage: NOQUOTA last auth: Sat Jun 30 19:02:16 2012 last auth ip: imap
The "limits:" item shows the status of several vpopmail flags which can be attached to each account. Two of these flags are used by qmailadmin, a web interface which allows mailbox owners to configure some things about their own mailboxes (i.e. their "real name", password, forwarding, and vacation auto-response message) and domain owners to create and administer mailboxes within their domains. These flags control which users have access to higher-level administrative functions.
There are two flags which are used by the rules system. One is called "qmailadmin administrator", which means they are able to administer mailboxes within their own domain. The other is called "system administrator", which means they are able to administer mailboxes in any domain on the system.
By default, any account with the username postmaster will automatically have domain-owner access, regardless of whether the "qmailadmin administrator" flag is set or not. Even if the flag is not set, postmaster accounts always have domain-owner access, both in qmailadmin and in this system. This seems to be hard-coded into vpopmail - as a test I tried removing both flags from a postmaster account, and vpopmaild still told me that the account had the "qmailadmin administrator" permission.
The "vmoduser" command is used to set or clear these flags.
To turn on the "qmail administrator" (domain owner) flag...
# vmoduser -a user@domain.xyz
To turn on the "system administrator" (machine owner) flag...
# vmoduser -S user@domain.xyz
To turn off all flags, including the qmail and system administrator flags...
# vmoduser -x user@domain.xyz
Note that if you change the access flags for a user who is currently logged into the system, they will need to log out and back into the system for the system to realize that a change has been made.
You will probably need at least one account with system administrator (i.e. machine owner) access. If the server isn't handling your own email, or if you don't want to use your own account for this (in order to limit the possible damage in case your own password is compromised) you may wish to add a dummy domain with a name like "domain.xyz" (i.e. a domain with a name which does not, and cannot, exist in the real internet), create a dummy mailbox like "admin" in that domain, give that account domain and system administrator access, and use that account to administer the system-wide rules.
# vadddomain domain.xyz
Please enter password for postmaster:
enter password again:
# vadduser admin@domain.xyz
Please enter password for admin@testdomain.xyz:
enter password again:
# vmoduser -aS admin@domain.xyz
At this point, the database and web interface should be working. You should use the web interface to create any rules you will want to enforce when the system goes live. Note that until you change qmail-smtpd to use the processing script (below), any changes you make will not have any effect on your incoming mail, so you may want to take this opportunity to get familiar with how to use the interface, using mailboxes at all three access levels - a normal user, a domain administrator, and a system administrator.
My suggestions are:
Unless you have a specific reason for changing them, don't change the phase order or the access levels for each phase. It won't hurt anything to change the phase descriptions.
Any RBL-reject and greylisting rules should be in the very last phase, so that users and domain owners are able to override them if they wish.
When you first set things up, add a Debug rule to one of the system-wide phases. It doesn't matter which phase you use, or where it is in the sequence, it only matters that it's in a system-wide phase, so that debugging will be turned on for every message.
Once you have the rules set up the way you want them, the last step is to add the filtering script to qmail-smtpd's SMTP processing. Again, this requires that you have patched qmail with Jay Soffian's RCPTCHECK patch, and that you are running qmail-smtpd from daemontools. The script also requires access to the database, but it does not need to access vpopmaild in order to work.
When qmail-smtpd receives a RCPT command from a client, if the RCPTCHECK environment variable exists and is not empty, it will save some values into environment variables and then run the script. In addition, there are other environment variables which are set by other programs, or by the run script for your qmail-smtpd service.
The script configures itself, and reads the data it needs in order to do the filtering, using environment variables. Those variables are:
G4DBCONN tells the script how to reach the database. It should have the exact same value that you stored in the web interface's .htaccess file. This variable is normally set in the qmail-smtpd run script.
G4DBUSER is the userid that the script will use to log into the database. For an engine like SQLite which doesn't use userids, this variable can be empty or unset. This variable is normally set in the qmail-smtpd run script.
G4DBPASS is the password that the script will use to log into the database. For an engine like SQLite which doesn't use userids, this variable can be empty or unset. This variable is normally set in the qmail-smtpd run script.
G4GREY24 affects how greylisting works. If this variable is set to "1", the greylisting function will only compare the first three octets of the IP address, rather than comparing the full IP address. This can be useful when receiving mail from large networks such as Google, AOL, or Facebook, which use clusters of machines (instead of single machines) to send their outbound mail. These clusters are generally in the same /24 network block. This variable is normally set in the qmail-smtpd run script.
TCPREMOTEIP contains the IP address of the SMTP client (i.e. the machine which is trying to send the message to you.) Note that this variable is set by tcpserver before it executes qmail-smtpd for each connection. Other variables may be set, DJB's web page has a list of them. This variable is set by tcpserver before it executes qmail-smtpd.
SENDER contains the envelope sender (i.e. the argument of the "MAIL FROM" command.) This variable is set by qmail-smtpd before it executes the processing script.
RECIPIENT contains the envelope recipient (i.e. the argument of the "RCPT TO" command.) The script removes any address extensions before processing anything (i.e. if a message is sent to "user-abc@domain.xyz", the script will look for rules owned by "user@domain.xyz".) This variable is set by qmail-smtpd before it executes the processing script.
HELO contains whatever value the client sent as part of its HELO/EHLO command. The script doesn't use this, however if you plan to extend the script or write your own RCPTCHECK handler, this value will be available. This variable is set by qmail-smtpd before it executes the processing script.
In addition, if you are using my combined patch, these variables will also be available:
USE_FD4 (in the combined patch version 7.08 and later) is set to "1" if qmail-smtpd is listening on file descriptor 4 for the script to send a custom error message back for the client. This variable is a flag to tell the RCPTCHECK program that it's okay to send error messages. If this variable is not set, the script will not try to send a custom error message and qmail-smtpd will send the client a generic message instead. This variable is set by qmail-smtpd before it executes the processing script.
SMTP_AUTH_USER (in the combined patch version 7 and later) contains the userid from a successful AUTH command. If the user has not authenticated, this variable will not exist (unless it was already set when qmail-smtpd started, which shouldn't happen.) The script doesn't use this, however if you plan to extend the script, this value will be available. This variable is set by qmail-smtpd before it executes the processing script.
After saving the environment variables, qmail-smtpd executes the program specified in the RCPTCHECK variable. You can set this value in the "run" script for your qmail-smtpd service, you can set it in a tcpserver access control file, or (if you're using my combined patch) you can conditionally set or unset the variable when a successful AUTH command is processed, by setting an AUTH_SET_RCPTCHECK or AUTH_UNSET_RCPTCHECK variable.
The processing script itself is "lib/rcptcheck.pl" in the download. (The ".htaccess" file in the main directory prevents web browsers from being able to access the file.) You can install it by copying or moving the script into /var/qmail/bin if you like having all of your binaries together, or you can create a symlink from /var/qmail/bin/rcptcheck.pl to the script, or you can just point the RCPTCHECK variable directly to the script within the web document root... whatever you are comfortable with doing on your own server. (On my own servers, I created a symlink in /var/qmail/bin pointing to the script within the web site, since the directory on the web site is under subversion control and I found that I wasn't always remembering to copy the new script into place when I updated it.)
The important part is that the RCPTCHECK variable needs to contain the full pathname to the script.
Wherever you put it, you need to make sure that the userid as which qmail-smtpd runs (which is qmaild by default) has permissions to read and execute the script itself, and that it has at least "execute" permissions on every directory from the script back up to the root of the filesystem (i.e. "/var/qmail/bin", "/var/qmail", "/var", and "/".)
Before you can run the script, you will need to configure it. This is done by arranging things so that qmail-smtpd runs with the following environment variables set to appropriate values (see the list above for descriptions of what the values should be.)
Before you activate the processing script, you may want to add a debugging rule to one of your system-wide phases. This will make it easier to see what's going on by watching the qmail-smtpd log file.
Some people like to set environment variables using the tcpserver access control file (i.e. the cdb file that tcpserver's "-x" option points to), by adding environment variables to the entries like this:
If your tcpserver access control file contains a lot of entries, having to add this list of variables to the end of several lines can get very tedious. Because of this, and because a lot of the things for which I used to use the tcpserver access control file can now be done by this system, I prefer to edit the qmail-smtpd service's "run" script to make changes like this, and only use the tcpserver access control file for things which can't be done here (such as setting things like GREETDELAY or SPFBEHAVIOR.)
For example, my /service/qmail-smtpd/run script contains the following lines:
RCPTCHECK=/var/qmail/bin/rcptcheck.pl
G4DBCONN="DBI:Pg(PrintError=>0):database=rules"
G4DBUSER=g4web
G4DBPASS=web
G4GREY24=1
export RCPTCHECK G4DBCONN G4DBUSER G4DBPASS G4GREY24
If you edited the tcpserver access control file, rebuilding the cdb file will effectively "activate" the processing script. Any messages which arrive from an IP address matched by the line(s) you edited will be processed by the rcptcheck.pl script.
If you changed the "run" script, restart the service.
# svc -t /service/qmail-smtpd
After restarting the service, watch the log file for log messages from the processing script. If you see any errors, be ready to un-set the RCPTCHECK variable (and restart the service, if necessary) to pull the script out of the processing stream while you debug the problem.
# tail -F /service/qmail-smtpd/log/main/current | tai64nlocal
At this point, the system should be running. If you run into any problems, please refer to the troubleshooting page.