qmail validrcptto.cdb Patch

I have been maintaining a combined set of qmail patches for the past few years, for use on my own server and for the servers I build for my clients. I found a set of patches that I liked and rolled them into one single patch file, and kept a copy of it on my web site for my own use (to download when installing a server for a client) and put together a small web page with links to the original patches, in case anybody else liked the combination I was using and wanted more information about one part of it.

I was also pleasantly surprised to find that my combined patch file was being used as part of the qmailrocks instructions- it turned out that my combination of patches was exactly what the qmailrocks people were looking for, and I certainly don't mind sharing it with them. In fact, I have joined their mailing list, and the feedback from other people who are using the patches has prompted me to do a lot of updates, this particular patch being one of them.

I was asked about adding a patch to verify the RCPT TO argument, and reject messages which were being sent to non-existent addresses. At first I was not interested in such a patch, because it opens the server up to a harvesting attack- something I used to see happening in my own logs before I started using qmail. However, I figured that if I were able to find something, or write something, with a counter to hang up on obvious harvesters, it probably wouldn't be so much of an issue.

I looked at three different patches which verify RCPT TO before accepting the command, and decided to go with the validrcptto patch. It seemed to be the most flexible of the three- it uses a text file with a list of email addresses, which means that by building the text file however you want/need it, you are not limited to any one mechanism for mailboxes- you can use system accounts, vpopmail accounts (stored in vpasswd.cdb files or in a mysql database), or whatever mechanism your system uses.

However, coming from an ISP background, I questioned the efficiency and scalability of using a text file as the storage mechanism. An examination of the code showed that it was reading the entire text file into memory, one line at a time, and building a hash table of the entries. While the hashing may be a good idea when you are expecting hundreds or thousands of RCPT TO commands over the life of the program, the fact is that each instance of qmail-smtpd normally handles one message- which means that because the entire file has to be read for each message, there is only a very marginal advantage to using a memory hash instead of just doing a linear search of the file.

For a small server, it's not really a big deal... but for a busy ISP's mail server, handling thousands of messages per hour, having to read and hash the entire text file (which could potentially have thousands of lines) just adds up and becomes a drag on the server.

So I thought about how to improve the situation, and the idea of using a cdb file instead of a text file seemed like a natural fit.

At first I made the changes to let it work with a cdb file, and added the counter function to hang up on the caller after ten invalid RCPT TO commands. I figured that some people may want a higher or lower limit, or may not want a limit at all, so I wrote in a check for the VALIDRCPTTO_LIMIT environment variable. If it doesn't exist, it uses a limit of ten. If it exists and contains a number, it uses that number as the limit. Setting a limit of zero disables the count entirely (making it possible to conduct a harvesting attack against the server.)

I tested it on my own server, and then released it as part of my combined patches. Then I discovered that because the patch doesn't correctly handle "-default" aliases, it was preventing TMDA from working correctly. Of course my next step was to add in the logic to handle these "-default" aliases, and release another version of my combined patch.

As part of testing the "-default" code, I added some debugging messages which told what addresses were being searched for in the cdb file. I found that I liked the idea of being able to see the RCPT TO argument and whatever it matched in the database in the logs, so I left the messages in there but disabled them by default. The messages can be enabled by setting the VALIDRCPTTO_LOG environment variable to a non-zero number.

Then I realized that the only way people would be able to use the cdb functionality, or the "-default" alias handling, would be if they were using my combined patch set. Different people have different needs, my patch set isn't for everybody... so I figured it would be nice to build a patch file which just adds the validrcptto.cdb and "-default" functionality, without all of the extra stuff that my combined patch uses.

2007-07-31 I was discussing this patch with Nigel Mundy, and his suggestions led me to the idea that if a server has multiple SMTP services (one for normal port-25 traffic, one for SSL traffic on port 465, etc.) it might be nice to be able to selectively enable and disable the validrcptto.cdb functionality. Taking it one step further, some people might want to use different cdb files for different services.

I now have a "version 2" of the patch. The only difference is that instead of using the hard-coded name "control/validrcptto.cdb" for the file, it now uses the value of a "VALIDRCPTTO_CDB" environment variable. (Reminder- if you add this variable to your SMTP service "run" script, don't forget to export it.)

For those who may be using the validrcptto.cdb functionality as part of my combined patch, this updated version is included in version "6cg" and later of the combined patch.

2012-07-04 At some point, I had added some functionality to the combined patch so that if the validrcptto.cdb entry had a data value starting with "-", messages sent to that recipient would be rejected. This allows you to reject "user-xyz@domain" while still accepting "user-default@domain".

While working on an update for the mkvalidrcptto script today, I noticed that this functionality is missing from the stand-alone patch here.

I've produced a new stand-alone patch which includes this functionality. I have verified that it compiles cleanly, however I have not done any further testing on it, since I don't have access to a machine which actually uses the stand-alone patch. If you download and use this patch, please let me know whether or not it works correctly for you.

In addition, while preparing a patch for netqmail, I found that the new validrcptto.cdb.3.patch applies cleanly to the netqmail-1.06 code as well. Because of this, I am no longer offering netqmail-specific versions of this patch.

Downloads

For people using djb's original source code, and netqmail version 1.06:

If you are building netqmail, the patch is designed to be applied AFTER the ./collate.sh command has been run when setting up netqmail-1.06. The procedure will look like this:

> ./collate.sh

You should see 7 lines of text below. If you see anything
else, then something might be wrong.
[1] Extracting qmail-1.03...
[2] Patching qmail-1.03 into netqmail-1.06. Look for errors below:
    24
[4] The previous line should say 24 if you used GNU patch.
[5] Renaming qmail-1.03 to netqmail-1.06...
[6] Continue installing qmail using the instructions found at:
[7] http://www.lifewithqmail.org/lwq.html#installation

Above is the normal netqmail install procedure. Before you continue, apply the validrcptto.cdb patch like so...

> cd netqmail-1.06
> patch < ../validrcptto.cdb.3.patch

... and THEN continue with the directions on the lifewithqmail web site.

Example

I received an email from somebody who didn't seem to understand how the VALIDRCPTTO_LIMIT feature works... either that, or my explanation above isn't very clear (which is entirely possible- my best language is perl, not english.)

The following example illustrates how the limit works, with no VALIDRCPTTO_LIMIT set (meaning the code will default to using a limit of 10.)

% telnet a.mx.jms1.net 25
Trying 209.114.200.128...
Connected to a.mx.jms1.net.
Escape character is '^]'.
220 a.mx.jms1.net NO UCE ESMTP
EHLO testing
250-a.mx.jms1.net NO UCE
250-STARTTLS
250-PIPELINING
250 8BITMIME
MAIL FROM: <jms1@jms1.net>
250 ok
RCPT TO: <badguy1@jms1.net>
553 sorry, this recipient is not in my validrcptto list (#5.7.1)
RCPT TO: <badguy2@jms1.net>
553 sorry, this recipient is not in my validrcptto list (#5.7.1)
RCPT TO: <badguy3@jms1.net>
553 sorry, this recipient is not in my validrcptto list (#5.7.1)
RCPT TO: <badguy4@jms1.net>
553 sorry, this recipient is not in my validrcptto list (#5.7.1)
RCPT TO: <badguy5@jms1.net>
553 sorry, this recipient is not in my validrcptto list (#5.7.1)
RCPT TO: <badguy6@jms1.net>
553 sorry, this recipient is not in my validrcptto list (#5.7.1)
RCPT TO: <badguy7@jms1.net>
553 sorry, this recipient is not in my validrcptto list (#5.7.1)
RCPT TO: <badguy8@jms1.net>
553 sorry, this recipient is not in my validrcptto list (#5.7.1)
RCPT TO: <badguy9@jms1.net>
553 sorry, this recipient is not in my validrcptto list (#5.7.1)
RCPT TO: <badguy10@jms1.net>
421 too many invalid addresses, goodbye (#4.3.0)
Connection closed by foreign host.

As you can see, the first nine attempts return an error message but allow the client to try again, but the tenth attempt causes the server to disconnect from the client without any further discussion.

Creating validrcptto.cdb

The /var/qmail/control/validrcptto.cdb file should have a key for every valid email address (or alias) on the server. The value attached to the key is ignored. The keys in the file MUST BE LOWERCASED. If you have any domains for which the RCPT TO check should not be done, or for which every email address should be considered valid (i.e. if you have a catch-all address specified) you should list this domain as "@domain.name" in the file.

You can create this file in any manner you choose. You can write a script which generates the list of email addresses, or you can maintain the list as a text file by hand. The cdbmake-12 program, included with djb's cdb package, will convert a text file with one email address (or "@domain" entry) per line, into the proper cdb file format.

Let me say this again: You can create the file in any manner you choose. Just because I wrote a "mkvalidrcptto" script does not mean that my script is the only one that will work- any process which generates a simple list of email addresses, with "@domain" to accept mail for an entire tomain, will work.

And for the benefit of those who need this to be explicitly stated... the final validrcptto.cdb file needs to be placed in the /var/qmail/control directory, and needs to have permissions which make the file readable to the userid that qmail-smtpd runs as.

My mkvalidrcptto script

The mkvalidrcptto script was last updated on 2012-09-16.

I have written a script called mkvalidrcptto, which can be used to generate the required cdb file. It scans your system and generates a list of valid email addresses. It reads system accounts (and matches them with the domain names listed in "control/locals"), including any ".qmail" files in their home directories, as well as virtual domains configured using the "control/virtualdomains" file (as used by vpopmail.) It recognizes vpopmail, and when it sees a virtual domain managed by vpopmail with "bounce-no-mailbox" on the vdelivermail line, it runs "vuserinfo -D ___" to get a list of every valid email address in the domain (which means it even works if you are using vpopmail with mysql.) It also reads the /etc/aliases.cdb file created by the "fastforward" program.

If you are using vpopmail and want messages addressed to a non-existent mailbox to not be accepted, you must use "bounce-no-mailbox" (or "Set catchall bounced", if you use qmailadmin) for the domain. If you use "Set catchall email deleted", messages for non-existent mailboxes will be accepted from the internet and then deleted without being delivered to any mailbox.

The output of this script is plain text, with one email address (or "@domain" entry) per line. This format is suitable for use with the original validrcptto patch (if you happen to be using that version.) It can also be piped through the cdbmake-12 program from djb's cdb package to produce the cdb file needed for my patch.

The mkvalidrcptto web page has examples of how to use the script.

The qmail-updater web page describes a daemontools service which uses mkvalidrcptto to update the validrcptto.cdb file whenever any other process on the system requests it. This makes it possible for vpopmail (when compiled using the "onchange" patch) to trigger a rebuild whenever mailboxes or domains are added, changed, or removed.

Similar Patches

In the interest of completeness, I do need to mention that there are other patches out there which do the same basic job (make qmail-smtpd reject RCPT addresses where the domain is on the local server but the mailbox isn't.) This is a list of some of them, with notes as appropriate:

• chkuser is probably the most well-known patch out there. I personally don't like it because:

  • It only works with vpopmail- if you use system mailboxes, vmailmgr, or some other virtual-domain scheme, you're out of luck.
  • It compiles vpopmail's code into qmail itself, which means that every time you upgrade vpopmail, you have to compile qmail again.

  However, a lot of people are using vpopmail, and never upgrade it once it's installed. If this is you, the chkuser patch may be a better fit for your server, since it doesn't rely on any external process to rebuild a database file whenever a mailbox is added, removed, or changed.

• The qmail-realrcptto patch copies logic from qmail-send, qmail-lspawn, qmail-getpw, and qmail-local into qmail-smtpd, so that if a local delivery (i.e. one for a domain in control/locals or control/virtualdomains) would eventually bounce due to a missing .qmail file, then that recipient address is rejected during the SMTP or QMTP protocol conversation.

  I remember looking at this one a while back, but it turns out that because qmail-smtpd normally runs as the qmaild user, it didn't have permissions to read the .qmail files it needed in order to tell whether an address was valid or not. Some people get around this on a pure-vpopmail system by running qmail-smtpd as the vpopmail user, but I'm not comfortable doing that- it's too easy for a bug in a script to accidentally wipe out other peoples' email.

• Of course I should mention the original validrcptto patch, which was my starting point for this patch. It works in a simliar manner, but instead of using a cdb file, it uses a flat text file- which is okay for small servers, but if you have more than a few hundred mailboxes, the fact that every execution of qmail-smtpd has to read and parse the entire text file into memory will make it run very slowly. It also didn't know how to properly deal with "-default" addresses.

This page has a list of other patches which do recipient checks in a variety of other ways.

I'm sure there are other similar patches out there- if you know of any which are not listed here, please email me at the address at the bottom of the page and let me know.

Mailing List

I am running two mailing lists regarding this patch. One is a public two-way list for discussing the patch, asking questions, reporting bugs, and suggesting new features. The other is a one-way announcement list, where I send out notices about new versions of the patch, as well as any major changes on the web site.

http://qmail.jms1.net/lists/ has more information about the lists.

Notes

Use with qpsmtpd

I hadn't thought about doing this, but it seems the idea of using a validrcptto.cdb file is catching on. Robin Bowes has written a plug-in for qpsmtpd (a replacement for qmail-smtpd written entirely in perl) that checks RCPT TO arguments against a validrcptto.cdb file, just as my patch does for the standard qmail-smtpd program. If you use qpsmtpd, or are just curious about it, this is the web page for his plug-in.

Use with Exchange Server

I have a client who uses Exchange Server to manage their mailboxes, along with a shared calendar and a set of company-wide storage folders. I've been trying to explain to them that this is a bad idea, but their CEO is fully "bought into" the Microsoft propoganda machine- it took the users filling up the MS-Access file(!) which contains all of the messages and stored items, and the entire server being offline for two days' worth of manual repairs, for him to understand that this is not a good idea.

Many people are stuck with a similar situation- they need to set up qmail as a front-end server to protect an existing MS Exchange Server. The normal scenario, and what I'm doing with this one client, is that messages arriving from the Internet are sent to the qmail machine, which does anti-spam and anti-virus checking... and which could also use the validrcptto.cdb mechanism to refuse any messages sent to non-existent email addresses, if there were a way to get a list of the valid email addresses from the Exchange server.

As it turns out, there is a way to do it, for systems where the Exchange server is part of an Active Directory tree. Because AD is actually an LDAP database, and because there is existing code to work with LDAP on just about every operating system on the planet, it's possible to write a script to do a series of LDAP queries and print out the email addresses contained in the entries.

Brian Landers has written such a script, he calls it adexport (local copy.) His script was written to work with sendmail, however because its output is fairly generic, it's very easy to use the same output to build a validrcptto.cdb file for use with qmail.

I have written a small wrapper around Brian's script, called adexport-go. It's designed to be run periodically (i.e. from a cron job, or within a sleep-loop in a daemontools service.)

If the Exchange server is the ONLY source of email addresses for your validrcptto.cdb file, this script can be used to build the actual validrcptto.cdb file that your system uses. Otherwise, you can modify the script so that the list from the Exchange server is added to a list from some other source (the output of "mkvalidrcptto", for example) and that result can be run through cdbmake-12 to produce the validrcptto.cdb file.

2007-11-14 Jason Haar (the author of qmail-scanner) pointed out on the qmail list that giving the qmail server a cached copy of the list of valid email addresses is a HUGE performance boost over some other methods which query the LDAP server in real time, as messages arrive.

Other notes

2005-09-26 I have received an email from Mauro Cicognini, pointing out a typo in the original patch file which was here. The die_vrt() function was missing the flush(); _exit(1); code at the end. This means that when the VALIDRCPTTO_LIMIT has been reached (i.e. 10 invalid recipients) and qmail-smtpd should hang up the connection, the connection does not in fact die.

I have corrected the patch file on the web site, if you downloaded it before this notice was placed on the site, you should download it again in order to get the corrected version.

Mauro- THANK YOU for pointing out the typo. It's not something I would have caught, I don't use the stand-alone patch like this- I actually use my combined patch instead. This was written as part of the combined patch and then "broken out" to become a stand-alone patch.

2005-10-20 I received an email from Patrick (aka "marlowe") which asked for clarification about when the validrcptto.cdb check is done versus when the normal rcpthosts check is done. For people who care about such things, the normal rcpthosts check is done first- if an email arrives where the recipient's domain name is not listed in the rcpthosts or morercpthosts.cdb file, the RCPT TO command will be refused before the validrcptto.cdb file is ever checked.

2006-01-22 I received an email from Carl de Kock asking about how this patch interacts with the RELAYCLIENT variable. To answer the question, if a client is coming from an IP address which has RELAYCLIENT permission, or if they have sent a successful AUTH command during their current connection, the validrcptto.cdb check is bypassed and they can send mail to any address at your local domains, even if they are not listed in the validrcptto.cdb file.

Why is this useful? He wants to have a few "internal only" email addresses, where people within the office can send mail to it, but people outside the office can't. He is using the validrcptto.cdb patch with a manually edited text file, and by simply not adding those email addresses to his validrcptto.cdb file, his server will not accept mail for those addresses unless the sender has RELAYCLIENT or has done a successful AUTH command.

Copyright 1995-2024 John M. Simpson KG4ZOW (email) (PGP key)
Last updated 2012-07-05 00:23:24 +0000