http:// qmail.jms1.net / scripts / grey3

grey3

The grey3 script is my first attempt at a three-tuple greylisting system.

This is not a "finished" product, and unless you know enough about Perl, SQL, and qmail to understand how it actually works, you should not use it on a live server.

Background

The idea behind greylisting is simple- if a legitimate mail server is told that the message it's trying to deliver is being refused, but not permanently, the sending server will try to send the same message again after a certain length of time. A spam-sending program, on the other hand, will usually give up right away, because they have nine million other people to victimize. By deliberately making each NEW sender wait a short while, you end up trimming out a lot of spam. And by remembering the servers which actually DID come back after being told to wait, you avoid having to inconvenience those (presumably legitimate) servers by making them wait again, the next time they have a message for you.

It's not a foolproof method of avoiding spam, but neither is it totally useless. When I started running jgreylist on my own server, the volume of spam did go down- not drastically, but noticeably.

My existing jgreylist script is a one-tuple greylisting system. (The word "tuple" is a database term meaning "piece of information", such as a field.) It works by remembering the IP addresses of the sending systems, and making each one wait a certain length of time, the first time they send a message to you. It's simple, and it does work- although if the sending IP is the mail server of a large ISP (such as "aol.com") you end up accepting mail from ALL of their users, to anybody on your server, after that initial timeout is done. This may or may not be a good thing, depending on your server.

Many people will tell you that a three-tuple system, which works based on the sending IP, the sender's email address, and the recipient's email address, is better than a one-tuple system. I tend to agree, but I can also see where some people, running small servers, might not want to mess with setting up a three-tuple system, especially since jgreylist is there and it's so easy to set up and maintain (no database involved.)

The mechanism which makes jgreylist work, inserting it into the chain of programs which run before qmail-smtpd, cannot be used for a three-tuple system, because when it runs, the sender and recipient addresses haven't been sent yet. The RCPTCHECK patch, added in version 7.07 of my combined patch, adds a "hook" to qmail-smtpd which makes an external program be executed after each RCPT command in the SMTP exchange. If this program also knows the sender address and the client's IP, it can be used for a three-tuple system.


The grey3 script

The grey3 script implements a simple three-tuple greylisting system. It stores the greylist status in a PostgreSQL database, although with a few tweaks it could probably be made to work with other database engines as well.

I used PostgreSQL because (1) it's the one I am most familiar and comfortable with, and (2) it has a built-in "inet" data type for IP addresses and blocks, and the operators to make it very easy to work with IP blocks (for example, find all rows where a certain IP field is within a given netblock.)

Setting up the database

On my server, I created a database called "grey3", and a PostgreSQL user called "qmaild" (which is the userid as which my SMTP services run.) I then ran the following queries to create the tables:

CREATE SEQUENCE greylist_seq ; CREATE TABLE greylist ( id int not null unique default nextval('greylist_seq') , seq int not null default 9999 , sender varchar(250) not null , recipient varchar(250) not null , ip inet not null , black boolean not null default false , white boolean not null default false , first timestamp not null default now() , last timestamp null , count_pre integer not null default 1 , count_msg integer not null default 0 , primary key ( sender , recipient , ip ) ) ; CREATE SEQUENCE log_seq ; CREATE TABLE log ( id integer not null primary key default nextval ( 'log_seq' ) , dt timestamp not null default now() , disp varchar(10) not null , sender varchar(250) not null , recipient varchar(250) not null , ip inet not null ) ; GRANT SELECT,UPDATE ON greylist_seq TO qmaild ; GRANT INSERT,SELECT,UPDATE ON greylist TO qmaild ; GRANT SELECT,UPDATE ON log_seq TO qmaild ; GRANT INSERT ON log TO qmaild ;

After creating the tables, I then installed the "grey3" script itself in the "/var/qmail/bin/" directory, with my other qmail-related programs.

File: grey3
Size: 6,384 bytes
Date: 2008-09-20 01:54:14 +0000
MD5: c425eac433bcfd7a8a46e0cf213c658e
SHA-1: 7d40690e2c61469236d6984e4e6c5a19b90ec9b6
RIPEMD-160: caeebaa4f9a0bb74466b0a04960634fca962251c
PGP Signature: grey3.asc

The last step was to make qmail-smtpd use it. Of course, I was using version 7.07 of the combined patch, so I had the RCPTCHECK functionality available. I added the following to the "run" script for my qmail-smtpd service:

RCPTCHECK="/var/qmail/bin/grey3"
export RCPTCHECK

Then I restarted the service (using the "svc -t" command) and did some testing, while watching the logs. It worked, for the most part.


Issues

The problems I've had with it so far are:


The Future

I am working on a better version of the idea- one which keeps custom rules in a separate table, which is parsed before any greylisting decisions are made. Custom rules will be available to match sender addresses and IP addresses, and I'm working on a web interface which will allow mailbox owners, domain administrators, and machine administrators to all create rules which affect their own mailboxes, all mailboxes in their domains, and all messages on the system (respectively.)

It uses vpopmaild to verify the users' access to the system, AND to tell whether the user is authorized to modify rules for their own mailbox, their domain, or the entire system. If a user is authorized at more than one level (i.e. a domain administrator will be able to write both domain-wide rules AND edit mailbox-specific rules on behalf of their users) there is a mechanism to set the "focus" of the web interface on a specific mailbox, on a domain (where domain-wide rules can be edited, but mailbox-specific rules cannot), or on the entire system (where system-wide rules can be edited, but domain-specific or mailbox-specific rules cannot.)

I'm not ready to release the new stuff yet, but as I make progress with it (it's a spare-time thing, between client projects) I might be looking for people to help me test it. Contact me if you're interested, but be aware that it's not an "open beta" program- I will be looking for people who are already familiar with their systems- I want real bug reports, I don't want to spend a lot of time explaining to a "newbie" how to set it up and get it working. It may also be a month or two before I ask anybody to do anything.