http:// qmail.jms1.net / starttls.shtml

Issues with STARTTLS

Some people seem to have trouble understanding how the STARTTLS command works, and how it interacts with other things like the recordio program, or programs like the "greylite" greylisting program, which works by monitoring the data transfer between the SMTP client and server.

STARTTLS is an extension to the SMTP protocol. It is officially defined in RFC 2487. It's not very difficult to understand- it looks like this:

Client sends --->

EHLO hostname





STARTTLS

nôðûCF ~!ãP‰ ƒu¶ 1!Ž„)âh% RïÐ9÷èIªQé
´>€‰¹·" · YPY IÈLé\ ¼ß“½BHºº i(ø÷p„OÀ



r©>Š¨,‰œ ³CzoÂÙÛW y¡5–Œ˜åòEH â[È•KøŽÞ Í
q/­;T¨Z VƬÖmë ¡‹ 1<^ ªñK¸à *u–²Èzuíî
Ž´æ àÁSØÓ HBPAcE4, íD6¼‹1ÿ~ü¿ ©½Ï¶Ría


...
<--- Server sends
220 server.xyz NO UCE

250-server.xyz NO UCE
250-STARTTLS
250-SIZE 0
250-PIPELINING
250 8BITMIME

220 Ready for TLS...


[ôj/Šÿý Ý´Ï„ å:Ó3 Ÿ8d£è›[8 v€\6EÙzQ
ª8ð B· Ú} Ï5¡ÏÌí±¦f G.·oº{X1Æ Ój1kÁŽ‰Rá¥
°ºÉr ¢³Qî $ÝïakΣR l ˜†Æ™Ø•G š—Vü[ö½šµ³



¾ ›"Ó õÙ&ï îÙW­¿.è¿´ 'ö8zIÈ ©6 ÆóñŽK~Nî{Ÿ
‰`&Ÿ¥ˆëÆ U(0ÔàE¾®ê <ˆ¾‹–P¯0 ÜáŸM”Í Ýix
...

As you can see, after the client sends "STARTTLS" and the server responds with a 220 message, everything after that is encrypted.

What this means is that programs like "greylite", which work by monitoring the data stream between the client and server, will not work if the client and server are using STARTTLS to secure the connection.

I should explain further: now that I've actually tried it on a test SMTP service on my own server, I can tell you that greylite "works", in as much as the connection goes through un-hindered (greylite dumps the connection into a binary pass-through mode after the server's response to the STARTTLS command.) However, greylite is not able to inspect the MAIL or RCPT commands within a STARTTLS connection, and is therefore unable to do the job for which it was written. My apologies to anybody who may have been confused by the language which was here before.

It also means that, since the encrypted data stream is just as likely to contain a "0x00" byte as any other, and the C language treats the zero byte as an "end of string" marker, programs such like "recordio", which involve themselves in the data stream between the client and qmail-smtpd, are likely to become confused.


greylite

You may have noticed me mentioning a program called "greylite" a few times above. This isn't because I'm a big fan of the program- in fact, when I first wrote this page, I hadn't even tried it.

Bugs

I have since tried it, and I've found a few bugs in it- nothing really serious, just a few things which the author probably didn't think about while writing it. (From greylite version 2.3)

I am looking at the program with an eye towards fixing the problems and using it on my own server. I will detail any further bugs I find on this page.

The author's slander

Apparently, at some point in the past, the author of greylite received error reports from two different "qmailrocks" users, who were having trouble with greylite, when "used with the JMS patch". His response was to post this on his web site:


From http://mij.oltrelinux.com/net/greylite/doc/faq.html, 2008-03-08 0500 GMT

When I first emailed him about the statement on his web site, his response tried to claim that he wasn't blaming my patch for the problem. Read the text above, and you decide if you believe this or not.

I don't really have a problem with that, however... the facts speak for themselves. What I have a problem with, is the fact that he added this to his web page, WITHOUT HAVING EVER CONTACTED ME. He never asked me for any help in resolving the issue, he never even contacted me to TELL me that there was an issue- he just put a notice on his web page which recommends that people avoid my patch.

I have asked him three times why he never contacted me about it. He refuses to answer the question. The only conclusion I can draw is that he is more interested in pointing fingers at somebody else, than he is in figuring out what the real problem is, and providing his users with a solution to their problems.

In fact, the original version of the web page you are reading right now was written, at least in part, to illustrate the real cause of the problem that his users (at least, those of his users who are also qmailrocks users) were reporting.

From what it says on his web site, it looked to me like the problem was with the STARTTLS command itself. I assumed that's what it was, because recordio has the same issue.

However, I was wrong about that. He says, and I have verified in the code, that greylite handles the STARTTLS command by dumping the connection into a binary pass-through mode (the "proxy_data()" function,) and before this check is done, if the server had responded with an error message (status code starting with 4 or 5) it loops back to wait for the next command.

So it doesn't look like that was the cause of the problems.

He also states...

On Mar 9, 2008, at 08:00 , John Simpson wrote:
> ...
> please explain what you mean by the term "qmailrocks plain"- are
> you trying to claim that there's a version of qmailrocks floating
> around which doesn't have my patch as part of it? if so, send me
> the URL where i can see it- because both the "stable" and
> "development" releases on http://www.qmailrocks.org/ COME WITH MY
> PATCH PRE-INSTALLED.
> ...

I don't use qmailrocks thus I could only rely on user feedback. As
per the investigation, you know that it is very difficult to drive
people to do it, especially when they are not proficient with the
software. In my about 20 emails with 2 different people showing the
same problem, I obtained that qmailrocks patched with your patch shows
the behavior, qmailrocks in which your patch is not applied does not,
and qmailrock + your patch contacted via tcp does work again.

Unless I'm missing something here, he's admitting that he relied on bug reports from people who "are not proficient with the software" (which is about 95% of qmailrocks' users.) He also admits and demonstrates that he knows nothing about qmailrocks, and obviously he knows nothing about my patch.

And yet somehow, this clown feels justified in giving people advice about my patch?

I have explained to him that qmailrocks was built around my combined patch, that there is no such thing as "qmailrocks in which your patch is not applied", and that because of this, the original bug reports were probably wrong. However, he continues to use the same excuse, over and over again- which only demonstrates that he's either ignoring me, he doesn't want to admit that he made a mistake, or he truly doesn't understand that he's wrong.

In any event, he's making himself look like an idiot.

Of course, since he didn't tell me about the original problems when they happened, and doesn't seem to want to give me any details about them now, I don't have any way to know what the original problems were. I can, however, tell you that I have tried greylite on my own server, with version 7.05 of my combined patch, and it does work.

I suspect the real problem is that the people who reported these issues had already installed greylite and had it working, and then tried to upgrade to a current version of the patch without understanding the issues involve with upgrading... and instead of reading my web site for help, or emailing me directly to ask for help (I would have directed them to my web site), or even asking for help on the qmailrocks mailing list (where other people would have directed them to my web site), they decided to ask the author of greylite- and rather than trying to understand the real problem, he took them at their word (even though he admits that he knew they were "not proficient with the software") and blamed it on my patch.

All of which leads back to the main problem- he never contacted me about the problem at all.

STARTTLS is useless?

I should also point out that the comment "the STARTTLS command is useless after all" on his web page, is totally wrong. I'm sorry, but I have a client (an ISP) with several thousand customers who would probably disagree with you. Their MUAs don't support any kind of encrypted SMTP other than using STARTTLS, and my client (the ISP) wisely refuses to allow the AUTH command on a channel which is not encrypted.

In fact, the opposite seems to be true- not only for SMTP, but for several other protocols as well (POP3, IMAP, XMPP, LDAP, etc.) STARTTLS even has its own RFC, which is part of the "standards track", meaning it is recognized as the best way to do the job. Just because there are other ways to do encrypted SMTP (i.e. running normal SMTP within an SSL tunnel on port 465, which was being done before STARTTLS existed, and is still being done by many people today) doesn't mean those ways are necessarily better. (For the record, my own server supports BOTH ways, STARTTLS and SSL-SMTP.)

I honestly have no idea why he would say this, unless he has some personal stake in seeing people not use STARTTLS. Maybe the fact that it prevents greylite from working?

Conclusion

He didn't see fit to tell me when he felt there was a problem with my code, and he obviously doesn't want to talk to me about this in a mature or professional manner, so to put it very bluntly, I give up. Some people just plain don't want to learn, and don't care about bad-mouthing other people. If he happens to visit this web page and find the bug reports above, more power to him.

And that's really sad- because, other than the minor problems I've found so far, greylite seems to be a pretty good program. I like the idea of applying different rules based on the country in which the IP address lives, for example. I'm thinking seriously about using it on my own server, instead of my own jgreylist program (which only looks at the source IP, not the sender or recipient.) I will, of course, fix the program if I do so.

Either that, or I'll just write a new greylisting program, which uses the "monitor the packets" idea, but which IS compatible with the STARTTLS command (if qmail-smtpd is using my combined patch, of course.)

Advice

If you are using my patch because it was included as part of qmailrocks, and you haven't done so already, you should seriously look into upgrading to a current version of the patch and away from the garbage that the qmailrocks installed on your machine.

And yes, if you think that upgrading will be "hard", then that's probably because you don't understand how qmail works in the first place, and in that case you're right- it WILL be hard.

However, if you're willing and able to spend the time and effort to learn how qmail was designed to work, and you are familiar enough with *nix system administration to be trusted with running a server in the first place, then you CAN upgrade to a much more secure server- and by the time you're finished, you will have a much better understanding of how your server works... which is a good thing to have, when your server breaks.

And regardless of how you added STARTTLS support to your SMTP service... be aware that using recordio will cause the connection to break soon after the STARTTLS command has been sent and accepted, and greylite will not be able to read the sender and recipient from the connection and therefore will not be able to do its job.

If you are using my patch, version 5f or later (qmailrocks uses version 5, you should upgrade), creating a "DENY_TLS=1" envirionment variable will make qmail-smtpd not advertise or accept the STARTTLS command. You may want to do this for any SMTP service on port 25 with which you plan to use recordio, greylite, or any other program which involves itself in the connection.

And if you're setting up an AUTH-only STARTTLS service on port 587, you don't need to use a greylisting program to begin with- even if the spammers DO connect to your port 587, if they don't have a valid ID and password for your server, they won't be able to send mail (and if they DO have a valid ID and password, you have bigger problems.)