amavisd-new + qmail QMQPqq
Some 'big picture' ;).
I will deal here only with essential things which definitely have to be done in order to run amavisd-new with qmail. However, this WILL NOT assure top performance - please see suggestions later on.
amavisd-new QMQPqq patch is considered to be 'stable'. Its announcement was postponed for 2 months - I want to be sure it runs smoothly. No changes were made since the first version of the patch. Mark pointed out that function 'if_exists_in_array' is not necessary (behaviour similar to grep) - yes, he definitely is right but I don't have time to wipe it out. The patch itself is really trivial so be my guests...
QMQPqq protocol does not exist. This name was chosen by me as qmail uses some simplified QMQP protocol for qmail-qmqpc.
QMQPqq could be easily patched to qmail-smtpd but most users do not need that tight integration and I don't want to bother people with so much patching and compiling. People who really need that are able to patch it on their own (and suffer the consequences if their code is broken).
QMQPqq allows for early rejection of spam/viruses. As qmail-queue/qmail-qmqpc do not know other codes than 'K' (ok), 'Z' (temp) and 'D' (perm) we can't inform smtp clients about the _exact_ reason why their mail was refused (if it was). Such behaviour would need tighter integration with qmail-smtpd and additional patches (which I don't want to post (yet) because of 'see above'). Therefore use of D_REJECT for spam and D_BOUNCE for viruses/banned is suggested.
- qmail
- virus scanner (optional but recommended)
- amavisd-new qmqpqq patch (diff) - to be used with amavisd-new-20030616-p9
- amavisd-new qmqpqq patch (diff) - to be used with amavisd-new-20040701
(note: patch is included in amavisd-new package so use that one if possible)
- operating system of Your choice (FreeBSD, of course :)
- some time (20 minutes should be most sufficient)
http://cr.yp.to/daemontools/install.html
http://cr.yp.to/ucspi-tcp/install.html
Fetch qmail distribution from http://cr.yp.to/qmail.html.
Unpack into temporary directory.
Fetch qmail-queue.patch from the place you love the most or follow the hyperlink to fetch it from my site.
Patch qmail:
patch < qmail-queue.patch
Now we can use different qmail-queue according to the one set in the enviroment (or default if none).
'The qmail system is heavily partitioned for security; it does almost nothing as root', each its part runs as a different user - therefore You will need to create them:
group nofiles
members: alias, qmaild, qmaill, qmailp
group qmail
members: qmailq, qmailr, qmails
File INSTALL.ids in the qmail source directory deals with this problem. For FreeBSD You may use this script (for suffix enter an empty line).
Now create directory:
mkdir /var/qmail
cd to qmail src directory and edit files:
conf-qmail (configuration file for qmail directory):
change directory to reflect Your directory, e.g.:
/var/qmail
This is the qmail home directory. It must be a local directory, not
shared among machines. This is where qmail queues all mail messages.
The queue (except for bounce message contents) is crashproof, if the
filesystem guarantees that single-byte writes are atomic and that
directory operations are synchronous. These guarantees are provided by
fixed-block filesystems such as UFS and by journaling filesystems. Under
Linux, make sure that all mail-handling filesystems are mounted with
synchronous metadata.
conf-groups (configuration file for qmail groups):
change groups to reflect Your groups, e.g.:
qmail
nofiles
These are the qmail groups. The second group should not have access to
any files, but it must be usable for processes; this requirement
excludes the ``nogroup'' and ``nobody'' groups on many systems.
conf-users (configuration file for qmail users):
change users to reflect Your users, e.g.:
alias
qmaild
qmaill
root
qmailp
qmailq
qmailr
qmails
The qmail system is heavily partitioned for security; it does almost
nothing as root.
The first eight lines of this file are the alias user, the daemon user,
the log user, the owner of miscellaneous files such as binaries, the
passwd user, the queue user, the remote user, and the send user.
qmail-qmqpc.c (source for qmail-qmqpc):
change
#define PORT_QMQP 628
to some nonprivileged port (mostly above 1024 - we don't want to run amavisd-new as root), e.g.:
#define PORT_QMQP 10628
Now You are ready to compile and install qmail:
make setup check
After compilation and installation type:
./config
You should see something like this:
Your hostname is some.host.com.
Your host's fully qualified name in DNS is
some.host.com.
Putting some.host.com into control/me...
Putting host.com into control/defaultdomain...
Putting host.com into control/plusdomain...
Checking local IP addresses:
127.0.0.1: Adding localhost to control/locals...
192.168.0.1: Adding some.host.com to control/locals...
If there are any other domain names that point to you,
you will have to add them to /var/qmail/control/locals.
You don't have to worry about aliases, i.e., domains with CNAME records.
Copying /var/qmail/control/locals to /var/qmail/control/rcpthosts...
Now qmail will refuse to accept SMTP messages except to those hosts.
Make sure to change rcpthosts if you add hosts to locals or virtualdomains!
If it prints something like this, You have to setup Your host in DNS correctly:
Your hostname is
some.host.com.
soft error
Sorry, I couldn't find your host's canonical name in DNS.
You will have to set up control/me yourself.
If You
are done, setup alias files (replace 'solko' with your login or mail address).
If the address begins with a letter or number, you may leave out the ampersand
(man dot-qmail):
cd ~alias
echo "&solko" > .qmail-root
cp .qmail-root .qmail-mailer-daemon
cp .qmail-root .qmail-postmaster
cp .qmail-root .qmail-virusalert
cp .qmail-root .qmail-spamalert
Create cdb file for tcpserver:
/etc/tcp.smtp, e.g.:
127.0.0.1:allow,RELAYCLIENT=""
127.10.10.10:allow,RELAYCLIENT="",QMAILQUEUE="bin/qmail-queue.orig"
192.168.0.1:allow,RELAYCLIENT=""
:allow
You will probably want to relay mail for this host as well for some subnet, e.g.:
127.0.0.1:allow,RELAYCLIENT=""
127.10.10.10:allow,RELAYCLIENT="",QMAILQUEUE="bin/qmail-queue.orig"
192.168.0.:allow,RELAYCLIENT=""
:allow
will relay mail for localhost and whole subnet 192.168.0.0/24.
Make it:
tcprules /etc/tcp.smtp.cdb temporary_file < /etc/tcp.smtp
Now setup supervise dirs. Either look at this page or use this script which does almost the same.
Using script for qmail:
chmod 0700 qmail_make_supervise.pl
./qmail_make_supervise.pl
You are now ready to create supervise dirs. First, I kneed to know few things:
Where did You install qmail? (e.g. /var/qmail - default)
/var/qmail
Where do You want to keep Your log files? (e.g. /var/log/qmail - default)
/var/log/qmail
What IP address will Your qmail listen to? (e.g. 192.168.0.1, type 0 (default) for all interfaces)
0
And what port? (e.g. 25 - default)
25
Where is the cdb file for tcpserver? (e.g. /etc/tcp.smtp.cdb)
/etc/tcp.smtp.cdb
What suffix will You use? (e.g. )
Assuming
no suffix!
OK, so You want me to create files as follows:
Directory: /var/qmail
Log directory: /var/log/qmail
IP address: 0
Port: 25
Control acces file for tcpserver: /etc/tcp.smtp.cdb
Suffix:
y/n?
y
Let's have a look at control files for qmail in /var/qmail/control:
locals:
put here Your local domains/hosts or nothing - depends on Your needs.
rcpthosts:
add domains/hosts You wish to accept mail for (one per line).
defaultdomain and plusdomain:
change to Your needs, however, defaults should be ok for a mailgate setup.
qmqpservers:
add IP address of amavisd-new, e.g.:
127.10.10.10
smtproutes:
content depends on Your decision:
1. Will it be a mailgate?
add domain and IP address (optionally followed by a port - defaults to port 25) of the mailhost, e.g.:
domain.com:192.168.0.27:10025
which tells qmail to deliver all mail for domain.com to 192.168.0.27 on port 10025. I recommend setup like this. If You want to relay another domain, just add another record to this file. All other domains will be delivered according to MX lookups.
2. Will it be the mailserver itself?
You do not need to add anything, just adjust locals a virtualdomains and You are done here.
Now cd to /var/qmail/bin directory and do the qmail-queue magic:
mv qmail-queue qmail-queue.orig
ln -s qmail-qmqpc qmail-queue
There are some variations of how one could setup qmail-queue things to achieve the same goal but I decided to post this one for the utmost simplicity. Explanation follows.
Many people want
to scan mail originating from localhost, too. So it's easier to scan
_everything_ by default and run amavisd-new on separate IP address (127.10.10.10 - or choose any other ip which is never ever likely to be listed in some of your favourite ordbs).
When qmail-smtpd sees this IP address it will run qmail-queue.orig and the mail
is queued. Yes, people could still send unchecked mail via 127.10.10.10... but I am
not that paranoid. (note: when linking qmail-queue to qmail-qmqpc local mail forwarded by qmail-local will be scanned twice as when qmail-local encounters forward instructions it will reinject mail back to qmail-queue which happens to be linked to qmail-qmqpc (thanks to Budi Ang for pointing this bug out - I posted the solution some time ago at amavis mailing list but since I was asked to include it here as well here it is:
edit qmail-local.c,
find void mailforward(recips)
and right BEFORE the if (qmail_open(&qqt) == -1) temp_fork(); add this line:
if (!env_put("QMAILQUEUE=bin/qmail-queue.orig")) temp_nomem();
or something simillar which suits your needs))
Now don't forget to add IP address 127.10.10.10 to your loopback interface (and don't forget to include it in your startup).
On FreeBSD one could use:
ifconfig lo0 inet 127.10.10.10 netmask 255.255.255.255 alias
and edit rc.conf:
ifconfig_lo0_alias0="inet 127.10.10.10 netmask 255.255.255.255"
This should not be a very difficult task, right? Just a 'seek and destroy' mission ;). But be sure to kill the right process and disable proper startup file. Differs from system to system.
Don't forget to adjust mail wrapper (/etc/mail/mailer.conf in FreeBSD) or something like that to be able to send mail from this host.
Accepting mails
from openrelay server is probably not a good idea so You _would_ like to use
this. I know You would, trust me. Edit Your /var/qmail/supervise/qmail-smtpd/run
file and add rblsmtpd, e.g.:
before adding rblsmtpd:
#!/bin/sh
QMAILDUID=`id -g qmaild`
NOFILESGID=`id -g qmaild`
exec /usr/local/bin/softlimit -m 2000000 \
/usr/local/bin/tcpserver -H -R -v -P -x /etc/tcp.smtp.cdb \
-u $QMAILDUID -g $NOFILESGID 0 25 \
/var/qmail/bin/qmail-smtpd 2>&1
after adding rblsmtpd:
#!/bin/sh
QMAILDUID=`id -g qmaild`
NOFILESGID=`id -g qmaild`
exec /usr/local/bin/softlimit -m 2000000 \
/usr/local/bin/tcpserver -H -R -v -P -x /etc/tcp.smtp.cdb \
-u $QMAILDUID -g $NOFILESGID 0 25 /usr/local/bin/rblsmtpd -r relays.ordb.org \
/var/qmail/bin/qmail-smtpd 2>&1
Now check files /var/qmail/rc and change them to something reasonable - but I think You should be happy with the script defaults ;)
We are done with qmail.
I am using NOD32 client/server version. It performs great and this is not some kind of advertising. It is fast and efficient. And as far as I know the daemon nod32d has never ever crashed since the time I started it for the first time ;).
But You have to use Your favourite one - amavisd-new supports a variety of them.
Most operating systems have its own port or package. If not, check http://www.ijs.si/software/amavisd/INSTALL.
After extraction please DON'T forget to apply QMQPqq patch, e.g.:
patch < amavisd-new-qmqpqq.patch
Now You are ready to configure amavisd-new. This will not be given here in much detail as there are other very good documents dealing with it, look at amavisd-new documentation section.
Section I - Essential daemon and MTA settings
find
# MTA SETTINGS, UNCOMMENT AS APPROPRIATE,
# both $forward_method and $notify_method default to 'smtp:127.0.0.1:10025'
and add this below:
# USING qmail QMQPqq
$forward_method = 'smtp:127.10.10.10:25';
$notify_method = $forward_method;
***
Then do not forget to edit amavisd (added in amavisd-new-20030314-p1):
find
# to reduce the likelyhood of a qmail bare-LF bug (bare LF reported
# when CR and LF are separated by a TCP packet boundary) one may use
# this 'while' line, instead of the normal one:
### while (defined($_=$msg->getline)) {
while ( $msg->read($_,16384) > 0 ) {
$smtp_handle->datasend($_)
or die "sending mail body from file\n";
}
change
# to reduce the likelyhood of a qmail bare-LF bug (bare LF reported
# when CR and LF are separated by a TCP packet boundary) one may use
# this 'while' line, instead of the normal one:
while (defined($_=$msg->getline)) {
# while ( $msg->read($_,16384) > 0 ) {
$smtp_handle->datasend($_)
or die "sending mail body from file\n";
}
***
Section II - MTA specific (defaults should be ok)
Find and change like this:
$inet_socket_port =
10628; # accept on this local TCP port
$inet_smtp_port = undef; # don't accept SMTP/LMTP
$inet_qmqpqq_port = 10628; # accept QMQPqq on this local TCP port
$inet_socket_bind = '127.10.10.10'; # limit socket bind to loopback interface
@inet_acl = qw( 127.10.10.10 ); # allow SMTP access only from localhost IP
If you wish to accept SMTP/LMTP traffic besides QMQPqq, just change like this:
$inet_socket_port = [10024, 10628]; # accept on these local TCP ports
$inet_smtp_port = 10024; # accept SMTP/LMTP on this local TCP port
Above seen
configuration is not very user friendly so it will be changed in a very near
future realease of amavisd-new.
And the change has finally come! Starting with amavisd-new-2.1.0-rc3 one have to use $protocol = "QMQPqq" instead of variables $inet_smtp_port and $inet_qmqpqq_port (look into sample config file after patching).
Section VII - External programs, virus scanners
find
@av_scanners = (
.
.
.
change
Well... uncomment and comment... the ones You want or don't want.
in my case:
['ESET Software NOD32 - Client/Server Version', '/usr/local/nod32/nod32cli',
'-a -r -d recurse --heur standard {}', [0], [10,11],
qr/^\S+\s+infected:\s+(.+)/ ],
You are definitely NOT done yet! Either read through amavisd-new documentation section or through amavisd.conf and amavisd itself. They are both HEAVILY commented.
Edit (create) /var/qmail/control/timeoutsmtpd and add
30 which means a 30 second timeout on qmail-smtpd connection. I really don't know whether this is good or not but it helps/helped me a lot to prevent some nasty DOS attacks at tcpserver.
Take a look at /var/qmail/supervise/qmail-smtpd/run:
#!/bin/sh
QMAILDUID=`id -g qmaild`
NOFILESGID=`id -g qmaild`
exec /usr/local/bin/softlimit -m 2000000 \
/usr/local/bin/tcpserver -H -R -v -P -x /etc/tcp.smtp.cdb \
-u $QMAILDUID -g $NOFILESGID 0 25 /usr/local/bin/rblsmtpd -r relays.ordb.org \
/var/qmail/bin/qmail-smtpd 2>&1
and change like this:
#!/bin/sh
QMAILDUID=`id -g qmaild`
NOFILESGID=`id -g qmaild`
exec /usr/local/bin/softlimit -m 20000000 \
/usr/local/bin/tcpserver -c 20 -H -R -v
-P -l qmail-change-me-host -x /etc/tcp.smtp.cdb \
-u $QMAILDUID -g $NOFILESGID 0 25 /usr/local/bin/rblsmtpd -r relays.ordb.org \
/var/qmail/bin/qmail-smtpd 2>&1
The '-m 2000000' parameter tells softlimit to limit memory usage to 2000000 bytes (approximately 19 MB). qmail-qmqpc stores incoming mail in memory so we need to increase limit for qmail-smtpd so the mail fits in. Decide on the maximum size of mail You want to accept and put the number into /var/qmail/control/databytes. Then add approximately 2 MB to this number and feed it to softlimit, e.g.:
15 MB is maximum for the mail so /var/qmail/control/databytes will contain 15728640 (1024 * 1024 * 15) and 'softlimit -m' line in /var/qmail/supervise/qmail-smtpd/run will be 'softlimit -m 20000000'.
The '-c 20' parameter tells tcpserver how many concurrent connections it should accept. Use something reasonable for Your site. Don't forget that every spawned qmail-smtpd process will need an amavisd-new child and every spawned amavisd-new child will need another qmail-smtpd process to forward the mail. If world was a nice and peacefull place following that math we would end up with C qmail-smtpd processes while having C/2 amavisd-new children ($max_servers).
But world is an ugly place so we can receive (example only) 20 connections to qmail-smtpd from outside clients while having -C set to 20 and $max_servers set to 10... amavisd-new would be then unable to forward scanned mail and... and we would be very very unhappy (yes, in real time world example one would 'only' need to wait until one side times out (nevertheless, it's easy to produce a simple dos attack in this way)). No, most of people will never encounter such problems but it could happen.
And yes, we are clever so we can insure that this can never ever happen by creating separate qmail-smtpd just for our beloved amavisd-new.
***
One approach is to 'reuse' what we have just installed:
cd /var/qmail/supervise
mkdir qmail-smtpd-just-for-our-lovely-amavisd-new
cd qmail-smtpd-just-for-our-lovely-amavisd-new
cp -p ../qmail-smtpd/run .
mkdir log
cp -p ../qmail-smtpd/log/run log/
edit run:
#!/bin/sh
QMAILDUID=`id -g qmaild`
NOFILESGID=`id -g qmaild`
exec /usr/local/bin/softlimit -m 2000000 \
/usr/local/bin/tcpserver -c 10 -H -R -v
-P -l qmail-change-me-host -x /etc/tcp.smtp-just-for-our-lovely-amavisd-new.cdb \
-u $QMAILDUID -g $NOFILESGID 127.10.10.10 10025 \
/var/qmail/bin/qmail-smtpd 2>&1
'Only' 2000000 bytes for softlimit are ok now.
The '-c 10' parameter should reflect $max_servers in amavisd.conf.
Create /etc/tcp.smtp-just-for-our-lovely-amavisd-new:
127.10.10.10:allow,RELAYCLIENT="",QMAILQUEUE="bin/qmail-queue.orig"
:deny
and make it
tcprules /etc/tcp.smtp-just-for-our-lovely-amavisd-new.cdb temporary_file < /etc/tcp.smtp-just-for-our-lovely-amavisd-new
We want to listen only on 127.10.10.10 on port 10025 (be sure to change $forward_method/$notify_method in amavisd.conf to reflect port change ('smtp:127.10.10.10:10025' in this case)).
No need to run rblsmtpd.
We can now remove
127.10.10.10:allow,RELAYCLIENT="",QMAILQUEUE="bin/qmail-queue.orig"
from /etc/tcp.smtp and make with tcprules to reflect change.
edit log/run:
#!/bin/sh
exec /usr/local/bin/setuidgid qmaill /usr/local/bin/multilog t s2500000 /var/log/qmail/qmail-smtpd-just-for-our-lovely-amavisd-new
to reflect proper logs location.
***
The '-l qmail-change-me-host' tells tcpserver not to look up local host name in DNS and use that name instead. Use something reasonable for Your site.
The '-H -R -P' parameters are nicely explained at tcpserver manpage. I always like to speed things up.
amavisd.conf:
Section I - Essential daemon and MTA settings
find
$max_servers = 2; # number of pre-forked children (default 2)
$max_requests = 10; # retire a child after that many accepts (default 10)
change
$max_servers = 10; # number of pre-forked children (default 2)
$max_requests = 350; # retire a child after that many accepts (default 10)
$max_servers should be set according to cpu/ram/your sanity.
$max_requests performs better for me/others with a higher number.
Section II - MTA specific (defaults should be ok)
find
$unix_socketname = "$TEMPBASE/amavisd.sock"; # amavis helper protocol socket
#$unix_socketname = undef; # disable listening on a unix socket
# (default is undef, i.e. disabled)
# (usual setting is $TEMPBASE/amavisd.sock
change
# $unix_socketname = "$TEMPBASE/amavisd.sock"; # amavis helper protocol socket
$unix_socketname = undef; # disable listening on a unix socket
# (default is undef, i.e. disabled)
# (usual setting is $TEMPBASE/amavisd.sock
We do not need a socket.
Section III - Logging
find
$DO_SYSLOG = 1;
change
$DO_SYSLOG = 0;
What for?! We have
/var/amavis/amavis.log.
find
#$log_level = 2;
change
$log_level = 2;
It is really a good idea
to log. This log level is nice.
Section IV - Notifications, quarantine
find
$viruses_that_fake_sender_re = new_RE(
qr'nimda|hybris|klez|bugbear|yaha|braid|sobig'i );
add the new ones
$viruses_that_fake_sender_re = new_RE(
qr'nimda|hybris|klez|bugbear|yaha|braid|sobig|lirva'i );
This is _only_ an
example, you have to keep your virus list up-to-date on your own!
find
$final_virus_destiny = D_BOUNCE; # (defaults to D_BOUNCE)
$final_banned_destiny = D_BOUNCE; # (defaults to D_BOUNCE)
$final_spam_destiny = D_BOUNCE; # (defaults to D_REJECT)
$final_bad_header_destiny = D_PASS; # (defaults to D_PASS), D_BOUNCE suggested
change
$final_virus_destiny = D_BOUNCE; # (defaults to D_BOUNCE)
$final_banned_destiny = D_BOUNCE; # (defaults to D_BOUNCE)
$final_spam_destiny = D_REJECT; # (defaults to D_REJECT)
$final_bad_header_destiny = D_PASS; # (defaults to D_PASS), D_BOUNCE suggested
Reject spammers on the fly! Bounce viruses/banned.
Section VII - External programs, virus scanners
find and change
$sa_local_tests_only = 1; # (default: false)
We do want only local tests.
Fire up your qmail:
ln -s /var/qmail/supervise/* /var/service/
and amavisd-new:
e.g. /usr/local/etc/rc.d/amavisd start for FreeBSD installation.
Don't forget to substitute right directories. Your qmail should start within few seconds.
If something do not work as You expected, check logs, read this again and start questioning me.
Whole this needs a better documentation (as always). There are many possible approaches to failovers (in this kind of setup if amavisd-new is down, all mail is being temporarily rejected), one could even easily implement a simple loadbalancing method via adding more servers to control/qmqpservers (simple patch required). Really a lot of possibilites, unfortunately, I lack the time to write about any/all of them...
If You find this document helpful, PLEASE SEND me some feedback to make it better. Thanks!
Copyright: copy&paste as You like (2004).
solko -at- imladris -dot- sk