Copyright © 2004-2006 Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".
qSheff is an advanced content filter for the qmail e-server. It can scan for viruses and spams. qSheff replaces qmail-queue and gets the e-mails received by qmail-smtpd or qmail-inject before qmail-queue. After scanning the content, if the e-mail is allowed, qSheff releases it to the queue and allows it to go on-route. By intercepting e-mails before they enter the queue, qSheff reduces the e-mail server load.
qSheff has more features than alternative content filters and performs better. qSheff's most important features are:
Direct connection to ClamAv Antivirus daemon over socket.
Executing antivirus program
Filtering by header lines.
Filtering by body.
Filtering by header lines.
Regular expressions support.
Attachment blocking by filenames and extensions
White/Black list support (network, IP, domain and e-mail address based blocking or permitting)
Could save all incoming/outgoing mail trafic or just rejected emails
Single line logging
Blackhole -- option to return nothing when a virus or spam is found
Custom message return option instead of qmail "permanently" error.
Subject tagging for spam mails.
Could filter or skip local users.
Supporting qmailqueue patch.
Blocking e-mails of broken headers
Easy installation by the GNU autotools
Monitoring SPAM traffic with MRTG
Extendable, fast and simple code
Easy debugging, bug tracking
SMTP from Network from local
remote mailserver
|
|
tcpserver / inetd MUA
| |
| |
qmail-smtpd qmail-inject
| |
| |
-------------- --------------
| |
| |
| |
qmail-queue
|
|
qmail-send
The detailed qmail picture can be found here : http://www.nrg4u.com/qmail/the-big-qmail-picture-103-p1.gif.
Remote mail (via qmail-smtpd) and local mail (via qmail-inject) is queued by delivering the mail to qmail-queue. qmail-queue gets the e-mails from standart input. In compliance with this, qmail-smtpd and qmail-inject writes the e-mails to standart output.
When qSheff is running, the install script move qmail-queue to qmail-queue.orig(If the qmailqueue patch installed qmail-smtd gets qmail-queue/qmail-qsheff path from the environment. See the qmailqueue patch). qmail-queue is linked to qSheff. If you enable qmailqueue patch, qmail-queue will not be moved. qmail executes qSheff instead of qmail-queue and then qSheff delivers email to qmail-queue. By both of these ways, the e-mails sent by qmail-smtpd and qmail-inject are received by qSheff. A system where qSheff is running looks like Figure 2.
SMTP from Network from local
remote mailserver
|
|
tcpserver / inetd MUA
| |
| |
qmail-smtpd qmail-inject
| |
| |
-------------- ---------------
| |
| |
| |
qSheff
|
|
qmail-queue
|
|
qmail-send
After running its own modules, qSheff does one of the following :
A spam or virus has been found, returns a PERMANENTLY REJECTED error. (1)
An error has occured while qSheff was running, it returns a TEMPORARY REJECTED error.
A spam or virus has been found, but enable_blackhole = 1 feature is active, it returns a SUCCESS message.
The e-mail was not stuck on any filter, thus it is delivered to qmail-queue.
If the qSheff configured with --enable-subject-tag option, qSheff will deliver the spam message with a tagged subject.
qSheff has been written compliant to standards so that it could be run on different platforms. Some of the software was developed on Solaris 8, the other part on FreeBSD 4.10. It was tested (without any problems) on the following platforms :
Solaris 5.8 (gcc version 3.4.2)
OpenBSD 3.5 (gcc version 3.3.2)
FreeBSD 4.10 (gcc version 2.95.4 20020320)
FreeBSD 6.x (natural support)
GNU istanbuLX/Linux (gcc version 3.4.2)
Red Hat Enterprise Linux ES release 3 (gcc version 3.2.3 20030502)
GNU/Linux Debian (gcc 3.3.5, Debian 1:3.3.5-12)
SuSe 10.0 (gcc version 4.0.2 20050901)
Fedora Core 4
Installing qSheff is a pretty easy process. First of all, remove any filtering software (like RAV, qscanq, simscan or an old veersion of qSheff) you may have installed before. A qmail-queue without a filter should look like this :
# ls -l /var/qmail/bin/qmail-queue -rws--x--x 1 qmailq qmail 12396 Nov 28 13:18 /var/qmail/bin/qmail-queue
PLDaniels ripmime should be installed on your system for qSheff to work. Ripmime tears the e-mail apart and saves its headers, data and attachments to different files. You can get this software from http://www.pldaniels.com/ripmime/.
# wget http://www.pldaniels.com/ripmime/ripmime-1.4.0.2.tar.gz # gunzip ripmime-1.4.0.2.tar.gz # tar xvf ripmime-1.4.0.2.tar # cd ripmime-1.4.0.2 # make # make install # ripmime -V v1.4.0.2 - Oct 23, 2004 (C) PLDaniels http://www.pldaniels.com/ripmime
ClamAv antivirus is optional but highly recommended. Other antivirus and content filtering softwares are optional. qsheff.conf file configured to enable clamd by default.
ClamAv: http://www.clamav.net/
Use clamd to get more performance. qSheff has an internal clamd client connects clamd over UNIX a domain socket. It doesn't execute any binary like clamdscan or clamscan. That increace system performance and decrease resource usage.
You can get qSheff from its official web page (http://www.enderunix.org/qsheff/) or SourceForge project page (http://sf.net/projects/qsheff/).
qSheff has some great configure parameters. Before the installation, check them and decide which of them you plan to use.
# ./configure --help --enable-debug Enable Debugging --enable-backup-all Enable backup --enable-subject-tag Enable Subject Tag --enable-qmailqueue-patch Enable qmailqueue_patch --enable-local-users Filter local users --enable-custom-error Return custom messages to senders --enable-custom-filter Enable user defined filter --with-clamd-socket Path to clamd socket, default=/tmp/clamd --with-maxfiles Maximum files in a directory (OS limit), default=32000 --with-qmaildir Define qmail directory, default=/var/qmail --with-qmailgroup Define qmail group, default=qmail --with-max-bodyline Maximum lines in message body will be filtered, default=40 --with-clamav Enable ClamAv
Some of these parameters conflict. For that you should understand well, what behaviours they have.
--enable-debug: This option enables debug information (-g flag) for gdb debuggers and some usefull qSheff outputs for system administrators.
--enable-backup-all: Use this option to save all incoming and outgoing mail taffic. qSheff uses /var/qsheff/backup directory to save backups. If you enable this option, qSheff will disable quarantine feature for performance reasons, because of double backup.
--enable-custom-filter: Enables a user-defined binary to filter message. If you want to enable this option, you should set program path and return values in the qsheff.conf.
--enable-subject-tag: With this option, qSheff never rejects emails, but alters subject and adds ***SPAM*** tag. The customers could write a simple subject filter to move tagged emails to Trash. MS Outlook, Thunderbird and most of the MUAs support filter feature. If you enable this option, custom error patch will be disabled.
--enable-qmailqueue-patch: This option enables Bruce Guenter's qmailqueue patch which gets QMAILQUEUE environment variable and executes it. So you don't need to make a link to qSheff. This option is also enables Baris Simsek's qsheff patch which includes custom error patch and qmailqueue patch. qSheff patch executes qSheff directly. Never looks for QMAILQUEUE variable. It is more secure and easier.
--enable-local-users: This option enables local users to filter. qSheff 2.0 doesn't filter local users by default. Some big ISPs would like this option.
--enable-custom-error: qmail returns "permanently" error for spammers. Most of the senders doesn't understand what it means. This option enables custom error patch which provides more clear message for senders. Note that, qsheff patch includes this feature too. You should use this option also for qsheff patch. Return messages are hard-coded but qSheff provides you to append your company name to the end of the message. See orgame variable in qsheff.conf.
--with-clamd-socket: Specifies clamd socket path. If you want to use ClamAv daemon, you must specify socket file path. You can learn it with netstat command or simply check the clamd.conf file (LocalSocket option).
# netstat -an | grep clamd dfdf9dc0 stream 0 0 dfe20040 0 0 0 /tmp/clamd
In this case, /tmp/clamd is the socket file.
--with-maxfiles: Maximum files in a backup sub-directory. See also --with-backup-all option. Operating systems has a limit for number of files in a directory. After it reach this limit, qSheff will fail while storing backups. backup and quarantine directories have sub-dirs from A to F.
--with-qmaildir: If you installed qmail a directory other than /var/qmail, define it with this option.
--with-qmailgroup: If you use group id other than 'qmail', define it with this option. Check it by id qmaild command.
--with-max-bodyline: qSheff 2.0 reads message line by line. To prevent against the DoS attacks, use this option. If an attacker sends emails about 100.000, this will reduce your system resources. With this option qSheff will read only the number of defined lines. It is 40 lines by default.
--with-clamav: Enables Clamav and if you haven't installed ClamAv under the system directories like /usr or /usr/local, you must define clamav installation path. qSheff searchs clamav library or clamav.h in this prefix.
qSheff binary is qmail-qsheff. It resides in the qmail/bin directory.
A simple installation procedure is looks like below:
# wget http://www.enderunix.org/qsheff/qsheff-2.0.tar.gz # gunzip qsheff-2.0.tar.gz # tar xvpf qsheff-2.0.tar # cd qsheff-2.0
qSheff has a GNU autotools compatible installation. So you can install it simply by following way:
# ./configure --enable-subject-tag --with-clamav=/opt/clamav --with-clamd-socket=/var/run/clamav/clamd # make # make install # /usr/local/etc/qsheff/install-wrapper.sh
If you want to change return messages to senders edit main.h before the compile. Note that, to use custom messages you have to patch qmail with qmail-custom-error or qsheff_patch.diff.
# ls -l /var/qmail/bin/qmail-queue lrwxr-xr-x 1 root qmail 27 Dec 16 10:48 /var/qmail/bin/qmail-queue -> /var/qmail/bin/qmail-qsheff # ls -l /var/qmail/bin/qmail-qsheff -rws--x--x 1 root qmail 18123 Dec 16 10:48 /var/qmail/bin/qmail-qsheff # ls -l /var/qmail/bin/qmail-queue.orig -rws--x--x 1 qmailq qmail 12396 Nov 28 13:18 /var/qmail/bin/qmail-queue.orig
If the qmail/bin directory looks like the above listing, then the installation is successfully done.
Note that, if you enabled qsheff patch or qmailqueue patch, qamil-queue will not a symbolic link. It will seem like below:
# ls -l /var/qmail/bin/qmail-q* -r-s--x--x 1 root qmail 38123 Feb 26 13:49 /var/qmail/bin/qmail-qsheff -rws--x--x 1 qmailq qmail 15276 Feb 26 13:43 /var/qmail/bin/qmail-queue
To upgrade, use following way:
# qmailctl stop (whatever you use to start/stop qmail.) # /usr/local/etc/qsheff/uninstall-wrapper.sh # mv /usr/local/etc/qsheff /usr/local/etc/qsheff-1.0 # netstat -an | grep clamd (If you are planning to use clamav) e5091000 stream 0 0 e58d0cc0 0 0 0 /tmp/clamd qSheff installation procedure described above. # qmailctl start
qSheff configurations files are PREFIX/qsheff.*. PREFIX is /usr/local by default.
qsheff.conf
qSheff's main configuration file. You can enable or disable different features here.
QSHEFFDIR: qSheff data directory which includes backup, quarantine, spool, tmp. Don't touch spool and tmp.
LOGFILE: Name of the qSheff log file.
debug_level: The log level. qSheff will log entries has debug level less than this value. It is 99 by default, so qsheff logs everything. Levels are below:
level Identifier Comment 0 ERR Internal error. 3 VIRUS Antivirus software found a virus. 5 CUSTOM Custom filter program caught a spam pattern. 7 WBLIST White/black list module caught a spam pattern. 11 SPAM Spam found in header or body. 13 ATTACH An prohibited attach file is caught. 15 HEADER Problem with header. 17 EMPTY Mail header is empty. 21 QUEUE Succeed/safe email.
enable_blackhole: If set to 1, qSheff won't let qmail to return a REJECT message when a virus or spam has been found. Nowadays, viruses generates random from adresses and spread. This kepts the e-mail server too busy to send error messages to unreal addresses. This feature allows you to silence qmail when a spam or virus has been identified.
paronia_level: This is a feature not applicable to 2.0 version but it is reserved for the next version. It defines how qSheff will react when an error occurs. The values can be 0, 1 or 2. If set to 0, if a problem occurs in any module (e.g spam scanning), it acts like nothing has happened and moves along to the next module continuing its process. If set to 1, only errors in antivirus modules will cause a termination and a TEMPORARY error return. If set to 2 (paronia_level) any error will cause a TEMPORARY error return.
drop_empty_line: Drop mails, if the From or To is empty.
enable_quarantine: When set to 1, spam or virus containing e-mails will be stored in WORKDIRPREFIX.
enable_wblist: When set to 1, enables the White/Black list module.
enable_header_filter: When set to 1, enables the internal filter engine for header lines.
enable_body_filter: When set to 1, enables the internal filter engine for body.
enable_clamd: When set to 1, enables internal client for clamd antivirus.
enable_custom_prog: When set to 1, enables the custom program in the specified path to scan e-mails.
enable_attach_filter: When set to 1, filters attachments by extensions (qsheff.attach)
PROG variables define the software, its parameters and the job it will be run for. Here, you can choose your alternative softwares. You should learn and write the normal runtime, error and spam/virus identification return values here. You can learn the return values of your own softwares from the software vendor or its official web site.
qsheff.rules
The rules for qSheff's internal filter engine are defined here. An example file :
type:(rule1)(rule2)...(ruleN)
b:(mature woman)(hot)
h:( important)
h:(^Subject: [Vv][aA@]gra)
a:(hot)(teen)
b is for .body., h is for .header. and a is for .all.. These specify filter type.
qSheff supports regular expressions. http://www.enderunix.org/qsheff/docs/regular.expressions.txt
Rules in same line are connected with AND logical operation. Each line is connected with OR logical operation. This is more flexible and more sensitive scan engine. For example: sex may be innocent in a form asks for age, sex, occupation and so... You can define multilevel and selective rule like:
b:(hot)(sex)
qSheff will only stop emails contain 'sex' and 'hot' words in the subject line. You can use more level to get more sensitive filter. I recommend that, define level 2,3 or maximum 4 rules.
IMPORTANT: To primarily identify the spam e-mail by its subject or header line is important for performance. Because the internal filter only scans the 256 character-wide subject line. If the e-mail is cut off at that point, it will not be scanned by a spam or antivirus software which scans the whole message and its attachments. This results in a big performance improvement. For instance, a virus has a distinctive subject line, appending it to the rule file would be better. Because it will be identified and stopped at the internal filter stage. If you don't; the internal filter and then the spam program will scan it. At the end the antivirus program will run and identify the virus. All of these programs will scan the whole content and result in a performance loss. Thus improving the internal filter rules is more important than improving the spam database. qSheff is designed to first filter the subject and then scan the whole content for you to benefit from this performance increase.
qsheff.wblist
File format:
WBL_COMMAND WBL_RULE DIRECTION
There are three different commands : ACCEPT, DISCARD ve REJECT.
E-mails matching an ACCEPT rule are only scanned for viruses. email enable_subject_filter and enable_spam_prog values are assumed as 0 for them.
E-mails matching a REJECT rule are sent back to the sender with a PERMANENTLY REJECTED error.
E-mails matching a DISCARD rule are deleted and nothing is returned to the sender.
The rules may be written in the 4 forms below :
1. user@domain.com
2. @domain.com
3. 192.168.22.10
4. 10.45.
First and second rules are applied to the from and to headers of the e-mail. If there is a match, the specified command is executed. The first rule matches an exact e-mail address. The second rule matches a partial address after the @ symbol. Meaning, all the e-mails from or to the given domain name are matched.
Third and fourth rules are for IP addresses and are applied to the recvfrom (the IP of the server which delivered the e-mail) part of the e-mail. Third rule matches an exact IP address. Fourth rule matches all the IPs in the network.
IMPORTANT: Don't forget that there is a dot '.' symbol suffix in the fourth rule. A "10.15" rule includes a "10.154.1.1" IP too. However "10.15." only includes IPs in the "10.15.X.Y" network.
Examples:
ACCEPT 192.168.135.166 OUT ACCEPT 192.168. OUT ACCEPT 212.6 IN ACCEPT user1@domain.com IN DISCARD @domain2.com IN DISCARD 81.213.179.56 IN DISCARD 81.3 IN DISCARD 81.113. IN REJECT user2@domain3.com IN REJECT 81.213.80.72 OUT
After installing qSheff, audit the log files to ensure everything went as planned :
# tail -f /var/log/path/to/current @4000000041c18ca305190bcc new msg 68931 @4000000041c18ca30527ec64 info msg 68931: bytes 1204 from <0000truenetworking@onelist.com> qp 31459 uid 1009 @4000000041c18ca305502d14 starting delivery 302397: msg 68931 to local rtxo.net.tr-85858585@rtxo.net.tr @4000000041c18ca30553ea1c status: local 1/400 remote 20/400 @4000000041c18ca305f34314 delivery 302397: success: did_0+0+1/ @4000000041c18ca305f8e864 status: local 0/400 remote 20/400 @4000000041c18ca305fd591c end msg 68931
# tail -f /var/log/qsheff.log 31/01/06 00:24:05: [qsheff], QUEUE, queue=q-1138659839-993801-71574, recvfrom=209.31.154.42, from=`rwatson_@_FreeBSD.org', to=`perforce_@_freebsd.org', subj=`PERFORCE change 90729 for review', size=30442,, 31/01/06 00:39:03: [qsheff], SPAM, queue=q-1138660742-5382-71745, recvfrom=66.225.220.97, from=`paypal@email.paypal.com', to=`cayci_@_enderunix.org', subj=`Account Review Team', size=6225, spam=` href="http://80.154.19.133/www.paypal.com" ', rule=`(PayPal)(security)(credit)(card)(href)' 30/01/06 19:09:43: [qsheff], VIRUS, queue=q-1138640822-504446-67100, recvfrom=64.90.164.206, from=`ismo@_enderunix.org', to=`qmail@lists.enderunix.org', subj=`[qmail] Fw: Sexy', size=184089, prog=`clamd', virus=`Worm.VB-9' 01/03/06 16:43:33: [qSheff] SPAM, queue=q-1141224210-661393-72042, recvfrom=, from=`simsek', to=`simsek', subj=`Baris', size=120, spam=`Subject: Baris', rule=`(Subject: Baris)' 01/03/06 16:43:49: [qSheff] QUEUE, queue=q-1141224219-42781-72070, recvfrom=, from=`simsek', to=`simsek', subj=`simsek', size=121, error=`qmail-queue', hint=`No such file or directory'
qsheff.log file is pretty clear and detailed. The output is suitable for 'cat' and 'cut' processes. Thus system administrators can write scripts to pull statistics from these logs or take precautions. qsheff-mrtg.sh is a few-line script that produces mrtg output from this log file.
qsheff-mrtg.sh
qsheff-mrtg.sh is a script to produce output suitable for Tobias Oetiker's MRTG software. These outputs can be feeded to MRTG for web-based monitoring of spam traffic. MRTG should be installed for this feature.
MRTG web page: http://people.ee.ethz.ch/~oetiker/webtools/mrtg/
There is a sample spam traffic output at http://www.enderunix.org/qsheff/index.php?sect=screenshot⟨=en
To run properly, qsheff-mrtg.sh appends qsheff.log to qsheff.log.cumulative and empties qsheff.log. Thus qsheff.log will be emptied on every MRTG run. The system administrator will only be able to view the recent (since last MRTG process) logs in qsheff. Older logs can be found in the qsheff.log.cumulative file. Since qsheff.log is periodically emptied, the log rotation software should check qsheff.log.cumulative instead.
Installation steps are very simple.
# wget http://www.enderunix.org/qsheff/qsheff-mrtg-20041213.tar.gz # gunzip qsheff-mrtg-20041213.tar.gz # tar xvf qsheff-mrtg-20041213.tar # cd qsheff-mrtg
Place qsheff-mrtg.cfg and qsheff-mrtg.sh in a directory. For example /root/scripts directory. Edit qsheff-mrtg.cfg to assign Workdir and Target values. Workdir, should be in your web server's DocumentRoot :
WorkDir: /usr/local/apache/htdocs/qsheff-stats Target[spam]: `/root/scripts/qsheff-mrtg.sh`
A cron job should be set up to continuously monitor the traffic.
# crontab -e */5 * * * * /usr/local/bin/mrtg /root/scripts/mrtg.cfg
A cron job to run every 5 minutes has been set.
Primarily, try asking your questions about qSheff on the qsheff e-mail list. To subscribe, send an empty e-mail to qsheff-subscribe@lists.enderunix.org and reply to the e-mail sent to you in return. You can then send your questions directly to the list adress qsheff@lists.enderunix.org .
The list is archived at http://news.gmane.org/gmane.mail.qmail.qsheff.
You can contact the authors for problems you can't find a solution in the mailing list.
Please read INSTALL and UPGRADE files before asking a question.
451 mail server temporarily rejected message (#4.3.0)
This is very generic error message. There may be 2 main reason: 1. Binaries or qmail-queue link is not exist in /var/qmail/bin/ directory. 2. qsheff doesn't work properly.
qmail-inject: fatal: qq trouble in home directory (#4.3.0)
This happens mostly when the qmail-queue.orig or qmail-qsheff has wrong permissions. Check it:
# ls -l /var/qmail/bin -r-s--x--x 1 root qmail 32920 Jul 4 13:43 qmail-qsheff lrwxr-xr-x 1 root qmail 27 Jul 4 13:43 qmail-queue -> /var/qmail/bin/qmail-qsheff -r-s--x--x 1 qmailq qmail 15332 Jun 29 01:43 qmail-queue.orig
Is it possible to use Spamassassin?
Yes sure. Here is document written by Omer Faruk Sen.
http://www.enderunix.org/qsheff/docs/spamass-qsheff.en.html
What are the log means?
QUEUE: Succeed/safe email. FILTER: Custom filter program caught a spam pattern. HEADER: Mail header is empty. WBLIST: White/black list module caught a spam pattern. SPAM: Spam found in header or body. ATTACH: Caught prohibited attach file. VIRUS: Antivirus software found a virus. ERR: Internal error.
How can I use regular expressions?
You need some googling ;) or http://www.enderunix.org/qsheff/docs/regular.expressions.txt. You can test your rules by using echo and grep couples.
# echo containsxab | grep [xy]ab containsxab # echo $? 0 (found) # echo containsxab | grep [y]ab # echo $? 1 (not found)
How can you contribute to the project?
Use qSheff, test it, promote the results.
If you are using qSheff and find it successful, inform the authors.
Translate this document to your native language.
Meet the authors' book needs. (About computer sciences.)
qSheff originally developed by Baris Simsek (simsek at enderunix). He is currently maintaining the source.
http://www.enderunix.org/simsek/
Some parts of the code(W/B list module) is written by Atilim Boy (aboy at enderunix).
I'd like to especially thank Murat Balaban for his suggestions, Omer Faruk Sen for his feedbacks and EnderUNIX Software Development Team for their friendship.
And thanks to Professor Huseyin Yuce who is using qsheff on a high volume mail server.
And thanks to Doruk Fisek who has translated this manual to English.
16 Dec 2004, b$