Anti-Spam Evaluation Guide

• Two measures of accuracy should be used to determine which anti-spam filtering solution is right for your site: spam detection rate and false positive rate.
• An anti-spam filter should have high spam detection accuracy. This accuracy number is usually expressed as the percentage of spam messages the filter correctly identified. A low spam detection rate will allow many spam messages through to user mailboxes.
• An anti-spam filter should have high spam detection accuracy. This accuracy number is usually expressed as the percentage of spam messages the filter correctly identified. A low spam detection rate will allow many spam messages through to user mailboxes.
• Anti-spam filters should have a very low number of false positives. False positives are legitimate messages that are incorrectly identified as spam. Even small amounts of false positives may lead to the loss or delay of important email messages.
• Some anti-spam filters have a very high spam detection rate and a large number of false positives, while others have a small number of false positives and a low spam detection rate. A good anti-spam filter strikes a balance between the two extremes.
• An anti-spam filter shouldn’t immediately discard messages that it identifies as spam. Even the most accurate anti-spam filters available may accidentally misclassify a message. Users should be able to review and retrieve any message addressed to them that was marked as spam.
• An anti-spam filter should be able to filter a spam message the first time it encounters it – an effective filter shouldn’t require a human to recognize each individual spam attack that is occurring and write a special rule for it. Some products cannot filter spam messages until a human writes a special rule for each individual spam attack and distributes it to all systems running the filtering product. Thousands of spam messages can pass through the filter unchecked during the time it takes for the attack to be recognized, a rule to be written, and the rule to be distributed.
• Rules should not be removed from the anti-spam filter after only a short period. If a rule is removed from the filter, it will not be able to recognize spam messages that have been delayed for several hours.

• An anti-spam filter should have the ability to be tailored to fit your site’s definition of spam. At the same time, a filter should be effective without requiring hours of configuration and tweaking.
• System administrators should be able to control each individual rule used by the anti-spam filter to determine if a message is spam. Many anti-spam filters only allow administrators to control rules in blocks, if at all.
• How “spammy” a message has to be before an action is taken should be configurable. Each user should be able to change the minimum score required to quarantine a message addressed to them.
• System-wide blacklists and whitelists should be available so system administrators can block all incoming mail from known spammers.
• Each user should have their own individual whitelist and blacklist that they control. If only system-wide lists are provided, they can quickly become unmanageable with the sheer volume of whitelist and blacklist entries.
• The anti-spam filter’s configuration and management interface should be intuitive and user-friendly. Managing the filter shouldn’t require hours of the system administrator’s time.
• Users should not have to install additional software on their desktop systems to perform configuration tasks. Several anti-spam filters require email client plug-ins to be installed for per-user blacklist and whitelist support. These plug-ins only support a small subset of mail clients, and provide no functionality for webmail users.

• An anti-spam filter is responsible for more than just deciding if a message is spam – it also needs to allow both users and administrators to see why a message was classified as spam. This information can be critical in determining if a rule should be modified or not.
• A user or system administrator should be able to tell why a message was filtered simply by looking at it. Very few anti-spam filters provide information about why a message was classified as spam, and most of those require the system administrator to search through the log files looking for the message in question. Providing the information in the headers of the message itself allows a quicker turn-around time to “why was this filtered?” questions.
• The product should provide succinct but useful log files. There should be one entry in a master log file for each message examined by the filter, and it should contain information about which rules were used to classify the message.
• An anti-spam filter should provide basic statistics such as the number of messages scanned and the number of messages filtered to demonstrate the product’s Return on Investment (ROI). Web-based graphical reports make it easy to see how effective an anti-spam filter is.

• Several filtering methods are commonly used by anti-spam products. A product should use methods that allow a rich feature set while balancing accuracy and system resource consumption.
• Anti-spam filtering products that only use one filtering method are easier for spammers to circumvent than products that use a mix of methods. By mixing several classes of methods, such as heuristic rule sets and Bayesian filtering, a filtering product increases its accuracy and prevents circumvention.
• Avoid products that depend on easily circumvented methods such as signature checking and challenge/response.
• Some products go overboard and use too many spam filtering methods. Remember that each filtering method has an associated cost in both CPU and memory usage. Products with too many methods can cause unacceptable message delays and overload a mail system.

• Email is a highly visible application to both internal and external users. Message processing delays will be quickly noticed, so an anti-spam product should not become a bottleneck.
• An anti-spam filtering product should be scaleable. The volume of email sent over the Internet continues to expand, requiring constant increases in mail server capacity.

• Your site’s email messages are private communications that are handled by systems residing on your network.
• The system administrator should have the option to control if updated spam definitions are automatically installed on the system. Automated updates are convenient and save administrator time, but some sites may have a security policy that requires them to manually inspect each new anti-spam rule or definition before it is put in place.
• An anti-spam filtering product should send no information about your site to anyone without your explicit permission. Some anti-spam products send statistical information back to the company that developed them without your permission. If this information is maliciously intercepted or misused, your filtering system may be compromised.
• Anti-spam proxies should support Transport Layer Security (TLS), which allows sensitive email messages to be strongly encrypted during transmission over public networks.

• One of the main reasons for purchasing an anti-spam filtering product is to regain time lost to spam. An effective solution should have minimal administration requirements.
• A good anti-spam solution will allow the administrator to empower users to manage their own spam messages and filter settings. This frees up the system administrator for more important tasks, and gives end users direct and immediate control over how the spam filter deals with their messages.
• The system administrator should have the option to have updated anti-spam rules automatically installed. This guarantees that new spam messages don’t slip by the filter while the system administrator is busy with other tasks.
• End-users should be able to preview and release their own quarantined messages. If a user has to ask a system administrator to release a quarantined message, it pulls the system administrator away from more important tasks and increases the time delay in receiving the message.
• End-users should have their own whitelist and blacklist that they can manage. End-user requests to add or remove list entries at even a small organization can quickly overload a system administration team. In addition, system-wide whitelists and blacklists can quickly become too large to manage effectively if they contain entries for each user.

A typical corpus testing setup requires two systems, one of which is running the anti-spam filter. The other system is used to send the messages to the filter via SMTP (a simple Perl script works well for this). At the end of each test run, the anti-spam filter should be able to generate a statistical report that will show the number of messages identified as spam.

The first step in performing this test is to select a group of volunteer users whose mail will be “forked” to the non-production system. These users shouldn’t receive any mail that the IT staff isn’t allowed to read (for example, the human resources director probably isn’t a good test user). On the non-production system, install the anti-spam filter and set up an account for each test user.

On your production MTA, create an alias for each of the test users that sends one copy of any incoming message to their “real” account, and one copy to the corresponding account on the test system. For example, if PMDF is your production MTA you would create an entry in the aliases file for each test user that looks something like:

john.doe: jdoe@example.com, jdoe@test.example.com

IT staff can log into the various user accounts on the test system to determine how the filtering solution will perform for actual users at your site.

Users who are not part of the test group will not notice anything different about their mail messages. Test users can set up rules inside their mail client to filter spam messages into a special folder based on the presence of certain headers inserted by the anti-spam filter.