Evaluating E-Mail Security Solutions

Selecting an antispam solution is a major undertaking even for seasoned IT professionals. How do you make the best choice when hundreds of solutions are on the market? How do you ensure that you get the best value for your investment? And, how do you guarantee that you continue to maximize value throughout the life of the product?

In this article, we examine some of the key factors to consider when selecting an antispam solution. Follow on articles will provide more insight and information to help you make an informed choice.

Hint - It's not AntiSpam!

Suprisingly, antispam capabilities should not be your primary consideration! Certainly everyone wants unsolicited trash removed from their in-basket - but not at the expense of legitimate e-mail.

The goal of any e-mail security solution is the accurate, reliable and consistent delivery of legitimate e-mail. Antispam capabilities are secondary!

People usually don't mind the odd bit of spam as long as they can trust a product to always deliver e-mail from established peers. We refer to the predictably correct handling of e-mail from established peers as consistency.

Your customers do business with you becuase you deliver consistent, predictable value. People who travel often visit familiar restaurants chains because they know what to expect. And, consistency should be the primary criteria in selecting an e-mail security solution.

Suprisingly, most vendors don't publish consistency data. This lack of hard data makes it especially challenging to satisfy yourself that a product is capable of predictably consistent filtering. Unless a vendor provides some guarantee of performance, it is safest to assume that consistency isn't a feature of a product under consideration.

If the vendor doesn't offer consistency metrics (and you are still interested), look for Black & White capabilities. Black lists are lists of unwanted senders while White lists are lists of wanted senders. Black lists are mostly ineffective (as spammers regularly change addresses). White list may be effective in preventing unwanted rejects but won't help much in stopping spam. This is becasue White Lists are very easy for spammers to defeat.

Measuring Filter Effectiveness

Four key metrics should be considered when selecting an antispam solution:

1. Consistency
   A product should learn your e-mail peers and never, ever reject e-mail from them (exception: viruses). Any consistency rating below 100% is not acceptable.
2. Accuracy
   The percent of time the antispam solution gets it right. Products that don't offer at least 95% filter accuracy are probably not worth considering.
3. False-positives
   The rate at which wanted messages are incorrectly rejected as spam. This value should be as close to zero as possible. Products with false-positive Values above 1% probably don't belong on your short-list.
4. False-negatives. This is the amount of spam that is incorrectly allowed through the filter. This is the least critical metric but still worth investigating.

There is an inverse relationship between accuracy and false-positives. Some vendors are so aggressive in scanning for spam that they incorrectly reject large amounts of legitimate e-mail as spam. In NetworkWorld's December 2004 study of antispam solutions the product with the highest overall filter effectiveness (99%) also had an unacceptably high false-positive rating (5.52%). If you used this product, over one in twenty legitimate e-mail messages would be rejected as unwanted. No business could tolerate this type of behavior.

Appliances, Server Software or Desktop Solutions

Spam, viruses and other malware are not only unwanted - they are dangerous and pose a significant risk to your business. To minimize risk, it makes sense to block unwanted traffic as far away from your users and mail servers as possible.

Appliances

Because appliances include hardware, you don't need to provision expensive server systems to run your security solution. Appliances should be easy to install, configure and run, and should offer administrative and reporting tools.

Appliances are the only class of antispam protection that removes work from your e-mail server (thus freeing it to handle more users and legitimate e-mail).

E-Mail Server Software Solutions

Server software products are usually fast and easy to install. They are easy to run and are tightly integrated into your mail server. Server software solutions may even be inexpensive (because you've already bought the hardware and operating system) and can be effective. They are most attractive to smaller deployments where cost and ease of use are key.

The major disadvantage to server software products is that spam and viruses actually make it to your mail server. Your hope is that your e-mail security product will stop them before any damage is done. If your server software product misses a malicious message, your mail server is put at risk.

Another concern is the amount of resources that e-mail server based antispam products steal from your mail server. Mail servers are expensive to buy and license. Any product that forces you to buy more hardware than you need or license more users than you need is costing you $$ well beyond the cost of the software.

Because of the risks and hidden costs of server based e-mail protection, we do not believe this approach is effective for any but the smallest organizations.

Desktop Antispam Solutions

Desktop e-mail security is unattractive to business because the labor cost involved with these products usually far exceeds the benefits provided. Even worse, by blocking spam and viruses at the desktop, you are allowing unwanted and malicious traffic to travel through your mail server - putting it at significant risk.

Know Your Rights - Read The Fine Print

• License Start & End Dates
  Does your right to use the product expire? If it does, it is likely that you will have no residual value beyond the expiry date. That may force you into another expensive acquisition or a costly renewal agreement.

  With software only products, the residual value of the product on the expiry date is most likely $0.00.

• Per User Licenses
  Does the product you are considering charge by the user? If it does, you had better find out what the vendor considers a user. E-Mail aliases, generic e-mail accounts, e-mail addresses being forwarded and other items may all count as users to your vendor. If they do, you may be forced to buy many more licenses than you have people in your company.

  If you are willing to consider per-user licensing, look to buy blocks of user licenses rather than exact license counts. That way, you can add staff or e-mail addresses without having to take out your check book.

• Is Everything Included
  Verify that everything you need is in the base price. Satisify yourself that you get maintenance, support, updates, and new features (including antivirus protection) before you buy. By not doing your homework up front, a good deal may not look that way after you've had to purchase expensive upgrades and add-ons to get the protection you need.

• Support & Update Services
  Because the spammmers constantly change their strategies, you need to ensure that the product you select remains effective. The best way to do this is to subscribe to the vendor's support and update services.

  Look at the timeliness of updates. Who updates your product (you or the vendor). If it is you, have you budgeted administrator time for this task?

• Do Your Homework
  Product accuracy and effectiveness claims are usually derrived from tests performed by the vendor or someone the vendor has contracted. As a result, you should treat all vendor claims with a degree of suspicion.

  Verify consistency, accuracy and effectiveness claims by checking published reviews. NetworkWorld's December 2004 study of antispam solutions is still the most thorough review conducted. In this study, NetworkWorld reported accuracy scores on some products that were as much as 5% below the vendor's claim.

• Get Independent Verification
  A great source of information on how a product might perform is a customer who is currently using that product. Look for customers in your industry who have at least one years experience with the product (so that the initial euphoria has had a chance to wear off). Ask them about effectiveness, updates, false-positives, accuracy, server resource consumption, user involvement and administrator overhead. Be wary of products that require a lot of human intervention.

  Even this isn't a perfect predictor of what you can expect because spammers can and do target different companies in different ways.

• Calculate TCO
  Once you've created your short list of potential vendors, estimate the Total Cost of Ownership for each product. Estimate how much mail and spam your users handle each day. Use the accuracy rating, consistency and false positive rates along with your average salary costs to determine how much time and money you will save by blocking unwanted messages. Then reduce your savings by the cost to purchase, license, deploy, administer and work with each solution. Assign high costs to any false-positive estimates you have (due to the potential impact that a false-positive may have on your business).

  THe result should be the true cost of ownership and realized benefits of a solution.

And The Winner Is...

Spam Calculators

__________
©2006 by Larry Karnis and XPMsoftware.com. All rights reserved. All products are trademarked by their respective owners. Please feel free to contact the author with your questions and comments.

Larry Karnis