URL/Phone Number Filtering

Question: What does nearly all spam have in common?
Answer: It wants you to click on a (URL) link, call a phone number or reply to an email.

Therefore, the heart of SPAMSteward is a database of all the URL links (domains, sub-domains and IP addresses) and phone numbers that appear in the body of recent spam, and which our research indicates belongs to spammers. This list is updated every 10 minutes, and receives about 1000 new entries every day.

This URL/Phone# Filter is the first level of filtering performed, and by itself blocks about 95% of all spam. All new entries are made or reviewed by our staff to ensure accuracy; as a result it blocks less than 1 in 100,000 legitimate emails.

For example, here are the names of a recent week's top-12 spamming domains:

the-rx-group.com, optinlistservices.com, zsupper.com
enhancement4u.com.br, digital-wholesaler.com, dailyherbals.com
herbliss.net, financeadvise.com, amazinghealthgroup.com
procare-pharmacy.com, pillsdirect.net, alwaysfeelgreat1.com

When in doubt about a domain, our staff visits the website, checks the contact information for real addresses, and reviews the privacy policy to see if the domain shares or uses shared email addresses. (In most cases, there is no home page, or there is no contact address, or there is no privacy policy.)

Phrase Filtering

Although SPAMSteward does not filter based on simple phrases or "trigger" words, it checks for distinctive long phrases found in spam.

The filtering used by other anti-spam systems is often based on individual words (e.g. "viagra") or short phrases. This is very error-prone and is sure to block legitimate email, such as newsletters from your bank, and messages from medical patients and legal clients. Therefore, it is not used by SPAMSteward.

The Phrase Filter helps block new spam campaigns. We have seen the same spam message sent from many different countries, from many different IP addresses and containing different URLs. However, the message itself is the same, and can therefore be blocked by its content.

Here are a few examples of the phrases we are filtering:

Diplomas from prestigious non-accredited universities
To grab a FREE ID#, simply go to the link
Genuine lasting results! Our pills will
our Doctor Approved formula offers a fast and natural way

These phrases are carefully hand-selected to ensure they are extremely unlikely to occur in a legitimate email.

Pattern Filtering

Spammers employ various HTML tricks and techniques to defeat simple anti-spam filters. For example, to mention "viagra", the email source might contain:

v<hjhd>1<!iuiod>a</duhz>g<!mncvmx>r</qsjf>a

These tricks have no effect on our service; we can check URLs, phone numbers and phrases no matter how hard spammers try to hide them.

However, the Pattern Filter will block any email which uses HTML tricks and other techniques which we are sure only spammers use. This helps us block new spam campaigns before our tools and staff add them to our primary databases. While we consider our exact Pattern Filtering rules to be proprietary, we will say that we currently only employ about 25 "sure" rules including:

• Numerous HTML comments between individual letters (as in the example above), will be blocked.
• A URL specified as an IP address encoded in hex character will be blocked.

We regularly add temporary Pattern Filters to block " strange" spams which often don't contain any real content. We suspect these are attempts to harvest email addresses for the real spam message. We then remove these temporary filter when they are no longer needed.

Many of our customers give us permission to monitor all spam blocked by our various filters. This lets us closely double-check emails blocked by our Pattern Filtering to ensure it is not blocking legitimate emails.

Additional Filtering

The "Additional Filtering" is an additional (small) set of rules that allows each customer to fine-tune our service, selecting some spam filtering rules which might not be suitable for all customers. The rules are organized into "Recommended", "Optional" and " Aggressive" categories.

For example, the legitimate domain www.geocities.com hosts free websites. Unfortunately, it is often abused by spammers with links to pornography sites. Therefore, a Recommended rule is that any email with a link to a geocities website will be blocked. In the unusual case that a customer needs to allows such emails, this rule can be disabled.

The Optional category includes blocking all emails that are in Chinese, Korean or Russian. If no one in your company reads these languages or expects such emails, it is reasonable to block them.

The Aggressive category is designed to block additional porn. Family domains may want to enable them; however, we recommend that these be reviewed as they may block an occasional legitimate email.

Our service will block over 98% of spam even if none of the Additional Filters are enabled. However, the "Geocities" filter is highly recommended for blocking extremely vulgar spam.

Country-of-Origin Filtering

To understand how Country-Of-Origin Filtering and Realtime Blacklists work, it is necessary to understand a little bit about mail servers and IP addresses. When an email is sent, the mail server (computer) that is sending it can be identified by its "IP" address. This is a number such as 63.233.144.100. Every mail server has a unique number and it cannot be forged or faked. While the number can be changed by the owner, it can only be changed to one of the few (typically 2 - 256) that have been assigned to the owner. Of the 2 billion possible numbers, an International organization assigns blocks of IP addresses to various countries, which in turn assigns smaller blocks to major communication companies, which in turn assign small blocks to individual owners.

Therefore, when our service receives an email, it knows the IP address of the originating (sending) mail server, and via an internal database lookup, which country this mail server is located in.

It is well known that a huge amount of spam originates in China, Taiwan, Brazil and Argentina. It is clear to us that there are businesses in these countries that primarily just send spam. South Korea has a huge number of "open relays" which are computers that have been hacked and are controlled by spammers. Therefore, blocking email from countries notorious for sending spam is an effective filtering method.

Which countries to block, if any, is a business decision.

SPAMSteward may add additional countries to the list in the future; in this case they will initially be unblocked for all existing customers.

Realtime Filtering Blacklist

Several organizations and companies are constantly identifying the mail servers which are actively sending spam. They create "real-time blacklists" (RBL) of the IP addresses of these mail servers which are updated daily, even hourly. We have chosen four of the better known RBLs for selection within the SPAMSteward service. (By default, all are selected.) Our main criteria was to choose RBLs which are least likely to block legitimate email. For this reason, the commonly used ordb.org list of "open relays" is not used; it lists many mail systems in the US and Canada which have never sent spam.

A good blacklist not only quickly adds a new spam source, but also removes it when it is no longer sending spam. Spam is often sent through "open relays", which are legitimate mail servers which have been (trivially) "hacked" by spammers. Typically, the owner of the mail server learns of this within a day and then fixes the problem. Therefore, mail systems should not be blacklisted any longer than necessary.

Although the RBLs used by SPAMSteward are the most accurate available, they are more likely to block legitimate emails than any other filter level. For that reason, many customer chose to either " Reject" (to sender) or review any emails block by the RBL Filter level. As shown near the top of this page, the RBLs only block about 1% of the spam blocked by this service. If you disable all the RBLs, our service will typically still block 98% of spam. (However, blocking "only" 98% still means receiving twice as much spam as when you're blocking 99%.)