Processing Issues faced by Email Verification Systems

rob_au on 2003-10-07T22:51:33

Following the distributed and coordinated attack on anti-spam service providers over the last month, this journal post is intended to provide an overview of one such attack on the company which I work for, Bluebottle, an email verification technology company, and highlight the core challenges faced by email verification systems in handling email.

Over the last month, the Internet community has been witness to the cessation of services by a number of anti-spam providers as a result of continual forged email and denial of service attacks. The most notably of these service closures was that of Osirusoft, distributor of one of the larger open-relay blacklists, which resulted in operator Joe Jared adding a blacklist for ‘the world’ in order to highlight the problem facing this service. Following the closure of this service, a number of other anti-spam providers, including monkeys.com and compu.net, have been similarly targeted resulting in service cessation.

Recent reports from Steve Linford of anti-spam services provider Spamhaus, suggest a correlation between the source of these denial of service attacks and Windows machines infected with the SoBig virus. Similar observations have been made by Matt Sergeant of message services provider MessageLabs, who reported that the profile of denial of service attacks against anti-spam services appears to match that of machines infected with the SoBig virus.

The effect of this concerted effort against anti-spam service providers has also been experienced by email verification technology company Bluebottle (http://www.bluebottle.com).

Email verification, also known as challenge response, is an anti-spam technique which requires the original sender of an email message to reply to a one-time challenge issued by the mail recipient prior to the delivery of email. The original email message is only delivered to the mail recipient following the successful fulfilment of the details of the challenge request. This challenge may take the form of necessitating the original sender to reply to a specific email address, click on a web link or supply additional information about the intended mail recipient.

This approach to protecting mailboxes against spam however has not without criticism – Some of the major criticisms oft made of email verification systems are that of the additional mail traffic generated by verification requests, the likelihood of misdirected verification requests and placing of an additional burden on the mail sender in order to ensure delivery.

Such concerns regarding email verification are valid and indeed great care must be taken when implementing an email verification system to ensure that the best principles for such systems are adhered to – These principles are outlined in the paper “Proper principles for Challenge/Response anti-spam systems” (http://www.templetons.com/brad/spam/challengeresponse.html) by Brad Templeton, Chairman of the Board of the Electronic Frontier Foundation (EFF) and include:

• Ensuring that the action(s) required in order to fulfil the verification request are easy to complete and accessible to all users.
• Ensuring that a verification request is never sent to a reply of a private message originally sent by the user employing email verification.
• Avoiding verification requests being sent in response to public messages such as those received via mailing lists or newsgroup gateways.
• Avoiding the issuing of verification requests in response to error messages and other verification request.
• Provision of the means by which users can regularly check to see what messages have been held for delivery pending verification.

Such concerns had been considered and incorporated into the Bluebottle product which additionally incorporated tagged addresses, similar to those employed by the Tagged Message Delivery Agent (TMDA) (http://www.tmda.net), for the handling of verification requests and error mail identification.

The result has been an anti-spam system that has proven to be highly effective at protecting mail accounts in manner minimising false-positive attribution of mail messages as spam and recognising the fundamental issue with spam being consent rather than content.

This in turn has appeared to make Bluebottle a target for attack over the last two months by individuals and groups of individuals intent on creating havoc for providers of anti-spam services. This attack took on a similar form to that employed against monkeys.com and compu.net and in turn resulted in ignorant administrators similarly blocking legitimate mail from Bluebottle users as a result of spam sent using forged and non-existent Bluebottle addresses.

This type of attack is known as a “joe job” whereby the spam message is fashioned to appear that it originated from an alternate source. This term originated from an attack on Joe Doll, proprietor of Joe’s Cyberpost (http://www.joes.com) – This web site, first online in 1994, had offered free web pages to any user who agreed to abide by the rules of conduct which included “good netiquette when publicising your page”. In 1996, after terminating a users’ account for the sending of unsolicited messages to newsgroups and email recipients, Joe Doll found that a large number of mail messages were being sent in a manner which made it appear that it originated from his web site. The result was that Joe Doll was inundated with complaints from newsgroups and email account holders and was eventually targeted in a Denial of Service (DoS) attack over a period of ten days – Further details on this attack on Joe’s Cyberpost can be found at http://www.joes.com/spammed.html.

As a result of the manner by which attack is fashioned in addition to all unsolicited email sent to email recipients appearing to be from Bluebottle, all bounce messages from undeliverable addresses are similarly returned to the Bluebottle mail servers. This combined with the roll that tagged addresses play in the Bluebottle email verification system lead to a scenario where significant delays were encountered in normal mail delivery.

This number of bounce messages returned to the Bluebottle mail servers and the load which this placed on normal mail processing became so great that eventually Bluebottle was required to disable all mail verification on user accounts in order to allow the delivery of normal mail in a timely fashion.

This incident highlights the core issue faced by email verification services – The requirement to balance the cost of processing mail messages in a timely and centralised fashion whilst still ensuring that the best principles associated with email verification implementation are adhered to. In the instance of Bluebottle, where the message load was increased as the result of a concerted effort by those intent on interfering with the operations of anti-spam providers, the time cost of processing these messages was too great to bear – The result was an untenable situation where Bluebottle was faced with either delaying normal user mail by unacceptable standards or disabling email verification. The latter of these options was chosen in order to ensure that whilst potentially allowing unwanted and unsolicited messages, Bluebottle users could also receive their normal mail messages in a timely fashion.

From this incident however comes a better understanding of the Bluebottle technology and product. Development work is currently being performed to improve the structure of the Bluebottle email verification technology and provide a more scalable platform upon which a consent-based spam protection system can be based. With the level of unsolicited and unwanted mail messages on the Internet growing, such an understanding of email verification technology will be critical for any consent-based spam protection system to succeed.