Email Blocklists for Real-Time Detection and Mitigation

Email is a zero-tolerance medium, explains Patrick Ben Koetter, Board Member of sys4 AG and Leader of the eco Email and Anti-Abuse Competence Groups . In interview with eco International's Lars Steffen, he provides insights into the recently published advice for mail server administrators on selecting suitable blocklists, developed by the eco Email Competence Group, and discusses the increasing importance of IP reputation for the successful delivery of commercial emails.

Lars Steffen: Patrick, you have recently been involved in updating the eco Email Competence Group white paper on blocklists. What have been the big changes?

Patrick Ben Koetter: Firstly, there has been a change in terminology, moving away from blacklist and whitelist to blocklist and allowlist. It doesn’t hurt us to do this, and it does others good. This terminology issue was beginning to emerge at the M3AAWG meeting in Canada the year before last. So we took the opportunity to change our usage in the updated paper.

Apart from that, we sharpened some terminology that we thought was needed. The basic message of the paper – that email blocklists exist, that they are useful, and how to ideally choose a good list – still remains in the new version. What has also remained is the basic message about why we might reject a particular list because we don’t think it’s appropriate. The basic finding is that some lists we had hoped would improve over time have unfortunately not improved. We continue to believe that there are a few list operators that simply should not be used.

Steffen: What are the reasons for not including or for criticizing certain list operators?

Koetter: Email is a zero-tolerance medium. Everyone thinks email is kind of stupid, but if it doesn’t work for a second, everyone thinks it’s even stupider. For that reason, great care needs to be taken by anyone who implements filters to try to let through only the stuff that an email recipient wants. You want to avoid the possibility of these filters being hypercritical and filtering out things that shouldn’t be filtered out. When this happens, you want to be able to turn that off as quickly as possible. This is where the wheat is separated from the chaff in terms of good and bad blocklists.

Blocklists are decidedly practical. They are created by having sensors all over the world that perceive whether someone is sending spam or not. These blocklists are updated to provide real-time information about which computers you should avoid communicating with because they might be sending spam.

At the same time, if someone is mistakenly put on such a list, then there needs to be an effective and efficient delisting process. If this does not exist, then this is a list you should not use. So one reason not to use a list would be, for example, if there is no explanation of how to get off this list again. Another reason would be that the people who edit this list may only edit it periodically, and therefore it may not be acted upon quickly enough.

And an even more important reason would be if someone says: If you want to get off this list right now, you have to pay us money. Such providers exist, and we do not recommend using them, as there is a conflict of interest in this case. One might assume that the operators are not interested in updating the list quickly. And that they have an interest in listing everyone in the world if possible, because then everyone in the world will want to pay money to get off the list. The suspicion here is that this is their business model, and that’s not on. You have to stay extremely far away from such things.

Steffen: Has the need for blocklists increased in recent years? 

Koetter: For basic understanding: Depending on which statistics you look at, between 97 and 99.9 percent of all emails sent around the world are spam. This raises two problems. First: I want to filter out the spam. And secondly, if you think about how much load this generates: I want to take as much load off the servers that have to process the spam as possible. So, on the one hand, there is this huge amount of spam, and, on the other hand, there’s an insanely large load on the mail servers that have to filter it out. So you want to find out as quickly as possible whether you want to talk to the computer that is currently connecting to you or not. And the best means of choice to decide this are simply IP-based blocklists. You can look at these lists and say: I know the IP, it’s on the list. I don’t want to talk to you. For this reason, blocklists still have their justification.

Now, fortunately, there is an increase in the use of IPv6 on the Internet. The number space in IPv6 is so much larger than in the IPv4 network, meaning that it is not possible to build blocklists that would block all these machines and IP addresses. No one can put that many hard drives anywhere. This is why blocklists for IPv6 are not a means of choice. You can’t write one big enough. That’s why IPv6 turns the principle around and goes over to so-called allowlists. Here, the logic is: I know this IP address and I know that it has a good reputation by now, so I allow it to send to me. Will there be a lot of use of IPv6? Yes, hopefully. Is this already happening today? Not really yet. That’s why we’ll be dealing with IPv4-based mail servers for a very long time to come. And for this reason, we will continue to need IPv4-based blocklists for quite a long time.

Steffen: You mentioned updates to provide real-time information. Is this really necessary or just a nice-to-have?

Koetter: Spam and ransomware attacks are passed on insanely fast. Abuse cases that we have today spread rapidly on the net and disappear again. Many malware campaigns we see pop up somewhere for 10 minutes, cause a lot of damage, and disappear. Reactions need to be quick – because within 10 minutes, the damage has already been done. Ideally, you will have already noticed with the help of a blocklist that you do not want to communicate with the affected computers and will have reacted accordingly. Which is why it’s increasingly important that we have real-time lists that can be queried instantly. And that includes being able to query lists easily and in fractions of a second. I know from my experience with Abusix that, on average, there is about a second between a honeypot registering something and the entry going on the blocklist.

Steffen: And how quickly does the information reach those who use the list?

Koetter: That’s the interesting point. This is covered in our paper. From a purely legal point of view, it is quite questionable to use a mail server that is located in Germany to query a service that is not on your platform. Because it means that you pass on an IP address of a device to a third party, and an IP address is definitely something that can be considered as personal data from the point of view of data protectionists. So if you query a service in America with your mail server, they learn a lot about our communication habits. This is a difficult issue from a data protection perspective. That’s why the big providers have developed processes that copy the data from the supplier over to their own platform as quickly as possible and use it there.

However, these procedures are no longer fast enough. You have to be able to query the data live because a lot happens in just one minute. This is why, in addition to what can be achieved with blocklists, people have also started to create so-called reputation-based systems. This means that the significance of an IP address is nowhere near as important today as it once was in determining the overall impression of whether to accept an email. In the past, a mail server was configured to check whether it allowed an IP address or not. This was a binary criterion: yes or no. Today, we’re moving towards a situation where the mail server says: No, we won’t decide so early. We will let the email through. We will look at the blocklist and check whether the address is listed or not. And we will check other parameters – for example, whether this domain has behaved properly in the past or whether it has sent junk. How can I tell that it is the right domain? Are the systems that are currently contacting me even allowed to send stuff to me on behalf of this domain? And so on.

All of these things are now lumped together and then a weighted decision is made. This so-called reputation of a system has become much more important and prominent. By the way, this is also the way we get a handle on IPv6-based mail servers. We know hardly anything about their IP address. But we know if the sender domain is OK, if the sender domain allows what this IPv6 address sends, etc. This all feeds in to the reputation.

The good thing is, reputation is something that spammers can’t build up quickly. When you’ve misbehaved three times, you’re just branded. But if you behave well all the time and then suddenly have a outlier, others can let it pass, because it’s not your standard behavior. This is the reason why reputation-based systems will be the ones to win the race in the long run. This is also the reason why we have been dealing with the whole topic of sender authentication in the Email Competence Group this year at eco. Meaning SPF, DKIM, DMARC. It’s all about sender reputation. 

Steffen: Do you usually use a list, or a provider, or do you put together a portfolio? And if so, how do you do something like that? If you look at the paper, you see different lists from one provider. How does this work? 

Koetter: There is this saying: “You get what you pay for”. This is actually true in the area of blocklists as well. If it is really important to you that you receive as little spam or malware as possible, then you need to sign up for a subscription so that you can query this data live. You might get the data elsewhere, but it might be 5 or 10 minutes old. And in 10 minutes, the critical cases are already through. The free lists do help, but they are nowhere near as effective as lists you pay for.

And if you want to think about it further, the two big players on the market right now are Abusix and Spamhaus, and if you take something from them, you are guaranteed not to be doing anything wrong. On the other hand, if you go for providers that don’t have a delisting process, or even want to take money for delisting, then you can be sure you’re doing it wrong. There are now also so-called meta lists provided by specialized service providers. These focus on different aspects of abuse and filter, for example, only email systems that are not RFC-compliant, or include only systems whose sender domain has been abused, or include only mail systems that have sent emails that contained links to malware. The big operators take this knowledge from the smaller, specialized lists and consolidate that into one big list for themselves.

What also happens, again and again, is that people create lists based on private projects, and these are not always continuously maintained, because of the amount of work involved. As soon as the enthusiasm fades, then the list ceases to be up-to-date. It makes sense not to include such lists, because they no longer do anything but cause harm. So you should check your lists regularly. If you run a mail server privately, do what you want. If a business is going through it, it’s best to pick lists where you can get a guarantee of performance. Such lists do not cost a fortune, but they do cost. We are, after all, dealing with a commercial Internet. If your business depends on these systems working, then it’s worth it.

Patrick Koetter is an email expert, and a board member of sys4 AG, which specializes in email, DNS, and the development of highly secure platforms and services. He contributes his knowledge and experience to eco as an expert and as Leader of the Email and Anti-Abuse Competence Groups.

Lars Steffen is Director International at eco – Association of the Internet Industry (international.eco.de), the largest Internet industry association in Europe. At eco, he coordinates all international activities of the association and takes care of the members from the domain name industry.