DNSBL checking order

[Black20VT]

Hi there,

Just following up on my email, as I'm wondering if there's more than one DNSBL to check against, what order the checks take place?

Is it a case of checking them all and the first one to come back with a positive match, that's what it's rejected on?

Then no match, it'll wait for them all to respond before moving onto the next step in the delivery process?

Thanks.

[Seth]

There is no enforced order; that is we aren't using an ORDER BY on the SQL query when we retrieve settings from the database. Further to that, we query all configured DNSBLs simultaneously in the background and wait for responses within a 5 second window, so ultimately they are processed in the order by which the fastest responses are received.

[Black20VT]

Thanks for the response Seth.

Is it possible to have them sorted and to check in a specific order, or even better, is it possible for the logs to show all matches against the DNSBL's? That way, it's possible to see which lists are being utilised and are effective at their job and which aren't being called upon, or matching. This may also result in a user dropping lists to check against.

At the moment, it's pretty much a race to respond first to steal the glory without getting a true reflection and how effective each DNSBL is.

[Seth]

Thanks for the response Seth.

Is it possible to have them sorted and to check in a specific order, or even better, is it possible for the logs to show all matches against the DNSBL's? That way, it's possible to see which lists are being utilised and are effective at their job and which aren't being called upon, or matching. This may also result in a user dropping lists to check against.

At the moment, it's pretty much a race to respond first to steal the glory without getting a true reflection and how effective each DNSBL is.

It does not scale to query and wait individually when you are processing as many connections as we are. If we were to I/O lock and wait on a list to finish each result alone, the servers would quickly be overrun by waiting connections.

If you put DNSBL in "Enabled (tag)" mode you'll see them all as X-Rollernet-Dnsbl header entries. Choosing this option delays DNSBL lookups until after DATA since it's adding headers.

[Black20VT]

I guess there would be a delay in waiting for each DNSBL to respond in order to add to the logs every match, but isn't it generally only seconds? Could you not just report those that responded within the allocated 5 seconds in the logs? If there's a DNSBL that always fails to meet that time, users (or yourselves) would see and generally remove that DNSBL as it's a wasted query?

Alternatively, in terms of the second choice/option/idea, running through them in order (giving each DNSBL a few seconds to respond, or delay the starting of each subsequent check by a few seconds), my simplistic thoughts were that if the message were to match in DNSBL priority 1, then there's no need to query DNSBL priority 2, 3.... Basically, it runs through them in order, but with a minor delay between each DNSBL check, but as soon as it hits, it stops, or logs?

It's just I'm keeping an eye on logs and how effective respective DNSBL's are. Along with checking the logs, I then also double check matches and some that are missed against MX Toolbox to see which other DNSBL would've caught it. Because of this, I've removed SORBS, which was a little to aggressive.

I hope that makes sense...

[Black20VT]

If you put DNSBL in "Enabled (tag)" mode you'll see them all as X-Rollernet-Dnsbl header entries. Choosing this option delays DNSBL lookups until after DATA since it's adding headers.

Forgot to mention this part. I would do this, but I do like the blocking as it prevents unnecessary mail into users email boxes, storage space and processing, but would just like to see which DNSBL matches. Also, I'd have to go amend the scripts for each user one by one to filter into Junk, rather than just dropping into the Inbox.

Infact, I think that might be a suggestion - Domain configurable sieve scripts...

[Seth]

I guess there would be a delay in waiting for each DNSBL to respond in order to add to the logs every match, but isn't it generally only seconds? Could you not just report those that responded within the allocated 5 seconds in the logs? If there's a DNSBL that always fails to meet that time, users (or yourselves) would see and generally remove that DNSBL as it's a wasted query?

Alternatively, in terms of the second choice/option/idea, running through them in order (giving each DNSBL a few seconds to respond, or delay the starting of each subsequent check by a few seconds), my simplistic thoughts were that if the message were to match in DNSBL priority 1, then there's no need to query DNSBL priority 2, 3.... Basically, it runs through them in order, but with a minor delay between each DNSBL check, but as soon as it hits, it stops, or logs?

One second passes and we miss hundreds (or at peak, thousands) of connections. On someone's personal server a few more seconds may not matter, but for us it's a denial of service. We could add more mail servers to reduce the number of concurrent connections each one needs to take, but the mail services aren't growing enough right now to justify that expense.

It's just I'm keeping an eye on logs and how effective respective DNSBL's are. Along with checking the logs, I then also double check matches and some that are missed against MX Toolbox to see which other DNSBL would've caught it. Because of this, I've removed SORBS, which was a little to aggressive.

I hope that makes sense...

I understand the goal, but we can't afford it in terms of I/O time. I've wanted to add a scoring option to DNSBL for a while, and some of the code and database changes have already been made, but I can't activate it because the same basic problem applies. Right now the only option is to run all lists in the background unless it's in tag mode. Tag mode is different because there is time to wait while the remote server is sending the message body.

I can add an ACC tool that runs IP addresses from the logs through the DNSBL config. The only drawback is that it's possible for an IP address to become listed or de-listed between connection time and the time you ran the tool. The benefit, of course, is that you won't have to enter IP addresses by hand and you can test DNSBL settings without affecting the live ones.

[Seth]

If you put DNSBL in "Enabled (tag)" mode you'll see them all as X-Rollernet-Dnsbl header entries. Choosing this option delays DNSBL lookups until after DATA since it's adding headers.

Forgot to mention this part. I would do this, but I do like the blocking as it prevents unnecessary mail into users email boxes, storage space and processing, but would just like to see which DNSBL matches. Also, I'd have to go amend the scripts for each user one by one to filter into Junk, rather than just dropping into the Inbox.

Infact, I think that might be a suggestion - Domain configurable sieve scripts...

That's not entirely impossible because the "include" extension does have a global provision. We would need to add an option in the ACC to move or make available an existing script to the global store. There's no security though, so if someone managed to guess your global script name then they'd be able to use it too. Although I can't imagine why someone would want to use a Sieve script they don't know what it does.

[Black20VT]

Thanks for the feedback Seth. I fully appreciate your situation and understand the detrimental effect it could have on your service, but just wanted to ask if there were any options/solutions. Thanks for looking into it.

With regards to the ACC tool, that would be a nice feature that people could use when needed. I guess people would only generally use it whilst making changes to their DNSBL, which I'm currently doing, to see their respective effectiveness. I generally check within a few hours, rather than days or weeks, so hopefully, the lists would remain quite accurate. Although interestingly enough, I see different results from MX Toolbox than DNSBL.info - To me, MX Toolbox seems the more accurate, well, matches what Rollernet is reporting

The ACC tool is not essential for me, as MX Toolbox works fine, so if resource is better used eslewhere, that's not an issue.

Thanks again Seth!