Switching…

Whilst I personally haven’t seen any false positives, some of the people who use my setup have seen them (only a few) and they needed help. So I updated the training script I provide to include the retraining of false positives. One of the Forum users “bohica”, aka Tim, already made a script himself which he posted on the forum. But it missed out on make sure the email being fed for retraining was indeed tagged as Spam by DSPAM. It’s no use feeding correct negatives als false positives.

There are two variables added to the script to indicate which folder the false positives are located and if they should be deleted afterwards. Read the documentation on the script before you use it ! You can download the script from there as well.

I’ve been reading up on the topic of greylisting and the techniques behind it and found out that it doesn’t quite work if you have a backup MX mailserver that does not have the same policy as your primary server. That is the situation I’m in and I think most of you as well.

If you have greylisting implemented on your primary server any email arriving will be checked against a stored list of previous encountered mailservers and mailadresses. If the combination is not known the sending mailserver will be asked to try again later. Most spammers will leave it at that. If you have received an x amount of mails from the same mailserver and mailaddress that combo will be whitelisted and always accepted directly.

The problem with backup MX servers is that they will be whitelisted and you need to accept all mail from that server because it will keep trying (as it is intended to do) ! If you don’t have control over the backup MX server and they don’t have a greylisting policy like you have or don’t do spamfiltering you have a backdoor that is wide open for spammers.

Read more on the subject in this whitepaper which describes it all in more detail.

Before we can use/implement greylisting we need to turn of our current backup MX and find someone else who will use the same policies and will want to be our backup MX, or just live without a backup mail server…

I would like to know your thoughts on this. What can I/we do to work this out ?

After installing DSPAM again on a different machine (someone else’s) I found there where some minor errors in the install documentation. Also other people left some comments in the blog about problems they had and posted questions in the DSPAM forum.

The things I’ve changed where:

• adding the ‘sudo chmod o+x /usr/local/bin/dspam‘ command to prevent the error ‘fatal: pipe_command: execvp /usr/local/bin/dspam: Permission denied‘.
• I’ve change the DSPAM startup documentation, there was a problem with starting DSPAM if MySQL wasn’t running yet. So I set the correct dependancies in the StartupParameters.plist and now that works. I also added the complete startup scripts as a downloadable file.
• The DSPAM training script is now working without any errors, sorry about that one…

Next up is maildrop and to write a script that will take care of false negatives (mail indicated as spam that really isn’t). When that is done we’ll pickup on greylisting as that seems to be something people really want.

While installing the DSPAM software at one of the people who use my setup (yes I will help you install if the task might look to daunting for you) I considered making the training script more dummy proof. In the first version of the script you had to separate out the tagged spam from the false negatives (missed spam email) because the script didn’t separate them. But the new version does, you can just use the Junk filter in your mail program to move everything to the Junk folder and only the unidentified spam will be fed to DSPAM for retraining.

Read all about it on the script training page.

Next up will be to deliver the identified spam email into a quarantine folder which will empty all email after a certain time. As we still need to check if DSPAM was too eager in to identify regular mail as spam, although I haven’t seen this happen on my server.

Just finished writing the next part of the DSpam documentation that was needed. It still needed a proper startup and shutdown procedure. So I started in writing a shell script that would take care of it and use that in the launchd scripts which would take care of starting DSpam at boot time.

It turned out to be simpler than I thought, I only needed to uncomment: ServerPID /var/run/dspam.pid in the dspam.conf file to get a proper pid file which I could use to feed the kill command as Mac OS X hasn’t got the killproc command. When I had this figured out the rest was simple. Read the results on the starting DSpam page and use it to your liking.

I’ve decided to go public with my current version of my installation guide for DSPAM. Although it’s not quite finished and it still has some rough edges and I still need to add some features. It’s polished enough to get you a working version running on your mail-server which will eliminate quite a surprising amount of spam. You’ll need to train DSPAM to get some good results, use it a week and you’ll be amazed by it’s performance.

For instance: almost all stock selling spam (the ones with the image spam) get’s tagged as spam . If a new version of spam appears I just need to train DSPAM with a few examples and from then on they are identified as spam.

The setup I’ve currently chosen is to include DSPAM as a content filter for Postfix. This means that mail enters Postfix, is then fed through DSPAM which tags the emails and feeds it back into Postfix to have the emails delivered into the users maildir. Spam is indicated by some header tags but also the tag [SPAM] is added to the subject to be able to use a filter on my email client. DSPAM is trained by putting the missed spam messages into the users Junk folder on the IMAP server. A script that runs overnight feeds them to the DSPAM training program.

One of the features I want to add in the near future is to use maildrop to drop spam messages into the users Junk folder.

Read all about installing DSPAM using my existing mailserver setup in the documentation.

I hope you enjoy the benefits of DSPAM as well as I do.

My apologies if you encountered any problems because the site was off-line. It’s because I was stupid enough not to use a more secretive password with characters, digits and use upper- and lowercase, I just had an easy common word which was easy to remember for me. I noticed for some time now someone trying to use SSH to gain access to my computer but never thought it to be harmful. I seen it at other sites as well. Until yesterday, they had guessed correctly ! Someone got access to my computer via SSH using my root account and installed a simple PHP script that started to send out a massive mailing (I haven’t counted them) with the postcard virus.

I was very lucky to discover it very quickly because I was fiddling with my new spam filter when I noticed that my logfile was filling up rather quickly with strange messages to email addresses I never used before. First I thought some of my others users was doing this but it kept on going so I stopped Postfix and started investigating.

I quickly found out what was going on. Cleaned the postfix queues, which where huge, and restarted Postfix. Scanned the drive for all files changed after 17:00 and located the script and removed it. Now I’ve changed all passwords, checked all user accounts and closed down SSH access until I can find a better, more secure way, of accessing this machine remotely from the outside.

Because of the spam being send out, my ISP got notified about it and blocked my internet access today without me knowing it. Which is a good case if you are on the receiving end of spam, but I solved the problem and didn’t know about it. Next time, I hope never, I will email my ISP that I solved the issue so they don’t need to block me again. I do wish that other providers would block their users if they send out spam, there would be a lot less spam.

Again my apologies for being off-line and even more if you received any of the spam being send out from my computer.

I dropped the test setup of Spammassassin on my production server to accommodate a setup of DSPAM. There is no scientific proof that either one is better, at least I couldn’t find it.

One of the major reasons to switch for me was the web interface that comes with DSPAM. With that you can enable users to train their spam filter, check the quarantined messages, identify the false positives (messages tagged spam and that aren’t) and correct them. One other, not so important, reason is that I found. When researching spam filters I read that spammers adapt their strategies to the counter measures spam filters develop. But they adapt only to the most used spam filters. It’s like virus-writers targeting Windows users. At least that is the theory.

I’ve got DSPAM running for two days now, with some hurdles, it’s not running in it’s complete and proper form but it is working. I’ve started with a blank corpus and started training from the beginning and I’m already getting good results. I even opened up my Postfix configuration to be less strict so I receive more spam then normal. What I was unable to achieve with my Spamassassin setup, filter the image spam, is working now with DSPAM. After training DSPAM with three image spam messages it currently blocks them, which is a sign of more promising results in the future.

I will write proper documentation on how to set it up, but before I do that I need to get a proper bullet proof working setup with the web-based administration (which I haven’t looked at yet). If you want to know what I’ve done to get my current setup please read my entries in the forum. Please be careful and only try to do this if you know what you are doing. I don’t understand all the finer details yet, but I’m learning as we go along. I’ll keep you posted on any progress.

Just a small post to let you know about some updates that where released today. First the Courier IMAP server has had a minor upgrade which solves some minor issues (which hadn’t troubled me yet). But it’s good to know and I’ll be upgrading later this evening. You can get the new version 4.1.1 from http://www.courier-mta.org/download.php.

Second update is a new update of the Akismet plug-in for WordPress. This plug-in is a real time saver as is keeps 99% of all spam from showing up in my blog as you might have noticed. It’s up to version 1.15 and the changes are more cosmetical than functional and you can get it here. If you are running a WordPress blog you really can’t do without.

Today is my birthday, a day you do not intend to spend behind a computer screen, but I couldn’t resist to tell you about the success I had yesterday evening. I’ve got spamassassin working on my mail-server. The install is still very rough and not something I’m keeping as is. But it works ! I still need to train the filter a bit more and have been feeding my personal mail archive through it as well as some archives from the Spam Archive. (more…)