Patrick Felt

i’ve finally gotten the antivirus stuff to go.  there were some things that kinda messed me up for a little bit, but we got those straightened out.  the worst of all of them was the eicar test string. it really doesn’t match unless it is in a file!  i spent hours trying to figure out why my test messages all got scanned negative for viruses when they had the eicar string in them.  it is because i wan’t mime encoding the eicar string, i was only pasting it directly into the email.  i guess i need to sleep more.

anyhow, we had a long chat on irc about what should happen with the antivirus scanning.  what should the agent do and what shouldn’t it do.  the commit i just made (r84) works against clamd.  i’ve tested it localhost only not a remote clam, but i think it will work ok ( i just don’t have a remote clam to play with ).  this version doesn’t have all the cool connection pooling stuff that i want to add, nor am i sure that it will work against any other scanners.   the current store configuration document enables clamd by default and enables notifying the recipient that a virus was sent (this code will probably disappear).  i debated doing that stuff before committing this, but decided against it for several reasons.

1. i was 3 days late getting it done for m2
2. we talked a little about adding something like milter support to smtp so that we can do antispam and antivirus at smtp time instead of letting the message hit the queue.  even if we don’t do that i’d like to allow for smtp time scanning perhaps via a library or something ( i’m just rambling here ).
3. it is 2:30 am and we are having friends over tomorow afternoon for some killer barbecued pork……  mmmm…..

as for the missing semaphore that locked up on me, that is as designed.  as i said before, the code for antispam is quite different than the code for antivirus.  the antivirus stuff blocks on a semaphore until there is something to do, presumably for scanners that require scanning of each mime part separately (currently coded as everything except clam).

oh, and one other note:  when coding/debuging antivirus code, it is IMPORTANT to have clamd running and listening on a tcp port….

i finally had some time to sit and play with the antivirus agent.  i got the config in place and it reads the config out great.  there are 6 configuration parameters

• enabled (boolean)
• flags (integer)  — defined in avirus.h:AVirusFlags
• patterns (string) — i’m not sure what this does atm
• queue (integer) — defaults to queue 0
• host (string) — host ip address or hostname
• port (integer) — port to connect to for clamd

the unfortunate thing with this agent is that it is different from the antispam agent in terms of codepath.  as i started debugging it, i hit an odd do{} that had a semaphore in it that i never seemed to signal.  because of that, i hang.  obviously there is a bug in there somewhere.

things that need to happen:  convert this to use the new style of queue reading (like the new smtp stuff), convert it to use connection pooling like antispam does.  convert it to use something like ParseHost from antispam.  figure out officialName.  antivirus when detecting a virus removes the mail from the queue and creates a new one in its place from postmaster.

so in essence there is a little too much for me to be able to get this thing straightened out by wednesday unfortunately.  i really, really, really wanted to get it done, and had it been more like antispam it would be, but there are portions of the code that i haven’t yet looked over (and as such don’t fully understand).   i don’t think that this is really in a state to commit until i at least fix the semaphore problem.  i’d stay up longer to do it tonight (”it has personally insulted me” <an old co-worker>), but i have an eye appt in the early am (*shudder*) and i need to get some sleep

today was a fun day.  when i finally got to start coding i decided to finally commit the smtp patch that i worked on for so long.  it helps prevent a buffer overflow (or it should at least).  it was a long time coming and seems to work ok for me in my limited testing environment.  i hope i found all the little gotcha’s, but how often does that ever happen ?    there is still a little work left on the smtp side of things.  i should go over a couple more functions to see if they are ok with how they read in the envelope from the queue agent.  i’d like to switch them all over to using the full connio and storing it in memory rather than reading off the wire as it goes.  this should prevent other possible memory overruns at the expense of a little bit of memory (the envelopes however are not generally that large).

on the antispam front there was a bit of additional work that i completed last night and today.  the commit log for the antispam stuff was a little long and i fear that i messed up the cia-2 bot which is kind enough to post the full commit log into the irc chat.  amongst the changes were the notable dropping of the dropthreshold and headerthreshold parameters.  we decided that the antispam agent itself should not decide on dropping the mail but should allow the user to specify what to do with the spam via the rules agent (which could probably use some looking over soon).  the configuration for the antispam agent now is quite small.  there are only three things that i don’t currently handle properly wrt the configuration, but they can come later.  the first is that the spam host configuration parameter (host) in the old code could have returned more than one item.  the current implementation only expects one line (a string).  this should be changed to allow for the use of more than one spamd agent if desired.  the other two are, incidentally, identical in implementation of the configuration.  they are the allow and deny lists for the agent itself.  the agent could be configured to assume a connecting ip address is a spammer and automatically dropping the mail.  i’m not sure if we want to keep this in the system yet.  the code is still there to check the lists, however they will always contain no items in them,, so it only provides a minor slowdown atm.

hopefully i can get to taking a look at antivirus now that antispam mostly works (at least for me). that way we can get out m2 quicker.

ok, i know. A POST FROM FAT??!!! and now here is another on the same day. i just couldn’t stop hacking and got most of antispam running. basic functionality exists and seems to work against spamassassin. there are some configuration settings that i don’t set yet but here are the ones that are available atm:

• enabled : boolean
  • are antispam services desired (this does not cause the agent to start automatically, that setting is in the manager, this just allows one to turn on or off the service)
• timeout : integer
  • connection timeout to the spamd daemon
  • defaults to 20 ms
• header_threshold : double
  • spam threshold to add header fields denoting spam checking and results
  • defaults to -9999
• drop_threshold : double
  • threshold to drop the message from the queue entirely
  • defaults to 9999
• quarantine_queue : integer
  • (i’m not sure what this setting does atm, but it was there)
  • defaults to queue 0
• feedback_enabled
  • (i’m not sure yet what feedback is)
  • defaults to false

there is code to do the hostname stuff, but as i look at it it is probably incorrect. in either case, it does connect to the spamd on localhost’s default spamd port and scans the mail. as this is night number two of 3:00am bedtimes, i’m gonna head out for now, but i’m coming back to this later today

it’s been a long while since i last blogged on what was going on in the bongo world. i’ve been pretty busy of late at work with a big project, and as such bongo time has been a little limited. nevertheless, i’ve been doing some good work in several areas of late.

the first is smtp. rprice from novell (now messaging architects) and i were chatting one day and he alerted me to a possible bug in smtp that would need some attention. i’ve spent a lot of time in re-working portions of smtp to hopefully fix the problem and will release the patch for it shortly. i’m waiting for ma to release their patch purely from a courtesy point of view (and because i like rprice so much

the issue actually should affect any queue agent, but currently smtp is the only queue agent that we have that functions (more on that in a minute). it turned out to be not as bad as i thought because i was having issues understanding one particular function that is quite complex in what it is trying to do and what buffers it is trying to use. anyhow, the new stuff should do a better job albeit using slightly more memory and possibly just a hair slower. i guess it remains to be seen though….

antispam. it has been on the plate for some time. i stayed up wayyyyy too late last night playing with it. i learned json and jpath (neither of which i had really used before) and i started playing with the store and alex’s json config parsing stuff. after a while, with alex’s help, i finally got stuff going and am about half done with the config stuff. i’m gonna have to re-work it slightly. i was trying to consolidate two config functions into one, but i think i’ll just leave it in two. so basically the re-work of the current code is just a copy and paste over to a different function. so far it is going really well, and i hope to have antispam running before the weekend is done.

on another front, i saw the mockup for the hawkeye (which i hadn’t seen at all up to this point). i think it is seriously amazing! we have one amazing product. this thing is gonna be so amazing. i can’t wait to see what comes out of it.