Adding TLS support to Log4j SMTP Appender

SMTPAppender that comes with log4j is a pretty useful Appender, allowing easily to start getting email alerts for errors in your application. But, this class is missing one important property (well, maybe more than one...): TLS support. Most modern mail servers use TLS for sending mails. TLS (Transport Layer Security) is a secure way for transfering information between two machines. For example, if you are using Google Apps (or even if you have a regular Gmail account) and you would like to use your account (user name and password of course) to send mail using SMTPAppender you won't be able to do it. That is because, Google allows sending mails only using TLS.

In this post: "Sending Email alerts with Log4j – Controlled Alerts" I showed how log4j SMTPAppender class can be extended to allow email alerts to be more controlled. In this post: "Sending SMS alerts with Log4j using ipipi.com", I showed how log4j SMTPAppender can be extended to send SMS error messages using ipipi.com service.
Both posts use this class: BaseFilteredSMTPAppender as a basic class for adding more neat capabilities to SMTPAppender.

I will show you how this class (BaseFilteredSMTPAppender) can be easily changed, to add the SMTPAppender TLS capabilities.
Unfortunatly, SMTPAppender class was not designed so well. It does not allow any control over the properties used for the creation of javax.mail.Session instance. In order to add TLS support to mail sending, we simply have to add to the javax.mail.Session class the following property:

props.put("mail.smtp.starttls.enable","true");

Where props is a simple Property instance containing mail properties.
And then we get the session instance, for example, by doing:

Session session = Session.getInstance(props);

But, as was said before, we have no access to the Properties instance, and therefore cannot add the TLS property.

Luckily, SMTPAppender does contain a protected method, for getting the Session instance: createSession. We can override this method and create a Session instance that contains TLS support. I copied the code of this method exactly as it was on the original SMTPAppender, and simply added the TLS property (note that TLS support is added only if user actually use the TLS property in the appender definition). This is how the class BaseFilteredSMTPAppender looks after adding the TLS support:

import org.apache.log4j.net.SMTPAppender;
import javax.mail.Authenticator;
import javax.mail.PasswordAuthentication;
import javax.mail.Session;
import java.util.*;
public abstract class BaseFilteredSMTPAppender extends SMTPAppender {
  private int timeFrame;
  private int maxEMails;
  protected long timeFrameMillis;
  protected Boolean isTLS;
  protected List<Date> exceptionDates = new ArrayList<Date>();
  public int getTimeFrame() {
    return timeFrame;
  }
  public void setTimeFrame(int timeFrame) {
    this.timeFrame = timeFrame;
  }
  public int getMaxEMails() {
    return maxEMails;
  }
  public void setMaxEMails(int maxEMails) {
    this.maxEMails = maxEMails;
  }
  public void setTLS(boolean isTLS) {
    this.isTLS = isTLS;
  }
  @Override
  public void activateOptions() {
    super.activateOptions();
    timeFrameMillis = timeFrame * 60 * 1000;
  }
  @Override
  protected Session createSession() {
    Properties props = null;
    try {
        props = new Properties (System.getProperties());
    } catch(SecurityException ex) {
        props = new Properties();
    }
    if (getSMTPHost() != null) {
      props.put("mail.smtp.host", getSMTPHost());
    }
    Authenticator auth = null;
    if(getSMTPUsername() != null && getSMTPPassword() != null) {
      props.put("mail.smtp.auth", "true");
      auth = new Authenticator() {
        protected PasswordAuthentication getPasswordAuthentication() {
          return new PasswordAuthentication(getSMTPUsername(), getSMTPPassword());
        }
      };
    }
    if (isTLS != null && isTLS)
    {
      props.put("mail.smtp.starttls.enable","true");
    }
    Session session = Session.getInstance(props, auth);
    if (getSMTPDebug()) {
        session.setDebug(getSMTPDebug());
    }
    return session;
  }
  protected void cleanTimedoutExceptions() {
    Date current = new Date();
    // Remove timedout exceptions
    Iterator<Date> itr = exceptionDates.iterator();
    while (itr.hasNext()) {
      Date exceptionDate = itr.next();
      if (current.getTime() - exceptionDate.getTime() > timeFrameMillis) {
        itr.remove();
      } else {
        break;
      }
    }
  }
  protected void addException() {
    exceptionDates.add(new Date());
  }
  protected boolean isSendMailAllowed() {
    return exceptionDates.size() < maxEMails;
  }
}

In order to use the Appender, only one additional parameter has to be added to the parameters already used and shown in the previous 2 blogs dealing with log4j SMTPAppender: TLS.

In order to make things a little interesting I will show you log4j configuration example using XML file instead of properties file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">  
   <appender name="console" class="org.apache.log4j.ConsoleAppender">
      <param name="Target" value="System.out"/>
      <layout class="org.apache.log4j.PatternLayout">
         <param name="ConversionPattern" value="%d{HH:mm:ss,SSS} %-5p [%c{1}] %m%n"/>
      </layout>
   </appender>
    <appender name="email" class="com.bashan.log4j.appender.FilteredSMTPAppender">
        <param name="BufferSize" value="10"/>
        <param name="SMTPHost" value="smtp.gmail.com"/>
        <param name="SMTPUsername" value="username@gmail.com"/>
        <param name="SMTPPassword" value="password"/>
        <param name="TLS" value="true"/>
        <param name="TimeFrame" value="10"/>
        <param name="MaxEMails" value="2"/>
        <param name="From" value="username@gmail.com"/>
        <param name="To" value="anotherUsername@gmail.com"/>
        <param name="Subject" value="Server Error"/>
        <layout class="org.apache.log4j.PatternLayout">
          <param name="ConversionPattern" value="%d{HH:mm:ss,SSS} %-5p [%c{1}] %m%n"/>
        </layout>
    </appender>  
    <root>
      <priority value="warn"/>
      <appender-ref ref="console"/>
      <appender-ref ref="email"/>
   </root>
</log4j:configuration>

This file can be dropped on your root src directory and log4j will know to find and load it automatically.

Note that TLS property is not mandatory. If you don't need TLS support, you can simply not add it to the appender properties.

Sending SMS alerts with Log4j using ipipi.com

In the post “Sending Email alerts with Log4j” and in the post “Sending Email alerts with Log4j – Controlled Alerts” I wrote about a simple way of sending email alerts using the ready log4j appender SMTPAppender. But, there are times in which an organization wants to send system alerts directly to a cell phone, in order to be notified about a problem as soon as possible.

ipipi.com is SMS service allowing sending SMS messages to almost every cell phone in the world. There is a support for sending SMS messages over SMTP protocol. Extending log4j SMTP appender to send SMS alert messages is easy task.

There is only one thing important to notice: Sending error message as SMS has to be much shorter than sending email. SMS messages can have very few characters. For this reason log4j SMTP appender has to be altered in order to send only the error string written to the log and the first line from the exception stack trace. This is done in the following class: FilteredShortSMTPAppender. Note that this class extends BaseFilteredSMTPAppender from the post “Sending Email alerts with Log4j – Controlled Alerts”:

import org.apache.log4j.spi.LoggingEvent;
import org.apache.log4j.helpers.LogLog;
import java.util.Date;
import javax.mail.Multipart;
import javax.mail.Transport;
import javax.mail.internet.MimeMultipart;
import javax.mail.internet.MimeBodyPart;

public class FilteredShortSMTPAppender extends BaseFilteredSMTPAppender {
  @Override
  public void activateOptions()
  {
    super.activateOptions();
    setBufferSize(1);
  }

  @Override
  protected void sendBuffer() {
    cleanTimedoutExceptions();
    if (isSendMailAllowed()) {
      try {
        MimeBodyPart part = new MimeBodyPart();
        StringBuffer sbuf = new StringBuffer();
        String t = layout.getHeader();
        if (t != null) {
          sbuf.append(t);
        }
        LoggingEvent event = cb.get();
        sbuf.append(layout.format(event));
        if (layout.ignoresThrowable()) {
          String[] s = event.getThrowableStrRep();
          if (s != null && s.length > 0) {
            sbuf.append(s[0]);
        }
        t = layout.getFooter();
          }
        if (t != null) {
          sbuf.append(t);
        }
        part.setContent(sbuf.toString(), layout.getContentType());
        Multipart mp = new MimeMultipart();
        mp.addBodyPart(part);
        msg.setContent(mp);
        msg.setSentDate(new Date());
        Transport.send(msg);
        addException();
      }
      catch (Exception e) {
        LogLog.error("Error occured while sending e-mail notification.", e);
      }
    }
  }
}

The class FilteredShortSMTPAppender allows to send email alerts in a short version. It can be used exactly the same as FilteredSMTPAppender, besides the parameter: BufferSize is no longer needed, since we do not send log history with the mail message.

After we have the class FilteredShortSMTPAppender we can easily extend it to send SMS email alerts over SMTP using the ipipi.com service:

import org.apache.log4j.helpers.LogLog;
import java.io.IOException;
import java.util.Properties;

public final class IPIPISmsOverSmtpAppender extends FilteredShortSMTPAppender {
  private static final String KEY_HOST = "host";
  private static final String TO_SERVER = "to.server";
  private String toServer;

  @Override
  public void activateOptions() {
    Properties properties = new Properties();
    try {
      properties.load(IPIPISmsOverSmtpAppender.class.getResourceAsStream("ipipi.properties"));
      setSMTPHost(properties.getProperty(KEY_HOST));
      setFrom(getSMTPUsername() + "@" + getSMTPHost());
      toServer = properties.getProperty(TO_SERVER);
      setTo(parseAddress(getTo()));  
    }
    catch (IOException ioe) {
      LogLog.error("Failed loading IPIPI service properties", ioe);
    }
    super.activateOptions();
  }

  private String parseAddress(String addressStr) {
    String[] addresses = addressStr.split(",");
    StringBuffer sb = new StringBuffer();
    for (String address : addresses) {
      sb.append(address).append("@").append(toServer).append(",");
    }
    return sb.substring(0, sb.length() - 1);
  }
}

Note that the information regarding the ipipi.com service is on external file named ipipi.properties. This file should be located on the same package as the IPIPISmsOverSmtpAppender class:

host=ipipi.com
to.server=sms.ipipi.com

This class extends the capabilities of the class FilteredShortSMTPAppender to allow adding cell phone numbers instead of email addresses. In your appender configuration file you can simply insert cell phone numbers and this code will know to convert the cell phone numbers to email addresses used for ipipi.com convention. The service allows sending SMS message to almost any cell phone on the planet by using the cell phone owner as the email address prefix. for example, you can simply send SMS message to a cell phone by emailing to: 972541234567@ipipi.com. Of course, that the service costs money, and in order to send SMS messages you must first register to the service and buy a package of SMS messages. Your registered username and password is used as login details for ipipi.com SMTP server.

This is an example of the IPIPISmsOverSmtpAppender log4j configuration:

log4j.rootLogger=INFO, a, sms
log4j.appender.a=org.apache.log4j.ConsoleAppender
log4j.appender.a.layout=org.apache.log4j.PatternLayout
log4j.appender.a.layout.ConversionPattern=%d{HH:mm:ss} %-5p [%c{1}]: %m%n
log4j.appender.sms=com.bashan.log4j.IPIPISmsOverSmtpAppender
log4j.appender.sms.SMTPUsername=bashan
log4j.appender.sms.SMTPPassword=bashan
log4j.appender.sms.TimeFrame=10
log4j.appender.sms.MaxEMails=30
log4j.appender.sms.To=972541234567,972542345678
log4j.appender.sms.layout=org.apache.log4j.PatternLayout
log4j.appender.sms.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

Note for few things:

• Phone numbers must contain country prefix.
  
• TimeFrame and MaxEMails parameters from the class: FilteredSMTPAppender exist on this appender as well.
  
• BufferSize parameter is no longer used.

Sending Email alerts with Log4j – Controlled Alerts

In one of my recent posts I wrote about sending email alerts using log4j. The SMTPAppender supplied with log4j is nice, but it a lacks of a basic important feature, when it comes to sending mail alerts: mails sent are not controlled. On every exception logged by log4j a mail message is sent. This is not always a good thing, since there are times servers may encounter massive amount of exceptions in a short period of time. When such a thing happens we don’t always care about all the exceptions, usually because when big amount of exceptions happen on a short period of time, there is a high probability they are all of the same type and happen from the same cause.

For example, the server may be trying to send some information over the network to a remote machine. If the server fails, it tries to send the information again after one second. Suppose the network is going down for 30 minutes. This will cause the server to send big amount of email alerts. When actually it is necessary to receive only one email alert notifying about the problem.

For this reason, I wrote an email alerts appender that extends the capabilities of the SMTPAppender supplied by log4j, to add basic level of controllability over the mail alerts send by log4j. The appender simply gives adds 2 new parameters:

• TimeFrame: Time frame in minutes.
• MaxEmails: Maximum allowed email alerts.

Both parameters restricts the maximum allowed email alerts that can be send in a given time frame.

For example, you can define that the maximum amount of emails in 30 minutes is 10. In a case of a severe server problem that generates big amount of email alerts, the appender simply stops sending mails after 10 messages were already sent. After 30 minutes, the server will allow mails again. This is not an ideal solution, but it is simple and neat, and it answers most of the needs from a basic alerts mechanism.

The appender is constructed from 2 classes:

• BaseFilteredSMTPAppender: which is an abstract class that defines the basic behavior of the controlled smtp appender.
• FilteredSMTPAppender: which extend BaseFilteredSMTPAppender to create the actual controlled filtering of email alerts.

This is the code of the BaseFilteredSMTPAppender:

import org.apache.log4j.net.SMTPAppender;
import java.util.Date;
import java.util.List;
import java.util.ArrayList;
import java.util.Iterator;
public abstract class BaseFilteredSMTPAppender extends SMTPAppender {
  private int timeFrame;
  private int maxEMails;
  protected long timeFrameMillis;
  protected List<Date> exceptionDates = new ArrayList<Date>();
  public int getTimeFrame() {
    return timeFrame;
  }
  public void setTimeFrame(int timeFrame) {
    this.timeFrame = timeFrame;
  }
  public int getMaxEMails() {
    return maxEMails;
  }
  public void setMaxEMails(int maxEMails) {
    this.maxEMails = maxEMails;
  }
  @Override
  public void activateOptions()
  {
    super.activateOptions();
    timeFrameMillis = timeFrame * 60 * 1000;
  }
  protected void cleanTimedoutExceptions()
  {
    Date current = new Date();
    // Remove timedout exceptions
    Iterator<Date> itr = exceptionDates.iterator();
    while (itr.hasNext())
    {
      Date exceptionDate = itr.next();
      if (current.getTime() - exceptionDate.getTime() > timeFrameMillis)
      {
        itr.remove();
      }
      else
      {
        break;
      }
    }
  }
  protected void addException()
  {
    exceptionDates.add(new Date());
  }
  protected boolean isSendMailAllowed()
  {
    return exceptionDates.size() < maxEMails;
  }
}

And the code of the FilteredSMTPAppender which is much simpler and contains only the actual filtering:

import org.apache.log4j.net.SMTPAppender;
public class FilteredSMTPAppender extends BaseFilteredSMTPAppender {
  @Override
  protected void sendBuffer()
  {
    cleanTimedoutExceptions();
    if (isSendMailAllowed())
    {
      super.sendBuffer();
      addException();
    }
  }
}

And log4j configuration is very similar to the one already posted on the previous blog dealing with email alerts, besides adding the 2 new parameters: TimeFrame, MaxEmails. Here is an example if how it looks:

log4j.rootLogger=INFO, a, email
log4j.appender.a=org.apache.log4j.ConsoleAppender
log4j.appender.a.layout=org.apache.log4j.PatternLayout
log4j.appender.a.layout.ConversionPattern=%d{HH:mm:ss} %-5p [%c{1}]: %m%n

log4j.appender.email=com.bashan.log4j.FilteredSMTPAppender
log4j.appender.email.BufferSize=10
log4j.appender.email.SMTPHost=mysmtp.mailserver.net
log4j.appender.email.SMTPUsername=myusername@mycompany.com
log4j.appender.email.SMTPPassword=mypassword
log4j.appender.email.TimeFrame=30
log4j.appender.email.MaxEMails=10
log4j.appender.email.From=admin@mycompany.com
log4j.appender.email.To=me@mycompany.com
log4j.appender.email.Subject=My Module Error
log4j.appender.email.layout=org.apache.log4j.PatternLayout
log4j.appender.email.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

That’s it. Now you can get email alerts, without your mail box being blocked… ;-)

Sending Email alerts with Log4j - SMTPAppender

Log4j is a great open source logging framework. It offers easy, modular and extensive way of adding logging capabilities to your application. But, the truth is, I don’t really like log files. I prefer the task of analyzing thousands lines of logs as last option. I always prefer to know about problems as soon as they happen. The sooner the better… Luckily, log4j supplies out of the box Appender for sending email alerts. If you use log4j in your application. You can easily configure log4 to send email alerts for all your error level errors.
By default mails are sent only for error level logs, but if you insist you can configure it to lower level logs (it is not a good idea doing this. You don’t want your application to start sending tons of emails…). The Appender used for sending mail is called: org.apache.log4j.net.SMTPAppender
Note that older versions of log4j may not have this Appender.
This is a typical properties file adding SMTPAppender:

log4j.rootLogger=INFO, a, email
log4j.appender.a=org.apache.log4j.ConsoleAppender
log4j.appender.a.layout=org.apache.log4j.PatternLayout
log4j.appender.a.layout.ConversionPattern=%d{HH:mm:ss} %-5p [%c{1}]: %m%n
log4j.appender.email=org.apache.log4j.net.SMTPAppender
log4j.appender.email.BufferSize=10
log4j.appender.email.SMTPHost=mysmtp.mailserver.net
log4j.appender.email.From=admin@mycompany.com
log4j.appender.email.To=me@mycompany.com
log4j.appender.email.Subject=My Module Error
log4j.appender.email.layout=org.apache.log4j.PatternLayout
log4j.appender.email.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

Don’t forget that in order to send mail using java you will have to add: mail.jar and activation.jar to your class path.
The nice thing in this Appender is that you can send email alerts containing more than just your Exception. You can also add lines that were logged before the exception. This will make it much easier to understand the cause of your exception. The number of log lines that will be sent can be determine by “BufferSize” property. For example: if BufferSize=10, then your email will also contain the 9 lines logged before the exception.
If your SMTP mail server is not on the same network of your server, you would probably won’t be able to send emails without authentication. You can easily overcome this problem by simply adding username and password properties:

log4j.appender.email.SMTPUsername=myusername@mycompany.com
log4j.appender.email.SMTPPassword=mypassword

Note that also here, older versions of log4j may not have support for “username” and “password” properties.
That’s it. you can now check you mail box. You should be able to get email alerts every time an exception is logged on your application. You can also make a fake test to your configuration. This will make it easier to see how your mail will look:

log.info("some fake info");
try
{
throw new Exception("some fake exception");
}
catch (Exception e)
{
log.error("Fake exception occurred", e);
}

I found only one disturbing this with this Appender: It doesn’t send the mail on a different Thread. This causes the the current thread to be stuck for a second when sending mail. I don’t find it as a big problem, since usually Exceptions don’t happen much often (at least on production environments where this Appender will mostly be used). But once such exception happens and it is being repeated in a loop, the system may hang and unpleasant things might happen. On the other hand, sending the mail on a different Thread won’t cause the current thread to be stuck, but may flood the system with big amount of mails.