Sending SMS alerts with Log4j using ipipi.com

In the post “Sending Email alerts with Log4j” and in the post “Sending Email alerts with Log4j – Controlled Alerts” I wrote about a simple way of sending email alerts using the ready log4j appender SMTPAppender. But, there are times in which an organization wants to send system alerts directly to a cell phone, in order to be notified about a problem as soon as possible.

ipipi.com is SMS service allowing sending SMS messages to almost every cell phone in the world. There is a support for sending SMS messages over SMTP protocol. Extending log4j SMTP appender to send SMS alert messages is easy task.

There is only one thing important to notice: Sending error message as SMS has to be much shorter than sending email. SMS messages can have very few characters. For this reason log4j SMTP appender has to be altered in order to send only the error string written to the log and the first line from the exception stack trace. This is done in the following class: FilteredShortSMTPAppender. Note that this class extends BaseFilteredSMTPAppender from the post “Sending Email alerts with Log4j – Controlled Alerts”:

import org.apache.log4j.spi.LoggingEvent;
import org.apache.log4j.helpers.LogLog;
import java.util.Date;
import javax.mail.Multipart;
import javax.mail.Transport;
import javax.mail.internet.MimeMultipart;
import javax.mail.internet.MimeBodyPart;

public class FilteredShortSMTPAppender extends BaseFilteredSMTPAppender {
  @Override
  public void activateOptions()
  {
    super.activateOptions();
    setBufferSize(1);
  }

  @Override
  protected void sendBuffer() {
    cleanTimedoutExceptions();
    if (isSendMailAllowed()) {
      try {
        MimeBodyPart part = new MimeBodyPart();
        StringBuffer sbuf = new StringBuffer();
        String t = layout.getHeader();
        if (t != null) {
          sbuf.append(t);
        }
        LoggingEvent event = cb.get();
        sbuf.append(layout.format(event));
        if (layout.ignoresThrowable()) {
          String[] s = event.getThrowableStrRep();
          if (s != null && s.length > 0) {
            sbuf.append(s[0]);
        }
        t = layout.getFooter();
          }
        if (t != null) {
          sbuf.append(t);
        }
        part.setContent(sbuf.toString(), layout.getContentType());
        Multipart mp = new MimeMultipart();
        mp.addBodyPart(part);
        msg.setContent(mp);
        msg.setSentDate(new Date());
        Transport.send(msg);
        addException();
      }
      catch (Exception e) {
        LogLog.error("Error occured while sending e-mail notification.", e);
      }
    }
  }
}

The class FilteredShortSMTPAppender allows to send email alerts in a short version. It can be used exactly the same as FilteredSMTPAppender, besides the parameter: BufferSize is no longer needed, since we do not send log history with the mail message.

After we have the class FilteredShortSMTPAppender we can easily extend it to send SMS email alerts over SMTP using the ipipi.com service:

import org.apache.log4j.helpers.LogLog;
import java.io.IOException;
import java.util.Properties;

public final class IPIPISmsOverSmtpAppender extends FilteredShortSMTPAppender {
  private static final String KEY_HOST = "host";
  private static final String TO_SERVER = "to.server";
  private String toServer;

  @Override
  public void activateOptions() {
    Properties properties = new Properties();
    try {
      properties.load(IPIPISmsOverSmtpAppender.class.getResourceAsStream("ipipi.properties"));
      setSMTPHost(properties.getProperty(KEY_HOST));
      setFrom(getSMTPUsername() + "@" + getSMTPHost());
      toServer = properties.getProperty(TO_SERVER);
      setTo(parseAddress(getTo()));  
    }
    catch (IOException ioe) {
      LogLog.error("Failed loading IPIPI service properties", ioe);
    }
    super.activateOptions();
  }

  private String parseAddress(String addressStr) {
    String[] addresses = addressStr.split(",");
    StringBuffer sb = new StringBuffer();
    for (String address : addresses) {
      sb.append(address).append("@").append(toServer).append(",");
    }
    return sb.substring(0, sb.length() - 1);
  }
}

Note that the information regarding the ipipi.com service is on external file named ipipi.properties. This file should be located on the same package as the IPIPISmsOverSmtpAppender class:

host=ipipi.com
to.server=sms.ipipi.com

This class extends the capabilities of the class FilteredShortSMTPAppender to allow adding cell phone numbers instead of email addresses. In your appender configuration file you can simply insert cell phone numbers and this code will know to convert the cell phone numbers to email addresses used for ipipi.com convention. The service allows sending SMS message to almost any cell phone on the planet by using the cell phone owner as the email address prefix. for example, you can simply send SMS message to a cell phone by emailing to: 972541234567@ipipi.com. Of course, that the service costs money, and in order to send SMS messages you must first register to the service and buy a package of SMS messages. Your registered username and password is used as login details for ipipi.com SMTP server.

This is an example of the IPIPISmsOverSmtpAppender log4j configuration:

log4j.rootLogger=INFO, a, sms
log4j.appender.a=org.apache.log4j.ConsoleAppender
log4j.appender.a.layout=org.apache.log4j.PatternLayout
log4j.appender.a.layout.ConversionPattern=%d{HH:mm:ss} %-5p [%c{1}]: %m%n
log4j.appender.sms=com.bashan.log4j.IPIPISmsOverSmtpAppender
log4j.appender.sms.SMTPUsername=bashan
log4j.appender.sms.SMTPPassword=bashan
log4j.appender.sms.TimeFrame=10
log4j.appender.sms.MaxEMails=30
log4j.appender.sms.To=972541234567,972542345678
log4j.appender.sms.layout=org.apache.log4j.PatternLayout
log4j.appender.sms.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

Note for few things:

• Phone numbers must contain country prefix.
  
• TimeFrame and MaxEMails parameters from the class: FilteredSMTPAppender exist on this appender as well.
  
• BufferSize parameter is no longer used.

Sending Email alerts with Log4j – Controlled Alerts

In one of my recent posts I wrote about sending email alerts using log4j. The SMTPAppender supplied with log4j is nice, but it a lacks of a basic important feature, when it comes to sending mail alerts: mails sent are not controlled. On every exception logged by log4j a mail message is sent. This is not always a good thing, since there are times servers may encounter massive amount of exceptions in a short period of time. When such a thing happens we don’t always care about all the exceptions, usually because when big amount of exceptions happen on a short period of time, there is a high probability they are all of the same type and happen from the same cause.

For example, the server may be trying to send some information over the network to a remote machine. If the server fails, it tries to send the information again after one second. Suppose the network is going down for 30 minutes. This will cause the server to send big amount of email alerts. When actually it is necessary to receive only one email alert notifying about the problem.

For this reason, I wrote an email alerts appender that extends the capabilities of the SMTPAppender supplied by log4j, to add basic level of controllability over the mail alerts send by log4j. The appender simply gives adds 2 new parameters:

• TimeFrame: Time frame in minutes.
• MaxEmails: Maximum allowed email alerts.

Both parameters restricts the maximum allowed email alerts that can be send in a given time frame.

For example, you can define that the maximum amount of emails in 30 minutes is 10. In a case of a severe server problem that generates big amount of email alerts, the appender simply stops sending mails after 10 messages were already sent. After 30 minutes, the server will allow mails again. This is not an ideal solution, but it is simple and neat, and it answers most of the needs from a basic alerts mechanism.

The appender is constructed from 2 classes:

• BaseFilteredSMTPAppender: which is an abstract class that defines the basic behavior of the controlled smtp appender.
• FilteredSMTPAppender: which extend BaseFilteredSMTPAppender to create the actual controlled filtering of email alerts.

This is the code of the BaseFilteredSMTPAppender:

import org.apache.log4j.net.SMTPAppender;
import java.util.Date;
import java.util.List;
import java.util.ArrayList;
import java.util.Iterator;
public abstract class BaseFilteredSMTPAppender extends SMTPAppender {
  private int timeFrame;
  private int maxEMails;
  protected long timeFrameMillis;
  protected List<Date> exceptionDates = new ArrayList<Date>();
  public int getTimeFrame() {
    return timeFrame;
  }
  public void setTimeFrame(int timeFrame) {
    this.timeFrame = timeFrame;
  }
  public int getMaxEMails() {
    return maxEMails;
  }
  public void setMaxEMails(int maxEMails) {
    this.maxEMails = maxEMails;
  }
  @Override
  public void activateOptions()
  {
    super.activateOptions();
    timeFrameMillis = timeFrame * 60 * 1000;
  }
  protected void cleanTimedoutExceptions()
  {
    Date current = new Date();
    // Remove timedout exceptions
    Iterator<Date> itr = exceptionDates.iterator();
    while (itr.hasNext())
    {
      Date exceptionDate = itr.next();
      if (current.getTime() - exceptionDate.getTime() > timeFrameMillis)
      {
        itr.remove();
      }
      else
      {
        break;
      }
    }
  }
  protected void addException()
  {
    exceptionDates.add(new Date());
  }
  protected boolean isSendMailAllowed()
  {
    return exceptionDates.size() < maxEMails;
  }
}

And the code of the FilteredSMTPAppender which is much simpler and contains only the actual filtering:

import org.apache.log4j.net.SMTPAppender;
public class FilteredSMTPAppender extends BaseFilteredSMTPAppender {
  @Override
  protected void sendBuffer()
  {
    cleanTimedoutExceptions();
    if (isSendMailAllowed())
    {
      super.sendBuffer();
      addException();
    }
  }
}

And log4j configuration is very similar to the one already posted on the previous blog dealing with email alerts, besides adding the 2 new parameters: TimeFrame, MaxEmails. Here is an example if how it looks:

log4j.rootLogger=INFO, a, email
log4j.appender.a=org.apache.log4j.ConsoleAppender
log4j.appender.a.layout=org.apache.log4j.PatternLayout
log4j.appender.a.layout.ConversionPattern=%d{HH:mm:ss} %-5p [%c{1}]: %m%n

log4j.appender.email=com.bashan.log4j.FilteredSMTPAppender
log4j.appender.email.BufferSize=10
log4j.appender.email.SMTPHost=mysmtp.mailserver.net
log4j.appender.email.SMTPUsername=myusername@mycompany.com
log4j.appender.email.SMTPPassword=mypassword
log4j.appender.email.TimeFrame=30
log4j.appender.email.MaxEMails=10
log4j.appender.email.From=admin@mycompany.com
log4j.appender.email.To=me@mycompany.com
log4j.appender.email.Subject=My Module Error
log4j.appender.email.layout=org.apache.log4j.PatternLayout
log4j.appender.email.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

That’s it. Now you can get email alerts, without your mail box being blocked… ;-)

Sorting large text files

Sorting a very big text file is a problem in terms of memory resources. Ideally, if all data could load to memory, it was an easy task: simply sort data in memory and write it to a new file. But, there are times we need to sort a really big file and the physical memory is not enough.

I once had to write such a large file sorter, and had a hard time finding something ready on the web. Therefore, I wrote a sorter of my own. The logic behind sorting a very large file is not hard to understand once you get the point. In general, the steps needed are:

Phase 1: Split big file to several small sorted files

1. Read n lines from file to memory.
2. Sort lines in memory.
3. Write sorted lines to a temp file.
4. Repeat step 1 while reading next n lines to memory and writing sorted lines to a new temporary file.

Phase 2: Merge smaller files to a new sorted big file

1. Find the smallest line among all files.
2. Write line to a new file (this file is the new merged file).
3. Move to the next row in file (the file that had the smallest line).
4. Repeat step 1 until all lines in all files were read.

Here is the code for such a file sorter:

import java.io.*;
import java.util.*;
public class FileSort
{
  protected String filenameToSort;
  protected String filenameSorted;
  protected Comparator comparator;
  protected int maxCapacity;
  public FileSort(String filenameToSort, String filenameSorted, Comparator comparator, int maxCapacity)
  {
    this.filenameToSort = filenameToSort;
    this.filenameSorted = filenameSorted;
    this.comparator = comparator;
    this.maxCapacity = maxCapacity;
  }
  public void sort() throws IOException
  {
    BufferedReader bufferedReader = new BufferedReader(new FileReader(filenameToSort));
    String line = null;
    int fileIndex = 0;
    BufferedWriter bufferedWriter;
    do
    {
      List<String> lines = new ArrayList<String>(maxCapacity);
      for (int i = 0; i < maxCapacity; i++)
      {
        line = bufferedReader.readLine();
        if (line == null)
        {
          break;
        }
        else
        {
          lines.add(line);
        }
      }
      Collections.sort(lines, comparator);
      bufferedWriter = new BufferedWriter(new FileWriter(filenameToSort + ".tmp" + fileIndex++));
      for (String _line : lines)
      {
        bufferedWriter.write(_line);
        bufferedWriter.newLine();
      }
      bufferedWriter.flush();
      bufferedWriter.close();
    }
    while (line != null);
    bufferedReader.close();
    mergeFiles(fileIndex);
  }
  public void mergeFiles(int numFiles) throws IOException
  {
    BufferedWriter bufferedWriter = new BufferedWriter(new FileWriter(filenameSorted));
    List<MergeFile> mergeFiles = new ArrayList<MergeFile>();
    for (int i = 0; i < numFiles; i++)
    {
      mergeFiles.add(new MergeFile(filenameToSort + ".tmp" + i));
    }
    do
    {
      // Find smallest line
      Iterator<MergeFile> iterator = mergeFiles.iterator();
      MergeFile minMergeFile = null;
      while (iterator.hasNext())
      {
        MergeFile mergeFile = iterator.next();
        if (mergeFile.line != null) // File has more lines
        {
          if (minMergeFile == null || comparator.compare(mergeFile.line, minMergeFile.line) < 0)
          {
            minMergeFile = mergeFile;
          }
        }
        else // No more lines in file. No need to iterate it
        {
          mergeFile.removeFile();
          iterator.remove();
        }
      }
      // Write smallest line to file
      if (minMergeFile != null)
      {    
        bufferedWriter.write(minMergeFile.line);
        bufferedWriter.newLine();
        minMergeFile.readLine();
      }
    }
    while (mergeFiles.size() > 0); // As long as there are files to read
    bufferedWriter.flush();
    bufferedWriter.close();
  }
  private final class MergeFile
  {
    BufferedReader bufferedReader;
    String filename;
    String line;
    boolean isReadNextRow = true;
    public MergeFile(String filename) throws IOException
    {
      this.filename = filename;
      bufferedReader = new BufferedReader(new FileReader(filename));
      readLine();
    }
    public void readLine() throws IOException
    {
      isReadNextRow = (line = bufferedReader.readLine()) != null;
    }
    public void removeFile() throws IOException
    {
      bufferedReader.close();
      new File(filename).delete();
    }
  }
}

Note for several things on this file sorter:

1. It doesn’t limit the number of simultaneous open files. It shouldn't be much of a problem, since the maximum open files limit is pretty high and also can be configured (at least on Linux machines). If such limit exists you can always merge a group of files at a time (the size of the group is the maximum allowed open files). Then merge the new grouped files to a single file.
   
2. This file sorter does not define the logic in which the lines are sorted.It should be supplied using a Comparator.
   
3. The maximum number of rows to read to memory is also defined passed as a parameter to the sorter.

Suppose you would like to sort several big files to a single huge sorted file. The FileSort class can be easily extended to support sorting multiple big files. It can be done by first merging all files to a single file. Then sorting it using the FileSorter:

public class MultipleFileSort extends FileSort {
    private static final int BUFFER_SIZE = 1024 * 4;
    protected String[] filesToSort;
    public MultipleFileSort(String[] filesToSort, String filenameSorted, Comparator comparator, int maxReadLines) {
        super(filenameSorted + ".tmp", filenameSorted, comparator, maxReadLines);
        this.filesToSort = filesToSort;
    }
    public static void mergeFiles(String[] files, String filenameMerged) throws IOException {
        // Open all files for reading
        InputStream[] inputs = new InputStream[files.length];
        OutputStream os = null;
        try {
            for (int i = 0; i < files.length; i++) {
                inputs[i] = new FileInputStream(files[i]);
            }
            // Open file for writing
            os = new FileOutputStream(filenameMerged);
            byte[] buffer = new byte[BUFFER_SIZE];
            int len;
            for (InputStream is : inputs) {
                while ((len = is.read(buffer)) > 0) {
                    os.write(buffer, 0, len);
                }
            }
        }
        finally {
            for (InputStream is : inputs) {
                FileUtils.close(is);
            }
            FileUtils.close(os);
        }
    }
    public void sort() throws IOException {
        FileUtils.mergeFiles(filesToSort, filenameToSort);
        super.sort();
        new File(filenameToSort).delete();
    }
}

And finally example of using the MultipleFileSort class:

    public static void main(String[] args) {
        try {
            MultipleFileSort multipleFileSort = new MultipleFileSort(
                    new String[]{"in_file1.txt", "in_file2.txt", "in_file3.txt"},
                    "out_sorted.txt", new Comparator<String>() {
                        public int compare(String o1, String o2) {
                            return (o1).compareTo(o2);
                        }
                    }, 1000);
            multipleFileSort.sort();
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }

Get Week and Month names in Java

More and more applications these days are becoming global. Globalized applications are usually multilingual. Most applications need somewhere a calendar input.The web offers a lot of calendar solutions. These solutions usually allows passing to the calendar information regarding the day and month names. This data can be easily extracted by Java without needing to maintain resources list for all desired languages.

The class that holds all this information is named: DateFormatSymbols. This class accepts the Locale as a parameter. The Locale will determine the language in which the week and month information will be shown.

Here is an example of how to get week names:

public static String[] getShortWeekDays(Locale locale)
{
  String[] weekDays = new DateFormatSymbols(locale).getShortWeekdays();
  String[] retWeekDays = new String[7];
  System.arraycopy(weekDays, 1, retWeekDays, 0, 7);
  return retWeekDays;
}

Note, that the first element in the array was ignored. Java returns the first item as empty, and the actual values are starting from index 1. That is to be correlated with Calendar.SUNDAY which returns the number: 1 rather than: 0. If you will want in some way to do: weekDays[Calendar.SUNDAY], it was smarter to leave the original array untouched.

In the same way it is very simple to get long week days:

String[] weekDays = new DateFormatSymbols(locale).getWeekdays();

Getting the short month names:

public static String[] getShortMonths(Locale locale)
{
  String[] months = new DateFormatSymbols(locale).getShortMonths();
  String retMonths[] = new String[12];
  System.arraycopy(months, 0, retMonths, 0, 12);
  return retMonths;
}

This time the returned original array starts from index: 0, but has an additional last empty element. This code fixes this issue, by ignoring the last element. In months Java has chosen to start the index from: 0 and in weeks from: 1. This is a bit inconsistent, and I believe that there will always be someone that can give a long speech for what were the reasons for choosing this way.

Anyway, here is a simple example application that outputs week and month names in both short and regular formats in US Locale:

public static void main(String[] args) {

  System.out.println(Arrays.toString(getShortWeekDays(Locale.US)));
  System.out.println(Arrays.toString(getLongWeekDays(Locale.US)));
  System.out.println(Arrays.toString(getShortMonths(Locale.US)));
  System.out.println(Arrays.toString(getMonths(Locale.US)));
}

And here is the output for this code:

[Sun, Mon, Tue, Wed, Thu, Fri, Sat]
[Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday]
[Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec]
[January, February, March, April, May, June, July, August, September, October, November, December]

RichFaces new TinyMCE Editor

JSF RichFaces released a new HTML/Rich Text editor on their latest version (3.3).Frankly, it is about time… It definitely took them a while. But better later than never…

The editor is a wrapper for the JavaScript WYSIWYG editor open source project.

Using the editor is very simple, assuming your project already using JSF RichFaces:

<rich:editor id="article" width="700" height="400"
    value="#{myBean.text}" required="true">
</rich:editor>

It is also possible to control the editor properties in the same manner TinyMCE editor is manipulated, buy simply adding editor specific properties. For example, in order to position the toolbar on the “top” and align the buttons to the left:

<rich:editor id="article" theme="advanced" viewMode="visual"
  width="700" height="400" value="#{myBean.text}" required="true">
  <f:param name="theme_advanced_toolbar_location" value="top" />
  <f:param name="theme_advanced_toolbar_align" value="left" />
</rich:editor>

Sending Email alerts with Log4j - SMTPAppender

Log4j is a great open source logging framework. It offers easy, modular and extensive way of adding logging capabilities to your application. But, the truth is, I don’t really like log files. I prefer the task of analyzing thousands lines of logs as last option. I always prefer to know about problems as soon as they happen. The sooner the better… Luckily, log4j supplies out of the box Appender for sending email alerts. If you use log4j in your application. You can easily configure log4 to send email alerts for all your error level errors.
By default mails are sent only for error level logs, but if you insist you can configure it to lower level logs (it is not a good idea doing this. You don’t want your application to start sending tons of emails…). The Appender used for sending mail is called: org.apache.log4j.net.SMTPAppender
Note that older versions of log4j may not have this Appender.
This is a typical properties file adding SMTPAppender:

log4j.rootLogger=INFO, a, email
log4j.appender.a=org.apache.log4j.ConsoleAppender
log4j.appender.a.layout=org.apache.log4j.PatternLayout
log4j.appender.a.layout.ConversionPattern=%d{HH:mm:ss} %-5p [%c{1}]: %m%n
log4j.appender.email=org.apache.log4j.net.SMTPAppender
log4j.appender.email.BufferSize=10
log4j.appender.email.SMTPHost=mysmtp.mailserver.net
log4j.appender.email.From=admin@mycompany.com
log4j.appender.email.To=me@mycompany.com
log4j.appender.email.Subject=My Module Error
log4j.appender.email.layout=org.apache.log4j.PatternLayout
log4j.appender.email.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

Don’t forget that in order to send mail using java you will have to add: mail.jar and activation.jar to your class path.
The nice thing in this Appender is that you can send email alerts containing more than just your Exception. You can also add lines that were logged before the exception. This will make it much easier to understand the cause of your exception. The number of log lines that will be sent can be determine by “BufferSize” property. For example: if BufferSize=10, then your email will also contain the 9 lines logged before the exception.
If your SMTP mail server is not on the same network of your server, you would probably won’t be able to send emails without authentication. You can easily overcome this problem by simply adding username and password properties:

log4j.appender.email.SMTPUsername=myusername@mycompany.com
log4j.appender.email.SMTPPassword=mypassword

Note that also here, older versions of log4j may not have support for “username” and “password” properties.
That’s it. you can now check you mail box. You should be able to get email alerts every time an exception is logged on your application. You can also make a fake test to your configuration. This will make it easier to see how your mail will look:

log.info("some fake info");
try
{
throw new Exception("some fake exception");
}
catch (Exception e)
{
log.error("Fake exception occurred", e);
}

I found only one disturbing this with this Appender: It doesn’t send the mail on a different Thread. This causes the the current thread to be stuck for a second when sending mail. I don’t find it as a big problem, since usually Exceptions don’t happen much often (at least on production environments where this Appender will mostly be used). But once such exception happens and it is being repeated in a loop, the system may hang and unpleasant things might happen. On the other hand, sending the mail on a different Thread won’t cause the current thread to be stuck, but may flood the system with big amount of mails.