Open Source Application Monitoring: Catching Exceptions

Imagine if you will, you are working on a new critical application and you need to write the output of your process calculations to a file. Simple stuff, a few lines of Java later you have

File file = new File("myfile.txt");
try {
  boolean fileCreated = file.createNewFile();
  log.debug("fileCreated = " + fileCreated);
} catch (IOException e) {
  log.error("Could not create file",e);
}

You even remembered to log the exception to the logs just in case there was a problem in production. A few weeks later the code ships and works perfectly for weeks until one day the network mount disappears and the application starts to throw exceptions.

Your applications logs then fill up with the exception message and stacktrace but no one realizes there is an issue until an angry customer rings up complaining that they never received their report.

A far worse scenario is that the exception occurs in production but the development staff decide that it is a “good exception” and that the best course of action is to ignore it. Forever! Well until the new guy starts and they have to explain that it is a “good exception”, and so are the following 600 exceptions.

I remember when I first heard the term “Good Exception”, I was working for startup in London over ten years ago. I was new to the company and the first phase of the application was already in production as part of a critical beta phase of the product. Each morning a developer would have to be in the office ready to deal with any issues that may arise from 6am.

One cold December morning I was in the office and as part of the morning grind I was going through a checkout of the application. Checkpoint number 27 was “Check application logs”. No more detail, so I jumped on to the application server and started to tail the logs and to my dismay hundreds of exceptions were being logged in realtime.

I spent the next hour trying to work out what was wrong with the application and what had changed to cause such an exception storm in production. At around 8:00am one of the developers who had the longest tenure in the team arrived in and calmly pointed out “oh those, those are good exceptions, you can ignore them. They occur the day after a billing cycle due to a bug in one the core components”.

Key lesson; Exceptions should be exceptional, if you get an exception in production you need to deal with it.

Exceptional Workflow

Exceptions are part of both the development process and the application monitoring process. An ideal flow is that once an alert is generated in production it is fed back into the development process as a potential bug fix or improvement. The key is to provide adequate monitoring of exceptions in production and to provide sufficient feedback into the development team.

How many of the applications you have worked on have had anything more than log level or log scrapping exception monitoring?

How many development processes have you seen that link production exception to bug fix and strive to fix as many exceptions as possible?

How many “good exceptions” were written to your logs in production since you started reading this post?

Baking Exception Monitoring

Personally I think one of the reasons for poor infrastructure in critical areas like this is down to the way different parts of the organizations are structured. In many large teams people are dedicated to different function of the application lifecycle. Developers are generally focused on the application business requirements and have unforgiving deadlines. Support teams have deadlines of a different type. They also tend to support many applications across a range of functions.

With the advancement of the DevOps movement these communities are starting to join forces and work on the infrastructure behind the applications. So one problem is certainly being addressed and will start to become more and more widespread in the next 2-3 years The other major factor is tool support. How many good modern tools are available for application monitoring that are quick to use and onboard? There are a number of interesting commercial startups in this space at the moment, AirBrake for example is used by a number of corporation to add monitoring support to their application.

Airbrake offers rich functionality but also supports almost all popular languages in it’s API arsenal. However it is hosted on their servers and this deployment configuration will not suit a large majority of application developers who build bespoke software for internal clients and are forbidden to publish information external regardless of content. Interestingly enough there is an open source alternative to AirBrake called ErrBit which is compatible with the AirBrake API.

It’s a ruby on rails application that can be easily installed on your local server or for the purpose of this blog I put it up on Heroku mainly for ease of use. Once you have installed ErrBit you can quickly post exceptions and stacktraces to the server and it has some basic workflow for your support staff to monitor and deal with the exception. It also has integration with some of the most popular Issue Trackers however there is currently no Jira support.

Installing ErrBit

This was the first time I used Heroku for anything even though I had heard great things. I had an account but it was unverified something that I over looked when I did my first installation. ErrBit needs MongoDB and to use MongoDB with Heroku you need to verify your account with a credit card. This surprisingly stop my application working for a while and it took me ages to notice the small error message in the install script. You have been warned!

To install the application you need to follow the simple steps from the github page https://github.com/errbit/errbit (you will need git and ruby installed locally)

Clone the repository

git clone http://github.com/errbit/errbit.git

Create & configure for Heroku

gem install heroku
heroku create example-errbit --stack cedar
heroku addons:add mongolab:starter
cp -f config/mongoid.mongolab.yml config/mongoid.yml
git add -f config/mongoid.yml
git commit -m "Added mongoid config for Mongolab"
heroku addons:add sendgrid:starter
heroku config:add HEROKU=true
heroku config:add ERRBIT_HOST=some-hostname.example.com
heroku config:add ERRBIT_EMAIL_FROM=example@example.com
git push heroku master

Seed the DB

heroku run rake db:seed

Pretty quick, well once you have a validated Heroku account. Once completed simply type

heroku open

And your new ErrBit install should be running. My instance is at ebit.herokuapp.com and you can use anyone@anyone.com/password to login

 

 

Once you have installed ErrBit you will need to configure your users and whatever applications you plan to monitor. Again straightforward, clicking “Add a new app” button will bring you to configuration screen And once you create the application record you will get the important application id You will need this later when publishing exceptions

Publishing Exceptions from Java

As I mentioned earlier ErrBit is compatible with all the language APIs that AirBrake provide and luckily for me there is an actively developed API for Java available at http://github.com/airbrake/airbrake-java. This will allow you to send Exceptions from you Java Server Appications, Mobile Applications and Desktop Clients. To start using it with maven add the following dependencies to your pom file

<project>
    <dependencies>
        <dependency>
        <groupId>io.airbrake</groupId>
        <artifactId>airbrake-java</artifactId>
        <version>2.2.0</version>
        </dependency>
    </dependencies>
</project>

Once I imported the libraries I saw a slight problem in how to override the url for communicating with the backend server. In the AirBrakeNotifier class, which is responsible for calling the server side rest api, the URL for AirBrake is hardcoded whereas I needed to override it for ErrBit. A quick solution was to create a new ErrBitNotifier class which takes the base url a construction argument.

import airbrake.AirbrakeNotice;
import airbrake.NoticeXml;

import java.io.*;
import java.net.*;

public class ErrBitNotifier {

    private final String baseUrl;

    public ErrBitNotifier(String baseUrl) {
        this.baseUrl = baseUrl;
    }

    private void addingProperties(final HttpURLConnection connection) throws ProtocolException {
        connection.setDoOutput(true);
        connection.setRequestProperty("Content-type", "text/xml");
        connection.setRequestProperty("Accept", "text/xml, application/xml");
        connection.setRequestMethod("POST");
    }

    private HttpURLConnection createConnection() throws IOException {
        return (HttpURLConnection) new URL(String.format("http://%s/notifier_api/v2/notices", baseUrl)).openConnection();
    }

    private void err(final AirbrakeNotice notice, final Exception e) {
        e.printStackTrace();
    }

    public int notify(final AirbrakeNotice notice) {
        try {
            final HttpURLConnection toairbrake = createConnection();
            addingProperties(toairbrake);
            String toPost = new NoticeXml(notice).toString();
            return send(toPost, toairbrake);
        } catch (final Exception e) {
            err(notice, e);
        }
        return 0;
    }

    private int send(final String yaml, final HttpURLConnection connection) throws IOException {
        int statusCode;
        final OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());
        writer.write(yaml);
        writer.close();
        statusCode = connection.getResponseCode();
        return statusCode;
    }

}

Perhaps the AirBrake API could potentially allow for custom configuration of the URL in the next revision. Once you have created a new ErrBitNotifier you can start publishing exceptions. Going back to our previous example

import airbrake.AirbrakeNotice;
import airbrake.AirbrakeNoticeBuilder;
import org.apache.log4j.Logger;

import java.io.File;
import java.io.IOException;

public class TestException {

    private static org.apache.log4j.Logger log = Logger
            .getLogger(TestException.class);

    public static void main(String[] args) {

        File file = new File("h://myfile.txt");
        try {
            boolean fileCreated = file.createNewFile();
            System.out.println("fileCreated = " + fileCreated);
        } catch (IOException e) {
            log.error("Could not create file",e);
            AirbrakeNotice notice = new AirbrakeNoticeBuilder("b4f7cb2020b2972bde2f21788105d645", e, "prod").newNotice();
            ErrBitNotifier notifier = new ErrBitNotifier("ebit.herokuapp.com");
            notifier.notify(notice);
        }

    }
}

This code will throw an IOException (well at least on my computer, since I don’t have a h drive!) and the exception will be seen on the ErrBit console It has the ability to spot duplication of exceptions and you can set it up to email you when the exception is generated.

Also the AirBrake API has log4j appender support but it is tied to the AirBrake public URL and I have left it out of the post. However it can be turned on by the following log4j configuration example

log4j.rootLogger=INFO, stdout, airbrake

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d,%p] [%c{1}.%M:%L] %m%n

log4j.appender.airbrake=airbrake.AirbrakeAppender
log4j.appender.airbrake.api_key=YOUR_AIRBRAKE_API_KEY
#log4j.appender.airbrake.env=development
#log4j.appender.airbrake.env=production
log4j.appender.airbrake.env=test
log4j.appender.airbrake.enabled=true

Conclusion

Application Exception Monitoring is an important part of your application lifecycle.

Exceptions should be easily visible to the support and development teams and your development process should look to address all exceptions in forthcoming sprints.

Exceptions should be used for exceptional cases only, any exception that is not acted upon in production is noise and creates confusion.

Tool Support is important in this area and ErrBit looks like a great multi-language tool that can help support your Exception Management workflow.

Advertisements

How Spring 3.1 Environments & Profiles will make your life better!

My goal of writing one technical blog post per week fell to the wayside around December mainly due to work related project time constraints. I have a four-day weekend and a sparkling new home office, which has to be used for something other than surfing hacker news. Its noon here and I have two hours to produce this blog post so here I go!

The software I write or design generally needs to be deployed in different configurations. These deployment configuration end points can be generalised into the following buckets

  1. Java Enterprise Containers (jboss, weblogic, tomcat, glassfish, etc)
  2. Standalone Java application
  3. GUI Applications
  4. Testing Frameworks

Ignoring GUI Applications for the moment (I might return to these later) the code is often the same between container, standalone and testing. This leads to a key design consideration or philosophy when designing and coding this “type” of software. The code I write needs to run perfectly and untainted in each scenario.

That’s crucial to quality and robustness! The problem is that there are environmentally aware resources that are configured depending on where the code is executed. When I am writing a unit test I will not (I know I could, but I think you are missing the point) have my datasource bound to a JNDI tree. Where as in the container I simple lookup the tree and ask can I have a datasource please.

And frameworks like Spring encourage this form of development, or at least have popularized patterns such as inversion of control. Instead of the executing code configuring the database or queue it is injected as run time and life is good again.

So a blog post to reiterate the same warbling around inversion of control? Not quite, at this point I have a piece of code

public class BusinessClazz implements SomethingReallyImportant {
    private DataSource dataSource;
    @Required
    public void setDataSource(DataSource dataSource) {
        this.dataSource = dataSource;
    }
}

The datasource is injected and the BusinessClazz is none the wiser about the origination of the datasource. Hey, I’m not the smartest guy in the world, but I’m certainly not the dumbest. I mean, I’ve read books like “J2ee Development Without Ejb” and “Expert one-on-one J2EE design and development”, and I think I’ve understood them. They’re about girls, right? Just Kidding. I create a datasource in spring and inject it

<bean name="myBusinessClazz" class="BusinessClazz">
   <property name="dataSource" ref="dataSource"/>
</bean>

My business service now is agnostic to the datasource origin and can now happily run in any of my deployment end points. But what about the datasource, how do we configure this to run anywhere? We are focusing on the datasource but this example can be applied to any component that changes between environments or runtimes.

A datasource typically requires two pieces of configuration. The first part of the puzzle is where is the database and how should I connect to it. I need hostnames, port numbers, service names, etc.

The second piece of configuration is how is it represented. Here we have a few options. I could create a simple one single connection to the database

 <bean id="dataSource" destroy-method="close" 
    class="org.apache.commons.dbcp.BasicDataSource">
    <property name="driverClassName" value="org.hsqldb.jdbcDriver"/>
    <property name="url" value="jdbc:hsqldb:hsql://localhost"/>
    <property name="username" value="sa"/>
    <property name="password" value=""/>
 </bean>

Or I could use apache pooling to create a pool of connections

<bean id="dataSource" 
   class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
  <property name="driverClassName" value="net.sourceforge.jtds.jdbc.Driver"/>
  <property name="url" value="jdbc:myserver"/>
  <property name="username" value="username"/>
  <property name="password" value="password"/>
  <property name="initialSize" value="2"/>
  <property name="maxActive" value="5"/>
  <property name="maxIdle" value="2"/>
</bean>

Or the container manages the connection and we look it up

<jee:jndi-lookup id="dataSource" jndi-name="java:mydatasource"/>

There are many other ways of configuring the datasource but this is enough for illustration purposes. Spring’s PropertyPlaceholderConfigurer class does a great job at handling properties that change between environments.

The problem is how do we seamlessly go between deployment types and pickup different styles of datasources?

Roll your own

This is not an old problem and has been solved in many different ways by many development teams over the last six or seven years. Each team would look at the problem and create a custom way of adding enough intelligence to the code to help alleviate the problem.

One of the biggest drivers behind making code run cleanly inside and outside of the container is to allow developers write unit tests that test true production code.

A typically example of a solution to this problem is to define datasources in a separate context files and create a naming convention to bring order to the chaos. For example if we had three modes of operation we would have three separate context files.

datasources-containerContext.xml
datasources-pooledConnection.xml
datasources-singleConnection.xml

Next comes custom infrastructure code that has to be used at every point of spring initialization throughout your application. It would leverage the ability to use ant style wildcard searches for context files across the deployed classpath.

ClassPathXmlApplicationContext containerContext = 
   new ClassPathXmlApplicationContext("**/**-containerContext.xml");
ClassPathXmlApplicationContext nonContainerContext = 
   new ClassPathXmlApplicationContext("**/**-pooledContext.xml");
ClassPathXmlApplicationContext testingContextContext = 
   new ClassPathXmlApplicationContext("**/**-singleContext.xml");

In a few hours you can have a system that is tolerant of each runtime and the code is engineered to take advantage of this. This roll your own approach has worked for several years but has the major disadvantage that each team and project has a different way of tackling the problem.

Enter Spring Profiles

As of Spring 3.1 there is a solution to this problem (If you do not have an upgrade path for the libraries your project depends on then you have a much larger problem!)

Spring has introduced the notion of Environments and Profiles across the container. Each application context has an Environment object which can be accessed easily

 ClassPathXmlApplicationContext classPathXmlApplicationContext = 
    new ClassPathXmlApplicationContext();
 ConfigurableEnvironment configurableEnvironment = 
    classPathXmlApplicationContext.getEnvironment();

Each environment can have a number of active profiles. Most Spring profile examples talk about profiles being of dev or prod. However I have a Spring solution to the environment type issues my problem is the different profiles for multiple runtimes. But this is the advantages of the implementation you are free to decide how you use it.

By default your beans have no profile and are loaded into the container. So lets start with example. Imagine if this is my current single connection datasource context

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="…">
 <bean id="dataSource" destroy-method="close" 
   class="org.apache.commons.dbcp.BasicDataSource">
   <property name="driverClassName" value="org.hsqldb.jdbcDriver"/>
   <property name="url" value="jdbc:hsqldb:hsql://localhost"/>
   <property name="username" value="sa"/>
   <property name="password" value=""/>
 </bean>
</beans>

As of Spring 3.0 there is a new xml application context class called GenericXmlApplicationContext which is an alternative to ClassPathXmlApplicationContext and FileSystemXmlApplicationContext.

The advantage of using GenericXmlApplicationContext is that it can be configured completely with setters rather than a single clunky constructor. Just remember to call refresh() once ready to instantiated the container.

Armed with GenericXmlApplicationContext we initialize the container with the following code snippet

GenericXmlApplicationContext ctx = new GenericXmlApplicationContext();
 ctx.getEnvironment().setActiveProfiles("standalone");
 ctx.load("*Context.xml");
 ctx.refresh();

Here I set the active profile to standalone. By convention in my project I will consider any code running outside the application container as “standalone” and anything inside the container as “container”. I can set multiple profiles here, for example I could have the following for setting it to standalone and activemq rather than MQSeries

ctx.getEnvironment().setActiveProfiles("standalone", "activemq");

Setting active profiles will have no effect on the current configuration context since I haven’t set a profile on the beans. So we change our configuration context to

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="…" profile="standalone">
  <bean id="dataSource" destroy-method="close" 
   class="org.apache.commons.dbcp.BasicDataSource">
   <property name="driverClassName" value="org.hsqldb.jdbcDriver"/>
   <property name="url" value="jdbc:hsqldb:hsql://localhost"/>
   <property name="username" value="sa"/>
   <property name="password" value=""/>
 </bean>
</beans>

These beans will now only be initialized if the active profile is set to “standalone”. Profiles are an attribute of beans rather than bean therefore you can not set individual beans to select profiles. In the older versions of Spring this would still leave the problem with multiple files and ant wildcards to select the correct context at runtime.

Spring 3.1 has introduced the ability to nest <beans/> within <beans/>. With a quick refactor we can now have a single datasource context file such as

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="…">
 <beans profile="standalone">  
 <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"> 
 <property name="driverClassName" value="org.hsqldb.jdbcDriver"/> 
 <property name="url" value="jdbc:hsqldb:hsql://localhost"/> 
 <property name="username" value="sa"/> 
 <property name="password" value=""/> 
 </bean> 
 </beans> 

 <beans profile="container">
 <jee:jndi-lookup id="dataSource" jndi-name="java:mydatasource"/>
 </beans>
</beans>

And I can quickly change profile to container by

ctx.getEnvironment().setActiveProfiles("container");

How to change profiles

As shown above you can change profiles programmatically by coding something like

ctx.getEnvironment().setActiveProfiles("container");

Another way to change the profile is to pass a system parameter at run time

-Dspring.profiles.active="standalone"

But you can also set it as part of the init parameter of your ear/war

<servlet>
 <servlet-name>dispatcher</servlet-name>
 <servlet-class>
   org.springframework.web.servlet.DispatcherServlet
 </servlet-class>
 <init-param>
   <param-name>spring.profiles.active</param-name>
   <param-value>production</param-value>
 </init-param>
</servlet>

What it means for you?

Hopefully this gives an insight into a new and powerful feature of Spring 3.1 and a feature that has been sorely missed over the 3 major revisions of Spring.