Monday, June 16, 2008

Twitter as a notifications hub

In HiveSight we're trying to keep eyes on our production environment as well as the status of services we need for development such as our continuous integration server and various other long running automation scripts we run on Amazon AWS.

We were looking for a way to easily see all the notifications as a single stream of data - so here is what we did:

First, we created a Twitter account where all the messages will go.
Then we made sure all notifications will be sent to the HiveSight Twitter account.
Now we can become followers of that account to receive updates.

Here are a few examples of what we used it for:

For continuous integration we're using Hudson - build results are sent to the twitter account using a plugin that was written by the Hudson community.

For production servers monitoring we're using nagios. Nagois is very easy to extend so we're using the twitter JSON API to update our account using the Linux curl command:
curl --basic --user "username:password" --data-ascii "status=Nagios: $NOTIFICATIONTYPE$ $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$"
We use the same method to send notifications from various EC2 automation scripts.

We also wrote a twitter-log4j extension that allows us to log errors from our applications directly to twitter. We open-sourced it so you can find it here

Now that we have all the messages in Twitter, we developed a very simple Ruby script that is executed every few minutes, checks if there are new messages that require immediate attention and sends and SMS if there are.

No comments: