R&D Lab - You pay for your mistakes

I made a big mistake a few months ago. I bought a few servers for us to use in our office - you know, it's important to have a few machines at your disposal so you can run intensive tasks on them or run all kind of different tests you couldn't otherwise do on your single development machine.

Makes sense right? Well, not in term of ROI and here is why:

I bought a fairly good machine with 4GB of RAM and an AMD 6400+ X2 CPU - that cost me about $1000. Let's assume the machine is utilized for about 8 hours a day (in reality I guess most lab machines are used much less). Assuming 250 work days a year that's 2000 CPU hours a year. Let's say the life expectancy of this machine is 3 years so for 6000 hours I paid $1000. That's about $0.16 a machine hour.

Now let's compare this to a machine I can rent from Amazon EC2 (prices). Assuming Moore's law will hold true for the next three years (limits of Moore's law) today's EC2 large instance (m1.large) will be equivalent to two of the machines I bought for my office (I'm assuming that Amazon are nice enough to lower prices for a given hardware configuration as it becomes older). Add to my in-house investment the electricity / cooling / management (as in personnel) and it will make my (already) bad investment even worse. Oh, and for my in-house machine I had to put the money upfront and not "pay as I go".

Now what happens If I need more machines at one time (for isntance, to do some tests before a release) and none at another time (like every night). Having these machines hosted on EC2 gives me the flexibility to use as many machines as I need and then don't pay for the time they sit idle.

The bottom line - If you're a small startup - don't buy lab machine. If you're a big company you could probably save thousands of dollars every month by hosting your lab on EC2.

Twitter as a notifications hub

In HiveSight we're trying to keep eyes on our production environment as well as the status of services we need for development such as our continuous integration server and various other long running automation scripts we run on Amazon AWS.

We were looking for a way to easily see all the notifications as a single stream of data - so here is what we did:

First, we created a Twitter account where all the messages will go.
Then we made sure all notifications will be sent to the HiveSight Twitter account.
Now we can become followers of that account to receive updates.

Here are a few examples of what we used it for:

For continuous integration we're using Hudson - build results are sent to the twitter account using a plugin that was written by the Hudson community.

For production servers monitoring we're using nagios. Nagois is very easy to extend so we're using the twitter JSON API to update our account using the Linux curl command:

curl --basic --user "username:password" --data-ascii "status=Nagios: $NOTIFICATIONTYPE$ $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$" http://twitter.com/statuses/update.json

We use the same method to send notifications from various EC2 automation scripts.

We also wrote a twitter-log4j extension that allows us to log errors from our applications directly to twitter. We open-sourced it so you can find it here

Now that we have all the messages in Twitter, we developed a very simple Ruby script that is executed every few minutes, checks if there are new messages that require immediate attention and sends and SMS if there are.

Amazon EC2 and S3

Amazon is hosting so interesting virtualization services called EC2 (Elastic Cloud 2) and S3 (Simple Storage Service).
EC2 is a hardware virtualization service - you basically create a virtual machine and run your software on top of it. You can create as many instances as you want and you only pay for what you use.
S3 gives you a storage API - here too you pay for the space you occupy.

There are some very interesting things which are being deployed on EC2/S3 for example:


NY Times Converting TIFF to PDF

Facebook applications

Hadoop