Graphite is a great APM supporting tool. It is basically (and broadly speaking) a python based server (Linux only AFAICT) with 3 components:
- Whisper: An RRD like database which is quite fast.
- Carbon: A high performance cache and network listener that listens for submitted data and stashes it in Whisper.
- Graphite: A web application that allows you to visualize and report data historically or in real time.
The data submission protocol and data format is very simple so wherever you collect performance data from, it's easy to deliver it to Graphite. You can optionally submit data through RabbitMQ but I have observed that the basic Text over socket is better performing since you can batch up many metrics in one go,while the Rabbit submission is one metric per message.
This python language is pretty interesting. Not that I am trying to learn it, but perusing the
.py files in the Graphite distro is edifying and for a java person, the general working bits-and-pieces of python is not hard to understand. (Now writing something useful.... different story).
After the umpteenth DZone posting from some code rock-star cum superhero admonishing us all to learn multiple languages, my feeling of inadequacy led me to consider delving into python. Sadly, I chose Erlang, having heard that women in bars are the most impressed with that. I'll save that story for another day, but suffice it to say that after I recovered from that blazing flame-out, I picked Groovy. Yes, it is still Java based, but the rock-stars let it slide because it has closures and befriends both sides of the
statically-typed-versus-dynamically-typed interstellar war. Apparently my prior list up until Groovy which was Java 1.3, Java 1.4, Java 1.5, Java 1.6 and bash (for a total of 5) was not passing muster.
I have been adapting OpenTrace to optionally send collected metrics to Graphite. Naturally, since I am A.R. about low level metrics, I publish the tracer's metrics through JMX.
The idea is the tracer accumulates metrics until
t time elapses, or
n metrics accumulate, and then the buffer is flushed up to Carbon. So can you see the metrics in real time ? Well yes. (Nice of you to ask). The Graphite Dev team explains:
How are Graphite graphs always real-time, even when carbon hasn't had time to write it's cached data to disk yet?
Upon receiving a rendering request, the Graphite webapp simultaneously retrieves data for the requested metrics from the disk and from carbon's cache via a simple cache query socket that carbon-cache provides. Graphite then combines these two sources of data points into a single series, which is then rendered. This ensures that graphs are always real-time, even when the data hasn't been written to disk yet.
I have been working on an Ubuntu 10.10 VirtualBox appliance that I can distribute because the installation of Graphite is a bit laborious, and each time I do it, I always forget one thing or another. The
installation documentation is decent, but there's a lot of moving parts here, and many of them differ by Linux distro, so I can't really fault the Graphite dev team in this regard. I saw something somewhere about an effort to put together Linux packages, but I can't find reference to it now.
At any rate, after the 23rd consecutive time of installing Graphite (and all the requisite packages) onto my Ubuntu guest, I think I finally nailed down a script that does most of the hard work, and it is here for your productivity and reading pleasure:
Updated: Friday May 20, 2011
