- Whisper: An RRD like database which is quite fast.
- Carbon: A high performance cache and network listener that listens for submitted data and stashes it in Whisper.
- Graphite: A web application that allows you to visualize and report data historically or in real time.
This python language is pretty interesting. Not that I am trying to learn it, but perusing the .py files in the Graphite distro is edifying and for a java person, the general working bits-and-pieces of python is not hard to understand. (Now writing something useful.... different story).
After the umpteenth DZone posting from some code rock-star cum superhero admonishing us all to learn multiple languages, my feeling of inadequacy led me to consider delving into python. Sadly, I chose Erlang, having heard that women in bars are the most impressed with that. I'll save that story for another day, but suffice it to say that after I recovered from that blazing flame-out, I picked Groovy. Yes, it is still Java based, but the rock-stars let it slide because it has closures and befriends both sides of the statically-typed-versus-dynamically-typed interstellar war. Apparently my prior list up until Groovy which was Java 1.3, Java 1.4, Java 1.5, Java 1.6 and bash (for a total of 5) was not passing muster.
I have been adapting OpenTrace to optionally send collected metrics to Graphite. Naturally, since I am A.R. about low level metrics, I publish the tracer's metrics through JMX.
The idea is the tracer accumulates metrics until t time elapses, or n metrics accumulate, and then the buffer is flushed up to Carbon. So can you see the metrics in real time ? Well yes. (Nice of you to ask). The Graphite Dev team explains:
How are Graphite graphs always real-time, even when carbon hasn't had time to write it's cached data to disk yet?Upon receiving a rendering request, the Graphite webapp simultaneously retrieves data for the requested metrics from the disk and from carbon's cache via a simple cache query socket that carbon-cache provides. Graphite then combines these two sources of data points into a single series, which is then rendered. This ensures that graphs are always real-time, even when the data hasn't been written to disk yet.
I have been working on an Ubuntu 10.10 VirtualBox appliance that I can distribute because the installation of Graphite is a bit laborious, and each time I do it, I always forget one thing or another. The installation documentation is decent, but there's a lot of moving parts here, and many of them differ by Linux distro, so I can't really fault the Graphite dev team in this regard. I saw something somewhere about an effort to put together Linux packages, but I can't find reference to it now.
At any rate, after the 23rd consecutive time of installing Graphite (and all the requisite packages) onto my Ubuntu guest, I think I finally nailed down a script that does most of the hard work, and it is here for your productivity and reading pleasure:
Updated: Friday May 20, 2011