Java run-time monitoring, Part 3

Part 3 is not really about monitoring Java itself, but rather how to monitor other resources using Java, plus how to manage and visualize collected data. There are several examples I would have liked to put in the article, but with the PDF version running at 73 pages, it got a bit too long.

Postcompilation instrumentation and performance monitoring

Part 2 of my article series on Java run-time monitoring was published today. Last week I reviewed methods of instrumenting java classes at the source level. While not ideal in light of the ubiquity of AOP techniques to inject instrumentation at runtime, if you have or want to do it, take a page from Log4J's book and make sure that the instrumentation is externally configurable (i.e. you don't need to re-compile to tweak it). Here's a quick list of features you might look for in your Log4J instrumentation analog:

• Verbosity: Yep, just like Log4J (and many other logging packages) the Level of loggers can be modified at runtime through JMX or simply by placing a watch on the XML configuration file.
• Thresholds: Automatically trace incidents when measurements exceed externally defined thresholds.
  
• Ranges: Automatically trace counts of measurements that fall into a value window as part of an externally defined and named range. (useful for SLAs where an operation's elapsed time might be classified as GOOD, WARN or CIRITCAL.)
  
• Rollups: Maintain aggregated performance statistics all the way up a hierarchical tree.
  

So in this week's article, I review methods of instrumentation that can be used in the post-compilation phase, or more simply, without futzing with the source code.

• Interceptors: J2EE and a bunch of other [not just] Java frameworks support the ability to inject interceptors into the application without ever touching the source code of your code application classes.
• Class Wrapping: Implement instrumentation in a class wrapper and then wrap your target class. You get the same functionality and instrumentation to boot.
• Byte Code Instrumentation: Inject instrumentation byte code directly into your already compiled classes. This can be done statically (i.e. you end up with the original jar and and instrumented jar) or it can be done dynamically at runtime when the classes are classloaded.

An additional approach which I did not really address in the article is the concept of class swapping. For example, if you wanted to know some low down performance and byte volume details of a network service, you could create a new class called com.myco.instrumentation.InstrumentedSocket which extends java.net.Socket and adds in some basic instrumentation like counting byte transfer rates. At class load time, you could swap out the implementation of all instances of java.net.Socket with your instrumented version.

Hopefully the article is useful to you.

Cheers.

Java Runtime Monitoring, Part 1

Part 1 of my developerWorks series on application monitoring was published today. My goal was to champion the use of fine grained monitoring of the JVM and application code instrumentation.

AutoDiscovery - [Trying to] Say No To Static Configuration

I am always on the lookout for ways to avoid configuration files and the like in favour of dynamic discovery. The JBoss HA-JNDI implementation is a great example where you can specify a set of properties and the ContextFactory will automatically discover a server matching your criteria.

I recently tired of creating yet another user database with roles, passwords, expiration and all the rest of it, so on a new project we started recently for an internal web application, I decided I was going to use the company's Active Directory server. Seemed like a genius idea. Single sign on for my users and I get to delegate all the afformentioned to a bunch of LDAP servers that someone else maintains.

Down the road a ways, I started thinking about which of 10 LDAP servers I should use. What if one of them goes down ? What if they change ? What if they add new servers ? It turns out (and perhaps you already knew this) that DNS will help you locate servers by service, not just by name. Sun provides a DNS Service Provider for JNDI which makes the process nice-n-easy. I could not get it to work exactly as advertised, but with a bit of jiggery pokery, it did the job admirably. In the example below, you simply pass in a peculiar looking string that specifies the service, protocol and domain you are looking for. In my case that string was _ldap._tcp.mydomain.com.

    /**
* Returns an array of located hosts for the given service descriptor.
* @param service String
* @return LocatedHost[]
* @throws NamingException
*/
public LocatedHost[] locateHosts(String service) throws NamingException {
   Properties dnsProperties = new Properties();
   DirContext ctx = null;
   ArrayList locatedHosts =  new ArrayList();
   dnsProperties.put(Context.INITIAL_CONTEXT_FACTORY,  "com.sun.jndi.dns.DnsContextFactory");
   dnsProperties.put(Context.PROVIDER_URL, dnsList);
   String[] hostAttributes = new String[] {"A"};
   LocatedHost locatedHost = null;
   try {
       ctx = new InitialDirContext(dnsProperties);
       DnsContext dnsContext = (DnsContext)ctx.lookup(service);
       Attributes attrs = dnsContext.getAttributes("");
       int i = attrs.get("srv").size();
       for(int c = 0; c < locatedhost =" "srv").get(c).toString();
           String parsedEntries[] = entry.split(" ");
           String hostName = parsedEntries[3];
           int port = Integer.parseInt(parsedEntries[2]);
           locatedHost.setHostName(hostName);
           locatedHost.setPort(port);
           Attributes hAttr = ctx.getAttributes(hostName, hostAttributes);
           locatedHost.setHostAddress(hAttr.get("A").get().toString());
           locatedHosts.add(locatedHost);
       }
       return (LocatedHost[])locatedHosts.toArray(new LocatedHost[locatedHosts.size()]);

   } catch (NamingException ex) {
       throw ex;
   } finally {
       try { ctx.close(); } catch (Exception e) {}
   }

A few things to note. This method is part of a class which takes a string of space separated DNS server URLs in the constructor. (Loaded into dnsProperties.) The format of the DNS URL is dns://. So to test on my [non DNS serving] laptop, I passed in "dns://127.0.0.1 dns://216.254.95.2".

What is retuned from the lookup entries is an SRV record which turned out to be returned as a string which looks like this:

0 100 389 serverAdc1.mydomain.com

If I iterate through all returned servers, it turns out we have way more than 10 Active Directory servers. More like 53 of them in my domain, so I get back 53 records which look more or less like the one above. Not sure what the two numbers are, but I think the 100 bit is intended to indicate some sort of preference, but every server has the same values, except the actual server name. The third value is the port, and fourth, obviously, is the server name. There is more detail on this in the internet draft.

Last problem: The list returned is in no particular order, so the first server returned is as likely to be in California than in New Jersey where I am. This is sort of mesmerizing in a distributed network transparency and location independence sort of way, but I quickly reaslized that actual authentication calls take significantly longer if the server is far away. I am still researching this issue, but it seems that there is no standard as to the sequence in which these records are returned. Accordingly, I also implemented a slightly less performant version of the code above that actually attempts an anonymous connection against each server as it is discovered. The elapsed time for the connection is measured, and the returned array is then sorted in ascending order of elpased time. Works great, but I would only run it once in a while......

My most recent exploration on this general subject has been Service Location Protocol. It really seems to fit the bill and should allow me to define custom services with all sorts of additional properties and then be able to automatically discover them. For example, if I want to discover the location of an HTTP Custom Service Invocation Client for Milkshake Delivery , the SLP server should be able to tell me where it is, provided that said service registered itself with the SLP server when it started.

All my SuSE Linux servers had SLP installed by default. However, the only Java API I have found is from OpenSLP and I cannot get it to work at all. I can use the slptool on my servers and find things like the SSH services, but my Java client just listens for 5 seconds and then gives up. If anyone has this working, I would sure appreciate a pointer or two.