Capture and expose flow exception statistics in Mule using Coda Hale metrics
This article describes how to use the Coda Hale metrics library to capture the counts and types of exceptions thrown as part of Mule ESB flow processing. Historical exception information is exposed via JMX or written to disk using CodaHale reporters as described on the CodaHale metrics web site. You can use this same technique in any Java application.
We add an instrumentation component to each mule Exception Flow. That component converts the wrapped exception stack to a counter name that is then created and incremented in the Coda Hale registry. I use the simple running counter because I don't find the native Coda Hale histogram data useful for this type of metric. You can use other metric types if you want more complex statistics.
Components
We use 1 injection component, 2 codahale components and a custom flow component to make this work:
- MetricsRegistry: A CodaHale singleton that maintains a reference to all statistics. It is injected into other components.
- MetricAgent: A Mule Agent that manages the lifecyle of the JMXReporter
- JMXReporter: A CodaHale reporter that exposes the MetricsRegistry via JMX. You can initialize any Reporter in place of or in addition to the JMXReporter.
- MetricRecordFlowException Component: A custom Mule component that is dropped into any Mule Flow exception block. This component creates a metric (counter) concatenating the Simple Name of each Exception in a wrapped exception block.
Initializing Coda Hale library
Metrics Registry and Flow Component
Instantiate the metrics registry as a singleton Spring Bean so that it can be injected into all other objects that wish to record metrics.
Instantiate a singleton MetricRecordFlowException component that can by inserted into exception flows, via spring bean ref, to record metrics.
...
Metrics Agent and JMX Reporter
Instantiate, MetricAgent, a custom Mule Agent that creates the JMX Reporter. This MuleAgent is called at application start up and instantiates the JMXReporter using the injected MetricsRegistry spring bean.
Note that we inject the app name so that we can add that to the JMX MBean domain name
...
MetricsRecordFlowException Component
Create a spring bean using this component. This lets us inject the Metrics registry Spring bean. An alternative would be to modify the MetricsRecordFlowException component to find the MetricRegistry through the Mule registry.
Operational Information
MetricRecordFlowException Flow Component
The component increments a codahale counter using the injected MetricRegistry. It creates a metric name including the names of all the nested exceptions in the raised exception.
Instrumenting Exception Flows
Metric component in flows
Drop a Java component into each Exception flow and refer to the spring bean using Mule's Spring bean reference syntax.
The XML that puts the component in the exception flow with the spring bean reference looks like:
Example Program
You can find an example program on github at https://github.com/freemansoft/Mule-demo. It is a Mule maven project originally built with Mule ESB 3.5.2. This program implements an HTTP endpoint that can throw 4 different types of exceptions from 4 different languages that run inside Mule.- Java
- Groovy
- JRuby
- Javascript
Each exception is caught by the Catch Exception Strategy. The strategy updates the appropriate counter, logs the exception and returns HTML that describes the exception stack trace. The base url returns a list of other test URLs. Connect to the server with this URL after starting the application in Mule or Anypoint studio. This also works for deployed applications.
http://localhost:8081/
Note: You can modify existing components or create new exception paths to generate other exception metrics. The program is also generally useful when wanting to learn how exceptions thrown by various languages are handled in mule.
Viewing the statistics
This sample exposes metrics via JMX. You can use any MBean viewer to see the statistics including most Java friendly monitoring tools. Ever Java JDK comes with jvisualvm that is capable of showing you JMX MBeans with the help of the MBean plugin. Run jvisuavm from the command line or your file viewer. I run the following command prompt on my windows machine
The following image displays the value for one of the counters. It shows that the exception originating with an IllegalAccessException has occured 13 times.C:\Program Files\Java\jdk1.7.0_55\bin>jvisualvm.exe
Conclusion
Teams can build production support, problem triage and troubleshooting aids into their applications by integrating Coda Hale metrics and Mule's exception handling strategies. These metrics can be exposed via files , JMX beans or other tooling to make them visible in a variety of environments.
Created 11/30/2014
All code samples were created using Mule ESB 3.5.2
Comments
Post a Comment