Sunday, November 30, 2014

Capture and expose system exception statistics in Mule using Coda Hale Metrics

This article describes how to use the Coda Hale metrics library to capture the counts and types of system exceptions thrown outside of Mule ESB flow processing.  Historical exception information is exposed via JMX or written to disk using CodaHale reporters as described on the CodaHale metrics web site. You can use this same technique in any Java application. 

We inject instrumentation at the system context level.  That component converts the wrapped exception stack to a counter name that is then created and incremented in the Coda Hale registry.  I use the simple running counter because I don't find the native Coda Hale histogram data useful for this type of metric. You can use other metric types if you want more complex statistics.


Components

We use 2 injection components 2 codahale components and a custom listener  to make this work:
  1. MetricsRegistry: A CodaHale singleton that maintains a reference to all statistics. It is injected into other components.
  2. MetricAgent: A Mule Agent that manages the lifecyle of the JMXReporter
  3. JMXReporter: A CodaHale reporter that exposes the MetricsRegistry via JMX. Instantiated by MetricAgent. Codahale supports other reporters in place of or in addition to the JMXReporter. 
  4. SystemExceptionMetricContextShim: A MuleContextAware class that injects the SystemExceptionMetricListener into the MuleContext
  5. SystemExceptionMetricListener: A custom class injected into the MuleContext as the ExceptionLister. This creates metric counters concatenating the Simple Names of each Exception in the stack. The failed class name is appended to the counter name if it is some type of Connection Exception

Initializing Coda Hale library

We use the same initialization configuration we used in the flow exception metrics example

Metrics Registry

Instantiate this component as a singleton Spring Bean so that it can be injected into all other objects that wish to record metrics. Note that we inject the app name so that we can add that to the JMX MBean domain name
...

JMX Reporter

Instantiate the JMXReporter using a custom Mule Agent that is injected with the MetricsRegistry spring bean.   
...

Injecting the Metric Listener into the MuleContext 

MuleContextAware / Mule Context

Create SystemExceptionMetricContextShim, a MuleContextAware component in spring that will get called by Mule when initializes the context.  This Shim injects the SystemExceptionMetricListener which actually counts all the exceptions. Create a exception listener spring bean as a singleton that we inject into the Shim.  This listener is set into the MuleContext by the shim.
...

The listener creates a metric name from the nested exception names and the Connection type if it is a connection exception:
...

Example Program

You can find an example program on github at  https://github.com/freemansoft/Mule-demo.  It is a Mule maven project originally built with Mule ESB 3.5.2.  This is the same demo used in the flow exception metrics example.

Th demo has a mis-configured HTTPS endpoint/connector that throws an System exception at deployment time.

Viewing the statistics

This sample exposes metrics via JMX.  You can use any MBean viewer to see the statistics including most Java friendly monitoring tools.  Ever Java JDK comes with jvisualvm that is capable of showing you JMX MBeans with the help of the MBean plugin. Run jvisuavm from the command line or your file viewer.  I run the following command prompt on my windows machine
C:\Program Files\Java\jdk1.7.0_55\bin>jvisualvm.exe
The following image displays the value for ConnectionExcepton raised in the HttpsConnector.  We get one exception for the endpoint on startup because of a configuration error.


 You could see higher numbers for connectors whose remote connections go up and down.

Conclusion

Teams can build production support, problem triage and troubleshooting aids into their applications by integrating Coda Hale metrics and Mule's exception handling strategies.  These metrics can be exposed via files , JMX beans or other tooling to make them visible in a variety of environments.

Created 11/30/2014
All code samples were created using Mule ESB 3.5.2

Capture and expose flow exception statistics in Mule using Coda Hale metrics

This article describes how to use the Coda Hale metrics library to capture the counts and types of exceptions thrown as part of Mule ESB flow processing.  Historical exception information is exposed via JMX or written to disk using CodaHale reporters as described on the CodaHale metrics web site. You can use this same technique in any Java application. 

We add an instrumentation component to each mule Exception Flow.  That component converts the wrapped exception stack to a counter name that is then created and incremented in the Coda Hale registry.  I use the simple running counter because I don't find the native Coda Hale histogram data useful for this type of metric. You can use other metric types if you want more complex statistics.


Components

We use 1 injection component, 2 codahale components and a custom flow component to make this work:
  1. MetricsRegistry: A CodaHale singleton that maintains a reference to all statistics. It is injected into other components.
  2. MetricAgent: A Mule Agent that manages the lifecyle of the JMXReporter
  3. JMXReporter: A CodaHale reporter that exposes the MetricsRegistry via JMX. You can initialize any Reporter in place of or in addition to the JMXReporter. 
  4. MetricRecordFlowException Component: A custom Mule component that is dropped into any Mule Flow exception block. This component creates a metric (counter) concatenating the Simple Name of each Exception in a wrapped exception block. 

Initializing Coda Hale library

Metrics Registry and Flow Component

Instantiate the metrics registry as a singleton Spring Bean so that it can be injected into all other objects that wish to record metrics. Instantiate a singleton MetricRecordFlowException component that can by inserted into exception flows, via spring bean ref, to record metrics.
...

Metrics Agent and JMX Reporter

Instantiate, MetricAgent, a custom Mule Agent that creates the JMX Reporter. This MuleAgent is called at application start up and instantiates the JMXReporter using the injected MetricsRegistry spring bean. Note that we inject the app name so that we can add that to the JMX MBean domain name
...

MetricsRecordFlowException Component

Create a spring bean using this component.  This lets us inject the Metrics registry Spring bean.  An alternative would be to modify the MetricsRecordFlowException component to find the MetricRegistry through the Mule registry. 
...

Operational Information

MetricRecordFlowException Flow Component

The component increments a codahale counter using the injected MetricRegistry.  It creates a metric name including the names of all the nested exceptions in the raised exception. 
  
...

Instrumenting Exception Flows

Metric component in flows

Drop a Java component into each Exception flow and refer to the spring bean using Mule's Spring bean reference syntax.




The XML that puts the component in the exception flow with the spring bean reference looks like:

...

Example Program

You can find an example program on github at  https://github.com/freemansoft/Mule-demo.  It is a Mule maven project originally built with Mule ESB 3.5.2.  This program implements an HTTP endpoint that can throw 4 different types of exceptions from 4 different languages that run inside Mule.

  • Java
  • Groovy
  • JRuby
  • Javascript
Each exception is caught by the Catch Exception Strategy.  The strategy updates the appropriate counter, logs the exception and returns HTML that describes the exception stack trace.  The base url returns a list of other test URLs.  Connect to the server with this URL after starting the application in Mule or Anypoint studio. This also works for deployed applications. 
http://localhost:8081/
Note: You can modify existing components or create new exception paths to generate other exception metrics. The program is also generally useful when wanting to learn how exceptions thrown by various languages are handled in mule.







Viewing the statistics

This sample exposes metrics via JMX.  You can use any MBean viewer to see the statistics including most Java friendly monitoring tools.  Ever Java JDK comes with jvisualvm that is capable of showing you JMX MBeans with the help of the MBean plugin. Run jvisuavm from the command line or your file viewer.  I run the following command prompt on my windows machine
C:\Program Files\Java\jdk1.7.0_55\bin>jvisualvm.exe
The following image displays the value for one of the counters. It shows that the exception originating with an IllegalAccessException has occured 13 times.



Conclusion

Teams can build production support, problem triage and troubleshooting aids into their applications by integrating Coda Hale metrics and Mule's exception handling strategies.  These metrics can be exposed via files , JMX beans or other tooling to make them visible in a variety of environments.

Created 11/30/2014
All code samples were created using Mule ESB 3.5.2

Sunday, November 16, 2014

Windows 8 tablet (battery) dies in 2 days because of Instant-On

I have a Windows 8 / 8.1 TW100 10" quad core tablet that leave at work as an inexpensive internet device.  The device is always dead with 0% battery when I arrive at work Monday morning. The thing has good battery life when I'm actively using it.  I have to charge the device and reset the clock every Monday. (The clock is dead because it relies on the main battery as the clock battery).  My IPad 2, by comparison, sits standby for days/weeks without draining the battery.

The TW100 uses Microsoft Windows 8 new Instant-On or Connected Standby to stay connected to the internet. This new SoC compatible standby mode uses a lot more energy than the standby on other tablet.  Note: This tablet has no applications installed.


You can use the "powercfg" command to analyze battery usage. It generates a nifty HTML file that describes battery usage across the last 3 days.
powercfg sleepstudy
 This chart shows the continual power drain on this tablet across the weekend.





The table after the chart shows that we were in the low power state 99% of the time.

powercfg creates a chart that shows the activity intervals on the tablet.  Windows 8 tablets wake up every 32 seconds.  Values other than 32 seconds mean some program is causing activity and using power. This chart shows that my tablet has no other activity.


The offender section shows which hardware causes the activity. The machine is 99% idle. We can see here that the I2C controller and SD card account for the < 1.5% active time.

powercfg also shows the programs that cause the activity. It finds no programs as the cause of my battery usage on this TW100 Windows 8 tablet.  This means all of this activity is related to the core functions of the Windows operating system, the SoC and this standby mode.

Conclusion

Windows 8.1 InstantOn (formerly InstantGo) provides a high level of connectivity for windows tablets when they are idle.  This is great when you are actually using your tablet or have it plugged in continually. InstantOn can completely eat the battery when the tablet is not actually in use, draining a full battery within a couple days.

The only way to guarantee a working tablet on Monday morning is to either leave it plugged in or power the machine off on Friday.  The TW100 cold boots in 5 seconds so a full powerdown isn't that big a deal.

Comment

I don't need for my tablet to communicate continually with the network when I'm not using it.  I wish Microsoft provided a way of adding additional power plans to tablets. They currently limit you to the InstantOn if it is installed by the OEM.

Sunday, November 2, 2014

Why does my Windows 8 (8.1) tablet have only one power plan?

I recently purchased a quad core Winbook TW100 Windows 8 tablet, similar to the TW801. The tablet gets OK active battery life running for 5 hours. It doesn't last as long on standby as I expected. It is often dead when I pull it out my backpack for a couple days.

Microsoft Windows uses "power plans" to manage the power consumption of windows based devices.  Power plan options that let the user trade off performance versus power consumption.

It turns out this is related to the way windows 8/8.1 manages power  for tablet and other always connected devices using a feature called instant on or InstantGo. These devices come with a single power plan and new sleep modes that are tightly integrated with the tablet/PC specific hardware.  The Windows Power Options panel shows only the Balanced power plan as seen in this image.

This feature, implemented by Tablet OEMs, lets Windows tablets / PCs wake up / receive notifications and other network traffic while the machine is ostensibly powered down.  Tablets and other lower power devices are highly integrated built with combined CPU / wireless and other functionality built into a single component called a System on Chip (SoC).

Microsoft provides information about this new sleep mode and how to study power usage on one of the windows blogs.