Enabling Microsoft Application Insights for Mule ESB monitoring

Microsoft Azure Application Insights requires Mule 3.7 or later. Application Insights depends on org.apache httpclient and httpcore versions that are first bundled with Mule 3.7

Application Insights is an Azure based application performance dashboard that can monitor applications deployed inside, or outside, Azure.  Application Insights SDKs are available for a variety of languages with a heavy focus on standard library web driven applications or services.

This blog entry describes how easy it is to enable Application Insights for a Mule ESB application that does not use any of the out-of-the-box supported web hooks. In this case, we monitoring the out-of-the-box JMX beans provided by Mule. Performance information is gathered by Application Insights where it is displayed in the Azure Portal.



Mule exposes performance data about applications and flows via JMX.  Any of this can be forwarded to the Application Insights Dashboard.

Steps

1. Create an Application Insights in Azure
2. Enable Application Insights in mule application.
3. Bind the Mule application and Application Insights with a shared Instrumentation Key
4. Select the Mule JMX data to be displayed in the ApplicationInsights dashboard.
5. Configure the Mule applicaton to send Mule JMX data to Application Insights
6. Run the Application
7. Configure the Application Insights instance dashboard to show the enabled Telemetry and metrics.

Create an Application Insights instance in Azure

Create an Azure Application Insights instance as described in this previous blog

Enable Application Insights

Create a Spring bean that loads the Telemetry Configuration. Put this Spring Bean in one of your xml configuration files.   Instructions can be found in this previous blog.

Bind a Mule application to Application Insights

Create an ApplicationInsights.xml file and add the InstrumentKey found in the Azure dashboard.   Instructions can be found in this previous blog  You can find an example ApplicationInsights.xml file below 

Select Mule application data to send to Application Insights

Determine which JMX beans you wish to expose to Application Insights and add those to our configuration. 

Use a JMX inspection tool to see information available in a Mule application.  We will JVisualVM in this example.   These screen shots are based on the example application below.

1. Run JVisualVM from a command prompt
2. Add the MBean Plugin and restart jvisualvm.  This has to be done only once.
3. Select  your running application in the left pane.
4. Select the MBeans tab at the top of the right hand pane.
5. Select Mule.<appname> in the right tab. This will show you the various Application and Flow level metrics that are available. This example app only has one flow so flow and application statistics will mostly be identical.

This target bean is nested two deep in the hierarchy. This makes for slightly complicated naming conventions in the JMX configuration.

Mule.muleappinsightexample:type=Flow,name=exampleflow








We will monitor the Average Processing Time , Max Processing Time and number of Processed Events for a the flow in this single flow application.

• ObjectName: Mule.muleappinsightexample:type=Flow,name=muleappinsightexampleflow
• .Attribute: AverageProcessingTime
• Average: Should stay within operation ranges.

• ObjectName: Mule.muleappinsightexample:type=Flow,name=muleappinsightexampleflow
• Attribute: MaxProcessingTime
• Raw number: grows without bounds over times

• ObjectName: Mule.muleappinsightexample:type=Flow,name=muleappinsightexampleflow
• Attribute: ProcessedEvents
• Raw number: grows without bounds over times

Configure Mule to send Mule application data

Add the monitors to the mule application ApplicationInsights.xml file.  This example monitors basic flow data.

... ...

Sample Application Flow

This example was written against a very simple Mule HTTP echo type application.

The mule corresponding Mule XML contains ApplicationInsights bootstrap Spring Configuration and the flow itself. It looks like:

... ...

WARNING: Mule flow name JMX beans include double quotes.  This means you must add escaped quotes to the name value.  This is done by wrapping the flow name with &quot;

Monitors in Application Insights

Charts can be created for any custom metric.  Just create a chart and then bind one or more Custom metric to the chart by selecting them in the chart details.

Each custom metric, can be added to the Application Insights dashboard. This image shows the three Mule Flow JMX values exposed by this example program.





















2015 12 23 Created 

Enabling Microsoft Application Insights for JMX monitoring of a Spring wired Java application

Application Insights is an Azure based application performance dashboard that can monitor applications deployed inside, or outside, Azure.  Application Insights SDKs are available for a variety of languages with a heavy focus on standard library web driven applications or services.

This blog entry describes how easy it is to enable Application Insights for a Spring wired Java application that does not use any of the out-of-the-box supported web hooks. In this case, we are enabling Java / JMX monitoring of a custom Spring application running in my home lab. Performance information is gathered by Application Insights where it is displayed in the Azure Portal.

I did this work as part of building a message driven application that had no true web interface. The application runs in a container that does not support tomcat or web filters normally used to enable Application Insight.

Microsoft does a good job of describing how to monitor Java Tomcat, Struts, Spring MVC and other standard web applications on their web site.

Steps

1. Create an Application Insights instance
2. Enable Application Insights in a Java application, binding the application and dashboard with a shared Instrumentation Key
1. Create a Spring Bean that enables Application Insights for a non-MVC/non-Struts application.
2. Create an ApplicationInsights.xml file that adds monitors for standard Java VM statistics exposed via JMX.
3. Run the Application
4. Configure the Application Insights instance dashboard to show the enabled Telemetry and metrics.

Create an Application Insights instance.

Step 1 is to create an Application Insights instance Use this Microsoft Azure blog article, Getting Started with Application Insights in a Java web project, to create an Application Insights instance for your new application. Application Insights gathers metrics/telemetry information from applications based on a set of monitors.  You can use the same Application Instance instance for any number of components if you manage the names of the monitors to be unique across the application components.  I am monitoring a single application for the purposes of this exercise.

Log into your Azure portal and create a new Java web application instance of Application Insights. My application isn't a true java web application but "Other" is experimental so I went with it. As of 2015 Dec 20, all Application Insights instances seem to be in Central US.

Find the Insights instance key.  This key binds your application to this instance/dashboard.

Do Not check this key into any public repository or expose this key. It can be used by any application to push events to your dashboard. Azure Application Insights is a metered service so messages cost you money in a non-development environment

Enable JVM metrics in a Spring Application

We are going to enable ApplicationInsights metrics for a standalone Spring MVC application or one that runs in a semi-custom platform like the Mule ESB which has its own filter/web configuration. This assumes that you are not able to enable monitoring through the web filter or other methods described in the Microsoft documentation.  The Application Insights team documentation describes how to monitor web applications running with a variety of containers and frameworks.  

Make sure to add the ApplicationInsights jar files to your project dependencies. The previously mentioned Microsoft Azure blog article describes how to do this.

Add a Telemetry Configuration Spring bean

Create a TelemetryConfiguration instance using Spring to call TelemetryConfiguration.getActive().  This will force the loading of the ApplicationInsights.xml configuration file that you put in src/main/resources.






... ...

Create an ApplicationInsights.xml file

This sample xml config file sends basic JVM statistics to the Applications Insight instance bound by the included key. This configuration gathers information about the number of loaded classes, the amount of heap memory used and the number of threads in this VM.

Microsoft examples show how to monitor heap and cpu. Any exposed JMX monitors can be added.  The following example adds an active thread count monitor.

You can use the JVisualVM MBean inspector to find other information that can be added to your ApplicationInsights dashboard. 

... ...

WARNING: Some JMX beans include double quotes in their paths, names or attirbutes. Elements with quotes in their name must be surrounded by escaped quotes to the name value.  This in the config file by wrapping with with &quot;

Run the Application.

The previous configuration enables debug output for testing. You should see trace data like the following show up in your logs if trace logging is enabled

AI: TRACE 21-12-2015 07:52, 17: Metric JMX: Heap Memory Usage-used, 4.2598904E7
AI: TRACE 21-12-2015 07:52, 17: Metric JMX: Loaded Class Count, 8150.0
AI: TRACE 21-12-2015 07:52, 17: Metric JMX: Thread Count, 53.0
AI: TRACE 21-12-2015 07:52, 17: Sent performance counter for 'Processor % Processor Time'(Processor, % Processor Time, _Total): '3.7053394317626953'<
AI: TRACE 21-12-2015 07:52, 17: Sent performance counter for 'Process IO Data Bytes/sec'(Process, IO Data Bytes/sec, javaw): '20.875185012817383'
AI: TRACE 21-12-2015 07:52, 17: Sent performance counter for 'Memory Available Bytes'(Memory, Available Bytes, ): '9.881853952E9'
AI: TRACE 21-12-2015 07:52, 17: Performance Counter: Process Private Bytes: 1.14066984E8
AI: TRACE 21-12-2015 07:52, 17: Performance Counter: Process % Processor Time: 0.06347814202308655


Configure the Dashboard

JMX beans show up in the Application Insights dashboard as custom metrics.  Add new graphs for those metrics or re-task the existing graphs to display your JMX information. The image to the right shows where JMX information appears as Custom metrics in the configuration screen.

You can pick the metrics in each graph by selecting the graph and then selecting the metrics you want for that graph in the Chart Details section to the right of the graph.

More in the future

This article is already pretty long.  I'll show how to add the metrics to the insights dashboard in a future posting.  This blog was driven by a desire to add Application Insights monitoring to a Mule ESB application.

2015 12 21 Created
2015 12 23 Added Escaped values and bean hierarchy GC example