Recent Knowledgebase Articles


JMX Troubleshooting

Using the collector debug facility, you can verify JMX connections, credentials, and counters.

Here is a JMX command you can test with against a common counter:

!jmx h=10.171.6.189 port=7091 "Catalina:type=GlobalRequestProcessor,name=*"

If the username and password are not specified, the command will rely on the inherited credentials set on the target host. These can be overridden by including them:

!jmx h=10.171.6.189 u=username p=password port=7091 "Catalina:type=GlobalRequestProcessor,name=*"

Deeper checks of MBEANS

When working on custom JMX datasources, you may need to first verify that data is being reported, or that the datapoints are configured correctly. It's a good practice to compare to a known working datasource for reference.

In this example we are looking at the "Cassandra JVM Heap and Threads and Uptime -" datasource. Below is an excerpt from the datasource to look at some of the datapoints it contains.

Referring to the datapoint HeapCommit, we can see the MBEAN identified as java.lang:type=Memory:HeapMemoryUsage.committed. These sections can be broken down into MBean Object = java.lang:type=Memory, Mbean Attribute = HeapMemoryUsage, and the additonal value commmitted. More information on how the MBean is structured can be found here.
Notice the object is separated from the attribute by a colon and the additional value is separated by a period, e.g. object:attribute.value

The following examples show a few different ways of getting this JMX information.

  • The first query is pointing to the Mbean object
  • The second query includes the Mbean object and the attribute.
  • The third, fourth and fifth queries demonstrate calling different values for that attribute.



So, the datapoint definition equivalent would be:

!jmx h=172.31.214.136 port=7199 u=username p=password "java.lang:type=Memory:HeapMemoryUsage.committed"

JMX unsuccessful checks

In this example, you can see what types of errors that can be expected when the host is not allowing connections, or perhaps JMX counters are not exposed, so this is when they would have to start looking more into their configuration and also verifying ports.



Sometimes, the JMX results will vary. You may get results back for some objects, but not others:
$ !jmx h=10.172.106.10 port=1099 u=monitorRole p=xxxxxx 'java.lang:type=*'
java.lang:type=* =>
MemoryPool
MemoryPool
MemoryPool
GarbageCollector
Runtime
Threading
OperatingSystem
GarbageCollector
MemoryPool
Compilation
MemoryPool
MemoryManager
MemoryPool
Memory
ClassLoading
MemoryManager

$ !jmx h=10.172.106.10 port=1099 u=monitorRole p=xxxxxx 'java.lang:type=Memory'
java.lang:type=Memory =>
ObjectPendingFinalizationCount
HeapMemoryUsage
NonHeapMemoryUsage
Verbose
ObjectName

$ !jmx h=10.172.106.10 port=1099 u=monitorRole p=xxxxxx 'java.lang:type=Memory:HeapMemoryUsage'
Insufficient roles/credentials for operation
java.lang.SecurityException: Insufficient roles/credentials for operation
at org.apache.karaf.management.KarafMBeanServerGuard.handleInvoke(KarafMBeanServerGuard.java:274)

The above results would indicate that Active Discovery is successfully able to read and discover instances, but on the provided credentials would lack permission to read the full value.