Using the collector debug facility, you can verify JMX connections, credentials, and counters.
Here is a JMX command you can test with against a common counter:
!jmx h=10.171.6.189 port=7091 “Catalina:type=GlobalRequestProcessor,name=*”
If the username and password are not specified, the command will rely on the inherited credentials set on the target host. These can be overridden by including them:
!jmx h=10.171.6.189 u=username p=password port=7091 "Catalina:type=GlobalRequestProcessor,name=*"
Deeper checks of MBEANS
When working on custom JMX datasources, you may need to first verify that data is being reported, or that the datapoints are configured correctly. It’s a good practice to compare to a known working datasource for reference.
Referring to the datapoint HeapCommit, we can see the MBEAN identified as java.lang:type=Memory:HeapMemoryUsage.committed. These sections can be broken down into MBean Object = java.lang:type=Memory, Mbean Attribute = HeapMemoryUsage, and the additonal value commmitted. More information on how the MBean is structured can be found here.
Notice the object is separated from the attribute by a colon and the additional value is separated by a period, e.g. object:attribute.value
The following examples show a few different ways of getting this JMX information.
- The first query is pointing to the Mbean object
- The second query includes the Mbean object and the attribute.
- The third, fourth and fifth queries demonstrate calling different values for that attribute.
So, the datapoint definition equivalent would be:
!jmx h=172.31.214.136 port=7199 u=username p=password "java.lang:type=Memory:HeapMemoryUsage.committed"
JMX unsuccessful checks
In this example, you can see what types of errors that can be expected when the host is not allowing connections, or perhaps JMX counters are not exposed, so this is when they would have to start looking more into their configuration and also verifying ports.
Sometimes, the JMX results will vary. You may get results back for some objects, but not others:
$ !jmx h=10.172.106.10 port=1099 u=monitorRole p=xxxxxx 'java.lang:type=*' java.lang:type=* => MemoryPool MemoryPool MemoryPool GarbageCollector Runtime Threading OperatingSystem GarbageCollector MemoryPool Compilation MemoryPool MemoryManager MemoryPool Memory ClassLoading MemoryManager $ !jmx h=10.172.106.10 port=1099 u=monitorRole p=xxxxxx 'java.lang:type=Memory' java.lang:type=Memory => ObjectPendingFinalizationCount HeapMemoryUsage NonHeapMemoryUsage Verbose ObjectName $ !jmx h=10.172.106.10 port=1099 u=monitorRole p=xxxxxx 'java.lang:type=Memory:HeapMemoryUsage' Insufficient roles/credentials for operation java.lang.SecurityException: Insufficient roles/credentials for operation at org.apache.karaf.management.KarafMBeanServerGuard.handleInvoke(KarafMBeanServerGuard.java:274)
The above results would indicate that Active Discovery is successfully able to read and discover instances, but on the provided credentials would lack permission to read the full value.